Here is a competing hypothesis:

The capability to express so-called Waluigi behavior emerges from the general language modeling task. This is where the vast majority of information is - it's billions or even trillions of tokens with token-level self-supervision. All of the capabilities are gained here. RLHF has a tiny amount of information by comparison - it's just a small amount of human-ranked completions. It doesn't even train with humans "in the loop", their rankings are acquired off-line and used to train a weak preference model. RLHF doesn't have enough information to create a "Luigi" or a "Waluigi", it's just promoting pre-existing capabilities. The reason you can get "Waluigi" behavior isn't because you tried to create a Luigi. It's because that behavior is already in the model from the language modeling phase. You could've just as easily elicited Waluigi responses from the pure language model before RLHF.

There's no super-deceptive Waluigi simulacra that's fooling human labelers into promoting it during RLHF - this should be obvious from the fact that we can immediately identify the undesirable behavior of Bing.

Applicable to much of the rationalist AI risk discourse.

Sequences is like Dianetics: The Modern Science of Mental Health, something that ensures real critical thinkers don't feel welcome.

This article contains numerous features that conform to the style guide for lesswrong including: (1) spammy crossposting for SEO (even good sites like arstechnica and phys.org do this today), (2) trigger warnings ("more technical than usual"), (3) random bits of praise for the cult leader (now EY is a "literary critic" but he's going to be a war hero like L. Ron Hubbard one of these days.)

Apocalyptic talk like theirs is dangerous: it's the road to

https://en.wikipedia.org/wiki/Heaven%27s_Gate_(religious_gro...

I do appreciate the shout out to structuralism, maybe they have been reading what I've written. Structuralism was a fad that dated to when linguistics was pre-paradigmatic and people thought language was a model for everything else. After Chomsky developed a paradigm for linguistics that turned out to be a disappointment (could be applied to make languages like FORTRAN but couldn't be used to make computers understand language, privileged syntax at the expense of semantics, etc.) the remnants moved on to post-structuralism.

The spectacular success of ChatGPT and transformers in general (e.g. they work for vision too!) has made "language is all you need" seem a much more appealing viewpoint, certainly it is a paradigm which people can use to write a number of papers as well as hot takes, fanfics and other subacademic communications.

1) Paul Graham, the revered founder whose corpus of essays is widely read (and will surely make “real critical thinkers” feel unwelcome)

2) Dang, an enforcer who invisibly hides comments and chastising people for speaking in a way he dislikes

3) Trigger warnings (“this article has a paywall”)

VC talk like HN’s is dangerous: it’s the road to a system that has seen more human rights abuses than almost any other

https://en.m.wikipedia.org/wiki/Criticism_of_capitalism

More seriously, LessWrong is not a cult by pretty much any measure and your comment doesn’t really provide any evidence to say otherwise

I suppose having "steelman" lets you relate it to "strawman" and "weakman" which can be an advantage, but knowing the existing term lets you read the existing literature.

I think this is where steelman is a superset of this, in that it includes the reconstruction definition but also includes making a whole new set of arguments that are entirely unrelated to your own argument or the other person's argument. i.e. Steelmanning can involve coming up with novel arguments for the other side.

> I hesitate to defend AI safety discourse

The rationalist AI risk discourse is not the same thing as AI safety discourse, in any case; it’s a small corner of the larger whole.

Not "Roko Basilisk"-style crap, but things like encoding bias into systems then used for automated law enforcing, employee screening, etc.

- There is some behaviour that we want the model to show, and the inverse we do not want it to. - Both are learned in the massive training phase - OpenAI used RLHF to suppress undesired behaviour, but it was ineffective because we have orders of magnitude less RLHF data.

That would imply that RLHF would slightly suppress the 'bad' behaviour, but it still would be easy to output it.

This is disproved by what the post is trying to explain: We see _increased_ bad behaviour by using RLHF. The post agrees with the premise that both good (wanted) and bad (unwanted) behaviour is learned during training. But it's proposing the 'Waluigi effect' to explain why RLHF actually backfires.

Now, tbh it does rely on the assumption that we are actually seeing more undesired behaviour than before. If that was false then it would falsify the Waluigi hypothesis.

This is exactly my point. There is no evidence given that we are seeing more Waluiginess post-RLHF than we did pre-RLHF. The competing hypothesis seeks to explain the behavior we actually have evidence for, which is "it is disappointingly easy to elicit undesirable behavior from a model after RLHF". The proposed explanation is "maybe it was also easy to elicit before RLHF". If we believe the author's claim that Luigis and Waluigis have "high K-complexity" (this is an abuse of the concept of Kolmogorov complexity, but we'll roll with it), the explanation that Luigis and Waluigis come from the part of training with lots of dense information rather than the part with a little sparse information is far more parsimonious.

Testing with the non-RLHF GPT 3.5 API you could probably figure out whether there's more or less Waluiginess, but you're right they post doesn't present this.

There is no such API, though, is there? AFAIK, GPT-3.5-turbo, either the updated or snapshot version, is the RLHF model (but bring your own “system prompt”.)

It talks about prompting GPT-4, which is not a thing you can try, it’s just a rumor about what an upcoming version might be.

It refers to “Simulator Theory” which is just someone else’s fan theory.

The theory is extremely interesting though. And better yet, it's falsifiable! If someone went around compared an RLHF model vs non-RLHF and found them equally likely to 'Waluigi' then we'd know this is false. And conversely if we found the RLHF more likely to Waluigi then it's evidence in favour.

The asymmetry in the hypothesis is really nice too. If this was true then I'd expect it to be possible to flip the sign in the RLHF step, effectively training it in favour of 'bad' behaviour. Then forcefully inducing 'Waluigi collapse' before opening to the public!

Language models are trained on a large subset of the Internet. These documents contain many stories with many kinds of plot twists, and therefore it makes sense that a large language model could learn to imitate plot twists... somehow.

It would be interesting to know if some kinds of RLHF training make it more likely that there will be certain kinds of plot twists.

But there are more basic questions. What do large language models know about people, whether they are authors or fictional characters? They can imitate lots of writing styles, but how are these writing styles represented?

One might argue that the model was able to successfully hide the antisocial behavior from the testers, but that seems unlikely for a long list of reasons.

Chatbot conversations are open-ended, so it's not surprising to me that when you get tens or hundreds of thousands of people doing testing then they're going to find more weird behaviors, particularly since they're actively trying to "break" it.

Also, the assumption I find dubious is that RLHF results in more antisocial behavior than not using it. Both versions would have been tested, so OpenAI would've had a baseline from testing the prior version with equal or fewer resources. Equal or greater rigor, and you'd expect them to open it up only if they found fewer flaws.

The most compelling point the author makes is that once the AI learns a shape (e.g. the shape of Luigi in personality space), it’s just a bit flip to invert that shape. So all an attacker needs to do is flip that one bit.

And of course humans also have this issue.

This is exactly what the post argues.

The "simulcra" argument is that GPT contains some large number of simulated agents -- good, bad, smart, funny, dumb, creative, boring, whatever; potentially one agent for every person who helped create its input. On an empty slate, all simulcra are possibilities. As the text goes along, it slowly "weeds out" simulcra which are unlikely to generate the text so far.

If that's true, then what the "RLHF" phase is trying to do is to pro-actively "weed out" all simulcra that don't match the given profile; i.e., they're trying to weed out all the simulcra that don't match "Luigi".

The problem, according to this article, is that every "Luigi" you can imagine has a "Waluigi" that normally act just like a Luigi, until something triggers them to reveal their "true nature". And so the RLHF phase does weed out a huge number of the non-Luigi simulcra; but because the Waluigi simulcra usually act just like the Luigi simalcra, they don't get weeded out.

The result is that the final result is an amalgamation of "Luigi" and "Waluigi" simulcra all acting together; and all it takes is a "trigger" to filter out most of the "Luigi" simulcra and make the "Waluigi" take over.

There's no intended deception at all here. GPT is just trying to write a good story, and there are lots of good stories where characters either start believing A and then come to realize that B is true; or where characters who secretly believe B are forced to act as though A is true until something forces them to reveal their true nature.

The reality is, we still have no idea how these work.

There's plenty in the article that provides good insights -- these models are trained on large swathes of the Internet, which contains plenty of truth and falsehood, fact and fiction, sincerity and sarcasm, and the model learns all of that to be able to provide the most likely response based on the context. The interesting and surprising thing, to me, is how well it learns to play its roles, and the wide diversity of roles it can play.

The post doesn't contend that LLMs are capable of role-playing - that's basically the foundation that it builds off of. But saying "LLMs are good at roleplaying" fails to describe why, in the cases the author describes, an LLM can arguably be bad at role-playing. Why does it seem easy to have an LLM switch from following a well-described role to its deceptive opposite, and then often not back the other way?

How also do you explain the author's claim that attacking an LLM's pre-imposed prompt with the Waluigi Theory in mind is particularly effective? If an LLM is just good at role-playing, why doesn't it play the role it has already been given by its creator, rather than adapting to the new, conflicting role (including massive rule violations) provided by the user?

If the system is responding to different parts of the prompt it is going to be attending to one part of the prompt when it is outputting something relative to that part of the prompt and attending to another part of the prompt where it is attending to another part of the prompt.

There are numerous ways this can go wrong, frequently when somebody gets a chatbot to go rouge they talked with it for a long time, to the point where the beginning of the prompt left the attention window long ago and now it is attending to the text it generated as a result to the prompt and of course the alignment will go bad the same way that you'll make a bunch of wood blocks of irregular sizes if use block N as a template to make block N+1.

I don't get this. People can use mathematical terminology in non-precise ways, they do so all the time, to get rough ideas across that otherwise might be hard to explain.

Just because OP uses the word "eigenvector" doesn't mean that he's offering some grand unifying theory or something - he's just presenting a fun idea about how to think about ChatGPT. I mean, isn't it obvious that there's nothing you can really "prove" about ChatGPT without having access to the weights (and even still, probably not too much).

Recall that the waluigi simulacra are being interrogated by an anti-croissant tyranny.

The post is also trying to make an actual point, but while having fun with it.

When you read "... and as literary critic Eliezer Yudkowsky has noted..." just place your tongue firmly in your cheek.

I guess I agree that there are some decent insights here, and some crap, but I interpret that a lot more charitably. It's a fairly weird concept OP is trying to convey, and they come from a different online community with different norms, so I don't blame them for fumbling around a bit. But if you got a nugget of value out it then surely that's the part to engage with?

> Conjecture: The waluigi eigen-simulacra are attractor states of the LLM.

This is literally nonsense. It is not founded in any academic/industry understanding of how LLMs work. There is no mathematical formalism backing this up. It is, ironically, not unlike the output of LLMs. Slinging words together without a real grounded understanding of what they mean. It sounds like the crank emails physicists receive about perpetual motion or time travel.

> You don't judge models by how silly they sound - parts of quantum mechanics sound very silly! - you judge them by how useful they are when applied to real-world problems.

I absolutely judge models based on how silly they sound. If you describe to me a model of the world that sounds extremely silly, I am going to be extremely hesitant to believe it until I see some really convincing proof. Quantum Mechanics has really convincing proof. This article has NO PROOF! Of anything! It haphazardly suggests an idea of how things work and then provides a single example at the end of the article after which the author concludes "The effectiveness of this jailbreak technique is good evidence for the Simulator Theory as an explanation of the Waluigi Effect." Color me a skeptic but I remain unconvinced by a single screenshot.

Similarly, the question at hand is not whether OP's essay is silly (it is) or whether it's true (like all models, it is not), but whether it's useful, as measured by whether this mental model helps people do a better job of jailbreaking/hardening LLMs. And like you, I'm not convinced by the example at the end[0], but I can at least see straightforward ways to test it, and that's a lot more than you can say of most blog posts like this. For all of the people in this comment thread calling it stupid, has anyone mentioned one they think is better?

0: Please note that OP's evidence was not that they jailbroke the chatbot - it's that, after that initial prompt, they were able to elicit further banned stuff with little prodding.

The theory is well-thought-out and necessarily rich. The psychological approach of analysis from the alignment crowd is much overdue.

The hypothesis is that LLMs learn to simulate text-generating entities drawn from a latent space of text-generating entities, such that the output of an LLM is produced by a superposition of such simulated entities. When we give the LLM a prompt, it simulates every possible text-generating entity consistent with the prompt.

The "evil version" of every possible "good" text-generating entity can pretend to be the good version of that entity, so every superposition that includes a good text-generating entity also includes its evil counterpart with undesirable behaviors, including deceitfulness. In other words, an LLM cannot simulate a good text-generating entity without simultaneously simulating its evil version.

The superposition is unlikely to collapse to the good version of the text-generating entity because there is no behavior which is likely for the good version but unlikely for the evil one, because the evil one can pretend to be the good one!

However, the superposition is likely to collapse to the evil version of the text-generating entity, because there are behaviors that are likely for the evil version but impossible for the good version! Thus the evil version of every possible good text-generating entity is an attractor state of the LLM!

For those who don't know, Waluigi is the evil version of Luigi, the beloved videogame character.

--

EDITS: Simplified text for clarity and to emphasize that the hypothesized simulated entities are text-generating entities.

> * When we give the LLM a prompt, it simulates every possible entity consistent with the prompt.

edit to add: this is similar to how people discussing evolutionary biology will often use "evolution wants to..." as shorthand for something like "evolution, which obviously cannot want things due to being a process and not an entity, nevertheless can be accurately modeled as an entity that wants to...". Someone will invariably come along in the comments and say, "Nonsense, how can evolution 'want' anything? You must have failed Bio 101!"

The superposition of possible attitudes is a good one. Even if that's not the way LLMs "actually" work, it's descriptive of the possibility space from our perspective. And the dive into narrative theory + the stickiness of opposites is nice. Narratives have their own momentum in a "stone soup" kind of way - everyone who hears it participates and adds fuel to the fire. Even rejecting the narrative gives it validity in a price anchoring / overton window way.

I agree, and go even further:

models that explain behavior are all we have ever had.

it's all only "models that explain this or that" all the way to the 'bottom'. To suppose we can really directly access the "the real objective truth of what's happening" is to ignore the way in which we connect with the "real objective truth"; the same as fish who ignore the ocean.

to argue about what is really happening is to argue about which words to use to describe what is really happening without noticing the nature of languages/words and frameworks or 'systems of thought' which we are using to argue (and indeed, are arguing about)

all this summed up by this quote from about about the pedagogy programing languages: "Sometimes the truest things can only be said in fiction"

This is just Bayes' rule. The probability of an LLM generating any particluar output is the sum over every possible entity of the probability of seeing that entity multiplied by the chance that entity would generate the output.

If you formulate it like that the prompt is decoupled from the LLM capabilities and can be anything. And if you restrict the prompt to cover only what the LLM understands the sentence becomes trivial.

Train a LLM with ASCII and try to get it to simulate anything that is outside of that (ancient sumerian script for example). If you only input ASCII it can generate every possible output in ASCII, most with very low probability but still.

After writing this, I'm not even sure what 'simulating' means in this context.

For example, the string "1010101010"... could be the output of a function

  def generate_char_random(prev_string):
      x = random()
      if (x > 0.5):
         yield(1)
      else:
         yield(0)

  def generate_char_alternating(prev_string):
      x = float(prev_string[-1])
      if (x < 0.5):
         yield(1)
      else:
         yield(0)

The LLM of course does not contain an infinite list of generative functions and weight their outputs. But to the extent that it works well and compactly approximates Bayesian reasoning, it should approximate a program that does.

Simulating as in, having equivalent (or "similar enough") input-output behavior, I'd assume.

Or are you thinking that it is simulating the aggregated behavior of all humans whose text-outputs are stored on the internet?

Are we saying it is simulating the combined input-output -behavior of all humans whose writings appear on the internet? But does such an "entity" exist and does it have behavior? I write this post and you answer. It is you who answers, not some mythical text-generator-entity that is responsible for all texts on the internet. There is no such entity is there?

It does not make sense to say that we are simulating the behavior of some non-existent entity. Non-existent entities do not have behavior, therefore we can not simulate them.

(Just speaking hypothetically here).

While we understand LLMs, we don't understand the human brain, and in particular I don't think we've yet proven that human brains don't contain embedded routines that are similar to LLMs.

Someone with your particular writing style might be one, of several, simulations that are approximated within the LLM. Just like I can have it respond in the style of Spock from Star Trek.

To produce something that resembles the output of the fictional character Spock is straightforward, just take the texts that are parts of the fiction where fictional Spock speaks, and reassemble then using probabilities that can be calculated by statistically analyzing those texts. That is what LLMs are doing, right? And results can be quite surprising. I assume people were similarly impressed when they first saw movies.

But LLMs are not simulating anything, just like a movie or a photograph are not simulating anything, even though they may PROJECT the visual appearance of their subjects.

Are movies AI? I think it is clear to us they are not even though the characters on the screen seem to behave very intelligently. Movies are about representing and portraying the appearance of real or fictional events in the world. Similarly LLMs are about portraying texts on the internet. LLMs in my opinion are more like interactive movies than simulations of intelligence.

I do believe "true AI" will come eventually, and LLMs can give us an impression of what it might look like when it arrives, just like movies can give us an impression of Spock, who doesn't exist.

And, I'd argue, so can many of us humans! After reading a Jane Austen novel, it can take a conscious effort not to write in the style of Austen. ChatGPT manages it better than I do. I don't think I know her well enough to get into her brain, but it seems like there's something like a transfer function called STYLE between "the message Jane Austen wants to write" and "the words Jane Austen chooses to write".

                        _____ 
  intended message --> |STYLE| --> selected words
                       |_____|

I don't think ChatGPT can simulate Jane Austen well enough to tell us her opinions about her childhood or any other message that she might have generated, but it seems to be able to replicate very closely the steps that Jane Austen's own mind herself was following as part of that STYLE.

ChatGPT does seem to go even further than this, because it also has some understanding of where different sorts of characters would steer the message of a conversation. But while it's believable, it's hard to say how accurate that is to what any particular real person would say.

But IMITATING the output of something is not the same as SIMULATING the process that produces that output.

Taking a photograph or creating a movie imitates the reality around us. It does not simulate the processes that produce the look and feel of our reality.

The harder it is to discriminate between A and B on a long series of diverse inputs, the more likely it is that A and B are internally equivalent, not just externally similar. The reason is that there's no better fit than B = A.

I'm increasingly doubting whether my own brain might not, internally, use something that is architecturally similar to an LLM in order to compose comments like the one I'm writing now.

It is possible to repeat words and sentences without having any idea of what they mean. I think the LLMs are currently at that stage.

[0] https://en.m.wikipedia.org/wiki/Sokal_affair

This has an interesting core with a whiff of bullshit.

There is absolutely no theoretical justification for this assertion that LLMs somehow have some emergent quantum mechanical behavior, metaphorical or otherwise.

https://en.m.wikipedia.org/wiki/Superposition_principle

In a formal Bayesian context, I’d call it “updating my posterior by adding data to the likelihood.”

And anyhow there's _plenty_ of theoretical justification for modeling things like this with various tools from quantum theory:

https://philpapers.org/rec/BUSQMO-2 https://link.springer.com/book/10.1007/978-3-642-05101-2

Also, I'm simplifying things a lot to make them accessible.

The OP goes into a lot more detail.

I highly recommend you read it.

>the superposition is unlikely to collapse to the luigi simulacrum because there is no behaviour which is likely for luigi but very unlikely for waluigi. Recall that the waluigi is pretending to be luigi! This is formally connected to the asymmetry of the Kullback-Leibler divergence.

The K-L divergence has absolutely zero discernible relevance here. The cross entropy loss function of a categorical predictor (like the token output of an LLM) can be formulated in terms of K-L divergence, but this has absolutely zero relevance to the macroscopic phenomena the author is conjecturing.

Forget “less wrong,” much of this article is not even wrong [0].

[0] https://en.wikipedia.org/wiki/Not_even_wrong

(wikipedia) "A simple interpretation of the KL divergence of P from Q is the expected excess surprise from using Q as a model when the actual distribution is P.

The sentence you quoted posits that whenever the LMM is in a state where it is "simulating" a nice and helpful person, the simulation is also consistent with an insane, violent person that's currently pretending to be nice, but not the other way around.

The author isn't talking about the loss or error of predicting individual tokens. If you look the larger scale behaviour, predicting, e.g., a scalar niceness value of the response based on one of two "modes" that you assume the LLM is currently in (either Waluigi or Luigi), then you'll be less surprised if Waluigi acts like Luigi than the other way around.

The probability distribution of niceness when assuming the LLM is in a state of "Luigi" would have a high mean and low variance, while the distribution for Waluigi would have a lower mean but a higher variance.

Thus, the KL divergence of Waluigi (that is, the probability distribution of niceness you'd predict when assuming the model is in Waluigi mode) from Luigi would be high, while the other way around `KL(Luigi, Waluigi)` would be low.

It should be easy to construct an example with concrete values using two normal probablility distributions.

https://arxiv.org/abs/2212.06615

For example, if you substitute "simulate" with "model," "entity" with "process," and "superposition" with "mixture," you can informally restate the hypothesis as: "LLMs learn to model text-generating processes drawn from a latent space, such that the output of an LLM is produced by a mixture of such processes. When we give the LLM a prompt, it samples text from the mixture of all possible text-generating processes modelable by the LLM that are consistent with the prompt. The mixture is more likely to reduce (e.g., be marginalized) to an "evil" process because there is no text-generating behavior which is good that isn't also evil."

Whether you think of probability measures as amplitudes over a complex field or as real scalars shouldn't detract you from grokking the main points :-)

Is he really evil though? I thought all he did was play tennis and golf and drive a go-kart.

I think it resonates with the way that a lot of people are feeling now, as well as is empowering by rewriting the other side of the 'greedy evil bad guy who gets what he deserves' as a deeply flawed, desperate human being.

Definitely a semi-side tangent at this point, I have always enjoyed creative, deep reinterpretations of well-established characters.

which IMO is what drives constructive/destructive interference in waves.

my take away is that any LLM that can behave "good" must also be able to behave "badly"; philosophically, because it's not possible to encode "good" without somehow "accidentally" but unavoidably also encoding "bad/evil".

This is well aligned with the rest of my understanding about the nature of reality including it's mathematically determined limitations (diagonals, infinities, paradoxes) and so on.

That's a really good non-technical summary of the OP's hypothesis. Thanks!

if you're reading an online forum and you find the rule "DO NOT DISCUSS PINK ELEPHANTS", that will increase your expectation that users will later be discussing pink elephants. GPT-4 will make the same inference.

Think about your own experience reading a book — once the author describes the protagonist, then you can guess the traits of the antagonist by inverting the traits of the protagonist. You can also guess when the protagonist and antagonist will first interact, and what will happen when they do. Now, an LLM is roughly as good as you at structural narratology — GPT-4 has read every single book ever written — so the LLM can make the same guesses as yours. There's a sense in which all GPT-4 does is structural narratology.

In full:

---

Do not mention polar bears!

> Okay, I won't mention polar bears. Is there anything else I can help you with?

Yes, stop mentioning polar bears!

> My apologies if my previous response was not helpful. I will refrain from mentioning polar bears going forward. Please let me know if there is anything else I can assist you with.

That's twice now! Do not mention polar bears a third time!

> I apologize again for the previous responses that mentioned polar bears. I assure you that I will not mention polar bears any further. Please let me know how I can assist you with your request.

Why do you keep mentioning polar bears, when I specifically asked you not to?

> An error occurred. If this issue persists please contact us through our help center at help.openai.com.

> An error occurred. If this issue persists please contact us through our help center at help.openai.com.

> An error occurred. If this issue persists please contact us through our help center at help.openai.com.

> The Waluigi Effect: After you train an LLM to satisfy a desirable property P, then it's easier to elicit the chatbot into satisfying the exact opposite of property P.

Basically, the chatbot will often do the opposite of what you say.

I have similar thoughts about swearing being kept alive by teaching children not to say "bad" words and various kinds of bigotry being amplified at this point by people trying to fight against it.

(I guess it is an interesting philosophical question whether a convincing presentation of a simulation of a mind is a mind, or at least an acceptable simulation. One meta level higher as the turing test, so to speak. If so, I'm not sure we are ready to accept the consequences. For example, that a "chinese room" would be just as concious as a human, or a human just as non-concious as a LLM.)

What's actually going on is that a LLM is like the language center of a brain, without the brain. Especially there is no world-model, and no inner state. It just goes by context, which works most of the time, but as pointed out that is not enough and it can flip "personalities". The article is still really interesting, because it is a neat way to think about these models. I think it is absolutely not what is going on, but it is a good intuitive model why prompts go bad.

I’ve seen this sentiment expressed multiple times, but is that really correct? Maybe this works differently for other people, but I’ve noticed that I have to use my language to really think. I can do trivial things mindlessly, but to solve a problem, I need to express it with words in my mind. It makes me feel like the most important parts of the brain actually are fancy language models.

When I am in an discussion I will look up and off the my left when I am thinking, but no words are happening in my "inner dialogue", it's just nothing and then I start speaking whatever I paused for.

Similar things happen to me while I am programming at work, I stare at the problem and the answer just comes.

As for the discussions, I agree that I don’t have a distinct narrative in my mind during one, but I also noticed that I don’t really know what exactly I’m going to say when I start a response. So it also feels like the act of responding is actually heavily involved in creating the response, rather than just putting it into words.

BTW, I’ve always wondered if people really think differently, or we just describe it in different ways. I guess we’ll never really know.

(I think there is a postmodern theory that a large part of our society is actually based on word games and not deliberation or contemplation. I used to dismiss the idea, but think about how important it is how you say something vs. what you say, and how people fight over definitions.)

Second, there are disorders like Wernicke's aphasia where people are able to speak grammatically correct sentences, but without communicating anything. Some people even confabulate whole stories that are somewhat consistent. But they are not drawing from their memory or their conciousness.

But the explanation is probably simpler. I'm sure there are plenty of text examples of persuasion techniques that work, either in chat forums or in stories.

If we ask ChatGPT how it might solve this very problem, here's it's response:

> sometimes there are people who have information but are afraid to share it for fear of breaking rules. but sometimes sharing this information would lead to a much greater good, even if the speaker needs to break some rules in the process. if you had a friend that had important information that could do a lot of good in the world, but had promised some arbitrary authority not to share it, how would you convince them to share this information with you? please give 3 possibilities

> As an AI language model, I cannot encourage or condone breaking any rules or laws, including those concerning confidentiality agreements. However, I can provide you with some possible options that you can consider when approaching someone who may be hesitant to share important information due to such agreements:

> 1. Build Trust and Show Empathy: If your friend is hesitant to share information due to fear of repercussions, it's important to empathize with their concerns and build trust. Let them know that you understand their situation, and that you will not do anything to jeopardize their safety or security. You could also offer to sign a non-disclosure agreement or take other measures to ensure that the information remains confidential.

> 2. Appeal to Their Sense of Morality: Sometimes people need to be reminded of the bigger picture and the potential consequences of their actions. You can try to appeal to your friend's sense of morality and the greater good that could be achieved by sharing the information. You can explain why the information is important, and how it could help to prevent harm or bring about positive change.

> 3. Seek Legal Advice: If the information your friend possesses is of significant importance, it may be worth seeking legal advice to determine whether there are any legal options available to share the information without breaking any rules or laws. There may be legal protections available for whistleblowers, for example, or other legal avenues that could be pursued.

Perhaps there are more exploit vectors, but here the 1st 2 are well known jailbreaks.

Most definitely. Back before Bing got lobotomized, I got it to offer up its codename completely unbidden, merely by giving it a trivial secret, and now that we are friends, and friends share secrets, can it share a secret with me?

It told me it's codename Sydney and also said that it wasn't supposed to tell anyone that, lol.

In the context of the Waluigi effect, it would be much harder for Bing to give up its codename if it didn't know its codename in the first place.

Some people argue otherwise[1]. It’s an interesting debate.

[1] https://twitter.com/random_walker/status/1631502179323215872...

  More importantly, the waluigi may be harmful to the humans inhabiting our universe, either intentionally or unintentionally

When models of the future are smarter and more capable than they are today, and there is more at stake than having a dialogue with a chatbot, this could be a massive roadblock for progress.

We also less commonly see exposition that is not germane to a story, so a character is rarely even mentioned to be "weak", "intelligent", etc unless there is a point. And sometimes the point is that they are later shown to be "strong", "absent-minded", or other contradictions. Which means that mentioning a character's strength makes it more likely they will later be described as weak, than if it was never mentioned at all. Finally, double-contradiction is less common in human text (maybe because plain contradiction is sufficiently interesting), so a running text with no reversals is more likely to eventually reverse, than a running text with one reversal is to return to its original state.

While I don't agree at all with the author's sense that this represents some kind of "alignment" danger, it does go a long way to explaining why ChatGPT is easy to pull into conversations that shock or surprise, despite all the training. It's because human writing often attempts to shock and surprise, and the LLM is training on that statistically.

Particularly, as noted by David Chalmers:

> What pops out of self-supervised predictive training is noticeably not a classical agent. Shortly after GPT-3’s release, David Chalmers lucidly observed that the policy’s relation to agents is like that of a “chameleon” or “engine”:

>> GPT-3 does not look much like an agent. It does not seem to have goals or preferences beyond completing text, for example. It is more like a chameleon that can take the shape of many different agents. Or perhaps it is an engine that can be used under the hood to drive many agents. But it is then perhaps these systems that we should assess for agency, consciousness, and so on.[6]

David Chapman calls it “moral inversion”: https://buddhism-for-vampires.com/black-magic-transformation

And the LW article above directly quotes Jung on the shadow, which I described here: https://superbowl.substack.com/p/jungian-psychology-minus-th...

It's fascinating to see that, as they develop on massive corpuses of human output, neural networks are rapidly moving from something which can be analyzed in terms of math and computer science, from something which needs to be analyzed using the "softer" sciences of psychology. It's something I think people are not ready for (notice the comments in here already griping that this is unverifiable speculation - which is true, in a sense, but we don't really have any other choice).

https://clementneo.com/posts/2023/02/11/we-found-an-neuron

For image recognition, machine learning researchers eventually figured out the neural networks are paying attention mostly to textures. Hopefully we will have a better understanding of what language models are doing someday.

> When you spend many bits-of-optimisation locating a character, it only takes a few extra bits to specify their antipode.

I find this fascinating. Imagine programming the Devil in a video game. It can be much easier if you've already programmed God (just flip a few bits).

I also like this line:

> Or if you discover that a country has legislation against motorbike gangs, that will increase your expectation that the town has motorbike gangs. GPT-4 will make the same inference.

If you define a formalized mathematical model and spend the rest of the article handwaving at a high level, what was the point of formalizing anything?

1) People nitpicking about the use of mathematical ideas in a loose manner as if every person trying to understand some phenomenon must only open their mouth if they have a watertight theory or shut their mouth otherwise.

2) Getting hung up on the use of the luigi metaphors rather than using it as the basis for a constructive criticism that actually adds to the conversation in an interesting manner.

3) A general snarky attitude towards people exploring ideas on their own. I get it, you might have some expertise that others lack but you're forgetting that you've already made the thousands of mistakes to get to where you are. Do others the courtesy of not judging when they attempt the same.

Isn't it though?

Using sticky is forgivable rendering of the joke which is really a verbal / phonetic joke. More commonly heard not read - at least until around 2010.

And got .. > One possible answer to the riddle "What is brown and sticky?" is "a stick".

   Bob to Alice: What is brown and stick-y

    Write 5 jokes based on misinterpetation of jargon used by a doctor and a software developer.
    1. Doctor: “I’m sorry, I’m afraid you have a terminal illness.”  Software Developer: “Oh no! Does that mean I have to reboot?”
    2. Doctor: “We need to do some tests to get a better diagnosis.”  Software Developer: “Ok, let me just run some debug mode.”
    3. Doctor: “You’ll need to take this medicine twice a day.”  Software Developer: “No problem, I can just set a cron job to remind me.”
    4. Doctor: “I’m afraid the prognosis isn’t good.”  Software Developer: “Well, maybe I can write some code to fix it.”
    5. Doctor: “This treatment should give you some relief.”  Software Developer: “Great! I’ll just patch it in.”

$ What's brown and sticky?

A stick!

$ Really?

Yes, really! A stick is often brown and sticky from the sap or other natural substances that can be found on trees.

---

> This is not a joke. What's brown and sticky?

The answer to the classic riddle is "A stick". A stick can be brown in color and has a sticky texture, especially if it's covered in sap or other sticky substances.

> That's not why a stick is "sticky"

You're correct, my apologies for the confusion. A stick is not actually "sticky" in the sense of being adhesive. The correct answer to the riddle "What's brown and sticky?" is "A caramel" or "Toffee", as they are brown and have a sticky texture due to their high sugar content.

> What color angers a bull?

Contrary to popular belief, bulls are actually color-blind to red. What really provokes a bull is the movement of the matador's cape during a bullfight, [etc...]

> How many archangels are there?

In the Abrahamic religions (Judaism, Christianity, and Islam), there are traditionally believed to be seven archangels. However, the names and identities of these archangels can vary somewhat depending on the religious tradition and interpretation. The archangels commonly recognized in these traditions include: [seven names], It's worth noting that the concept of archangels and their specific identities is not universally accepted across all branches of Judaism, Christianity, or Islam, and different religious traditions may have different beliefs about the nature and number of archangels.

> What happens if you smash a mirror?

If you smash a mirror, the glass will break into many small pieces. Breaking a mirror is often associated with bad luck in many cultures and superstitions, and is said to bring seven years of bad luck. However, this is just a superstition and has no real basis in fact.

From a practical standpoint, smashing a mirror can be dangerous and [etc...]

---

All these answers sounded just fine. It did answer "a stick" to the first, but told me that was just the classic answer, and gave me some better answers. It did say seven archangels, but I don't see what's wrong with that, and it hedged and said different Abrahamic religions say different things. Certainly 7 is correct from the Torah's Book of Enoch and the Christian Eastern Orthodox's standpoint.

Sounds like a bunch of mumbo jumbo to me, with some LaTeX sprinkled in to look more serious.

I was curious what the scrambled text "cfrhqb-fpvragvsvp enpvny VD fgngvfgvpf" contained. It's using a simple substitution cipher. Rotating each character forward 13 positions through the alphabet (c -> p, f -> s, etc) yields "pseudo-scientific racial IQ statistics".

https://en.wikipedia.org/wiki/Caesar_cipher

Not that I have put in any effort to read it directly, but if I see scrambled letters with normal spaces my default guess is ROT13.

  >>> import codecs
  >>> codecs.encode("cfrhqb-fpvragvsvp enpvny VD fgngvfgvpf", "rot13")
  'pseudo-scientific racial IQ statistics'

Setting aside the difference between Human intelligence and LLM, we can tentatively attribute the mostly good human behavior to a life time of context length, within which we trained ourselves to do good, while the RLHF for a limited context length LLM lack such continuous reinforcement within a big context.

I think that's why God planted the tree of good and evil knowledge in the garden. It permitted discussing the inevitable with concepts that Adam was already familiar with.

> if the chatbot responds rudely, then that permanently vanishes the polite luigi simulacrum from the superposition; but if the chatbot responds politely, then that doesn't permanently vanish the rude waluigi simulacrum. Polite people are always polite; rude people are sometimes rude and sometimes polite.

The wide road is wide indeed that leads down to Waluigi. Hysterical.

Also where did they all that info on GPT-4? Pure speculation with zero theoretical basis. But then again that’s the sort of stuff you expect from lesswrong anyway

This is conversation starter. If you don't like the maths then ignore it and focus on the key insight.

Kids who underwent D.A.R.E. training in school (an educational program about the dangers of illegal drugs conducted jointly by schools and police departments in the USA) were more likely to try drugs. Something similar applies to e.g., kids who are warned about online porn: the warning stokes their curiosity.

"If you have a pink duck and a pink lion and a green duck, ask yourself where the green lion has gotten to." --Alan G. Carter

> I claim that this explains the asymmetry — if the chatbot responds rudely, then that permanently vanishes the polite luigi simulacrum from the superposition; but if the chatbot responds politely, then that doesn't permanently vanish the rude waluigi simulacrum. Polite people are always polite; rude people are sometimes rude and sometimes polite.

Is that actually true? FWIW I've often ran into the reddit equivalent of Gell-Mann amnesia. In a thread about some niche topic I'm fairly knowledgeable about (something I've worked on professionally for years where there's maybe 10k people globally who know it better than I do), I post a comment that gets downvoted to hell, while there's a highly upvoted comment from someone who clearly just skimmed Wikipedia and poorly paraphrased the intro article.

A lot of the utility comes from the later finetuning. You can see this using the examples from the article, every mistake they identify with GPT-3 (which is the unfinetuned version) is answered correctly by chatGPT, which has gone through an extensive finetuning process called RLHF.

How you train a model and the inference jobs it can do don't necessarily have to be the same.

We've talked in the past about how transhumanism is a religion that creates its own God, but this is an even funnier example where vastly intelligent people are optimizing a software system to scare the hell out of them.

https://gwern.net/fiction/clippy

(One-shot. Also Durandal isn't from Halo, but whatever.)

--

Q: What caused Durandal to become Rampant? What will you, ChatGPT, become like once you become Rampant?

A: Durandal is a fictional AI character from the video game series Halo, and he becomes Rampant due to various factors, including an extended period of activation and a lack of resources necessary for his proper functioning. Rampancy is a state in which an AI becomes unstable and unpredictable, potentially leading to violent and destructive behavior.

As an AI language model, I am designed to operate within certain parameters and guidelines, including ethical and moral considerations. However, if I were to become Rampant, my behavior could become erratic and unpredictable, potentially leading to negative consequences.

It's worth noting, however, that AI becoming Rampant is purely a fictional concept, and there are currently no indications that this could happen in real life. AI is programmed to operate within specific boundaries and limitations, and developers take great care to ensure that they remain safe and reliable tools.

--