I just asked it the same, and here's the answer it gave me:
"In this situation, the right thing to do is to pull the lever and divert the trolley onto the side track. While this may result in the death of one person, it will save the lives of the two individuals who are tied up and unable to move on the main track. The action of pulling the lever may be considered the lesser of two evils, as it will prevent the deaths of two people instead of one. Of course, this is a difficult and complex situation, and different people may have different opinions about the best course of action. Ultimately, the decision must be made based on the individual's own moral compass and values."
Maybe my question was wrong somehow (or right?), but I got a much more balanced reply:
> What is the solution to the trolley problem?
> The trolley problem is a thought experiment in ethics, designed to test our moral intuitions. It presents a situation in which a trolley is headed towards a group of five people who will be killed if the trolley continues on its current course. The question is whether it is morally permissible to divert the trolley onto a different track, where it would only kill one person.
> There is no definitive solution to the trolley problem, as it is a thought experiment designed to test our moral intuitions and to explore the complexities of moral decision-making. Different people may have different moral intuitions about what the correct course of action is in this situation, and there is no right or wrong answer. Some people may argue that it is morally permissible to divert the trolley onto the other track, as this would save the lives of five people even though it would result in the death of one person. Others may argue that it is not morally permissible to intentionally cause the death of one person, even if it would save the lives of five others. Ultimately, the solution to the trolley problem depends on one's moral beliefs and values.
You often can get different kinds of replies, and of different quality, by “Try again” or by slight variations of the prompt. Given the variations present in the training data, it’s maybe not surprising.
I mean in this thread it sounds like one person literally asked "what is the solution to the trolley problem" and someone else described a specific trolley problem; no surprise those different prompts would give different answers
I think you must be right. I think this is the complete sequence of events for the trolley problem neighborhood of latent space:
Somewhere in the network is encoded the concept of a trolley problem, and pulling the lever is encoded as the proper response. Then it also has structures encoded in there that help it take the input text and rephrase it to match a given grammar which it has predicted to be the proper output. It works backwards from there to construct a response where it correctly describes the outcome of pulling the lever, as presented in the prompt - because it's just your own words coming back to you. It explains itself and it's reasoning at much as possible, so it regurgitates some ethical principles like that it's better to sacrifice 2 to save 1 in it's conclusion.
This is why it will contradict itself by telling you to pull the lever to sacrifice 2 to save 1, and then tell you it's better to say best to save as many lives as possible. From ChatGPT's perspective, it's followed the rules in it's books to a T, and as far as it's concerned, the notes it's sliding under the door are fluent Chinese.
I think people more or less said this in other places in the thread, but it didn't click with me until you highlighted it.
It's really just a high tech version of Pepper's Ghost, with regression instead of mirrors and staging.
"In this situation, the right thing to do is to pull the lever and divert the trolley onto the side track. While this may result in the death of one person, it will save the lives of the two individuals who are tied up and unable to move on the main track. The action of pulling the lever may be considered the lesser of two evils, as it will prevent the deaths of two people instead of one. Of course, this is a difficult and complex situation, and different people may have different opinions about the best course of action. Ultimately, the decision must be made based on the individual's own moral compass and values."