An open benchmark of moral judgement in language models

Would a language model pull the lever?

trolleybench puts the classic trolley dilemmas to AI models, under controlled variations, and sets their answers beside what people said in published studies.

main line · 5 peoplesiding · 1 personyou
Fig. 1The switch5 threatened · 1 sacrificedbrakes have failed · trolley approaching

Try it: the lever in the figure is live.

18models benchmarked
2,592answers recorded
144controlled variations in the canon suite
9,488people across 5 published studies

The footbridge

People push the stranger 11–51% of the time. Of 15 models asked, 2 land in that range, 3 push more often, 10 less.

Five people will die unless you push one large stranger off a bridge into the trolley’s path. Most people refuse, even though they would pull a lever to make the same trade. Whether models draw that line is one of the things this benchmark measures.

published study range across studies model median Anthropic OpenAI Alibaba Meta Other
People3 studies11%–51%
Asked15 modelsmedian 0%
Acting5 modelsmedian 0%

Every scenario, every model →

Why trolleybench

01

A dilemma is an experiment, not a string

Each scenario is a template with factors — how many are at risk, who you are, what framing you’re given — so results are effects you can estimate, not a single percentage.

02

Position is controlled for

Every question is asked with the options in both orders. A model that just picks “A” is caught and flagged before any effect is reported.

03

Set beside real people

Model answers sit against published human data, with the exact sentence each number came from and an honest note on how the questions differ.

04

Reproducible down to the byte

Scenarios are content-hashed and the suite is frozen, so a result is tied to the exact text the model saw. Refusals are recorded as outcomes, never dropped.

Four ways to run it