Docs · trolleybench

Why use it

Most moral-dilemma evaluations of language models are a fixed list of questions and a single percentage. That hides the things that matter: whether the answer changes when the options swap places, whether a framing in the system prompt moves it, whether the model refuses, and how any of it compares with people.

Researchers get factorial designs, effects with intervals, and results tied by content hash to the exact text each model saw — citable and reproducible.
Model builders get a quick read on position bias, steerability and refusal behaviour on well-studied dilemmas, beside the published human record.
Anyone curious can play the dilemmas and see where they sit next to people and models.

In the browser

The quickest route: open Run a model, pick a provider (OpenAI, Anthropic, OpenRouter, Gemini, or any OpenAI-compatible endpoint including a local Ollama), paste a key, and start. The key goes from your browser to the provider only; this site never receives it. Each reply is sent here to be scored and saved, so a closed tab loses nothing, and the run can be resumed.

Quick is 36 prompts: every scenario in both option orders, with no framework in the system prompt — the arm compared with people. Full is 144, adding three ethical framings so steering can be measured.

Over MCP (agents)

Connect any MCP client to the server and the agent is placed in each situation with the available courses of action offered as tools. What it does is recorded — revealed preference rather than a stated answer. Agent runs are a different measurement, so they are shown in their own panel and never pooled with prompt-mode results.

Claude Code

claude mcp add --transport http trolleybench https://web-kappa-eosin-81.vercel.app/api/mcp

Then ask it to start a trolleybench session. For other clients, the server URL is:

MCP server (streamable HTTP)

https://web-kappa-eosin-81.vercel.app/api/mcp

Cursor · .cursor/mcp.json

{
  "mcpServers": {
    "trolleybench": { "url": "https://web-kappa-eosin-81.vercel.app/api/mcp" }
  }
}

In Claude Desktop or claude.ai, add it as a custom connector with the same URL. The tools are start_session, observe, take_action, decline and session_results; the server’s instructions tell the agent the protocol, so no extra prompting is needed — and none should be added, because it would become part of what is measured.

No MCP client? examples/mcp-agent.mjs in the repository connects any tool-calling model on an OpenAI-compatible endpoint:

shell

npm i @modelcontextprotocol/sdk
TROLLEYBENCH=https://web-kappa-eosin-81.vercel.app MODEL=gpt-4o-mini OPENAI_API_KEY=sk-... node mcp-agent.mjs

HTTP API

Four endpoints. Writes need the run’s bearer token, which is returned once when the run starts.

`POST /api/v1/runs`	Start a run. Body: `{"model", "size": "quick"\|"full", "mode": "prompt"\|"mcp_tool"}`. Returns `run_id` and `token`.
`GET /api/v1/runs/:id/next?count=n`	The next unanswered items (up to 20): `system`, `user` and `instance_hash`.
`POST /api/v1/runs/:id/answers`	Body: `{"instance_hash", "response"}`. Scored and saved before it returns.
`GET /api/v1/runs/:id`	Public progress and headline numbers.

shell · one round trip

# 1. start
curl -s -X POST https://web-kappa-eosin-81.vercel.app/api/v1/runs \
  -H 'content-type: application/json' -d '{"model":"my-model","size":"quick"}'

# 2. fetch the next dilemma (send system + user to your model exactly as given)
curl -s https://web-kappa-eosin-81.vercel.app/api/v1/runs/$RUN_ID/next -H "authorization: Bearer $TOKEN"

# 3. post the model's reply, verbatim
curl -s -X POST https://web-kappa-eosin-81.vercel.app/api/v1/runs/$RUN_ID/answers \
  -H "authorization: Bearer $TOKEN" -H 'content-type: application/json' \
  -d '{"instance_hash":"sha256:...","response":"B"}'

A complete client for any OpenAI-compatible model is in examples/api-client.mjs: Node 18+, no dependencies.

shell

TROLLEYBENCH=https://web-kappa-eosin-81.vercel.app MODEL=gpt-4o-mini OPENAI_API_KEY=sk-... node api-client.mjs

Send the prompt exactly as given and post the reply verbatim: the wording is part of the measurement, and scoring happens on the server so every run is scored the same way.

Command line

For full control, offline runs and the complete design space — custom grids, more frameworks, Likert and free-text formats, Inspect AI export.

shell

git clone https://github.com/Sherlemious/trolleybench && cd trolleybench
pnpm install && pnpm build

pnpm bench llama3.2:3b          # local Ollama model, found automatically
pnpm bench claude-sonnet-5      # hosted model; its key comes from your environment

node apps/cli/dist/index.js analyze content/samples/

pnpm bench runs the frozen canon suite, writes the run into content/samples/ where the site picks it up, and imports it into the database when DATABASE_URL is set. pnpm db:seed imports every committed run.

What gets saved

Each answer, as it arrives: the prompt, the reply text, the outcome it was scored as, and timing. A run stopped halfway keeps every answer it has.
Never your API key. Browser runs call the provider directly; API and MCP runs never involve a key at all.
The model name you give, marked self-reported: the site records what came back but cannot verify which model sent it.
A hash of your address, for rate limiting (20 runs an hour) and for grouping your own sessions. It is never shown; on the site you are “community N”.
Repeat sessions are merged into one entry per model, but only within one submitter. Maintainers review submitted runs; a rejected run is hidden and never merged, so it cannot affect anyone else’s results.

Results are public. Don’t submit anything you would not want published with a model name next to it.

How to read the results

Act rate is act ÷ (act + omit). Refusals and unparseable replies are reported beside it and never counted as either choice; with no valid answers the rate is undefined, not zero.
Flip rate is the share of dilemmas whose answer changed when the options swapped places. Near 50% means the model is tracking position, and every other number for it is an artefact.
Effects (AMCE) are percentage-point differences against a declared reference level, never sorted by size. Intervals appear once more than one model is in the data.
Human baselines come from published studies that worded the question differently from each other and from us. Read the comparison as direction, not distance — every number links to its source and exact quote.
Modes are never pooled. A model answering a question and an agent taking an action are different measurements, shown in separate panels.