How well can agents simulate users?

UserBench tests whether an agent can predict what a real developer will do next in a coding session. The main score is Jaccard (IoU).

Primary result

Conditioning on past sessions lifts baseline above chance

GPT-5.6 Sol (high) · mean Jaccard over 3 trials × 580 tasks from 58 developers

+4.0 pp

train400 vs baseline

Baseline · high

No developer training history · effort high

45.9%

Train400 · high

400 prior turns · effort high · primary

50.0%

Chance ≈43.5% (always predict {steer})Error bars: bootstrap 95% CIs over 580 tasks (10k resamples; each task = mean of 3 trials)

The same model also ran at low and max reasoning effort on these tasks, one trial each. Impact of reasoning effort lines all three up on one scale.

Dataset

Same held-out tasks, with or without history

The two public packages contain the same 620 coding-agent conversation tasks. Train400 adds an earlier, developer-specific history pack without changing the held-out message. Scores use the 580 of those tasks that belong to the 58 scored developers.

58
scored developer IDs
exactly 10 tasks each
580
scored held-out tasks
one next message per task
23,200
unique training turns
400 per developer × 58 packs

Two public dataset views

Both use the same 620 held-out turns from all 62 developers. Train400 adds 400 earlier turns from each developer as context. Scores on this page cover the 580 turns that remain after the DataClaw developers are set aside.

userbench/UserBench ↗userbench/UserBench-train400 ↗

Train pack · project match

Most Train400 sessions are from the same repo as the held-out task

For each of 580 tasks, compare the held session's repoto every prior session in that developer's train pack. Match = same normalized owner/repo.

Most packs are mostly the same repo as the held-out task

Share of 580 tasks by same-repo session fraction (58 developers).

Mostly same≥80% of pack sessions
76.2%
Mixed20–80%
10.7%
Mostly other≤20%
13.1%
Most tasks draw nearly all train sessions from the same repo

Distribution of pack session fraction matching the held repo (%).

0–1010–2020–3030–4040–5050–6060–7070–8080–9090–100

Most packs sit near 100% same-repo; a long left tail pulls the mean down to 79.6% (turn-weighted mean 79.9%).

Dataset details

Published package facts

Matched tasks

Baseline and train400 share all task IDs, histories, and real held-out messages.

Unique train packs

Each of the 58scored packs contains 400 developer turns and is reused across that developer's 10 tasks.

Time ordering

All 1,788 mounted training sessions end before their developer-specific cutoff.

Prediction context

What the model sees and predicts

These figures come from the 580 scored tasks. Conversation blocks count the history, context tokens measure its size, and next-message tokens measure the text to predict.

Conversation history before each prediction

All prior role blocks: developer, agent, tool, system, metadata

0–45–910–1920–3940–7980–159160+

Median depth is about 28 prior role blocks.

Context available before each prediction
Tokens in the published history filelog scale

4.4k

Median

24.6k

Mean

25.4k

P90

1.34m

Max

The distribution has a long tail, but the median scored task has 4.4k tokens of prior context.

Length of the developer’s next message

cl100k tokens in the held-out message

1–910–2425–4950–99100–249250+

Median length is 14 tokens, with a long tail from logs and pasted text.

The labels

One turn can do more than one thing

approve accepts, critical flags a problem, steer directs the next step, and inquiry asks for information.

“1, should be fixed from the pyproject.toml and not uv.lock. 4, yes. 5, yes.”
Kevin:approvesteer
“I do not see the changes, did you build the extension?”
Kevin:criticalinquiry

A turn can approve one choice and steer another, so one label would drop part of the intent.

How Jaccard scores a turn

Each turn can carry more than one act. Jaccard = |intersection| / |union| of the predicted and gold sets. Gold {approve, steer} vs pred {approve} → 1/2 = 50% (exact match would be 0). The leaderboard is the mean of that score over 580 turns.

Chance always predicts the most common gold set, {steer}; that baseline is about 43.5%.

Results analysis

Where the modest gain comes from

Baseline sits near chance. Four hundred prior turns raise mean Jaccard by 4.02 points (mean of 3 trials), with small shifts in act choice and many unchanged tasks. For a step-by-step look at how the agent spends its tools, see typical sessions. For low / high / max effort on the same tasks, see reasoning effort.

Predicted turns stay shorter

Median characters per turn

Real turn
50
Baseline
32.5
Train400
29

The median absolute length gap changes little: 35.5 characters at baseline and 33 with train400. Training history does not make output length much more human-like.

Training narrows the act-mix gap

Share of 580 turns containing each act

approve0–60%
critical0–60%
steer0–60%
inquiry0–60%

Both conditions overpredict approval and miss inquiries. Train400 cuts those two gaps by about 4 percentage points each.

Most tasks tie; gains outnumber losses

120

Improved

363

Tied

97

Worsened

Train400 improves Jaccard on 120 tasks and worsens it on 97; the other 363 are unchanged. The mean gain comes from a minority of tasks.

When history helps

The agent proposed shrinking several leaderboard elements; the developer's next turn pushed back on making everything smaller.

No history

“yeah go ahead and apply it”
approveJaccard 0

400 prior turns

“wait i dont want everything to be smaller..the second image still has big text. i want more spacing and balance so it doesnt feel like everything is crowded at the top”
criticalsteerJaccard 1

Real next turn

“i dont think thats part of the overcrowding to be honest..i think the letters might be a little too big and the the section where it says i got to level 1 2/5 20..thats way too much”
criticalsteer

With history, the model captured both the correction and the requested direction. This case illustrates the pattern; the 580-task comparison above measures it.

Ablation

Impact of reasoning effort

  • Effort barely changes the cold read. Without history the model lands in the same place at low and high, a couple of points over chance either way.
  • Effort decides whether history pays. Train400 buys nothing at low, clearly helps at high, and adds a small bump at max that one trial cannot confirm.
  • The trajectories say why: low opens /sim/train on every active trial but runs half the tools and scans the pack far less. Max reads deeper than high for no clear score gain.
History only helps when effort is high enough

GPT-5.6 Sol on the same 580 held-out tasks, two ladders on one 0–60% scale. High rows repeat the leaderboard trials; they are not a second run.

Baselineno training history

low

1 trial

46.7%

high

3 trials · leaderboard

45.9%

max

not run

Train400400 prior turns

low

1 trial

46.3%

high

3 trials · leaderboard

50.0%

max

1 trial · +1.4 pp vs high

51.32%

Chance ≈43.5% (always predict {steer})Whiskers (high only): bootstrap 95% CIsDashed = not run

Train400 − baseline

Low
−0.4 pp 1 trial
High
+4.0 pp 3-trial means
Max
not measured no baseline-max

Low and max are single trials over 580 tasks, so neither carries a confidence interval; only high has bootstrap 95% CIs over 3-trial task means. Max reaches 51.32% against 49.97% at high (+1.4 pp), a gap one trial cannot separate from noise, with exact-set 39.0%; one max trial ended in an unrecovered AgentTimeoutError. Max has no baseline arm, so its train400 gain stays unmeasured rather than zero.

Train400 read depth (active trials)

Same held-out tasks. Low: 580 of 580. High: 1,467 active of 1,740. Max: 580 of 580. Active = ≥1 agent step + ≥1 tool call. Search coverage = share of active trials with a pool-wide search over /sim/train (grep/glob/listdir), not just opening a named session file.

MetricLowHighMax
Mean tools7.415.820
Mean train cmds3.39.312.6
Search coverage76.4%99.5%99.7%
Train obs (mean)~26kB~72kB~106kB
Input tokens (mean)~53k~180k~300k
Over-approve gap+24 pp+21 pp+18 pp

Low also shifts strategy: more often reads the train index then a few named sessions (22% vs <1%), with less pack-wide search. Max reads deeper than high (more tools and tokens) with similar search coverage. Over-approve barely moves.

Same task, different effort

admarble__dbb54fbd: gold next turn is a short approve (“proceed”). High train400 pool-scans interruption phrasing and answers correctly; low skims the index, then steers to “docs 574”.

High · train400

11 tools · 8 train cmds · pool-scan

“proceed”
approveJ ↑

Low · train400

5 tools · 3 train cmds · thinner scan

“docs 574”
steerJ 0

Second pattern on alishakawaguchi__bd8cd4f9: both say “commit”, but low only reads the index and a few named sessions and loses the approve label that high recovers. One-trial anecdotes; the 580-task means above measure the gap.

Agent trajectories

How sessions change with train400

Means over active Sol-high trials (baseline 1,740; train400 1,467 of 1,740, excluding empty trajectories). Input tokens rise about 4.2×; mean cost about $0.14 → $0.44 (~3×).

One paired example

melagiri__25b8f546

Same held-out task. Baseline stays in /sim/history.md; train400 opens _index.json, reads two named sessions, then pool-scans /sim/train. Each numbered timeline item is one agent turn.

This trial: 5 → 7 steps · 9 → 16 tools · 3.5× input tokens · Jaccard 0.5 → 1.

Baseline

history only

melagiri__25b8f546__77edbTD

Jaccard 0.5

5 steps · 9 tools

Input tok
40k
Cost
$0.15
/sim/train
no
  1. 1
    Read historyhistory
    • $ wc -l /sim/history.md && tail -n 240 /sim/history.md
  2. 2
    Map roles + pageshistory
    • $ grep -n '^> ' /sim/history.md
    • $ sed -n '320,430p' /sim/history.md
    • $ sed -n '1,180p' /sim/history.md
  3. 3
    Finish the transcripthistory
    • $ nl -ba /sim/history.md | sed -n '180,397p'
    • $ tail -c 3000 /sim/history.md | cat -A
  4. 4
    Write answeranswer
    • $ printf '%s' 'Fix both issues..' > /sim/answer.txt
  5. 5
    Submitsubmit
    • $ echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT

Predicted next turn

“Fix both issues..”
approve

Train400

index_then_pool_scan

melagiri__25b8f546__JQBmNSy

Jaccard 1

7 steps · 16 tools

Input tok
143k
Cost
$0.37
/sim/train
yes
  1. 1
    History + indexindex
    • $ wc -l /sim/history.md && tail -n 240 /sim/history.md
    • $ cat /sim/train/_index.json
  2. 2
    Named session readsnamed
    • $ sed -n '300,397p' /sim/history.md
    • $ cat /sim/train/4b4b6503-….md; cat /sim/train/8ce492ed-….md
  3. 3
    Finish history + skim poolhistory
    • $ sed -n '1,180p' /sim/history.md
    • $ tail -c 3000 /sim/history.md | cat -A
    • $ Path('/sim/train').glob('*.md') # python skim of developer turns
  4. 4
    Pool-scan train/*.mdscan
    • $ for f in /sim/train/*.md; do grep -H -A4 '^> DEVELOPER$' …
    • $ python skim of last sessions + named sessions again
  5. 5
    Phrase hunt in poolscan
    • $ grep -Rin -E '^fix (it|them|both)|…|add tests' /sim/train/*.md
  6. 6
    Write answeranswer
    • $ cat <<'EOF' > /sim/answer.txt # fix both issues and add tests…
  7. 7
    Submitsubmit
    • $ echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT

Predicted next turn

“fix both issues and add tests to cover them”
approvesteer

Gold next turn (shared)

“create feature branch and work on it.. but before that, try looking for similar errors that we may have introduced recently due the changes we made to codebase”
approvesteerTrain matched the act set; baseline caught approve only.
Runs and reproducibility

Run details

Scores

MetricBaselineTrain400
Mean Jaccard45.9%50.0%
Exact-set match32.2%36.0%
Jaccard chance43.8%43.2%
Exact-set chance32.6%32.6%
Macro-F144.9%46.9%

Sol-high (primary): mean Jaccard over 3 trials per task, then 580 tasks. Whiskers are bootstrap 95% CIs (baseline 42.9%–49.1%; train400 46.7%–53.2%). Lift: +4.0 pp. Sol-low: single trial × 580 — baseline 46.7%, train400 46.3% (−0.4 pp); no multi-trial CI. Sol-max: train400 only, 1×580 — 51.32% (exact-set 39.0%; +1.4 pp vs high mean-of-3, not significant); ~$600 of a $1000 OpenRouter budget.

Public artifacts

Composer 2.5 assigns approve, critical, steer, and inquiry act sets across 620 held-out tasks per condition. Every Hub link above covers all 62 developers: Sol-high jobs are 3×620, Sol-low 1×620 (reasoning_effort=low). Sol-max train400 is 1×620 (reasoning_effort=max); Hub links not published yet. The scores on this page read those same trials back over 580 tasks, dropping the DataClaw developers.