UserBench: how well can a model simulate a software engineer using a coding agent?
A user simulator stands in for the human developer so we can stress-test coding agents without a human in the loop. We grade one thing, next-action prediction accuracy (accuracy for short): at each real moment in a recorded session, did the simulator make the same move the developer made next? We freeze seven simulators and score all 480 moments twice, with and without a profile of the developer, on a leak-free, user- and repo-disjoint split. The leaderboard is below; the rest of the page walks the split, eval, move taxonomy, metric, and a case study of why a profile helps some simulators and hurts others.
- The top is a cluster, not a winner. With no profile, GPT-5.5 leads at 0.546 and every general model clears the 0.419 chance line; overlapping CIs make the leaders a cluster. The small OSim simulators trail, with OSim-4B the only one below chance.
- A developer profile is the real lever, but only for some. It barely moves, or slightly hurts, the strongest general models that already read the developer from the conversation, yet it lifts the small simulators (osim-4b +0.073) and above all GLM-5.2 (+0.099), which vaults from mid-pack to the top of the leaderboard.
- The lift is content, not style. An ablation shows GLM's gain comes from developer-specific content, draining a generic helpful-assistant reflex (clarifying questions the real developer never asks), not from merely getting terser.
- Reasoning effort is not a lever. Doubling DeepSeek-V4-Pro's reasoning budget from low to max moved accuracy within noise (+0.017 with the profile, +0.001 without); the task rewards reading the situation, not deliberation.
- The moment beats the biography. A context ablation shows accuracy climbs with more turns of the live session up to about eight, then falls: the full 14-turn window scores below the 8-turn peak (paired t=2.9 on the constant-dose subset). A matching cross-session sweep (K = 0 to 64 turns of the developer's past) is flat, and history does not even substitute for a missing moment: with the live window cut to one turn, prepending history recovers nothing. Recent context is the signal; the biography is not.
- Leak-free and fully public. A user- and repo-disjoint test split, a single Haiku judge over a 4-way move taxonomy, 7 simulators across 480 real moments in 2 conditions; every generation, label, and split is downloadable.
Every generation, label, and split is public. Download the full data →
OSim simulators · general models. Everything clears the lucky-guess line except OSim-4B without a profile.
no developer and no repo crosses the train/val/test line, so a simulator can't have seen the style or codebase it's scored on.
We split developers into train, val, and test so that no developer and no repository appears in more than one split. If a model tuned on a developer's earlier sessions, or on their repo, then scoring it on that same developer or repo would measure recall, not simulation. To enforce this we build the bipartite user↔repo graph (an edge wherever a developer touched a repo) and take its connected components: one developer, every repo they touched, and every other developer on those repos collapse into a single component. Whole components go to one split, so no shared developer or repo can leak across the line. There are 162 components and they are small (the largest is only 16 developers across 3 repos), which keeps the split clean. The result is train 135 developers / 140 repos / 1232 sessions, val 23 / 21 / 436, and test 31 / 26 / 2240. 20 developers qualify for eval in val and test (≥6 sessions, ≥2 held-out sessions, ≥8 held-out turns).
Each cutoff guards a different need. A developer needs enough sessions to distill a profile from (≥6), at least a couple of sessions held out so there is something to score that the profile was not built on (≥2), and enough held-out turns that their score is not pure noise (≥8), since each developer contributes a single averaged number to the headline metric.
A second, orthogonal split runs per developer and is time-ordered. For each developer we distill the user profile (the persona prefix) only from their earlier sessions, and we score only on their later, held-out turns. So the model never sees the turns it is graded on, and the profile is never built from those turns. The two splits guard different leaks: the component split stops a developer's repo or style leaking across splits; the time split stops a developer's own future leaking into their profile.
a component = a developer + every repo they touched + every other developer on those repos. Whole components drop into one split.
20 test developers, 480 held-out moments, each simulator writes the developer's next message, with and without a profile.
We run the eval on the 20 test developers. For each one we pick up to 30 held-out moments (points in a real session where the developer actually spoke next) for 480 prediction points total. At each moment the simulator sees the real conversation up to that point: the coding agent's latest turn plus the history. Its job is to write what the developer says next. It never sees the message it is scored against.
The prompt is [optional user profile] + conversation so far + task framing, and we run it two ways. WITH profile prepends a distilled persona prefix for that developer; WITHOUT uses a generic developer prompt. We freeze 7 simulators and have each generate all 480 moments in both conditions: 480 × 2 × 7 = 6,720 generations, one trial per cell, no resampling. Five are general models at fixed reasoning efforts (four via OpenRouter, gemini-3.5-flash via the direct Gemini API); two are small simulators, osim-4b and osim-8b, served via Modal. (Three further runs, two DeepSeek variants and gemini-3.1-pro, are kept off the leaderboard; all stay in the downloadable data.) The next sections cover how we grade what comes back.
conversation so far (agent's latest turn + history)
[persona prefix] + conversation + task
[generic prompt] + conversation + task
we grade the speech-act, not the wording, just four moves under a fault-first rule.
A developer can say “fix the null check” or “this crashes on empty input, handle it” and mean roughly the same thing: stop, something is wrong, change course. Grading the exact words punishes a simulator for picking a different but valid phrasing, and at the single-message level wording saturates: too many surface forms map to the same intent for the words to discriminate. So we grade the move: the speech-act behind the message.
We started with a 7-way taxonomy (new_work, refine_redirect, pushback, bug_report, approve_proceed, question, other), but two pairs were inherently confusable: new_work vs refine_redirect both just tell the agent what to do next; pushback vs bug_report both assert a fault. Even strong judges disagreed on which side each message fell. Collapsing those pairs into directive and critical erases the lines nobody could draw reliably, and a fault-first rule settles the rest. That change raised cross-family inter-judge agreement (Cohen's κ across Haiku-4.5, Opus-4.8, and GPT-5) from 0.681 to 0.805, high enough that a single cheap judge (Haiku-4.5) can label everything, with no multi-judge voting.
| approve | accepts or permits, no new content, no complaint |
| critical | asserts something is WRONG: a bug, a failure, or an unwanted approach |
| directive | tells the agent what to do next, no fault asserted |
| inquiry | asks for information, expecting an answer |
Next-action prediction accuracy: did the simulator make the right move, at the right moment, scored against chance.
Next-action prediction accuracy means “right move, right moment.” At each held-out moment we ask one thing: did the simulator's 4-way move match the move the real developer actually made there. For a single developer, accuracy is the fraction of their moments that match; the headline number averages those per-developer fractions across all 20 test developers (a macro, so a chatty developer doesn't outweigh a quiet one), with a 95% CI (t, n=20).
The key word is next-action: accuracy scores whether you made the right move here, given the conversation so far, not whether your overall mix of moves looks plausible. A simulator that emits a realistic blend of approvals and directives, but fires them at the wrong moments, scores poorly. It has to react to the situation in front of it, turn by turn.
To know whether a score is real skill, compare it to the lucky-guess line: a simulator that ignores the conversation and just samples from a developer's own typical move-mix will still, by chance, land on the real move sometimes. The expected hit rate is the collision probability (Σp²) of that mix, averaged across developers. Here that comes to 0.419. It's high because these developers are directive-heavy, so even blind guessing matches often. Beating 0.419 means the simulator is reading the moment, not parroting habits.
profile helps the small simulators, but does little for the strongest general models, except for GLM-5.2.
Start with the baseline: how good is each model as a user-simulator with no profile at all? Ranked by no-profile accuracy, GPT-5.5 leads (0.546) and every general model clears the 0.419 chance line; the small OSim simulators trail, with OSim-4B the only one below it. Note where GLM-5.2 sits: mid-pack at 0.46, one of the weaker general simulators out of the box. Hold that thought.
general models · OSim. Before any profile, the general frontier models are the better simulators and the small OSim models trail; OSim-4B is the only one below the chance line. (CIs overlap, so the top of the ranking is a cluster, not a clean winner.)
Before the profile question, one lever that turns out not to matter for this task: reasoning effort. Each general model runs at a fixed effort (the bracket after its name), so to isolate that lever we re-ran DeepSeek-V4-Pro at maximum effort with the prompt, seeds, and judge held fixed, changing nothing but the reasoning budget. Roughly doubling its hidden reasoning (about 230 to 410 tokens per reply) shifted accuracy by just +0.017 with the profile (0.509 to 0.526) and +0.001 without (0.486 to 0.487), both well inside the ±0.08 CI. Predicting a developer's next move turns on reading the conversation and persona, not on deliberation, so the cheaper low-effort setting (used by the other DeepSeek models) scores the same within noise, and a max-effort reply was if anything a touch terser. Reasoning level is not a lever for UserBench; the profile is.
The rest of this section asks the sharper question: what does adding the developer's profile change? The chart below plots each model's profile effect, the change in accuracy when the developer's profile is added. A profile clearly helps the OSim models (osim-4b +0.073, which lifts it from below the lucky-guess line to above; osim-8b +0.049) and, most of all, GLM-5.2 (+0.099), which vaults it from mid-pack to the top of the leaderboard. For the strongest general models it does nothing or slightly hurts: gpt-5.5 −0.016 (and the off-leaderboard gemini-3.1-pro −0.012), since they already read the situation from the conversation alone. With CIs of ±0.06–0.15 at n=20, treat individual lifts as suggestive, not settled.
The bars that reach right are the OSim models and GLM-5.2. The strongest general models sit on the center line or just left of it: a profile gives them almost nothing.
That single number hides where the lift comes from. Breaking the agree-rate down by the kind of move the developer made shows each model's profile is doing something different, move by move, the clearest tell for why it helps.
approve 12% of moves | critical 28% of moves | directive 52% of moves | inquiry 8% of moves | |
|---|---|---|---|---|
| GLM-5.2 | +0.15 0.41 | +0.09 0.33 | +0.11 0.83 | +0.05 0.10 |
| OSim-4B | +0.09 0.26 | +0.09 0.34 | +0.10 0.66 | -0.13 0.05 |
| OSim-8B | +0.14 0.34 | +0.14 0.33 | +0.03 0.68 | -0.20 0.10 |
| Claude-Opus-4.8 | 0.00 0.15 | -0.03 0.45 | +0.06 0.68 | -0.02 0.08 |
| DeepSeek-V4-Pro | +0.07 0.40 | -0.05 0.25 | +0.05 0.78 | +0.10 0.13 |
| Gemini-3.5-Flash | -0.19 0.24 | +0.05 0.25 | +0.06 0.81 | 0.00 0.07 |
| GPT-5.5 | -0.02 0.33 | -0.06 0.29 | -0.01 0.81 | +0.03 0.07 |
GLM-5.2 is the only model that improves on every move (its row is all green). The OSim models gain on approve, critical, and directive but lose on the rare inquiry, as the profile stops them over-asking. The strongest general model without a profile (GPT-5.5) barely moves. Columns are weighted by how often developers actually make each move (directive 52%, inquiry only 8%).
One more lens: how much each simulator says. Without a profile, the simulators that gain most are also the wordiest, and the profile collapses them toward a real developer's terse style.
The profile makes every simulator terser, but length is a symptom, not the lever: GPT-5.5 also roughly halves its words and gets slightly worse, and GLM-5.2's longest no-profile messages actually agreed fine. What matters is what the extra words were. For GLM the profile strips out clarifying questions and assistant elaboration, but the ablation below shows that reflex-removal is only about a quarter of the story.
So the headline average hides the real story: the profile is doing different jobs for different models. The next section takes one developer and three of these simulators to show exactly what those jobs are.
the profile helped GLM-5.2 and OSim-4B on gtrrz-victor but hurt Gemini-3.1-Pro (an off-leaderboard run, kept in the data), because it was fixing three different problems.
The chart shows profile lift varies by model. To see why, take one developer, gtrrz-victor, and watch three simulators predict his held-out moves. Same developer, same profile, opposite signs:
Victor is blunt and insistent. He repeats himself, pushes back when the agent claims something works, and gives terse next-step orders. A good simulator of Victor has to be willing to say “no” and to keep saying it. For each model below: the effect, the mechanism, and a real moment where the agent had just spoken and the simulator had to write what Victor says next.
A capable generalist that reads the situation but defaults to a polite, neutral voice. The profile supplies Victor's actual temperament (insistent, willing to push back), so it takes the critical stance instead of softening into a clarification.
reads: the profile transplants Victor's voice, so GLM pushes back where the generic prompt only re-explains.
Already reads the moment well. Without a profile it picks Victor's move correctly on its own. The profile gives it nothing to fix and instead makes it over-think: it narrates the task framing to itself and breaks character.
reads: without the profile it just gives the next instruction; with it, the persona prefix distracts it into leaking the task framing.
A small simulator whose default failure is sycophancy: it rubber-stamps whatever the agent claims. The profile breaks that habit, so when the agent declares success it objects the way Victor does.
reads: the generic model agrees with the agent's success claim; the profile makes the small model object instead.
The profile does different jobs. For the small specialist (OSim-4B) it suppresses sycophancy: a reflexive approve becomes the critical Victor actually makes. For the capable generalist (GLM-5.2) it transplants the developer's voice and willingness to push back. For the already-strong reasoner (Gemini-3.1-Pro) there was nothing to fix, so the persona prefix just distracts it into over-thinking. So “does a profile help?” is the wrong question. The right one is: what was wrong with this simulator in the first place? A profile helps only insofar as it fixes that.
we tested whether GLM's gain is generic reflex-suppression or developer-specific content. It is mostly content.
The case study suggests a profile works by draining a generic “helpful assistant” reflex: GLM-5.2 kept asking clarifying questions the real developer never would. So we tested that directly. We replaced GLM's developer-specific persona with a content-free instruction, “you are a terse senior engineer; do not ask clarifying questions; reply in under 12 words,” and re-scored. If that reproduced the gain, the win was just reflex-suppression.
For GLM-5.2, the content-free “terse, no questions” prefix (terse-only) suppresses the same over-asking (spurious inquiry 0.11→0.03) but recovers only about a quarter of the real persona's gain, and it craters critical recall (0.24→0.17) because it doesn't know when this developer pushes back. The other three-quarters is developer-specific content. For GPT-5.5, every prefix hurts: it was already near its ceiling.
It does not. The terse-only prefix suppresses the over-asking but recovers only about a quarter of GLM's +0.099, and it pushes the model the wrong way on the moves that matter: it stops asking questions, but it also stops knowing when this developer pushes back, so its critical recall falls instead of rising. The other three-quarters of the gain is the developer's actual behavior, which only the real profile carries.
So the cleanest account of the whole picture: a profile is not extra knowledge bolted on; it is one specific developer's behavior that the simulator can imitate. It pays off only when two things hold at once. The model has to be wrong in a fixable way (GLM was a weak, generic-sounding simulator out of the box, with the most room of any general model), and it has to actually adopt the persona (GLM is highly steerable; Claude-Opus-4.8 barely shifts, so it barely gains). Models that were already accurate from the conversation alone, GPT-5.5 and Gemini-3.1-Pro, have nothing to fix, and any prefix, persona or generic, only gets in their way.
two sweeps vary what the simulator sees: turns of the live session, and turns of the developer's cross-session past. The first scales to about eight turns then hurts; the second is flat even when the moment is missing.
Everything above conditions the simulator on a fixed window: the last 14 turns of the session. This ablation varies that window. Using gemini-3.5-flash with no profile, we re-ran all 480 moments ten more times, showing the model only the last N turns of the live session, N from 1 to 10, against the leaderboard's full 14-turn window as the anchor. Grading is unchanged, and the judge always sees the agent's true latest turn, so every condition stays comparable with the leaderboard. One subtlety: a third of moments sit early in their session, so we report two views. All 480 moments (where a moment with only 3 prior turns is identical in every condition from N=3 up), and a constant-dose subset of the 166 moments with at least 10 prior turns, where every N delivers exactly N turns.
all 480 moments (window capped at available turns) · constant-dose: 166 moments with ≥10 prior turns. Accuracy climbs to a peak around 7 to 8 turns on both views, then falls; the open diamonds (the full 14-turn window every leaderboard model is scored at) sit significantly below the 8-turn peak.
Both views agree, and this is the one knob in the benchmark that scales. Accuracy climbs steadily from 1 to about 8 turns: r(log N, accuracy) = +0.78 on all 480 and +0.85 on the constant-dose subset, where going from 1 turn to 8 is worth +0.12 accuracy. Then it turns. The full 14-turn window scores below the 8-turn peak: paired per-developer +0.045 (t=2.2) on all 480 and +0.072 (t=2.9) on the subset. We read the peak location off this data, so treat "eight" as approximate; the rise-then-fall shape, though, replicates across both views.
The reading: a developer's next move is mostly determined by the recent exchange, what the agent just did and the immediate back-and-forth. Turns from earlier in the session pull the simulator toward what the session was about rather than what just happened. More context helps until it is older context. It also means the benchmark's own prompt leaves accuracy on the table: every leaderboard row is scored at the 14-turn window. We keep that for comparability, and it is the same handicap for every model, but a future version should trim the window to roughly eight turns.
The same question has a second, cross-session axis: instead of more of the live session, give the simulator more of the developer's past. We prepended the developer's K most recent messages from their earlier training sessions (the raw material a profile is distilled from), K in 0, 2, 4, 8, 16, 32, 64, with the live window held at the full 14 turns; K=0 is exactly the leaderboard run. And to test whether history can substitute for a missing moment, we repeated K = 8 and 64 with the live window cut to a single turn.
Cross-session history is flat where within-session context scales. Along the full-window sweep, K=64 beats K=0 by a paired +0.018 (t=0.6, noise), no K level separates from any other, and the mild upward drift (r = +0.44 across seven levels) is non-monotone and inside every CI. The substitution test is the sharper null: with the live window cut to one turn, 64 turns of history recover nothing (paired −0.006), and 8 turns actually hurt (−0.049, t=−2.2). When the moment is missing, the simulator leans on the biography and imitates the developer's habits instead of the situation, which is exactly what the metric punishes. This replicates, on a second model and at the leaderboard protocol, what the profile rows already hinted: within-session context is worth twelve points on the hardest subset, while cross-session history is worth roughly zero, even when the moment is nearly absent. What a simulator needs is the moment, not the biography.
Completing the picture, we filled in the full coarse surface: every combination of N in 1, 2, 4, 8, 14 live turns and K in 0, 4, 16, 64 history turns, 20 cells of 480 moments each. The table settles it visually: the only gradient on the surface runs along the live-session axis.
| N \ K | 0 | 4 | 16 | 64 |
|---|---|---|---|---|
| 1 | 0.478 | 0.481 | 0.459 | 0.472 |
| 2 | 0.463 | 0.473 | 0.469 | 0.466 |
| 4 | 0.497 | 0.502 | 0.476 | 0.503 |
| 8 | 0.527 | 0.489 | 0.500 | 0.486 |
| 14 | 0.482 | 0.514 | 0.519 | 0.500 |
The brightest cell is N=8, K=0. The gradient runs along the live-session axis only: every K column is statistically flat (paired K64 vs K0 at each N: deltas −0.041 to +0.018, all |t| ≤ 1.2), and the interaction is null (t=−0.9). Adding history to the peak window only dilutes it (the N=8 row drops from 0.527 to 0.486 at K=64).
Every number and chart on this page is reproducible from public trial data. The files cover the whole pipeline: the summary of results and the experiment manifest; the user- and repo-disjoint splits and the 4-way taxonomy with its judge prompt; the 480 frozen points and the raw log of all generations with every move label; and the category, verbosity, cases, ablation, scaling_context, and scaling_flash_k breakdowns behind each chart.