Trial results: public, agent-readable
Raw files from the earlier v1 study: the 7 leaderboard simulators × ±profile over 480 held-out moments on a 20-developer, user- and repo-disjoint SWE-chat test split (the raw files also include 3 off-leaderboard runs: 2 DeepSeek variants and gemini-3.1-pro, which gemini-3.5-flash replaced on the leaderboard; 10 model conditions in total). The Sol-high results on the front page come from a later run; nothing here backs them. Files are public on Vercel Blob. Point your agent at the machine-readable index, or download files directly below.
curl -s https://wykde3bdnvkojpef.public.blob.vercel-storage.com/data/condagree_multi/index.json | jq . # the catalog curl -s https://wykde3bdnvkojpef.public.blob.vercel-storage.com/data/condagree_multi/summary.json # headline accuracy results curl -s https://wykde3bdnvkojpef.public.blob.vercel-storage.com/data/condagree_multi/raw.jsonl \ # every generation + move label (NDJSON) | head -1 | jq .
| File | What it is | Size |
|---|---|---|
| summary.json | Headline results: per (model, condition) next-action prediction accuracy (macro) mean + 95% CI + per-user array; lucky_guess line; real move distribution. | 8 KB |
| manifest.json | Experiment config: the 10 models with backend ids (OpenRouter / Gemini API / Modal) + reasoning efforts + concurrency, split file, taxonomy, judge model, n_per_user, n_points, git sha, timestamp. | 2 KB |
| taxonomy.json | The v2 4-way move taxonomy: categories (approve/critical/directive/inquiry), the exact classifier prompt body, OLD->NEW remap from the legacy 7-way, and inter-judge kappa. | 3 KB |
| splits.json | Train/val/test split with NON-OVERLAPPING users AND repos (connected components of the user<->repo graph); qualifying eval-users per split. | 8 KB |
| points.jsonl | The 480 frozen held-out prediction points (one JSON per line): {point_id, slug, repo, turn_index, prev_agent, real_text}. | 696 KB |
| raw.jsonl | Every trial (one JSON per line). Generation: {key, kind:'gen', point_id, slug, model, model_id, backend, effort, cond('distilled'|'generic'), text, ts, seed?}. Move label: {key:'lab:haiku:<hash>', move}. Join gens to labels via the 4-way classifier on (prev_agent, text). | 6.9 MB |
| cases.json | Case-study data for glm-5.2 / gemini-3.1-pro / osim-4b: per-developer accuracy ±profile (delta), and every moment where the profile flipped the move (with the agent turn, real message+move, and both ±profile generations+moves). | 556 KB |
| category_recall.json | Per-move agree-rate (recall) for all models, both conditions: of moments whose real move was X, the fraction the sim matched. Includes the profile delta per category. Drives the by-category heatmap. | 5 KB |
| verbosity.json | Average + median message length (words) per model, both conditions, vs the real developers; plus each model's accuracy no-profile/with-profile. Drives the verbosity chart. | 2 KB |
| ablation.json | Contrastive-prefix ablation for glm-5.2 + gpt-5.5: accuracy under no-profile vs the real persona vs a content-free terse style prefix, with spurious-inquiry rate, word count, and per-move recall. | 2 KB |
| scaling_context.json | Context-window ablation for gemini-3.5-flash (no profile): accuracy per N=1..10 prior session turns plus the full 14-turn anchor, per-developer values, on all 480 moments and the 166-moment constant-dose (>=10 prior turns) subset. Drives the context-window chart. | 13 KB |
| scaling_flash_k.json | Cross-session history sweep for gemini-3.5-flash (no profile): accuracy per K in {0,2,4,8,16,32,64} most recent train turns prepended at the full 14-turn live window (K=0 = the leaderboard generic run), plus K in {8,64} at a 1-turn live window (substitution test), per-developer values and paired tests. Drives the cross-session chart. | 6 KB |
| scaling_grid.json | The completed coarse N x K surface for gemini-3.5-flash (no profile): accuracy at every (N in {1,2,4,8,14} live turns) x (K in {0,4,16,64} history turns) cell, per-developer values, paired K-effect tests at each N and the N1-vs-N14 interaction test. Drives the grid table. | 12 KB |
raw.jsonl: every trial
NDJSON. Two record types. gen: {key, kind:"gen", point_id, slug, model, model_id, backend, effort, cond, text, ts, seed?}. Move label: {key:"lab:haiku:<hash>", move}. The label hash is over the (prev_agent, message) pair, so labels are shared across identical messages.
points.jsonl: the test set
The 480 frozen held-out moments: {point_id, slug, repo, turn_index, prev_agent, real_text}. Each is one real developer turn the simulators had to predict, with the agent’s prior turn held fixed.
summary.json: results
Per model/condition: accuracy macro mean + 95% CI + the 20 per-developer values, plus the lucky_guess line and the real move distribution. Models: deepseek-v3.1, deepseek-v4-flash, deepseek-v4-pro, gpt-5.5, claude-opus-4.8, glm-5.2, gemini-3.1-pro, osim-4b, osim-8b.
manifest · splits · taxonomy
manifest.json: full config (models, OpenRouter ids, reasoning efforts, concurrency, git sha). splits.json: the user- and repo-disjoint train/val/test split. taxonomy.json: the 4-way move taxonomy + classifier prompt + inter-judge κ.
Where everything lives
- Website: this site, a Next.js static export under web/ on the benchmark-website branch of cooperbench/user.skill.
- Benchmark code: the eval harness, the 4-way move taxonomy + judge, the analysis and ablation scripts, and the Modal serving for the OSim models. Kept in a private repo (cooperbench/user.skill); available on request.
- Data: everything on this page, public on Vercel Blob: the files above plus the machine-readable index.
Prompts are reconstructable from points.jsonl + each developer’s users/<slug>/ folder at manifest.git_sha via validate.build_prompt / osim_backend.build_osim_messages. Data derives from public SWE-chat (ODC-BY).