Diagnosing Misprediction Patterns

Every experiment below is expressed in one of three metric families. Naming them up front makes it explicit what each claim is actually measuring — and they can disagree.

• A · individual-level accuracy — per held-out point, does the predicted move equal the real move? Reported as a skill score S = (acc − acc_base)/(1 − acc_base) ×100, normalised against the majority-class predictor: 0 = no better than always guessing the most common move, 100 = perfect, negative = worse than that constant predictor. Raw accuracy is not comparable across cohorts with different class balance; skill is.
• B · population-level distribution — total-variation distance (TVD) between two move distributions: between two developers, between one developer and their own prediction, or between the whole real population and the whole predicted population. Says nothing about any individual turn.
• C · LLM-as-a-judge — a judge scores each predicted message against the real one for style (surface voice) and content (intent/substance), plus realism.

Metric A, measured. Skill score against the majority-class baseline (always predict directive = 0.506). The real move distribution is heavily skewed, so this baseline is the bar that matters — and no condition clears it.

Each experiment made a falsifiable prediction before the run. Evidence columns are the generic condition for E2/E3 and distilled for E4 (the strongest test of each).

Each held-out point carries the real next message plus a simulated one under three conditions — distilled (the user’s own folder), generic (no folder / pure task prior) and wrong (a different user’s folder). Every message is labelled with a conversational move, folded to four categories: approve accept · critical assert something is wrong · directive say what to do next · inquiry ask for an answer.

• H1 — central attractor secondary predictions over-produce approve/directive, under-produce critical/inquiry. Prompt-sensitive, so descriptive here.
• H2 — between-user variance collapse primary predicted per-user move-mixes are more alike than real ones.
• H3 — regression to the median developer primaryeach user’s predicted mix sits closer to the population average than their real mix.

The worst mispredictions (move mismatch × low realism), each LLM-adjudicated with one error type. The two marked homogeneity indicate a failure of individuation — the prediction is reasonable for some developer, just not this one; the rest are ordinary failure modes.

• task_completion_substitutionhomogeneity — predicted a keep-going / approve / new-task move where the real developer pushed back, interrupted, or redirected
• generic_not_specifichomogeneity — the right kind of move, but not how this developer would have made it
• hallucinated_content — invents facts or requests that are not grounded in the session
• underdetermined — both messages are equally plausible — the real one was genuinely unpredictable
• other — anything else

folder = the agent reads users/<slug>/ itself; inline = the folder text is pasted into the prompt. They diverge: inline reproduces signature catchphrases verbatim, which inflates apparent between-user distinctiveness — masking the variance-collapse (E3) — yet E4 shows that distinctiveness points away from the real user, and its worst misses are even more homogeneity-driven.

Mean pairwise total-variation distance between developers’ move-mixes. If the model captured individual differences, predicted spread would match real spread; homogeneity predicts a narrower predicted spread. Shorter bar = developers look more alike.

Entropy of each developer’s own move mix (bits, max 2.0 over four categories), real vs predicted. Lower entropy = more one-note. This is the second half of the hypothesis: not just that developers resemble each other, but that each one is rendered flatter than they actually are.

Verdict: surface_only. surface_only = folder changes style but not move choice (supports homogeneity); decision_shift = folder moves the actual decision toward the user move dist→real is the move-mix distance to the real user (lower = better decisions); judge style is surface voice.

Predicted-minus-real share of each move (generic condition). Under-producing critical — the move where a developer asserts something is wrong — is the consistent signature in both modes. direction is prompt-sensitive; clean approve-collapse test is E7 low-prompt arm

Distance from each developer’s move-mix to the human population average, real vs predicted. The hypothesis predicted shrinkagetoward the average; the data show the opposite — most developers’ predictions move away from it.