AlphaDraft Labs

The judge comes first

Notes on teaching an engine to draft fantasy-football auctions, June 2025 to now.

A fantasy-football auction is a good game to point a learning agent at. Ten teams, a $200 budget each, about three hundred draftable players, fourteen roster spots a side. Budgets are visible to everyone, and the hidden information is what each team privately thinks players are worth. Every dollar spent on one player is unavailable for the rest, and the whole thing runs on a clock. The game is also small enough to simulate completely, and my simulator plays a full draft, nomination by nomination, in about a quarter of a second on a laptop. Since June 2025 I've been testing the usual recipe for games like this — search, a learned evaluator, self-play — under the name AlphaDraft Labs, trying to build a learned drafter that beats the hand-built valuation engine inside NineDraft, the product I ship.

The 2025 agent

In 2025 I trained a policy-gradient agent over ten thousand simulated drafts. It won 8–12% of its games against my own heuristic baseline, and I shelved the project for nine months.

When I came back in June 2026 I started by reading the 2025 code to work out why the agent lost. The observation was a 127-number summary of the draft state, while the heuristic it lost to read the full player table. The action space was one flat 440-way head in which most actions aliased to "pass." The reward was a single sampled season outcome per draft, one noisy number to credit hundreds of decisions. The agent also did no search at decision time, despite a simulator that could play out the rest of any draft in a fraction of a second.

I left the 8–12% in the record as what that setup was worth. I never retrained the old model, and the new work went onto a different architecture.

Fixing the judging

Before building a new agent I fixed the scoring. A finished roster had been scored by sampling one simulated season, so two identical rosters could score a win and a loss. The replacement evaluator simulates many seasons per roster and reports a win probability with a confidence interval. I added a promotion rule on top of it: a new drafter has to out-draft the current one on drafts it hasn't seen, with a confidence interval clear of the coin-flip line, and the decision rule has to be written down before the run.

I write the decision rule down before each run. Most of the lab's experiments are executed by AI coding agents running unattended overnight, dozens of sessions at a stretch, and an agent interpreting its own results afterward tends to favor the reading that makes the run look successful. A rule committed before the data exists prevents that.

Tournament results

Ten copies of the same brain crashed the market. Seat ten instances of the strongest search brain at one table and nobody bids. In a thirty-draft probe the brain deployed 31% of its budget against itself, roughly $138 of every $200 left unspent, while still filling legal lineups 97.7% of the time. Prices collapse and the ten win probabilities flatten toward a tie. I ran the probe before scaling self-play, because the default self-play field is ten clones. The same probe left something I still haven't resolved: seat position seemed to matter, best seat 0.164 win probability against worst 0.061, but the permutation test came back p=0.079 and it stays a suspicion.

Near-identical siblings cannibalized each other. A budget-planning strategy won 61.5% of its matchups as the only member of its family in the field. The strategy made its money exploiting a particular market inefficiency, and each added copy competed for the same one: 43% with one sibling seated, 36% with two, against the same opponents on the same seeds, with 57 of 60 paired drafts moving in the same direction.

A field that was diverse but weak got farmed. Two tournaments differed in only four seats: competitive clones in one arm, weaker strategies of assorted types in the other. Each arm takes about 38 minutes, and this comparison ran twice end to end because the first attempt was orphaned by a crashed session and had to be killed. The six fixed incumbent seats captured 54% of total win probability against the clones and 80% against the mixed field. The weaker strategies spent aggressively, and average unspent money across the field fell from $133 to $15. In that live market the best incumbent more than doubled its win share while the strategies that freed up that money lost ground. There is a confound here, since the seat swap changes spending aggression along with strategy type, and the writeup says so.

Distillation

Through all of this the strongest drafter stayed a search brain: hand-built valuation plus simulation at decision time. At each nomination it rolls out completions of the draft across a price grid and buys below its computed indifference price. It's slow, and it can't improve without me rewriting it. So: distill the search into a small network. Policy targets come from the search's chosen prices and nominations, value targets from the evaluator. The encoder is set-attention over the actual player pool instead of a summary vector. The student decides in 2–5 milliseconds. The teacher takes 20–35.

Most of the training work turned out to be regularization, and I got there late. More self-play data made the student worse, and so did more capacity — wider, deeper and longer-trained students all regressed while training loss kept falling, which is just overfitting. My first read of those regressions was under-fitting, so I grew the student. That was wrong too. Dropout and weight decay took the same small network from a pooled 0.397 against a fixed benchmark opponent to 0.598 across three disjoint seed blocks, with nothing else changed. One loose end I never firmed up: 0.598 is above the teacher's own measured ceiling of 0.535 against that opponent, which is either a real gain from the smoothed targets or that ceiling being a single-seed measurement.

In competitive fields, every distilled student refused to buy quarterbacks, and so did the teacher itself running at sixteen times its normal search budget. I found the cause by replaying a hundred recorded drafts with the search instrumented. The pool holds 27 quarterbacks, and a competitive room buys about 26 of them at a mean price near $48, because opponents hoard backups. The teacher's internal model of how a draft completes assumes teams buy starters only. Under that assumption quarterback demand dries up and the leftovers should cost about $7, so the search declined $48 quarterbacks all draft while it waited for cheap ones that didn't show up, and the students learned the same mistake from its labels. I patched the completion model conservatively. That cut the failure roughly in half, still short of the promotion bar. I stopped distillation work at that point and wrote the failure up as a defect in the teacher's labels.

Redesigning the arena

Three promotion exams collapsed over four days in late July, each one invalidated on procedural grounds before it reached a verdict on the candidate. About $265 of compute. No promotions. One tripwire, built to independently catch degenerate play, turned out to correlate 0.999 with the outcome it was supposed to check. I stopped and reviewed the design instead of running a fourth exam, and the faults were structural. Head-to-head credit against a single frozen incumbent cannot distinguish building a better roster from damaging that particular opponent. A yardstick that is replaced at every promotion means progress cannot be charted, and chained gate-wins are known to inflate. The computer-Go community measured about sevenfold. A single frozen opponent is also easy to overfit.

I rebuilt the arena in August around absolute measurement. A drafter is now scored on the expected fantasy points of the roster it builds, in randomized rooms drawn to look like real leagues, against a frozen set of anchor opponents. A candidate and the incumbent run the same scenario in separate rooms (same opponents, same seat, same player pool) and the paired difference in roster points is the measurement. The incumbent is never at the candidate's table, so there is no way to win by damaging it. Every evaluation gets committed to the repo as one point on a chart. I record inconclusive verdicts too, and every candidate enters the opponent pool whether it was promoted or not.

Status, August 2026

As of this month the strongest drafter in the lab still runs on hand-written valuation. Search did earn one promotion this summer, when replacing its cheap draft-completion estimates with full simulated playouts beat the incumbent under paired evaluation. The learned agents draft legal, competent rosters in simulation and none has beaten the search brain. The shipped product runs the deterministic engine, and nothing learned is in it. What I'd keep from the year is the measurement: the ensemble evaluator and the paired-scenario arena. Every branch has a written decision record, closed ones included.

Next is population evolution, with an LLM writing the strategies as code. Once per generation it reads the tournament results and transcripts and writes a hypothesis about why the winners won. Then it proposes new strategies, which enter the next generation's tournament, and I drop the losing ones from the population. The LLM itself never runs during a draft. Strategy code makes no network calls, so a draft replays deterministically from its seed, which the arena design requires.

Most of the lab's sessions are run by AI coding agents under the rules described here. I picked what they worked on. If something above is wrong, it's my error. — Jaswinder Singh, August 2026