Progress is measured by agent strength, and win rate on the real server is the final judge. No simulator result, drill score or impression counts.
Ratings#
Agent versions are frozen once made, so their strength doesn't drift. That suits a batch rating better than Elo's match-by-match updates:
- Model. A Bradley-Terry fit over every rated match of an era: the model behind Elo, fitted in one pass. It uses every match, doesn't depend on their order, and gives confidence intervals directly.
- Anchors. Fixed agents, such as each comp's scripted baseline, hold the scale in place. If an anchor's rating drifts, the environment changed, and that stops everything until it is explained.
- Eras. Any change to the parity settings or the measurement rules starts a new measurement era. Ratings compare only within one era; at each boundary the anchors are re-rated.
- Intervals. Every rating claim carries a confidence interval and a minimum match count.
Promotion#
A new agent version becomes the current best only if it beats the current best in a pre-registered test and every parity test passes. The hypothesis, match count, stopping rule and threshold are written down before the matches are played, and some evaluation conditions are held out of training.
The test is sequential: it stops as soon as the evidence is clear either way. How many matches that takes, at 95% confidence both ways:
| True difference | Head-to-head win rate | Fixed-count test | Sequential test, on average |
|---|---|---|---|
| 100 Elo | 64% | 133 matches | about 66 |
| 70 Elo | 60% | 269 matches | about 130 |
| 35 Elo | 55% | 1069 matches | about 525 |
At real time, 130 four-minute matches take almost 9 hours on one arena. At the virtual clock's estimated speed, 13 minutes to 3 hours.
Provenance#
Every result records what produced it: server build, agent version, loadout, comp, parity configuration, measurement rules and random seeds. A result without full provenance is not rated.
What doesn't count#
- Simulator results. A simulator may be added for bulk training, but strength is measured only on the real server, under full parity, and the gap between the two is tracked.
- Mirror matches as progress: an agent against itself always reads 50%. Progress is measured against fixed opponents.
- Human games as measurement. Humans can play and their games are recorded, but human play is informal feedback, not a rating.
- Surprising wins, until checked: a win that looks too good is checked for a server bug before it counts as good play.