Progress is measured by agent strength, and win rate on the real server is the final judge. No simulator result, drill score or impression counts.
Ratings#
Agent versions are frozen once made, so their strength doesn't drift. That suits a batch rating better than Elo's match-by-match updates:
- Model. A Bradley-Terry fit over every rated match of an era: the model behind Elo, fitted in one pass. It uses every match, doesn't depend on their order, and gives confidence intervals directly.
- Anchors. Fixed agents hold the scale in place. The zero point has to be an agent whose results can go either way: one that presses random legal buttons, not one that never acts, since a team that never acts loses every match, and a record of nothing but losses can't fix its place on the scale. If an anchor's rating drifts, the environment changed, and that stops everything until it is explained.
- Lopsided results. A lineup that wins, or loses, nearly every one of its matches gets a one-sided bound ("at least 570 Elo above the anchor"), not a point rating with an interval: past a long run of wins the fit can no longer tell how far ahead a lineup is, only that it is at least that far.
- Eras. Any change to the parity settings or the measurement rules starts a new measurement era. Ratings compare only within one era; at each boundary the anchors are re-rated.
- Intervals. Every rating claim carries a confidence interval and a minimum match count.
Promotion#
A new agent version becomes the current best only if it beats the current best in a pre-registered test and every parity test passes. The hypothesis, match count, stopping rule and threshold are written down before the matches are played, and some evaluation conditions are held out of training.
The test is sequential: it stops as soon as the evidence is clear either way. How many matches that takes, at 95% confidence both ways:
| True difference | Head-to-head win rate | Fixed-count test | Sequential test, on average |
|---|---|---|---|
| 100 Elo | 64% | 133 matches | about 66 |
| 70 Elo | 60% | 269 matches | about 130 |
| 35 Elo | 55% | 1069 matches | about 525 |
At real time, 130 four-minute matches take almost 9 hours on one arena. At the virtual clock's speed as measured on the server's side, about 5 to 45 minutes; that leaves out the agents' own compute, so the real figure will be longer.
Provenance#
Every result records what produced it: server build, agent version, loadout, comp, parity configuration, measurement rules and random seeds. A result without full provenance is not rated.
What doesn't count#
- Simulator results. A simulator may be added for bulk training, but strength is measured only on the real server, under full parity, and the gap between the two is tracked.
- Mirror matches as progress: an agent against itself always reads 50%. Progress is measured against fixed opponents.
- Human games as measurement. Humans can play and their games are recorded, but human play is informal feedback, not a rating.
- Surprising wins, until checked: a win that looks too good is checked for a server bug before it counts as good play.