Earlier model leaderboard

What the benchmark asks

SupplyChainBench asks three things of an agent, and keeps their scores apart so unlike tests are never averaged together. Control: can it hold inventory and backlog cost down on the frozen 16-seed standard suite, which is the board above. Adaptation: five further suites change the demand law, the delivery times or the supply ceiling mid-run to see whether the agent notices and recovers. Learning: separate runs compare starting fresh each episode, carrying short notes between episodes, and updating weights across them.

Only the standard suite is ranked here. The others are reported in the repository against their own references, because a policy that looks strong under a fixed demand law is not thereby strong when the law moves.

How the score reads

Every policy is scored against a hindsight-perfect wholesaler on the same seed: the best feasible open-loop order sequence, computed after the fact with the demand already known. That reference costs 287.22 on average and is defined as 100. A score of 25 means the policy spent four times what perfect foresight would have.

Two reference points sit between the models and that ceiling. An adaptive base-stock heuristic — classical inventory control, no learning, no language — reaches 73.9. And a blind constant order of 18 units every week, which never reads the observation at all, reaches 19.82. Anything below that second line is losing to a policy with no inputs.

The gap

GPT-5.6 Sol and Grok 4.6 lead the fully clean API runs at 43.613 and 42.091 — roughly two-fifths of hindsight-perfect and still well short of the base-stock heuristic's 73.9. Claude Opus 5 scores higher at 51.282 on 15 of 16 seeds, but that bar is diagnostic: one protocol failure rebases the perfect reference and keeps it off the canonical rank. The trained adapter in this repository — a rank-16 LoRA on a 4B base model — reaches 20.636, clearing the blind constant-order baseline of 19.82 but sitting well below the latest frontier models.

Runs with any protocol failure stay in the full table rather than being dropped. Laguna finished 7 of 16 episodes, Nemotron 2 of 16, the DeepSeek V4 Flash 0731 rerun 11 of 16, and Muse Spark 1.2 completed none — each scored against a perfect reference re-based onto the seeds it actually finished, so those rows are small-sample or diagnostic. They are left visible because a benchmark that quietly removes the runs that broke is not a benchmark.

What training moved in the earlier evaluation

The adapter was trained with critic-free group-relative policy optimization on the earlier environment: roll the seat out several times per seed under common random numbers, grade each weekly decision by a six-week windowed return, normalize against its groupmates at the same timestep, and reweight only the tokens that carry the order digits. The 4B base weights stay frozen; the LoRA is the only thing the optimizer touches.

On the earlier 16 held-out seeds that takes the same base model from 6.73 to 20.64 — a 3.1× improvement — with mean local cost falling from 4264.7 to 1391.8 and 16/16 episodes protocol-clean before and after. What changed is the over-ordering habit that dominates cost in this game: mean weekly order in the rollouts fell from about 22.9 to about 17.5, against a real obligation near 16 units a week.

Two caveats belong next to that number. It clears the blind constant-order baseline by 0.82 points, which is not a comfortable margin. And both training runs used the same seed (20260808), so run-to-run variance is unmeasured.

Full table

Human baseline

This page is missing a line it should have. The browser build logs anonymous sessions so that human play can be scored on the same seeds as the models, and 23 sessions have been recorded, but 0 of them reached week 36. Every logged session was abandoned after the first order, which leaves no final cost to take a median over.

Rather than substitute an estimate, the line is left off the chart until the log contains finished episodes. The plumbing is in place; the data is not.

Protocol

• Controlled role Wholesaler; the two retailers, distributor and factory run fixed policies
• Topology Y — one wholesaler, two competing retailers
• Horizon 36 decision weeks, 3 settlement weeks, plus a terminal inventory-position charge
• Capacity Factory capacity 400; the prompt withholds both the capacity and the demand law
• Seeds 16, common random numbers, held out from training
• Score score = 100 × mean(perfect cost) / mean(policy cost), over protocol-clean episodes
• Cost $0.50 per unit held, $1.00 per unit backlogged, charged weekly
• Frontier models Run through OpenRouter at versioned API ids

A separate board exists and is deliberately not mixed with this one: the published Verifiers Hub package keeps factory capacity 22 for historical leaderboard parity, while the research board and public browser build run at capacity 400.

The archived board figures are transcribed from artifacts/live_y_capacity_400/evaluations/ in the project repository, where the same board is regenerated from the evaluation JSON so the published chart cannot drift from the runs. Nothing here is recomputed in the browser.