Consti Ertel
← Projects

Beer Distribution Game: A Supply-Chain Environment for Agents

Play the game Environment GitHub

Highlights

  • Rank-16 bf16 LoRA lifts Qwen3.5-4B from 7.80 to 51.47 benchmark points on 100 held-out demand sequences, ahead of GPT-5.6 Terra at 46.99; an independently frozen replication preserved the ordering at 51.47 / 47.30 / 7.74.
  • Frozen environment spec: Y-shaped chain, wholesaler seat, 36 decision weeks, integer orders 0–128, one-week order delay, two-week shipment delay, factory capacity 22 with proportional allocation under shortage, and local observations only.
  • The JavaScript port is parity-tested week by week against the Python environment, so the browser game, the recorded LLM traces, and the training environment all replay to identical trajectories from the same seed.

The question

The beer distribution game is the canonical demonstration of the bullwhip effect: a four-tier supply chain in which small swings in customer demand amplify into violent order oscillations upstream, purely because each tier decides under delay and partial information. That makes it an unusually honest test bed for agents. There is no adversary and no hidden trick, only a wholesaler who must place a replenishment order each week while shipments arrive two weeks late, orders propagate one week late, and the only visible signal is local inventory, backlog, and the order that just came in from downstream. A policy that looks intelligent in prose has nowhere to hide here: it either keeps holding and backlog cost down over 36 weeks or it does not.

Environment design

The environment is frozen and fully deterministic: the same seed and the same order sequence always produce the same trajectory, and invalid actions neither mutate state nor consume randomness. The controlled role is the wholesaler in a Y-shaped topology; retailer, distributor, and factory follow fixed base-stock policies so that a run measures the agent rather than the rest of the chain. Actions are integers from 0 through 128, exposed to models as a single `place_order(quantity)` tool call, with malformed output counted as an order of zero and logged as a format failure rather than silently retried. The factory has a capacity of 22 units and allocates proportionally under shortage, which is what makes over-ordering self-defeating instead of free. It is published as a Verifiers Hub environment and wrapped for PettingZoo, and ships alongside a dependency-free browser build so a human can play the identical seeds a model was scored on.

Scoring

Raw cost numbers are hard to read across difficulty settings, so the published score is anchored to a naive baseline that simply orders whatever quantity it observed downstream last period: score = 100 × naive_mean_cost / (naive_mean_cost + policy_mean_cost). The naive policy therefore always scores exactly 50, a zero-cost policy approaches 100, and a policy worse than naive decays toward zero rather than being clipped, which keeps failures legible instead of flattening them. The 100 benchmark seeds are fixed in the repository and never used for training, generation, or tuning, and the base-stock comparison policy has its target level tuned only on separate training seeds.

Fine-tuning result

A rank-16 bf16 LoRA adapter was trained on a single A40 GPU against the frozen environment and then evaluated on the held-out 100-seed set. It raises Qwen3.5-4B from 7.80 to 51.47 points, moving a model that mostly emitted unusable orders into a policy that beats both the naive anchor and GPT-5.6 Terra at 46.99. A second 100-seed replication, frozen separately after training, reproduced the ordering at 51.47, 47.30, and 7.74. The headline is less that a 4B model outscores a frontier model than that most of the gap was format and calibration rather than reasoning: the environment rewards a policy that holds a stable target position under delay, and that is a behavior a small model can be taught directly.

The negative result

The earlier recorded LLM wholesaler run is kept in the repository rather than quietly dropped. Across three Tier 5 development seeds it averaged a local cost of 1,111.8 ± 213.2 against 850.7 ± 326.1 for the paired adaptive base-stock policy — that is, the untuned tool-calling agent lost to a classical heuristic. Publishing it is the point: it fixes a concrete learning target, and it means the later fine-tuning gain is measured against a baseline that was allowed to win first.