Supply Chain RL Environment
I built SupplyChainBench, an RL environment and benchmark designed to test whether LLMs can learn to make better decisions when the consequences of their actions only become visible several steps later. Using the Beer Distribution Game, models must forecast demand, account for orders already in the pipeline, and make decisions whose effects unfold over several weeks. On 16 held-out stochastic benchmark seeds, Qwen3.5-4B scored 38.42 with a GRPO + LoRA adapter versus 11.78 untrained—a 3.3× improvement. The adapter was trained on the earlier environment and evaluated on the stochastic version without retraining.
Play the current stochastic environment: control the wholesaler for 36 weeks under uncertain demand and delayed deliveries. The current results evaluate the existing adapter without retraining on this version.
Teaching an LLM to Forecast the Future
One central pillar that our personal lives, as well as society, rest on is the ability to forecast the future.
This is an inherently difficult skill because the future is unknown, and forecasting it requires pulling together context and correctly assessing what is important and what is not.
I have tested LLM abilities on this before—for example, when creating financial models—and have not been particularly impressed. I think part of the problem is that many of the environments these models are trained on are too narrow.
So, I brainstormed what sort of task would test—and hence potentially teach—a model to become better at forecasting the future.
I thought back to a supply chain class I had taken and the Beer Distribution Game.
The Beer Distribution Game traditionally contains four roles: factory, distributor, wholesaler, and retailer, and is designed to teach the bullwhip effect. Orders and shipments arrive with delays, so to do well at the game, one needs to think and estimate several weeks ahead while also considering what happened in the past.
That made it an interesting environment for testing long-horizon decision-making.
I built this by starting with an open-source version of the game and then turning it into an environment and benchmark for LLM agents. In the current version, the model controls the wholesaler for 36 weeks, serving two retailers while facing delayed orders and shipments without being told the underlying demand pattern or supply limits.
I then thought about how to score it. The score is 100 times the best-found feasible reference cost divided by the policy cost on the same benchmark seeds. Lower costs mean a higher score, with the hindsight reference set to 100.
The next step was to train the model to get better. I used GRPO + LoRA. GRPO allows the model to compare different rollouts and learn which actions performed better, while LoRA allows me to efficiently change a relatively small number of model weights while leaving the base model frozen.
The particularly interesting part is the delayed reward. An order placed today does not reveal its full consequences immediately. Because of the delays built into the environment, I measure how each order affects costs over the following six weeks. The model therefore needs to learn not merely which action looks good now, but which action leads to better outcomes several weeks down the road.
The trained adapter also transfers better than the base model to the stochastic game. On the same 16 held-out benchmark seeds, Qwen3.5-4B scored 11.78 untrained and 38.42 with the GRPO + LoRA adapter, a 3.3× improvement, while mean cost fell from 4,742.3 to 1,453.6. The adapter was trained on the earlier environment and has not been retrained on the stochastic version.
The broader idea is not just forecasting. Many important capabilities require understanding how actions today affect outcomes much later. I wanted to see whether an environment built around those delayed consequences could actually teach a model to become better at that skill.
Highlights
- Seeded, replayable stochastic Y-topology environment: one wholesaler splitting a single inventory pool between two retailers, 36 decision weeks, delayed orders and shipments, local observations only.
- Browser build and Python/JS simulator share the same rules; the JS port is parity-checked against a Python oracle so human play and model traces replay identically.
- The stochastic benchmark evaluates 16 held-out seeds at factory capacity 400, withholding the demand law and supply limits from the model. The best-found feasible reference costs 558.44 on average; the adaptive base-stock reference costs 802.53.
- The existing GRPO + LoRA adapter transfers to the stochastic benchmark without retraining: Qwen3.5-4B scores 38.42 versus 11.78 untrained, a 3.3× improvement, with mean cost falling from 4,742.3 to 1,453.6.