SPADE: Self-Play in Adaptive Synthetic Executable Environments 💡 The… — AI & ML Papers — TG.ME

💡 The paper addresses the need for ever‑expanding, diverse training goals that can keep pace with a language model’s growing abilities. Existing collections of training environments are either hand‑crafted, generated once and frozen, or verified by a static system, so the distribution of tasks does not change as the learner improves. This limits the potential for open‑ended self‑improvement.

SPADE Self‑Play in Adaptive Synthetic Executable Environments proposes a self‑play reinforcement‑learning framework in which a single large language model assumes two complementary roles. The first role, the Environment Designer, writes complete, long‑horizon training environments as executable code that follows an OpenAI‑Gym‑style reset and step interface. The second role, the Reasoning Agent, interacts with those environments, learning to act, reason, and use tools over multiple steps. Both roles are stateful and involve multi‑turn interactions, allowing the same interface to cover pure reasoning problems as well as tool‑use scenarios.

A key innovation is the use of a regret‑based signal to guide environment creation. The Reasoning Agent’s regret is estimated as the difference between the reward it obtains when it receives privileged hints and the reward it obtains without those hints. The Environment Designer is trained to maximize this regret, thereby generating environments that sit at the edge of the agent’s current capabilities while remaining solvable. The authors find that two components are critical for success: grounding the Designer on documents sampled from a large pre‑training corpus, and providing the Designer with an accumulated memory of previously created environments so it can build on past experience.

Experiments scale the framework up to 30‑billion‑parameter models and compare against the strongest fixed‑environment baselines across eight held‑out benchmarks covering mathematics, science, code, and general reasoning. SPADE yields an average improvement of 5.3 points. In tool‑use settings it raises performance by 5.7 points on the BFCL‑v4 multi‑turn benchmark and by 13.9 points on ACEBench‑Agent. In game‑like environments the performance gap over baselines grows with model size. The results demonstrate that making environment design a learnable component enables a concrete step toward open‑ended self‑improvement for language models.