What does a world model need to preserve for its world to remain playable? In this experiment, the answer became very concrete: the player.

Small Worlds is an action-conditioned model of a 2D pixel-art environment, trained from scratch on generated data. It is small enough to run locally, but the most interesting result is about evaluation. A model can get better at reconstructing a frame while becoming worse at reconstructing the character you control.

A small world with a measurable problem

The controllable character occupies only 1.3% of the observation. That creates an imbalance when reconstruction is evaluated over the whole image. Most pixels belong to something other than the player.

A reconstruction that removes the agent completely can still score 0.987 on whole-frame accuracy. That number looks good until you try to use the environment. A scene without its player is not a useful reconstruction for control.

The baseline measurements make this failure visible. Between steps 1,000 and 3,000, an adapted general-purpose tokenizer improved its whole-frame accuracy from 0.4245 to 0.7656. Over the same interval, agent-only accuracy fell from 0.0151 to 0.0010.

Changing what the model preserves

The implementation uses two design choices to address this problem.

First, the latents retain spatial structure. The character has a location in that structure, rather than becoming a tiny contribution to a representation of the entire frame.

Second, reconstruction is treated as classification over the environment’s known 21-color palette. This keeps the output within the palette and allows class weights to express the importance of different pixels. It is a deliberate fit to this pixel-art setting, not a general solution for arbitrary video.

The complete system combines a tokenizer with a dynamics model. The tokenizer has about 2.94 million parameters; the dynamics model has about 1.46 million. Together they total roughly 4.4 million parameters and 17.7 MB of model weights.

Generating the training data

The dataset contains 2,025,385 frames across 35,000 episodes. It is generated from the project’s environment rather than scraped from existing videos.

The stored data includes frames, action vectors, rewards, episode identifiers, goals, and terminal flags. Generating the environment and its data makes it possible to connect what the model predicts with the actions and state that produced each observation.

The repository separates environment code, collection, tokenizer training, dynamics training, evaluation, and local play. It also releases checkpoints and provides a dataset on Hugging Face.

What the evaluation found

The project reports tokenizer agent accuracy of 0.9994 and dynamics rollout agent accuracy of 0.885. These measure different stages and should not be treated as interchangeable.

Action tests also check whether the predicted character moves in the requested direction. The reported measurements include displacement for left, right, and idle actions. This connects evaluation to the behavior that makes the model interactive.

The benchmark reports 101 frames per second on an M2 GPU and 26 on an M2 CPU, with 2.3 GPU-hours of training compute. Those numbers apply to this compact pixel-art environment and the documented benchmark setup. They do not describe general-world video generation.

The lesson I’m taking from it

An average over a frame can hide the thing that makes the frame useful. The right evaluation depends on what someone is supposed to do with the model.

Here, agent-specific measurements exposed a failure that whole-frame accuracy concealed. Building a small, controlled environment made that failure easier to inspect and gave the design changes a concrete target.

Try it and read more

The repository includes a local play script and released checkpoints:

python play_local.py --tok models/tokenizer.pt --dyn models/dynamics.pt

This is the 4.4M-parameter project. The separate 21M-parameter world model mentioned in my portfolio is not the model described here.