A world model can make good-looking predictions and still lose information that matters for acting in the world. In Mario, that information includes where the character is, how the camera is moving, and whether an action is likely to lead to a fall.

AQ-Mario explores this through an action-conditioned JEPA. The central question is whether the learned representation keeps useful state, rather than merely producing a low prediction loss.

The experiment

The project implements LeJEPA’s SIGReg objective in a video setting with actions. JEPA predicts representations instead of reconstructing every pixel. SIGReg regularizes the distribution of those representations to help prevent collapse, where different inputs end up with nearly identical embeddings.

The model has 9.8 million parameters and predicts five steps ahead in a 192-dimensional latent space. Its components include a ViT-Tiny encoder, a projection head, and an action-conditioned causal predictor.

The dataset contains 8,000 episodes from Super Mario Bros. level 1-1, with roughly 4.88 million observations. Half the episodes come from a jump-biased random policy and half from a trained PPO explorer. Console RAM provides state labels such as position, camera scroll, and whether the character dies within five steps.

Why the regularizer needed calibration

A loss value is difficult to interpret without knowing what it should look like under a healthy representation. The study derives and checks a finite-sample reference value for SIGReg, then compares measurements against it.

Batch size matters. Small batches make it harder for the statistic to distinguish some collapsed representations from healthy ones. That is particularly relevant for video, where each example contains several frames and memory limits can force smaller batches.

Consecutive frames also are not independent samples. Treating a batch of nearby frames as if every frame were independent can make a healthy representation look problematic. The implementation instead computes the statistic at each frame position and averages the results.

Preventing collapse was not enough

The project compares training with and without stop-gradient and investigates the effect of normalization. In the measured short runs, removing stop-gradient still allowed collapse. Increasing the regularizer alone did not remove the incentive to make prediction targets constant.

A second finding is more directly relevant to control: a representation can avoid collapse and still fail to retain a useful state variable. In the documented experiment, the SIGReg-only model did not collapse, but a frozen-encoder probe could not reliably recover the character’s height.

Adding small auxiliary heads improved state decoding. That is a useful practical result, but it needs the right interpretation. Those heads explicitly train the representation to expose the same kinds of information that the probes measure. It is not evidence that the unsupervised objective independently discovered a better representation.

Checking what a controller would need

The repository includes frozen-encoder probes, camera-aliasing checks, and a prediction of death within five steps. These provide a more concrete evaluation than prediction loss alone.

Checkpoint gates make those measurements part of the workflow. A model should be checked for the state information a downstream task needs before it is treated as a useful basis for planning.

What the results do and do not show

The reported findings come from a 4,000-step run, about 4% of the planned training schedule, on one level of one game. They are evidence about this experimental setting, not a general verdict on JEPA or a claim that the model has solved Mario.

The takeaway is a measurement habit: check what a representation retains. A healthy-looking embedding distribution and an improving training loss are useful signals, but neither replaces testing the information the task actually needs.

Code, data, and reports