NeurIPS 2026

Do multimodal models imagine electric sheep?

Santhosh Kumar Ramakrishnan* Carl Vondrick* Raja Giryes* Philipp Krähenbühl Vladlen Koltun

Apple

Abstract

Yes. We find that large multimodal models develop mental imagery when solving spatial puzzles, and they do imagine sheep when solving sheep puzzles. We fine-tune a Qwen3.5 VLM to solve twelve diverse visual reasoning tasks – including tangram, jigsaw, sokoban, 3D mental rotation, and rush hour – that require understanding geometry, spatial relationships, and the consequences of actions. By supervising the model to predict the open-loop sequence of actions to solve a puzzle from an initial state, we show that the model’s activations after each action encode meaningful visual information about the intermediate state. This finding suggests that an imperfect visual world model begins to form as a byproduct of learning to select correct actions, in the absence of any explicit visual supervision. Building on this observation, we study two techniques to sharpen and use the mental images formed by the model. We find that integrating as few as sixteen visual tokens per step into the chain of thought improves the average solve rate from 83% to 89%, with particularly strong gains on reasoning-heavy tasks such as jigsaw and 3D mental rotation.

A cartoon sheep, rotated and lying on its side
Left sheep In the VLM’s imagination
A cartoon sheep facing forward
Right sheep
Prompt

“Are these the same sheep?”

VLM
rotatey(30) rotatex(30) rotatey(30) rotatex(15) rotatey(15) rotatez(15) Answer“no”Mirror image

Spatial intelligence suite

Click any puzzle to play it yourself.

Mental imagery emerges

We train the model with action supervision only, freeze it, and fit a small probe that sees nothing but the VLM’s hidden states, at every layer, at the token that ends each action (not even the input image). The probe recovers piece positions, poses, and layouts the model was never asked to represent. Colors are often off, but the geometry is there.

Sharpening Visual Imagination

Two ways to sharpen these mental images: train the probe together with the model, or have the model write out each next state in its chain of thought.

Head to head

The same puzzle, solved by all three models.

When imagination misleads

Results

Solve rates (%) of Qwen3.5 9B fine-tuned with each method, 200 held-out puzzles per task. State CoT and Boxes CoT write each state as text in place of Visual CoT’s visual tokens: a compact JSON description of the state, or the locations of the objects in it.

Per task

Action Supervision alone produces useful mental imagery, and Visual Supervision refines it, improving the average solve rate from 82.9% to 86.3%. Next-state chain-of-thought performs better still: State CoT and Boxes CoT reach 87.3% and 86.7%, respectively, and excel on clearly discretizable puzzles such as tangram, sokoban, bloxorz, and rush hour. Visual CoT leads overall at 88.9% and excels where visual states are difficult to express in text—lifting jigsaw from 65.0% to 92.0% and 3D mental rotation from 49.5% to 75.0%. Its compact 16-token variant also reaches 88.5%, outperforming the higher-resolution variant on longer-horizon puzzles.

Citation

@inproceedings{Ramakrishnan2026,
  author = {Santhosh Kumar Ramakrishnan and Carl Vondrick and Raja Giryes and Philipp Kr{\"a}henb{\"u}hl and Vladlen Koltun},
  title = {Do Multimodal Models Imagine Electric Sheep?},
  booktitle = {NeurIPS},
  year = {2026}
}