Spatial intelligence suite
Click any puzzle to play it yourself.
Mental imagery emerges
We train the model with action supervision only, freeze it, and fit a small probe that sees nothing but the VLM’s hidden states, at every layer, at the token that ends each action (not even the input image). The probe recovers piece positions, poses, and layouts the model was never asked to represent. Colors are often off, but the geometry is there.
Sharpening Visual Imagination
Two ways to sharpen these mental images: train the probe together with the model, or have the model write out each next state in its chain of thought.
Head to head
The same puzzle, solved by all three models.
When imagination misleads
Results
Solve rates (%) of Qwen3.5 9B fine-tuned with each method, 200 held-out puzzles per task. State CoT and Boxes CoT write each state as text in place of Visual CoT’s visual tokens: a compact JSON description of the state, or the locations of the objects in it.
Per task
Action Supervision alone produces useful mental imagery, and Visual Supervision refines it, improving the average solve rate from 82.9% to 86.3%. Next-state chain-of-thought performs better still: State CoT and Boxes CoT reach 87.3% and 86.7%, respectively, and excel on clearly discretizable puzzles such as tangram, sokoban, bloxorz, and rush hour. Visual CoT leads overall at 88.9% and excels where visual states are difficult to express in text—lifting jigsaw from 65.0% to 92.0% and 3D mental rotation from 49.5% to 75.0%. Its compact 16-token variant also reaches 88.5%, outperforming the higher-resolution variant on longer-horizon puzzles.
Citation
@inproceedings{Ramakrishnan2026,
author = {Santhosh Kumar Ramakrishnan and Carl Vondrick and Raja Giryes and Philipp Kr{\"a}henb{\"u}hl and Vladlen Koltun},
title = {Do Multimodal Models Imagine Electric Sheep?},
booktitle = {NeurIPS},
year = {2026}
}





