Preprint, not yet peer reviewed
Unbundling Multimodal Vision Encoding
AI models that work with both images and text differ in how they take in a picture.
Why it matters Models that handle images and text did not need a vision component trained in advance, only a part of the model devoted to images, in a matched comparison.
Most pass it through a separate vision component trained beforehand; some train that component together with the language model, and some drop it. Published models also differ in data, design and computing budget, so comparing them cannot show which choice matters. Here every model was trained from scratch on the same data with the same computing budget. A vision component trained jointly did as well as a pre-trained one, so the advance training was not needed. What mattered was having part of the model devoted to vision: models without it fell behind even when enlarged to match.
Brown et al., arXiv cs.LG, 12 November 2026.













