Mitigating Negative Transfer in Cross-View Adaptation of Compact Multimodal Language Models

Introduction: Spatial adaptation of compact multimodal large language models (MLLMs) may improve selected cross-view abilities but can degrade baseline performance elsewhere, a form of negative transfer. We investigate whether a cross-view adapter that leaves base computation unchanged at initialization can limit this degradation without an external 3D encoder.

Methods: We introduce a cross-view adapter with a zero-initialized residual scale. Across three shared-weight recurrent cross-attention steps, projected tokens from each later view attend to preceding-view tokens, adjacent view pairs are processed sequentially before language decoding. Zero initialization leaves base-model output unchanged initially. Training used approximately 160,000 filtered SAT v2 examples. Gemma-4-E4B-it and Qwen2.5-VL-3B were compared with unadapted baselines and LoRA-only adaptation using the same data. From model-specific EASI checkpoint sweeps, we selected Gemma at 40,000 steps and Qwen at 10,000 steps because they provided the strongest joint performance on the intended multi-view and camera-motion submetrics; later checkpoints did not improve both submetrics. Evaluation used seven EASI benchmarks: BLINK, ViewSpatialBench, EmbSpatialBench, MMSIBench, MindCubeBench, SiteBenchImage, and VSI-Bench covering image and video-based spatial reasoning.

Results: On Gemma, the largest decrease among the seven benchmark-level scores was 0.63 percentage points on VSI-Bench, whereas LoRA-only adaptation reduced BLINK, MindCubeBench, and VSI-Bench scores by 9.26, 10.67, and 12.29 percentage points, respectively. On Qwen, the largest decrease was 2.69 percentage points on MindCubeBench with the proposed adapter versus 5.67 percentage points with LoRA-only adaptation among completed evaluations. On these tasks, Gemma and Qwen improved by 1.50 and 2.25 percentage points in multi-view reasoning and by 2.70 and 4.06 percentage points in camera-motion reasoning, respectively. Other changes were small, and some object-centric submetrics regressed.

Conclusions: Under the evaluated configurations, the proposed adapter provided a better retention-specialization trade-off than LoRA-only adaptation. The results support negative-transfer mitigation with targeted cross-view gains rather than broad spatial-reasoning improvement.