Zero-Shot, Few-Shot, and Chain-of-Thought Prompting for RSVQA with Open-Source Vision-Language Models

Remote Sensing Visual Question Answering (RSVQA) has become a challenging multimodal task that involves fusing spatial reasoning, domain knowledge, and natural language understanding. While early methods relied on supervised fusion architectures trained end-to-end, large Vision-Language Models (VLMs) have since opened new possibilities for addressing RSVQA problems without task-specific training, shifting much of the performance burden onto how these models are prompted. There is, however, little systematic investigation into how different prompting strategies affect performance on RSVQA benchmarks.

In this paper, we systematically compare three prompting strategies—zero-shot, few-shot, and chain-of-thought (CoT)—across three open-source VLMs (LLaVA, InternVL, and Qwen-VL) on the RSVQA-LR and RSVQA-HR benchmark datasets. Performance is measured across all standard question types: presence, count, comparison, and rural/urban classification. Beyond overall accuracy, we examine the effect of each prompting strategy on model performance per question type to identify which question types benefit most from structured prompting.

We find that CoT prompting is consistently more effective for count and comparison questions, where multi-step spatial reasoning is required, and remains competitive for binary presence questions, where reasoning demands are simpler. Improvements obtained with few-shot prompting are architecture- and exemplar-dependent. Notably, there are significant performance differences between prompting strategies across VLMs, and prompt sensitivity is model-dependent rather than universally predictable. These results indicate that effective prompting for RSVQA cannot be assumed to transfer across models or question types, and that prompt-strategy selection should be treated as an empirical, model- and task-specific design choice.