Intent First, Pixels Later: Separating Routing Policy from Action Spaces in Scientific Vision Agents

Introduction: Scientific image-analysis workflows require agents to coordinate heterogeneous capabilities such as description, detection, counting, segmentation, and quantitative measurement. However, providing an agent with a fixed set of actions seldom specifies how scientific intent maps to those actions. When semantic interpretation and visual execution are coupled, routing failures are difficult to distinguish from downstream perception failures. We investigate whether explicitly separating these stages improves the reliability and auditability of scientific vision agents.

Methods: We introduce ZETA, a scientific vision-agent framework governed by a “Route First” principle to determine the requested operation before invoking task-specific visual modules, including description, detection, counting, and the zero-shot Zenesis segmentation framework. Quantitative measurement is treated as a secondary deterministic operation over explicit visual artifacts. To study routing independently of execution, we compare two conditions: Action Space Only (the router receives the same fixed set of legal actions without explicit semantic selection rules) vs. Action Space + Routing Policy (the legal action space remains unchanged, but explicit intent-to-action semantics are provided).

Results: In an initial routing benchmark spanning counting, description, detection, segmentation, and text-only requests, explicit routing-policy specification substantially improved performance at semantically confusable action boundaries. Across 51 queries and 10 diverse images, models achieved peak gains where intents matched semantic operations: Qwen 3.8/3.6 detection rose 60% to 100% (segmentation 6.2% to 75%), Nemotron 3.5 reached 70% to 90% (segmentation 37.5% to 87.5%), and counting held at 100%.

Conclusions: These results show that constraining an agent to a valid action inventory is not sufficient for reliable scientific intent routing. Explicit routing-policy specification can improve route selection even when the available action space is unchanged. By separating routing decisions from downstream visual execution, ZETA also enables route-selection errors and execution failures to be analyzed independently, providing an auditable foundation for agentic image analysis.