Spatiotemporal Scene Graph for Scene-State Reasoning under Incomplete Observations in Repeated UAV Patrols

Unmanned aerial vehicles (UAVs) are increasingly used as mobile sensing platforms for surveillance, inspection, monitoring, and situational awareness in large outdoor environments. However, repeated patrols rarely observe the same physical space under identical conditions, due to variations in flight position, camera viewpoint, field of view, and resolution, as well as occlusion. A missed detection therefore does not necessarily indicate the absence of an object, and associating objects across patrols is unreliable when objects move or their appearance changes. These limitations hinder consistent reasoning about scene states over time.

We propose a spatiotemporal scene graph that accumulates incomplete observations from repeated UAV patrols. Observations are georeferenced to fixed spatial units using UAV positioning, depth, and visual cues, so that observations referring to the same physical space can be associated. Individual object observations are then matched across patrols based on spatial and visual consistency. Each spatial unit records observed objects, timestamps, positions, attributes, uncertainty, and mobility characteristics, and maintains one of three scene states: present, absent, or unobserved. When a unit's scene state is uncertain or its observations are insufficient, active re-observation is performed using an alternative position, viewpoint, or zoom level, driven by scene-state uncertainty rather than detector confidence alone.

Based on these accumulated observations, a large language model (LLM) interprets user queries about the observed environment by retrieving spatiotemporal evidence from the scene graph together with semantic concepts and relations from an ontology-based knowledge graph. These sources are jointly used to interpret queried entities and events and to perform spatiotemporal reasoning. The framework is implemented in Isaac Sim and validated through experiments on spatial association, scene-state estimation, spatiotemporal query accuracy, and active re-observation under repeated patrols with varying UAV positions and viewpoints.