In recent years, the global transition toward carbon neutrality has driven a rapid expansion of renewable energy systems. Agrivoltaics has emerged as a promising solution, combining energy production with agricultural use by elevating solar panels above the ground. However, this configuration increases vulnerability to extreme transient wind events such as downbursts, whose frequency and intensity are rising due to climate change. This work investigates the aerodynamic loading induced by downburst wind fields on agrivoltaic structures using Computational Fluid Dynamics (CFD). To this end, a novel multi-scale precursor methodology is developed to capture the interaction between a full-scale downburst and high-resolution agrivoltaic panel systems. Results show that the presence of panels and supporting structures generates strong flow channeling, concentrating peak wind velocities near the ground and amplifying structural risks. In open terrain environments, the horizontal configuration of solar panels emerges as the safest stow position, while negative tilt angles produce the highest aerodynamic loads. Although staggered panel layouts provide partial shielding, localized channeling effects can still occur. Furthermore, results reveal that downburst-induced loads on elevated agrivoltaic panels are nearly twice those associated with standard atmospheric boundary layer winds. These findings highlight the potential inadequacy of current design criteria for agrivoltaic structures under downburst conditions and stress the urgent need for dedicated modeling to ensure the resilience and safety of next-generation renewable energy systems.