Dynamics of Gradient Propagation in
Complex-Valued Neural Networks with Planted
Singularities

This reseach investigates the dynamics of gradient propagation in complex-valued neural networks (CVNNs) in the presence of planted singularities. Unlike real-valued neural networks, CVNNs operate in the complex domain, enabling the simultaneous representation of magnitude and phase information. This property makes them particularly suitable for applications such as radar signal processing, biomedical imaging, and quantum systems. However, training stability in CVNNs is often compromised by optimization instabilities arising near singular points in parameter space, where gradients may either vanish or diverge.

To address this issue, a rigorous mathematical framework is developed to characterize how the local behavior of activation functions near the origin governs gradient stability. Using Wirtinger calculus and asymptotic analysis of activation derivatives near singularities, the study establishes a trichotomy of gradient regimes: (i) exploding gradients when the derivative diverges, as observed in magnitude-gated activations such as modReLU; (ii) vanishing gradients when the derivative approaches zero, as in zReLU with dead zones; and (iii) bounded and stable gradients when the derivative converges to a finite constant, as in smooth activations such as the Cardioid. The analysis formally demonstrates that while network depth amplifies gradient behavior, it does not alter the underlying regime determined by the activation function.

The theoretical results are supported by extensive experiments on both shallow K-2-K architectures and deeper multi-layer CVNNs trained on synthetic complex-valued datasets. Empirical findings confirm that modReLU frequently leads to gradient explosion, zReLU exhibits prolonged training stagnation, Split ReLU shows irregular convergence due to non-smooth transitions, and the Cardioid activation consistently maintains bounded gradients and stable optimization. These results validate the predictive strength of the proposed theoretical framework and underscore its practical relevance.