This paper investigates the complex dynamical behavior of a discrete-time Cournot triopoly game in which firms pursue relative profit maximization under bounded rationality. Unlike the standard absolute-profit framework, relative profit objectives introduce strategic aggressiveness and stronger competitive interactions, significantly enriching the market dynamics. Starting from a nonlinear three-dimensional adjustment process, we derive the Cournot–Nash equilibrium and analyze its local stability through linearization and Jacobian-based conditions. Explicit analytical thresholds for flip (period-doubling) and Neimark–Sacker bifurcations are obtained, revealing the mechanisms through which stability is lost as key parameters, such as the adjustment speed and relative profit weight, vary.

Beyond local analysis, we explore the global dynamics of the triopoly system using numerical simulations, including bifurcation diagrams, phase portraits, and basins of attraction. The results uncover a wide range of behaviors, from stable equilibria and periodic cycles to quasi-periodic motions and chaos. To quantitatively characterize the complexity of these dynamics, spectral entropy analysis is employed, providing a robust measure that complements traditional bifurcation techniques and clearly distinguishes between regular and chaotic regimes.

Furthermore, an effective chaos control strategy based on parameter feedback is proposed to stabilize unstable dynamics and restore market equilibrium. The controlled system exhibits reduced complexity, confirmed by a significant decrease in spectral entropy. The findings highlight how relative profit considerations can destabilize oligopolistic markets, while appropriate control mechanisms can enhance stability. Overall, this study offers new analytical and numerical insights into triopoly competition, contributing to the understanding of nonlinear phenomena in economic systems.