In this study, we investigate a mathematical model describing the competition between two microbial species in a chemostat environment. One of the species is capable of producing a toxin that inhibits the growth of its competitor, while its own growth is negatively affected by substrate inhibition. The model is reduced to a planar system, and the existence and stability of all steady states are analyzed in detail with respect to the operating parameters of the chemostat.

When classical Michaelis–Menten or Monod growth functions are considered, the system admits a unique positive equilibrium. However, this equilibrium is shown to be unstable whenever it exists. By extending the model to include both monotone and non-monotone growth functions, we demonstrate the emergence of an additional positive equilibrium that can become stable for certain parameter values.

The resulting general model exhibits a wide range of dynamical behaviors, including stable coexistence of the two microbial species, multistability phenomena, and the appearance of stable limit cycles generated through supercritical Hopf bifurcations. Furthermore, homoclinic bifurcations are identified, highlighting the complexity of the system’s dynamics. An operating diagram is constructed to describe the long-term behavior of the system as the operating parameters vary, and to illustrate how substrate inhibition influences the formation and persistence of the species coexistence region.