This study investigates the flexural behavior of reinforced concrete beams with a layered steel-FRP hybrid reinforcement layout. In this system, FRP bars are placed near the tensile surface to improve durability and provide high tensile resistance, while steel bars are arranged in the inner tensile layer to introduce yielding behavior and enhance failure warning. Theoretical calculation models and failure mode criteria for FRP-steel hybrid reinforced beams were established based on the plane section assumption, with perfect bond assumed between the steel bars, FRP bars, and the surrounding concrete, and relative slip neglected. Based on the proposed model, 810 numerical cases were analyzed to evaluate the effects of reinforcement ratio, steel-to-FRP area ratio, concrete strength, steel yield strength, FRP tensile strength, and the effective depth of the inner steel bars. Spearman correlation analysis was employed to identify the governing parameters, and a multi-criteria decision-making method was used to support design optimization. Twelve beams were then tested under four-point bending to examine the predicted behavior and verify the analytical model. The results show that the total longitudinal reinforcement ratio governs the transition between failure modes. For the investigated design range, a reinforcement ratio of about 0.79% is identified as a practical threshold for achieving the desired ductile mode, characterized by steel yielding followed by concrete crushing. The response of the hybrid beams is stage dependent: the yield capacity is mainly associated with steel reinforcement, whereas the ultimate capacity is influenced by FRP reinforcement and concrete strength. Increasing the effective depth of the inner steel bars improves both strength and ductility. The optimized steel-to-FRP area ratio is approximately 1.0. Test results confirm that the proposed system provides a clear yielding stage and stable post-yield stiffness, with secondary stiffness ratios ranging from 0.19 to 0.36. The analytical predictions agree well with the experimental results, indicating that the proposed model can support the design of durable and ductile hybrid reinforced concrete beams.