Androgenetic alopecia (AGA) is a common dermatological condition that can affect psychological well-being and quality of life. Conventional assessment relies primarily on clinical visual examination or trichoscopic imaging, which may be difficult to access in remote or resource-limited settings. This study proposes FEv4, a lightweight Convolutional Neural Network (CNN) trained from scratch for automated classification of male androgenetic alopecia (MAA) from standard scalp photographs without requiring specialized imaging equipment. A dataset of 130 original scalp photographs representing 126 unique identities was curated from dermatology sources and categorized into four groups derived from the Norwood–Hamilton scale. To evaluate generalization to previously unseen individuals, all data partitioning was performed at the identity level, with data augmentation applied exclusively to the training partitions. Using identity-grouped 5-fold cross-validation, FEv4 achieved 71.5 ± 5.2% exact classification accuracy, 70.3 ± 5.8% macro-F1, 96.9 ± 2.7% within-one-stage accuracy, and an ordinal mean absolute error of 0.32 stages. Most classification errors (89.2%) occurred between adjacent Norwood–Hamilton categories, while no errors spanned three stages. Comparisons with lightweight architectures including SqueezeNet 1.1, MobileNetV3-Small, and EfficientNet-B0 demonstrated that FEv4 provides a favorable accuracy–latency trade-off, achieving 4.6 ms CPU and 0.5 ms MPS inference latency at a 256 × 256 input resolution. These findings suggest that a task-specific CNN can provide meaningful MAA severity classification from standard scalp photographs while maintaining low inference latency. The approach has potential applications in accessible telemedicine-based screening and monitoring, although further validation on larger, independently collected datasets is required to establish robustness across patients, imaging devices, populations, and real-world acquisition conditions.