Forecasting crop yield at an early stage of the growing season is crucial to reinforcing climate-smart agriculture and enhancing food security in nations with high vulnerability, including Bangladesh. This study creates an interpretable, ML-based framework for the early forecasting of potato yield and climate-driven recommendations using historical crop modeling, field production, climatic factors, and agro-meteorological data. Monthly data for rainfall and temperature were combined with yearly potato yield values covering the 1970/71 to 2024/25 agricultural years. The dataset was augmented through agronomically relevant approaches, such as noise injection, masking, mixup, pattern mixing, and shuffling, to create 21,440 records from the original 2,144 observations. The proposed framework consists of two complementary components. Prediction using KNN, SVR, and RF; and recommendation through RF feature importance, ANOVA F regression analysis to identify influential weather variables, and Pearson and Spearman correlation analysis with KDE to identify optimal climatic conditions. The analysis confirms that yield is highly sensitive to stage-specific climatic conditions in terms of tuber initiation and tuber bulking. Decadal climate indices indicated that rainfall in December, October, and January made large contributions (34%, 32%, and 24%, respectively) to yield variability, while excess rainfall had generally negative associations with yield. The ideal climatic window for high yield was low rainfall and cool temperatures, especially in November and December. From the three aforementioned models, RF provided the best predictive performance with a mean MAE of 0.0803 ton_ha⁻¹ and RMSE of 0.1014 ton_ha⁻¹ as compared to KNN and SVR. This provides a tangible framework for decision support that is interpretable and can be used to facilitate irrigation planning, climate risk mitigation, and precision agriculture driven by IoT networks in Bangladesh and comparable agro-climatic regions.