Smart weather data management based on big data analytics and artificial intelligence

Weather monitoring is essential for implementing sustainable agricultural practices. It can be used as an input in various tasks such as crop simulation and yield forecasting to name a few. Currently, weather data can be collected with high temporal resolution thanks to advances in technologies such as remote sensing and the Internet of Things (IoT) that enable low-cost and simple design and deployment of real-time weather data sensors. This data is not useful in its raw form. And generating it, without having the proper infrastructure and tools to process it, makes no sense. To this end, we developed a smart weather data management system powered by state-of-the-art machine learning and deep learning models, that offers several ways to derive actionable insights from this data. This cloud-based system consists of three layers: the data acquisition layer, the data storage layer, and the application layer. The data can be sourced from real-time IoT sensors, third-party services, or manually imported from files. It is then checked for errors such as temporal resolution correction and missing values before being stored using the MongoDB NoSQL database, known for its ability to deal with large quantities of complex and diverse real-world big data. The system provides several services related to weather data: i) forecast univariate time series ii) perform advanced analysis and visualization and study different relationships between meteorological data iii) use machine learning to estimate and model important climatic parameters and vi) scheduling early alert warnings by sending SMS or emails as an automatic trigger event. This system was first tested using data from meteorological stations from 2013 to 2020, collected in our study area, located 40 km east of Marrakech city in Morocco.