Bayesian Optimization-Driven U-Net Architecture Tuning for Brain Tumor Segmentation

Precise brain tumor segmentation from MRI scans is critical for clinical diagnosis, surgical planning, and treatment monitoring. However, determining an optimal deep learning architecture remains a significant challenge due to the high-dimensional hyperparameter space, variability in tumor morphology, and differences across MRI modalities. This paper presents a novel approach that integrates Bayesian Optimization (BO) to systematically tune the U-Net architecture for enhanced brain tumor segmentation performance. The proposed BO-UNet framework explores various encoder, bottleneck, and decoder configurations using a Gaussian Process-based surrogate model. The optimization is guided by a composite fitness function that averages the Dice Similarity Coefficient (DSC) and Jaccard Index (JI), ensuring accurate spatial overlap between predicted and ground truth masks. Experiments were conducted on two benchmark datasets: the Figshare Brain Tumor Segmentation (FBTS) dataset and the BraTS 2021 dataset, with a focus on Whole Tumor (WT) segmentation. The best architecture discovered—[64, 64, 64, 256, 64, 128, 256]—demonstrated superior performance. On the FBTS dataset, it achieved 0.9503 DSC and 0.9054 JI, while on BraTS 2021 it reached 0.9261 DSC and 0.8631 JI, outperforming several state-of-the-art deep learning models. Convergence plots and segmentation map visualizations illustrate the effectiveness of BO in guiding architectural evolution. These findings highlight the potential of data-driven optimization strategies for automatic model design in medical image analysis, particularly in domains requiring high precision and structural sensitivity.