From Stereo RGB Cameras to Spectral Imaging via Co-Design of Optical Filters and Spectral Super-Resolution

Spectral imaging provides detailed wavelength-dependent information that is essential for applications such as remote sensing, precision agriculture, medical imaging, and industrial inspection. Despite its advantages, conventional spectral imaging systems, including scanning-based devices and tunable-filter cameras, are limited by high cost, slow acquisition speed, and poor suitability for portable or dynamic imaging scenarios. Spectral super-resolution (SSR) addresses these limitations by reconstructing spectral information from RGB images. However, its performance is fundamentally constrained by the spectral response of conventional RGB sensors and the limited availability of annotated spectral datasets.

This paper presents a differentiable framework that jointly optimizes the spectral filters attached to a stereo camera pair and the SSR reconstruction network. By integrating optical filter design and network training into a unified end-to-end optimization pipeline, the proposed method automatically learns task-specific filter characteristics that maximize spectral reconstruction quality. To ensure the learned filters are physically realizable, the optimization incorporates smoothness and non-negativity constraints that promote manufacturable spectral responses. The framework is evaluated using representative state-of-the-art SSR architectures, enabling a comprehensive comparison of reconstruction accuracy and computational efficiency. Experiments across multiple spectral image datasets demonstrate up to 67% performance improvement for the optimized stereo configuration compared with its single-camera counterpart. Furthermore, cross-dataset evaluations confirm that the learned filters generalize effectively across different spectral image distributions, highlighting the robustness of the proposed approach. Finally, we assess the manufacturability of the optimized filters using a physics-based fabrication model and evaluate their robustness to realistic manufacturing tolerances, demonstrating their feasibility for practical deployment.

Overall, the proposed framework provides a principled approach to jointly optimizing optical hardware and reconstruction algorithms, advancing the development of accurate, robust, and manufacturable spectral imaging systems for real-world applications.