Crumb rubber modified asphalt (CRMA) contributes to sustainable pavement construction, yet its elastomeric components undergo substantial swelling and thermo-oxidative degradation during service. The opaque asphalt matrix hinders the in-situ characterization of swollen rubber (SR), leaving the relationship between SR degradation and the rheological deterioration of aged CRMA insufficiently understood. This study develops a microscopy-thermogravimetry method to isolate and characterize SR degradation. Based on the distinct aging characteristics of the asphalt and rubber phases, a multi-phase regeneration method combining an aromatic-oil rejuvenator, virgin rubber, and reactive isocyanate was proposed. The relationships among aging condition, material dosage, and binder properties were established through rheological, thermogravimetric, and infrared spectroscopic analyses. For the binder aged for 20 h in a pressure aging vessel (PAV), the dosages of the three regenerating agents were 5.44 wt%, 7.22 wt%, and 0.69 wt%, respectively. Microscopic observations revealed that PAV aging progressively fragmented the porous network of SR, while thermogravimetric analysis confirmed the accompanying degradation of its elastomeric components. This structural and compositional deterioration weakened the continuity and deformability of the rubber network, thereby reducing its contribution to binder elasticity and stress relaxation and directly increasing low-temperature creep stiffness. The proposed degradation index further identified the first 40 h of PAV aging as the critical stage of elastomer depolymerization, during which SR degradation was closely associated with the most pronounced rheological deterioration. The multi-phase intervention restored this micro-macro linkage by replenishing light fractions, reconstructing the swollen-rubber network with virgin rubber, and covalently reconnecting fractured polymer chains through isocyanate-mediated addition reactions. Consequently, the regenerated binder exhibited a creep stiffness modulus more than 10% lower than that achieved by conventional regeneration. This work clarifies how SR microstructural degradation governs the macroscopic rheology of aged CRMA and provides a phase-targeted approach for the high-value recycling of reclaimed CRMA binder.