A Reproducible Framework for Diagnosing LLM-based Rerankers in Cold-Start Recommendation: Cross-Dataset Analysis and Practical Guidelines

Cross-encoder rerankers based on large language models (LLMs) have shown promise in information retrieval, yet their effectiveness in cold-start recommender systems remains poorly understood. This paper presents a reproducible diagnostic framework to systematically evaluate LLM-based rerankers across two distinct domains: Serendipity-2018 (movies) and Taobao-Serendipity (e-commerce). Through rigorous multi-seed experiments with 500 users per dataset, we identify three critical failure modes that generalize consistently across both domains: (1) a retrieval coverage bottleneck, where recall@200 reaches only 0.109 versus 0.609 for catalog-aware baselines; (2) severe exposure bias, with rerankers concentrating recommendations on just 3 unique items compared to 495+ for random selection; and (3) poor score calibration between relevant and irrelevant items (Cohen's d = 0.11, Spearman r ≈ 0.004). Root cause analysis identifies out-of-domain transfer from MS-MARCO passage ranking as the primary driver of calibration failure, while FAISS embedding mismatch explains retrieval collapse. Surprisingly, simple popularity-based ranking outperforms LLM reranking by 33.5× on HR@10 (0.268 vs. 0.008, p < 10⁻⁹⁹), with consistent patterns replicated across both datasets. Pool size ablation reveals a counterintuitive finding: smaller candidate pools (K=200) yield 3.1× better performance than larger pools (K=1000) while reducing inference time by 4.1×. We propose actionable mitigation strategies — including hybrid retrieval combining ANN and BM25 (improving recall@200 from 0.109 to 0.234), optimized pool sizing, and ensemble scoring — and discuss their practical limitations and trade-offs. All experimental code, configurations, raw results, and step-by-step reproduction scripts are publicly released, enabling full replication and extension. This work establishes reproducible diagnostic baselines and provides practical guidelines for deploying neural rerankers in cold-start scenarios.