Asymmetric KL Regularization for Reasoning-Aware Reinforcement Learning in Large Language Models

Reinforcement learning (RL) has become an effective approach for improving the reasoning capabilities of large language models (LLMs), yet RL fine-tuning of reasoning models remains hampered by high gradient variance, policy instability, and the heterogeneous structure of long reasoning trajectories, which interleave exploratory intermediate steps with short, highly constrained final answers. Conventional RLHF/RLVR methods apply uniform Kullback–Leibler (KL) regularization across all generated tokens, which can needlessly restrict exploration during reasoning while offering too little control over final-answer precision and stability. We propose Asymmetric KL Regularization (AKL), a token-role-aware framework that applies different KL directions and strengths to reasoning and answer tokens. Token roles are assigned by combining lightweight positional, predictive-entropy, and verifier-based signals through a deterministic majority vote, requiring no additional supervision or learned classifier. AKL applies weak forward-KL regularization to reasoning tokens to preserve exploratory solution diversity and stronger reverse-KL regularization to answer tokens to encourage precise, stable outputs, integrating directly into PPO- and GRPO-style objectives with negligible overhead. We characterize AKL through the Bregman-divergence structure induced by the negative-entropy generator: under bounded log-density ratios and Lipschitz-continuous rewards, we derive an O(1/√T) expected-regret bound on answer tokens with a sharper dependence on the regularization parameters than the corresponding symmetric-KL bound. We evaluate AKL on GSM8K, MATH, AIME, MBPP, and HumanEval using Qwen2.5-Instruct models from 7B to 72B parameters. Across five random seeds, AKL improves accuracy by 2.5–4.1 percentage points, reduces answer-span length by 12%, and lowers gradient-norm variance by 18% relative to uniform-KL baselines. Controlled ablations isolate the contributions of KL direction, regularization strength, and token-role conditioning, showing that directionality itself—not merely stronger regularization—drives the gains. AKL offers a simple, theoretically grounded approach to balancing exploration during reasoning with precision at final-answer generation.