https://lineance.github.io/tags/rl/Recent content in Rl on Lineance LensitHugoenThu, 12 Mar 2026 00:00:00 +0000https://lineance.github.io/posts/the-evolution-of-policy-gradient-methods/Thu, 12 Mar 2026 00:00:00 +0000https://lineance.github.io/posts/the-evolution-of-policy-gradient-methods/<blockquote> <p>The Bias-Variance Tradeoff and Architecture Simplification in LLM Training</p> </blockquote> <hr> <h2 id="1-historical-trajectory">1. Historical Trajectory</h2> <p><img alt="diagram" loading="lazy" src="https://lineance.github.io/posts/the-evolution-of-policy-gradient-methods/teopgm-diagram.webp"></p> <p><strong>Core Tension</strong>: Unbiased high-variance Monte Carlo → Biased low-variance bootstrapping → Critic-free relative estimation → Fine-grained credit decomposition</p> <hr> <h2 id="2-reinforce-the-monte-carlo-foundation">2. REINFORCE: The Monte Carlo Foundation</h2> <p><strong>Problem</strong>: Direct policy gradient via likelihood ratio suffers from extreme variance.</p> <p>$$ \nabla_\theta J = \operatorname{E}<em>{\pi</em>\theta}\left[\sum_{t} G_t \nabla_\theta \log \pi_\theta(a_t \mid s_t)\right] $$</p> <p><strong>Challenge</strong>: $G_t = \sum_{k=t}^T \gamma^{k-t}r_k$ has variance $O(T)$. Unstable for long episodes.</p>