Beyond Euclidean Clipping: Overcoming Exploration Collapse in LLM RL via Riemannian Isometric Policy Optimization
This paper addresses a problem in reinforcement learning (RL) where algorithms with PPO-Clip suffer from "exploration collapse", where they fail to explore new areas of the environment. The authors propose a new method called Riemannian Isometric Policy Optimization (RIPO) that helps to balance exploration and exploitation by using a more accurate geometric metric.