[GRPO] why doesn't GRPO trainer use old policy for ration when num_iterations=1?

[JVP15]

When num_generations=1 (and steps_per_generation <= gradient_accumulation_steps), the GRPO trainer just sets old_per_token_logps to per_token_logps.detach(), so the ratio is always 1. I understand it means we don't have to keep the weights from the previous step in memory, but other than that, why do we do this and why does it work?

[JVP15]

Is this a case where (Pi_old) is a little ambiguous? As in, Pi_old in PPO is just the policy used to collect the data (instead of the policy weights from the previous step). So you try not to diverge too much from the policy that you used to generate data (and thus when you are only doing one optimization step on that data, you'll never need the policy ratio in the first place)?