Question regarding the effective batch size

[hank0316]

Hi @willccbb,

I have a question regarding how effective batch size and the number of global training steps per epoch are computed in the GPPOTrainer.

My setup:

• 6000 prompts for training
• num_generations=8
• gradient_accumulation_steps=8
• per_device_train_batch_size=8
• 2 GPUs for training

My current understanding is that the total number of global steps in one epoch can be calculated as:

total_global_steps = (#prompts) * num_generations / effective_batch_size

where:

effective_batch_size = per_device_train_batch_size * num_processes * gradient_accumulation_steps

Plugging in my settings:

total_global_steps = 6000 * 8 / (8 * 2 * 8) = 375

However, I noticed that the logs (e.g., on wandb) show that one epoch actually corresponds to 750 global steps in my case, which is double what I expected.

Could you clarify how the effective batch size and total steps per epoch are computed? Am I misunderstanding how batches are constructed in GPPOTrainer?

Thanks for your help!

[willccbb]

If you're using the default config, num_iterations was set to 2, i.e. each batch is used for 2 gradient steps. This is generally a good stability-efficiency tradeoff in my experience, but I agree it's confusing/unexpected behavior -- just updated the default to 1, which should show 375 steps in your case.

[hank0316]

@willccbb Thanks for your reply!

Actually, I have set num_iterations=1 as follows:

training_args=vf.grpo_defaults(run_name=args.run_name)
training_args.learning_rate=5e-7
training_args.num_iterations=1
training_args.per_device_train_batch_size=8
training_args.num_generations=8
training_args.gradient_accumulation_steps=8
training_args.num_train_epochs=2
training_args.max_steps=-1
training_args.save_steps=25
training_args.eval_strategy="steps"
training_args.eval_steps=25
training_args.max_completion_length=args.max_tokens
training_args.max_prompt_length=1024
training_args.temperature=args.temperature
training_args.top_p=args.top_p
training_args.mask_truncated_completions=True
training_args.max_concurrent=32  # the default 1024 will make evaluation timeout, as well as gibberish in response

I’ve also checked the config logged to wandb, and num_iterations is correctly set to 1.

I also tried the latest version of the repo, but the total number of steps (shown by the tqdm progress bar) remains unchanged (doubled as what I expected). I’m not sure what’s causing this issue. (BTW, I have also encountered the multi-epoch training failed issue. The execution just failed at the end of the first epoch)

Any ideas on what might be going wrong?

[hank0316]

I used examples/gsm8k.py for a quick experiment to reproduce this issue. I noticed that the total number of training steps remains unchanged, regardless of whether I use one or two GPUs.

Here’s the example script with my modifications (I just modify the training_args and n when loading datasets):

import verifiers as vf
from verifiers.utils.data_utils import load_example_dataset, extract_boxed_answer

"""
inference:
CUDA_VISIBLE_DEVICES=0 vf-vllm --model willcb/Qwen3-0.6B

training:
CUDA_VISIBLE_DEVICES=1 accelerate launch --num-processes 1 --config-file configs/zero3.yaml verifiers/examples/gsm8k.py
"""

dataset = load_example_dataset("gsm8k", split="train", n=96)
eval_dataset = load_example_dataset("gsm8k", split="test", n=10)


system_prompt = """
Think step-by-step inside <think>...</think> tags.

Then, give your final numerical answer inside \\boxed{{...}}.
"""
parser = vf.ThinkParser(extract_fn=extract_boxed_answer)

def correct_answer_reward_func(completion, answer, **kwargs):
    response = parser.parse_answer(completion) or ''
    return 1.0 if response == answer else 0.0

rubric = vf.Rubric(funcs=[
    correct_answer_reward_func,
    parser.get_format_reward_func()
], weights=[1.0, 0.2])

vf_env = vf.SingleTurnEnv(
    dataset=dataset,
    eval_dataset=eval_dataset,
    system_prompt=system_prompt,
    parser=parser,
    rubric=rubric,
)


model_name = "willcb/Qwen3-0.6B"
run_name = "gsm8k-grpo_" + model_name.split("/")[-1].lower()

model, tokenizer = vf.get_model_and_tokenizer(model_name)
training_args=vf.grpo_defaults(run_name=run_name)

training_args.per_device_train_batch_size=8
training_args.num_generations=4
training_args.gradient_accumulation_steps=4
training_args.num_iterations=1
training_args.max_prompt_length=512
training_args.max_completion_length=2048
training_args.max_steps=-1
training_args.num_train_epochs=2
training_args.save_strategy="steps"
training_args.save_steps=-1
training_args.save_total_limit=1

trainer = vf.GRPOTrainer(
    model=model,
    processing_class=tokenizer,
    env=vf_env,
    args=training_args,
    peft_config=vf.lora_defaults()
)
trainer.train() 

With these settings, I expect the global steps to be calculated as 96 * 4 / (8 * #GPUs * 4) * 2 epochs = 24 / #GPUs.
However, when running this script with either 1 or 2 GPUs, the progress bar always shows 24 global steps.

Is there something I might be missing in the configuration for multi-GPU training? Any advice would be appreciated!