How to launch GRPO with vLLM on multi-node slurm?

[pbelevich]

How to write sbatch script to run GRPO with vLLM on multiple nodes? What should be --num_processes? Is GRPOTrainer compatible with multinode training?

[Superskyyy]

GRPOTrainer is compatible with multinode training, but the current vLLM isn't. But it will be in near future.

[pbelevich]

@Superskyyy What do you by compatible? I managed to run GRPOTrainer mulitple nodes, but with a bottleneck of a single vLLM on cuda:7 on the first node and idling cuda:7 on the remaining nodes. But the performance is very low.

[Superskyyy]

@Superskyyy What do you by compatible? I managed to run GRPOTrainer mulitple nodes, but with a bottleneck of a single vLLM on cuda:7 on the first node and idling cuda:7 on the remaining nodes. But the performance is very low.

Exactly, the current vllm deployment method is suboptimal due to the limitations of how accelerate and deepspeed use pdsh to launch nodes, it will as you said waste a card on each node by setting cuda visible devices.

[shirinyamani]

Hi @pbelevich
for multi-node setup please refer to the guide here where we provide the slurm for multi-node setup.

[boren-ms]

Hi @shirinyamani

Thank you very much for sharing the guide on multi-node training.

I've successfully run a similar setup with 2 nodes for training and 1 node for vllm generation. However, when I scale the training nodes to 3 or 4, the training process gets stuck and I encounter warnings like the ones below.

Any suggestions or insights would be greatly appreciated.