Slurm (Soperator) Distributed Training Tutorials #5
Conversation
.DS_Store
Outdated
There was a problem hiding this comment.
MacOS artifact, to delete
| @@ -0,0 +1,33 @@ | |||
| #!/bin/bash | |||
|
|
|||
There was a problem hiding this comment.
add : "${HF_TOKEN:?provide your HF_TOKEN}" to force providing HF_TOKEN before launching script
| # pip install -e . | ||
|
|
||
| # Flux specific setup | ||
| python ./torchtitan/experiments/flux/scripts/download_autoencoder.py --repo_id black-forest-labs/FLUX.1-dev --ae_path ae.safetensors --hf_token <YOUR HF TOKEN> |
There was a problem hiding this comment.
replace <YOUR HF TOKEN> with $HF_TOKEN from the env var
| pip install -r requirements-flux.txt | ||
|
|
||
| # 5) Make required directories | ||
| mkdir -p output slurm_out |
There was a problem hiding this comment.
Replace
mkdir -p output slurm_out
cd ../../../..
with
cd $ROOT_DIR
mkdir -p outputs slurm_out
since dirs should be created in $ROOT_DIR
| nodes=( $( scontrol show hostnames $SLURM_JOB_NODELIST ) ) | ||
| nodes_array=($nodes) | ||
| head_node=${nodes_array[0]} | ||
| head_node_ip=$(srun --nodes=1 --ntasks=1 -w "$head_node" hostname --ip-address) |
There was a problem hiding this comment.
Replace
nodes=( $( scontrol show hostnames $SLURM_JOB_NODELIST ) )
nodes_array=($nodes)
head_node=${nodes_array[0]}
head_node_ip=$(srun --nodes=1 --ntasks=1 -w "$head_node" hostname --ip-address)
with head_node_ip=$(scontrol show hostnames $SLURM_JOB_NODELIST | head -n 1)
In general avoid using srun to get node IP since it launches a slurm job, use single scontrol command
| For running this workload, you will need to SSH to the login node of the Soperator cluster and clone this repository to the shared filesystem (by default, Soperator has `/` mounted as a shared filesystem). | ||
|
|
||
| ### 🔧 Setup the environment | ||
|
|
There was a problem hiding this comment.
Add instruction about providing HF_TOKEN before launching setup_flux.sh
| @@ -0,0 +1,24 @@ | |||
| #!/bin/bash | |||
|
|
|||
There was a problem hiding this comment.
add a check for HF_TOKEN env var
set -e
: "${HF_TOKEN:?provide your HF_TOKEN}"
| # | ||
| # This config is only tested on 2 nodes w/ 8 H100 machines. | ||
|
|
||
| root_dir: /root/ml-cookbook/slurm-recipes/torchtune/ # <-- Update this to your ml-cookbook/slurm-recipes root dir |
There was a problem hiding this comment.
root_dir: /root/ml-cookbook/slurm-recipes/torchtune/ # <-- Update this to your ml-cookbook/slurm-recipes root dir
Use YAML template .tpl rendering in setup.sh with the current $(pwd)
| # This config needs 8 GPUs to run | ||
| # tune run --nproc_per_node 8 lora_finetune_distributed --config llama3_3/70B_lora | ||
|
|
||
| root_dir: /root/ml-cookbook/slurm-recipes/torchtune/ # <-- Update this to your ml-cookbook/slurm-recipes root dir |
There was a problem hiding this comment.
root_dir: /root/ml-cookbook/slurm-recipes/torchtune/ # <-- Update this to your ml-cookbook/slurm-recipes root dir
Use YAML template .tpl rendering in setup.sh with the current $(pwd)
| # Download LLaMA-3 model to shared FS --> Syncs to all Soperoator nodes | ||
| tune download meta-llama/Llama-3.3-70B-Instruct \ | ||
| --ignore-patterns "original/consolidated*.pth" \ | ||
| --output-dir ./models/Llama-3.3-70B-Instruct |
There was a problem hiding this comment.
Maybe add a section which inserts $(pwd) as root_dir var in the YAMLs, for this you need to use .tpl files for generating final YAML configs for torchtune
This comment was marked as duplicate.
This comment was marked as duplicate.
Sorry, something went wrong.
This comment was marked as duplicate.
This comment was marked as duplicate.
Sorry, something went wrong.
Beginner friendly slurm recipes with torchrun, tested on 2 H100s
Recipes:
Each example contains its own README.md and setup script to setup distributed training.