[AutoDeploy]: Nemotron-6 8K/16K (ISL/OSL) on B200; BF16

[nzmora-nvidia]

🚀 The feature, motivation and pitch

capture nsys; compare to vLLM and identify issues.

• Create and share a perf table for various configuration points (same as H100 + 8K/16K)
• Share instructions on how to setup vLLM docker on B200
• Share traces
• Find root causes of bad AD perf, if any.

Alternatives

No response

Additional context

capture nsys; compare to vLLM and identify issues.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

[galagam]

Initial results on B200 single device below.
Configured vLLM to run with FlashAttention. Default on Blackwell is FlashInfer, but it causes the benchmark to crash.

isl/osl	num prompts	vllm	AD
1K	256	6.26	7.16
2K	256	3.34	3.59
4K	256	1.63	1.76
8K	256	0.81	>1 hour benchmark run time (timed out)
16K	256	0.36	N/A

With ISL==OSL==1K/2K/4K, AD outperforms vLLM.
With ISL=8K (and presumably 16K) - AD benchmark is very slow, eventually timing out when the slurm lease expired.

vLLM trace shows significant GPU idle times between iterations. The idle time increases proportionally to the ISL/OSL.
Need to understand whether vLLM should be configured differently, because this low utilization seems far from ideal.

The attached zip file includes both vLLM and AD traces.
trace_output_nemotron_b200_1k_1k_256.zip