[Dashboard] Add GPU component usage

[Bye-legumes]

Why are these changes needed?

Close #45755.
This PR addresses the need for enhanced GPU usage metrics at the task/actor level in the Ray dashboard. Currently, the Ray dashboard provides detailed CPU and memory usage metrics for individual tasks and actors, but lacks similar granularity for GPU metrics. This enhancement aims to fill that gap by introducing per-task/actor GPU utilization and memory usage metrics.

---

Area	Change	+ / –
dashboard/agent.py, dashboard/modules/stats_collector.py	Collect per-GPU SM, memory-used, memory-total and temperature using NVML (fallback to nvidia-smi --query-gpu if NVML is not available).	+307 LOC
dashboard/frontend/src/pages/node/Stats.vue	New GPU bars beside existing CPU/Mem charts; shows live %, absolute MiB and thermals with colour-coded alert gradients.	+62 LOC
dashboard/frontend/src/components/ResourceIcon.tsx	Adds gpu-core, gpu-mem icons and tooltip helpers.	+18 LOC
python/ray/dashboard/tests/test_gpu_stats.py	E2E integration test that spins up a fake GPU via CUDA_VISIBLE_DEVICES=0 + mock NVML bindings to assert Dashboard JSON schema and time-series values.	+20 LOC
Misc.	Typo fixes, pylint: disable=c-extension-no-member guards, build-time NVML check in setup.py.	–12 LOC

Related issue number

Close #45755.

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[Bye-legumes]

import ray
import torch
import os

# Initialize Ray, using all available GPUs
ray.init()

# Check if CUDA is available
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"CUDA device count: {torch.cuda.device_count()}")


@ray.remote(num_gpus=1)
class TorchGPUWorker:
    def __init__(self):
        assert torch.cuda.is_available(), "CUDA is not available"
        self.device = torch.device("cuda")
        print(f"Worker running on device: {self.device}")

    def matrix_multiply(self, size=20000):
        # Create two large random tensors on GPU
        a = torch.randn(size, size, device=self.device)
        b = torch.randn(size, size, device=self.device)
        result = torch.matmul(a, b)

        # Return just the norm to reduce transfer cost
        return result.norm().item()


if __name__ == "__main__":
    # Create an actor
    gpu_worker = TorchGPUWorker.remote()

    # Run a GPU task
    result = ray.get(gpu_worker.matrix_multiply.remote(2048))

    print(f"Result norm of matrix multiply on GPU: {result}")

[zhaoch23]

[zhaoch23]

Script to test:

import ray
import torch
import os
import time
# Initialize Ray, using all available GPUs
ray.init()

# Check if CUDA is available
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"CUDA device count: {torch.cuda.device_count()}")


@ray.remote(num_gpus=0.5)
class TorchGPUWorker:
    def __init__(self):
        assert torch.cuda.is_available(), "CUDA is not available"
        print(os.getenv("CUDA_VISIBLE_DEVICES"))
        self.device = torch.device("cuda")
        print(f"Worker running on device: {self.device}")

    def matrix_multiply(self, size=16384):
        a = torch.randn(size, size, device=self.device)
        b = torch.randn(size, size, device=self.device)

        torch.matmul(a, b)
        torch.cuda.synchronize()
        
        REPEATS = 6 

        start = time.time()
        for _ in range(REPEATS):
            c = torch.matmul(a, b)
        torch.cuda.synchronize()



if __name__ == "__main__":
    # Create an actor
    gpu_workers = [TorchGPUWorker.remote() for i in range(4)]

    # Run a GPU task
    for i in range(100):
        result = ray.get([worker.matrix_multiply.remote() for worker in gpu_workers])
        print(f"Result norm of matrix multiply on GPU: {result}")

[alanwguo]

let's remove pid to align with the Node CPU component graph.

[alanwguo]

can we use a different type here? Since gpu_memory_usage is of type Megabytes. It's very confusing for it to be a percentage and may introduce tricky bugs later.

[alanwguo]

What if we match the Ray Dashboard behavior where we show the total gpu utilization for gpu that the process attaches to. Not necessarily the utilization exclusive to that process.

The nvdia-smi parsing is a fragile. I don't know what backwards compatability guarantees nvidia-smi provides

[zhaoch23]

Do you mean we give up nvidia-smi? Or we implement a fallback strategy that use the pynvml to display the total gpu utilization if nvidia-smi is not available?

[alanwguo]

yes, lets remove the usage of nvidia-smi and just use pynvml all the time. We can add nvidia-smi at a later time if there is enough demand. But I think for most usecases the pynvml approach should be good enough.

[alanwguo]

Discussed offline. We will be adding the nvidia-smi dependency. We will add a test validating the output of nvidia-smi pmon. We will also update the ray dashboard UI to utilize the gpu utilization value from nvidia-smi instead of pynvml.

[alanwguo]

I did some manual testing and didn't seem to get any metrics for component_gpu_utilization even though I got metrics for component_memory_usage

In my worker node, this is the output i get from nvidia-smi:

$ nvidia-smi --query-gpu=index,name,uuid,utilization.gpu,memory.used,memory.total --format=csv,noheader,nounits
0, Tesla T4, GPU-9b0c908a-c921-4693-783f-534cb205ec77, 100, 9059, 15360
1, Tesla T4, GPU-15797035-31fa-a236-16b1-209b8dd896dd, 100, 9059, 15360
2, Tesla T4, GPU-f20b2bc8-2492-322f-551b-23e8ef87130f, 100, 9059, 15360
3, Tesla T4, GPU-2ac2bffc-c188-fa30-3710-21d904c37691, 100, 9059, 15360

$ nvidia-smi pmon -c 1
# gpu         pid   type     sm    mem    enc    dec    jpg    ofa    command 
# Idx           #    C/G      %      %      %      %      %      %    name 
    0       5569     C     94     84      -      -      -      -    ray::RayTrainWo
    1       5718     C     91     80      -      -      -      -    ray::RayTrainWo
    2       5716     C     91     81      -      -      -      -    ray::RayTrainWo
    3       5717     C     90     81      -      -      -      -    ray::RayTrainWo

$ curl localhost:8085/metrics | grep component_gpu
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  125k  100  125k    0     0  20.0M      0 # HELP ray_component_gpu_memory_usage GPU memory usage of all components on the node.
--:# TYPE ray_component_gpu_memory_usage gauge
--ray_component_gpu_memory_usage{Component="ray::RayTrainWorker",GpuDeviceName="Tesla T4",GpuIndex="2",IsHeadNode="",SessionName="session_2025-05-13_16-42-55_367950_2270",Version="3.0.0.dev0",ip="",pid="5716"} 9.495904256e+09
:-ray_component_gpu_memory_usage{Component="ray::RayTrainWorker",GpuDeviceName="Tesla T4",GpuIndex="3",IsHeadNode="",SessionName="session_2025-05-13_16-42-55_367950_2270",Version="3.0.0.dev0",ip="",pid="5717"} 9.495904256e+09
- ray_component_gpu_memory_usage{Component="ray::RayTrainWorker",GpuDeviceName="Tesla T4",GpuIndex="1",IsHeadNode="",SessionName="session_2025-05-13_16-42-55_367950_2270",Version="3.0.0.dev0",ip="",pid="5718"} 9.495904256e+09
--ray_component_gpu_memory_usage{Component="ray::RayTrainWorker",GpuDeviceName="Tesla T4",GpuIndex="0",IsHeadNode="",SessionName="session_2025-05-13_16-42-55_367950_2270",Version="3.0.0.dev0",ip="",pid="5569"} 9.495904256e+09

[zhaoch23]

I have fixed some potential parsing error. This is what is looks like on my side:

Please let me know if this fix works on your environment. @alanwguo

[jjyao]

Should we turn it on by default, otherwise the code path will never be tested and no one is going to use this feature

[jjyao]

Discussed offline: we will gradually roll it out and we also have the follow up to use c nvml directly instead of nvida-smi CLI

[jjyao]

Is it guaranteed to be integer, never float?

[zhaoch23]

They are int types according to NVML.

[jjyao]

what if these headers don't exist?

[zhaoch23]

pmon is expected to have all the following fields. Now a ValueError is raised and handled if the header or any of these fields are missing

[jjyao]

This is no longer needed since gpu_ids is already a dict

[zhaoch23]

One pid can be in multiple gpus. Here we create a flat map gathering by the pids among all gpus.

[jjyao]

This can just be moved to _generate_reseted_stats_record

[zhaoch23]

Unlike cpu processes, a gpu process is deemed stale as soon as it stops using gpu resources, even if it remains alive. To address this, we implemented a separate method for generating and resetting gpu statistics.

[jjyao]

Do we need all these code?

[zhaoch23]

Similarly to the above, this code tracks only processes with positive gpu utilization to identify stale gpu processes and generate corresponding reset records for them.

[jjyao]

LG. Just some naming nits.

[jjyao]

@zhaoch23 @Bye-legumes there are some lint failures.

[Bye-legumes]

Follow UP

use nvml with pybind to implement pynvml to get the GPU usage directly.