CVCUDA backend design

[AntoineSimoulin]

Summary

Users have to explicitly opt-in for those transforms. Here we provide the first building block for this interface. We add the functionals to_nvcv_image and nvcv_to_tensor to transform torch.Tensor to nvcv.Tensor. We also implement the corresponding class transforms ToNVCVImage and NVCVToTensor.

How to use

from PIL import Image
import torchvision.transforms.v2.functional as F

orig_img = Image.open("leaning_tower.jpg")
img_tensor = F.pil_to_tensor(orig_img)
nvcv_tensor = F.to_nvcv_tensor(img_tensor.cuda())
img_tensor = F.nvcv_to_tensor(nvcv_tensor)

Note

NVCV tensors are automatically converted to NHWC format. Contrary to torchvision convention, which relies on NCHW format.

Run unit tests

pytest test/test_cvcuda.py
...
37 passed in 0.15s

Differential Revision: D85862362

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/vision/9259

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 9 New Failures

As of commit e96659c with merge base 2fee489 ():

NEW FAILURES - The following jobs have failed:

• Tests / unittests-linux (3.10, linux.12xlarge, cpu) / linux-job (gh)
  RuntimeError: Command docker exec -t f2c6b7fe01335b936d4cdce3ce110cdbad73d110b7b63c37eb90262411ae46a1 /exec failed with exit code 2
• Tests / unittests-linux (3.10, linux.g5.4xlarge.nvidia.gpu, cuda, 12.6) / linux-job (gh)
  RuntimeError: Command docker exec -t ba13605280c606ae5d5ddb7ad9b7f450b0f7a1de441b3e826bd26bd114a6e175 /exec failed with exit code 2
• Tests / unittests-linux (3.11, linux.12xlarge, cpu) / linux-job (gh)
  RuntimeError: Command docker exec -t dc2399b3e1aff2c15775220676bad827e2ce35930a9bc1407a64cafde16c3b0e /exec failed with exit code 2
• Tests / unittests-linux (3.12, linux.12xlarge, cpu) / linux-job (gh)
  RuntimeError: Command docker exec -t 6ab7f3a742fa93ccbc6ccd6414b7c111d8c8ba3cc7214b811ed82fb5c1704f96 /exec failed with exit code 2
• Tests / unittests-macos (3.10, macos-m1-stable) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 2
• Tests / unittests-macos (3.12, macos-m1-stable) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 2
• Tests / unittests-windows (3.10, windows.4xlarge, cpu) / windows-job (gh)
  TypeError: results.testsuites.testsuite.testcase is not iterable
• Tests / unittests-windows (3.11, windows.4xlarge, cpu) / windows-job (gh)
  TypeError: results.testsuites.testsuite.testcase is not iterable
• Tests / unittests-windows (3.12, windows.4xlarge, cpu) / windows-job (gh)
  TypeError: results.testsuites.testsuite.testcase is not iterable

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[meta-codesync]

@AntoineSimoulin has exported this pull request. If you are a Meta employee, you can view the originating Diff in D85862362.

[AntoineSimoulin]

NOTES:

• Should we upload tensors .to(cuda) directly inside to_nvcv_tensor function?
• Should we keep CVCUDA specific functions inside isolated "_cvcuda" files or within existing files (e.g. "_geometry")?
• NVCV tensors are automatically converted to NHWC format. Contrary to torchvision convention, which relies on NCHW format. If we call to_nvcv_tensor and then nvcv_to_tensor, we might have an output tensor with a different shape.
• Are the image format from _infer_nvcv_format exhaustive and aligned with what is done for PIL?
• What is the difference between nvcv.Image and nvcv.Tensor?

[meta-codesync]

@AntoineSimoulin has imported this pull request. If you are a Meta employee, you can view this in D85862362.