when i use onnx and CUDAExecutionProvider , why onnx fp16 slow than fp32 on nvidia3090

[slantingsun]

Search before asking

• I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

python 3.8
cuda 11.3
onnxruntime-gpu 1.14.1

python export.py --weights ./best.pt --imgsz 960 --device 0 --half --simplify

Another question, will onnx using CUDAExecutionProvider be faster than pt file using gpu?
Thanks for your answer

Additional

No response

[UltralyticsAssistant]

👋 Hello @slantingsun, thank you for your interest in YOLOv5 🚀! This is an automated response to help you get started while an Ultralytics engineer reviews your issue.

Please visit our ⭐️ Tutorials for quickstart guides, including Custom Data Training and Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide a minimum reproducible example (MRE) so we can better assist you.

If this is a custom training ❓ Question, please share as much detail as possible—such as dataset image examples and training logs—and verify that you are following our Tips for Best Training Results.

Requirements

Python>=3.8.0 with all requirements.txt installed, including PyTorch>=1.8. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 runs in the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training, validation, inference, export and benchmarks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

An Ultralytics engineer will review your question and provide further assistance soon. Thank you for your patience! ⚡

[pderrenger]

FP16 performance can vary depending on your model architecture and GPU utilization - not all operations benefit from FP16 on RTX 3090, and some may even be slower due to conversion overhead. For NVIDIA GPUs, I'd recommend trying TensorRT export instead (--include engine) which typically provides better optimization than ONNX Runtime's CUDAExecutionProvider.

You can benchmark different formats directly using python utils/benchmarks.py --weights best.pt --device 0 to compare PT, ONNX FP32, ONNX FP16, and TensorRT performance on your specific model and hardware setup.

[gilligas]

Hey @slantingsun I found the same issue. I think the reason is that when you then pass a float16 numpy array in your onnx inference session it needs to convert to float32 to pass it to gpu which then it is converted back to float16. While I'm not sure this is the true reason why it happens, the solution was to make a custom model with self.model as as yolov5 and add _preprocessing_hook in such way that it expects uint8 as input and then convert to fp16 in the models graph.

Alternatively and maybe easier is that, I believe onnx allows you to pass torch tensors already on the GPU, so you can maybe to this uint8 to uint16 in the gpu using torch and then pass that array, already on the GPU to the onnx inference.

So in summary, the graph computation itself is faster but these convertions when you pass the image to the onnx engine are consuming a lot of time, since the float16 to float32 is done still on the CPU. Again, not sure if the explanation is correct but I believe it is smth like this.