Improving YOLOv5 Inference Speed on CPU for Detection

[gauricollab09]

Search before asking

• I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

Hi everyone,

I'm using YOLOv5 for a logo detection. On GPU (RTX A6000), the inference speed is excellent : around 30+ FPS. However, when running on CPU (a reasonably powerful machine), the inference speed drops significantly to about 1 frame every 2 seconds (~0.5 FPS), which is too slow.

Is there a way to speed this up on CPU? Even achieving 8–9 FPS would be a huge improvement. Are there any flags, quantization techniques or runtime options you recommend?

Any suggestions if you could give would be useful. Thanks in advance!

Additional

No response

[UltralyticsAssistant]

👋 Hello @gauricollab09, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials for guidance, including Custom Data Training and advanced topics like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide a minimum reproducible example (MRE) so we can help debug more efficiently.

If this is a custom training or inference ❓ Question (such as yours about CPU speed), please provide as much detail as possible, including:

• Model version and export format (e.g., PyTorch, ONNX, etc.)
• Example command(s) used for inference
• Specs of your CPU environment
• Any logs or error messages

You can also review our Tips for Best Training Results for general optimization advice.

Requirements

Python>=3.8.0 with all requirements.txt installed, including PyTorch>=1.8. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 runs in any of these up-to-date verified environments (with all dependencies, including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training, validation, inference, export and benchmarks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

This is an automated response. An Ultralytics engineer will also assist you here soon. 😊

[pderrenger]

For significant CPU inference improvements, I'd recommend trying these optimization approaches in order of expected impact:

First, export your model to ONNX format and use optimized inference engines like OpenVINO for Intel hardware which can provide substantial speedups on CPU. You can also try DeepSparse with Neural Magic's sparsified YOLOv5 models which achieves 2.8x-5.8x performance gains over standard ONNX Runtime on CPU.

Additionally, ensure you're using batch size 1 for lowest latency, consider model quantization (INT8), and if accuracy allows, try YOLOv5n which is specifically designed for lightweight deployments. The reference material shows OpenVINO achieving 23ms inference times on Intel i9 compared to 115ms with standard PyTorch on CPU.