We've just open-sourced a very competitive IO framework driven by io_uring !!!

[beef9999]

https://github.com/alibaba/PhotonLibOS

It's a production ready library while still being evolving. io_uring is working inside as the event engine, file io engine, and socket engine. Hopefully more people would join us...

Below are the performance data:

Disk IO

	IO Engine	IO Type	IO Size	IO Depth	DirectIO	QPS	Throughput	CPU util
Photon	io_uring	Rand-read	4KB	128	Yes	433K	1.73GB	100%
Photon	libaio	Rand-read	4KB	128	Yes	346K	1.38GB	100%
fio	libaio	Rand-read	4KB	128	Yes	279K	1.11GB	100%

TCP echo server

Client Mode: Streaming

	Language / Coroutine Type	Buffer Size	Conn Num	QPS	Bandwidth	CPU util
Photon	C++ / Stackful	512 Bytes	4	1604K	6.12Gb	99%
cocoyaxi	C++ / Stackful	512 Bytes	4	1545K	5.89Gb	99%
tokio	Rust / Stackless	512 Bytes	4	1384K	5.28Gb	98%
acl/lib_fiber	C++ / Stackful	512 Bytes	4	1240K	4.73Gb	94%
Go	Golang / Stackful	512 Bytes	4	1083K	4.13Gb	100%
libgo	C++ / Stackful	512 Bytes	4	770K	2.94Gb	99%
boost::asio	C++ / Async-callback	512 Bytes	4	634K	2.42Gb	97%
libco	C++ / Stackful	512 Bytes	4	432K	1.65Gb	96%
zab	C++20 / Stackless	512 Bytes	4	412K	1.57Gb	99%
asyncio	C++20 / Stackless	512 Bytes	4	163K	0.60Gb	98%

Client Mode: Ping-pong

	Language / Coroutine Type	Buffer Size	Conn Num	QPS	Bandwidth	CPU util
Photon	C++ / Stackful	512 Bytes	1000	412K	1.57Gb	100%
boost::asio	C++ / Async-callback	512 Bytes	1000	393K	1.49Gb	100%
evpp	C++ / Async-callback	512 Bytes	1000	378K	1.44Gb	100%
tokio	Rust / Stackless	512 Bytes	1000	365K	1.39Gb	100%
Go	Golang / Stackful	512 Bytes	1000	331K	1.26Gb	100%
acl/lib_fiber	C++ / Stackful	512 Bytes	1000	327K	1.25Gb	100%
swoole	PHP / Stackful	512 Bytes	1000	325K	1.24Gb	99%
zab	C++20 / Stackless	512 Bytes	1000	317K	1.21Gb	100%
cocoyaxi	C++ / Stackful	512 Bytes	1000	279K	1.06Gb	98%
libco	C++ / Stackful	512 Bytes	1000	260K	0.99Gb	96%
libgo	C++ / Stackful	512 Bytes	1000	258K	0.98Gb	156%
asyncio	C++20 / Stackless	512 Bytes	1000	142K	0.54Gb	99%

HTTP

	File Size	QPS	CPU util
Photon	4KB	114K	100%
Nginx	4KB	97K	100%

[YoSTEALTH]

Congratulation on launching your project. HTTP 114K is that per thread?

[beef9999]

Yes, it is

[YoSTEALTH]

nice

[axboe]

Thanks for sharing this! Is this something Alibaba uses in production or is moving into production? Would be nice if you could talk about that aspect a little bit.

[axboe]

Given the initial perf differences described, it must be writes.

[beef9999]

No, I was testing reads, because writing a tmpfs file would somehow enable io-wq. Don't want it to get involved.

The tests were running in a cloud VM, with CentOS kernel 5.15.6

[axboe]

Then I'm a bit puzzled at the gap. We've made improvements since 5.15, but none that would explain a gap this big. Can you try my test app I attached in here? You'd need to remove the ring registration and the 1 << 8 from the queue init to run it on 5.15.

[beef9999]

After changing 1 << 8 to 0 and comment out io_uring_register_ring_fd, I got:

4KB:
psync: 7384.34MB/sec
io_uring: 6248.50MB/sec

16KB:
psync: 8590.87MB/sec
io_uring: 8000.58MB/sec

These data seem to be fine. So the only variable between your test code and mine is the io_uring_register_files

[beef9999]

After removing io_uring_register_files

4KB:
psync: 7381.09MB/sec
io_uring: 6123.07MB/sec

16KB:
psync: 8581.70MB/sec
io_uring: 8019.70MB/sec