Do sequential io_uring_prep_send requests require linking?

[andyg24]

I already posted this as a discussion topic 5 days ago but didn't get any traction.

For operations where the order of completion is significant, such as send requests to a TCP socket, can I assume that requests will be processed (i.e. serialized) in the order of submission, at least when submitted to the same ring by the same thread? Or do I still need to use IOSQE_IO_LINK in that situation?

It would be good to clarify this in the man page, as I couldn't easily find the answer.

[jmillan]

I'm not an expert in io_uring but I'm quite certain that the SQEs are processed sequentially, or order of submission, yes.

[axboe]

The first issue attempt is always done in order, that's the only way we can pull SQEs off the SQ ring. If your send request completes inline (eg there's space in the socket to put the data in there), then it'll be done before you return. However, if the socket buffer is full, then an internal poll handler is armed and when POLLOUT is triggered we'll complete it. After this poll handler is armed, we will process the next SQE in the ring, if any.

In other words, the answer is "it depends". We always issue in order, but if requests need poll triggering to complete, or if they need to punt to io-wq for processing, then you can have later SQEs in the ring being issued and completed before prior SQEs are fully issued.

[isilence]

Rewording the answer, it's not safe to assume that SQEs will be processed in order. In practise you may notice that they're executed in order, but that's only until sth happens.

Let me add that IOSQE_IO_LINK doesn't work across different submission syscalls (`i.e. io_uring_submit() and others). So with in the example below there will be two separate not-linked requests.

sqe1 = prep_sqe();
sqe1->flags |= IOSQE_IO_LINK;
submit();

sqe2 = prep_sqe();
submit();

And since traversing the kernel's networking is pretty expensive, you'll be far better off, if instead of sending 2 requests, you'd squash them into one request by packing data into a single iovec array.

[andyg24]

Is it the case then that the only way to use io_uring_prep_send() correctly with a TCP socket is to wait for everything to complete before issuing another submit()?

Because even with IO_LINK or when using an iovec array, the order of serialization to the socket is not guaranteed across submit() calls.

If so, that seems to negate a lot of the benefit of using an async interface for networking (the fact that you have to drain everything before issuing another submit() call), especially when networking is mixed with disk IO on the same ring.

Could there be a mechanism where the order of serialization could be guaranteed across submit() calls? For example, if requests for a socket were previously added to io-wq for processing, could we somehow keep track of that and assign future requests to the same queue, even if the socket becomes unblocked?

[isilence]

Is it the case then that the only way to use io_uring_prep_send() correctly with a TCP socket is to wait for everything to complete before issuing another submit()?

Because even with IO_LINK or when using an iovec array, the order of serialization to the socket is not guaranteed across submit() calls.

Right, the most basic way is to have only one request in flight per socket at any point in time. And before queuing more data to a socket you'd wait for the previous completion for that socket.

If so, that seems to negate a lot of the benefit of using an async interface for networking (the fact that you have to drain everything before issuing another submit() call), especially when networking is mixed with disk IO on the same ring.

I don't think so, it might be a bit inconvenient for the userspace but doesn't change the math. First, you don't have to drain everything. It's just before sending more data to a socket you need to wait for the previous request completion for that particular socket. And it's still asynchronous, i.e. while waiting for a completion you can process other sockets / files / requests. There is no problem with disk IO whatsoever in that regard.

Also, getting a send request completion doesn't mean that the data was actually put onto the wire or delivered to the other end, it's just saying that you can queue more data, so there is no latency problem for that.

For example you can look up as it's implemented in folly. There is a class for socket, and when a user tries to send but there is a request in flight, it only add the data to the socket backlog. When the socket gets the send completion it'll send out all the backlog.

That works even better because you have more time to batch, which will ammortize the networking stack overhead by sending fewer requests with more data.

Could there be a mechanism where the order of serialization could be guaranteed across submit() calls? For example, if requests for a socket were previously added to io-wq for processing, could we somehow keep track of that and assign future requests to the same queue, even if the socket becomes unblocked?

That'd be complicated and at the same time can be easily and cheaply done in userspace

[andyg24]

Thank you, this is very helpful.

It didn't cross my mind that only a single request to a socket could be in flight at any given time. May I ask what happens when there are only a few bytes available in the socket's buffer and a large send request with MSG_WAITALL is submitted? Will I get a single completion event when the entire message is flushed to the socket's buffer? Will an intermediate copy of the message be made inside of io_uring (i.e. do I need to preserve the message buffer in userspace after submission, especially when using SEND_ZC)? Can a SEND request ever fail when used with MSG_WAITALL (assuming the socket is not closed, etc)?

Without MSG_WAITALL, I assume I will get a short write if the entire message cannot be flushed to the socket's send buffer.

[isilence]

May I ask what happens when there are only a few bytes available in the socket's buffer and a large send request with MSG_WAITALL is submitted? Will I get a single completion event when the entire message is flushed to the socket's buffer?

Yes, for OP_SEND there should be only one completion. And for SEND_ZC, unless there was an early fail it'll also post the notification CQE.

Note, there are features, e.g. multishot recv, that produce >=1 completions, but nothing like that for sends.

Will an intermediate copy of the message be made inside of io_uring (i.e. do I need to preserve the message buffer in userspace after submission, especially when using SEND_ZC)?

There won't be any copies, you have to keep the buffer alive until you get the completion back. And MSG_WAITALL nor any other flag changes that. It's fine though to free struct msghdr and/or the iovec itself once the submit syscall returns with !SQPOLL rings. With SEND_ZC you'd need to wait for the notification CQE before you can reuse / release buffers.

Can a SEND request ever fail when used with MSG_WAITALL (assuming the socket is not closed, etc)?

It can, and it can also be a short send, even though it's not very likely. Mostly because we can't guess where and how lower layers can fail, but we also can't do anything if allocations at the io_uring side fail.

Without MSG_WAITALL, I assume I will get a short write if the entire message cannot be flushed to the socket's send buffer.

right

[gootorov]

The first issue attempt is always done in order, that's the only way we can pull SQEs off the SQ ring. If your send request completes inline (eg there's space in the socket to put the data in there), then it'll be done before you return. However, if the socket buffer is full, then an internal poll handler is armed and when POLLOUT is triggered we'll complete it. After this poll handler is armed, we will process the next SQE in the ring, if any.

In other words, the answer is "it depends". We always issue in order, but if requests need poll triggering to complete, or if they need to punt to io-wq for processing, then you can have later SQEs in the ring being issued and completed before prior SQEs are fully issued.

Does that mean a request can be either fully completed, or not completed at all though?
Say, I'm sending messages over a single TCP socket. For simplicity, let's say messages are strings and they look like this:

• "HEADERAAAA"
• "HEADERBBBBBBBB"
  and so on

For each message, I'm issuing a vectored write SQE (io_uring_prep_writev()), where the io_vec for the message has two entries - header and the body. So, for "HEADERAAAA", I'd have a pointer to "HEADER" and a pointer to "AAAA".

With that setup, am I guaranteed to get either "HEADERAAAA" or "HEADERBBBBBBBB" on the wire in any order? Possibly split across TCP segments of course.
What I don't want to get, is something like "HEADBBBBBERAAAA" (and so on).
Also, let's say SQ_POLL is enabled.

I'm experiencing an issue right now, where I'm not able to parse messages on the receive side (they get corrupted). But this happens only when I'm blasting writes (10Gbps is what I'm getting at receive side). I'm not sure if I'm doing something wrong on the receive side, or send side, or if I misunderstood "atomicity" of io_uring.

Some things I've verified on the send side:

• cqe->res is always equal to the message size, short writes do not happen
• since I'm using SQ_POLL, I make sure the io_vec and the message itself are alive until CQE is returned
  This is on kernel 6.11, liburing is 2.8

[isilence]

In a general case the guarantee you get is ordering for data within a single request. They're not atomic, a request can decide to split the buffer into sub-chunks and send them separately. Short writes can happen as well. You can easily get sth like "HEADERHEADERABAB..." in your example. "HEADBBBBBERAAAA" is not possible though because the second request can't start writing "BBB" until it has written the first part of its buffer "HEADER".

That's the general rule, in specific cases the protocol might promise you atomicity / etc., but that's not the case for normal TCP. A good rule of thumb it to think of it as if you'd be doing write(2) / send(2) in parallel from multiple threads.

[gootorov]

I see, thank you.

This is a bit unfortunate. I've opened another issue for discussion. IMO, that complicates writev usage quite a bit. I hope this is something that could be improved.