Getting UDP recv_multishot to work with buffer rings

[mbr0wn]

I am trying to set up a multishot recv() call on a UDP socket. However, while the single-shot version works, the multi-shot version always fails. I have a hunch it's with how I'm setting up the buffer ring, but I can't see the problem. All comparisons to similar code snippets make it seem like this is correct (I tried to boil it down as far as I could):

#include <arpa/inet.h>
#include <liburing.h>
#include <stdio.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>

#define BUF_BGID 0
#define N_BUFS 128
#define PAGESIZE 4096
#define BUFSIZE 8192
#define PORT 49000

int main(int argc, char *argv[])
{
    struct sockaddr_in addr;
    int sock_fd;
    char* bufs; // All bufs go here
    int i;

    struct io_uring ring;
    struct io_uring_buf_ring* br = NULL;
        struct io_uring_sqe *sqe;
        struct io_uring_cqe* cqe;

    /* Set up socket */
    addr.sin_family = AF_INET;
    addr.sin_port = htons(PORT);
    inet_aton("0.0.0.0", &addr.sin_addr);

    /* Create UDP socket */
    sock_fd = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);
    bind(sock_fd, (struct sockaddr*)&addr, sizeof(addr));

    /* Create uring */
    if (io_uring_queue_init(N_BUFS, &ring, 0) < 0) {
        fprintf(stderr, "Could not init ring.\n");
        return 1;
    }


    /* Create buffers and buffer ring */
    if (posix_memalign((void**)&bufs, PAGESIZE, BUFSIZE * N_BUFS)) {
        fprintf(stderr, "Could not allocate memory.\n");
        return 1;
    }
    memset((void*)bufs, 0, BUFSIZE * N_BUFS);
    int err = 0;
    br = io_uring_setup_buf_ring(&ring, N_BUFS, BUF_BGID, 0, &err);
    if (!br) {
        fprintf(stderr, "Could not create buffer ring.\n");
    }
    io_uring_buf_ring_init(br);
    for (i = 0; i < N_BUFS; i++) {
        io_uring_buf_ring_add(br, bufs + i * BUFSIZE, BUFSIZE, i,
                              io_uring_buf_ring_mask(N_BUFS), i);

    }
    io_uring_buf_ring_advance(br, N_BUFS);

    /* Set up recv SQE */
    sqe = io_uring_get_sqe(&ring);
    io_uring_sqe_set_flags(sqe, IOSQE_BUFFER_SELECT);
    sqe->buf_group = BUF_BGID;
    io_uring_prep_recv_multishot(sqe, sock_fd, NULL, 0, 0); /* This doesn't work! */
    /*io_uring_prep_recv(sqe, sock_fd, bufs, BUFSIZE, 0);*/ /* This works! */
    if (io_uring_submit(&ring) < 0) {
        fprintf(stderr, "Failed to submit SQE.\n");
        return 1;
    }

    if (io_uring_wait_cqe(&ring, &cqe) < 0) {
        fprintf(stderr, "Could not get CQE.\n");
        return 1;
    }

    if (cqe->res < 0) {
        fprintf(stderr, "Bad CQE result: %s\n", strerror(-cqe->res));
        return 1;
    }

    fprintf(stderr, "Buffer index: %d\n", cqe->flags >> IORING_CQE_BUFFER_SHIFT);


    /* Now do handling and cleanup. */

    return 0;
}

This can be compiled and executed without modifications (gcc recv_multishot.c -luring).

With the multishot version, the return value cqe->res of the read call is always -EINVAL, but the single-shot version works, making me think it's not a problem on the socket, but with the provided buffers. Any suggestions where I'm going wrong?

Notes:

• Tested with Linux 6.8, liburing version 2.5 (on Ubuntu 24.04).
• I don't think this is an rlimit issue, I see the same behaviour as root
• I tried replacing the buffer ring with the older API (io_uring_prep_provide_buffers() and friends), but I get the exact same behaviour
• I've seen the recv-multishot test case, but nothing stands out to me as obviously different

[axboe]

You probably want to upgrade your liburing, that's ancient. Not sure what the problem is, I ran it here with a newer kernel (should not matter) and with current liburing, and it works fine for me.

General note - do io_uring_prep_foo() first, before fiddling with the SQE. The sqe prep may overwrite some bits you already set.

Here's running a slightly modified version of yours which loops getting CQEs until the mshot terminates:

axboe@m2max-kvm ~> ./mshot-rep
Buffer index: 0, res 8192
Buffer index: 1, res 8192
Buffer index: 2, res 8192
Buffer index: 3, res 8192
Buffer index: 4, res 8192
Buffer index: 5, res 8192
Buffer index: 6, res 8192
Buffer index: 7, res 8192
Buffer index: 8, res 8192
Buffer index: 9, res 8192
Buffer index: 10, res 8192
Buffer index: 11, res 8192
Buffer index: 12, res 8192
Buffer index: 13, res 8192
Buffer index: 14, res 8192
Buffer index: 15, res 8192
Buffer index: 16, res 8192
Buffer index: 17, res 8192
Buffer index: 18, res 8192
Buffer index: 19, res 8192
Buffer index: 20, res 8192
Buffer index: 21, res 8192
Buffer index: 22, res 8192
Buffer index: 23, res 8192
Buffer index: 24, res 8192
Buffer index: 25, res 8192
Buffer index: 26, res 8192
Buffer index: 27, res 8192
Buffer index: 28, res 8192
Buffer index: 29, res 8192
Buffer index: 30, res 8192
Buffer index: 31, res 8192
Buffer index: 32, res 8192
Buffer index: 33, res 8192
Buffer index: 34, res 8192
Buffer index: 35, res 8192
Buffer index: 36, res 8192
Buffer index: 37, res 8192
Buffer index: 38, res 8192
Buffer index: 39, res 8192
Buffer index: 40, res 8192
Buffer index: 41, res 8192
Buffer index: 42, res 8192
Buffer index: 43, res 8192
Buffer index: 44, res 8192
Buffer index: 45, res 8192
Buffer index: 46, res 8192
Buffer index: 47, res 8192
Buffer index: 48, res 8192
Buffer index: 49, res 8192
Buffer index: 50, res 8192
Buffer index: 51, res 8192
Buffer index: 52, res 8192
Buffer index: 53, res 8192
Buffer index: 54, res 8192
Buffer index: 55, res 8192
Buffer index: 56, res 8192
Buffer index: 57, res 8192
Buffer index: 58, res 8192
Buffer index: 59, res 8192
Buffer index: 60, res 8192
Buffer index: 61, res 8192
Buffer index: 62, res 8192
Buffer index: 63, res 8192
Buffer index: 64, res 8192
Buffer index: 65, res 8192
Buffer index: 66, res 8192
Buffer index: 67, res 8192
Buffer index: 68, res 8192
Buffer index: 69, res 8192
Buffer index: 70, res 8192
Buffer index: 71, res 8192
Buffer index: 72, res 8192
Buffer index: 73, res 8192
Buffer index: 74, res 8192
Buffer index: 75, res 8192
Buffer index: 76, res 8192
Buffer index: 77, res 8192
Buffer index: 78, res 8192
Buffer index: 79, res 8192
Buffer index: 80, res 8192
Buffer index: 81, res 8192
Buffer index: 82, res 8192
Buffer index: 83, res 8192
Buffer index: 84, res 8192
Buffer index: 85, res 8192
Buffer index: 86, res 8192
Buffer index: 87, res 8192
Buffer index: 88, res 8192
Buffer index: 89, res 8192
Buffer index: 90, res 8192
Buffer index: 91, res 8192
Buffer index: 92, res 8192
Buffer index: 93, res 8192
Buffer index: 94, res 8192
Buffer index: 95, res 8192
Buffer index: 96, res 8192
Buffer index: 97, res 8192
Buffer index: 98, res 8192
Buffer index: 99, res 8192
Buffer index: 100, res 8192
Buffer index: 101, res 8192
Buffer index: 102, res 8192
Buffer index: 103, res 8192
Buffer index: 104, res 8192
Buffer index: 105, res 8192
Buffer index: 106, res 8192
Buffer index: 107, res 8192
Buffer index: 108, res 8192
Buffer index: 109, res 8192
Buffer index: 110, res 8192
Buffer index: 111, res 8192
Buffer index: 112, res 8192
Buffer index: 113, res 8192
Buffer index: 114, res 8192
Buffer index: 115, res 8192
Buffer index: 116, res 8192
Buffer index: 117, res 8192
Buffer index: 118, res 8192
Buffer index: 119, res 8192
Buffer index: 120, res 8192
Buffer index: 121, res 8192
Buffer index: 122, res 8192
Buffer index: 123, res 8192
Buffer index: 124, res 8192
Buffer index: 125, res 8192
Buffer index: 126, res 8192
Buffer index: 127, res 8192
Bad CQE result: No buffer space available

[axboe]

Here's what I ran, only change is looping for CQEs and doing the prep side in a more solid fashion. The latter may make a difference on your old liburing.

#include <arpa/inet.h>
#include <liburing.h>
#include <stdio.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>

#define BUF_BGID 0
#define N_BUFS 128
#define PAGESIZE 4096
#define BUFSIZE 8192
#define PORT 49000

int main(int argc, char *argv[])
{
    struct sockaddr_in addr;
    int sock_fd;
    char* bufs; // All bufs go here
    int i;

    struct io_uring ring;
    struct io_uring_buf_ring* br = NULL;
        struct io_uring_sqe *sqe;
        struct io_uring_cqe* cqe;

    /* Set up socket */
    addr.sin_family = AF_INET;
    addr.sin_port = htons(PORT);
    inet_aton("0.0.0.0", &addr.sin_addr);

    /* Create UDP socket */
    sock_fd = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);
    bind(sock_fd, (struct sockaddr*)&addr, sizeof(addr));

    /* Create uring */
    if (io_uring_queue_init(N_BUFS, &ring, 0) < 0) {
        fprintf(stderr, "Could not init ring.\n");
        return 1;
    }


    /* Create buffers and buffer ring */
    if (posix_memalign((void**)&bufs, PAGESIZE, BUFSIZE * N_BUFS)) {
        fprintf(stderr, "Could not allocate memory.\n");
        return 1;
    }
    memset((void*)bufs, 0, BUFSIZE * N_BUFS);
    int err = 0;
    br = io_uring_setup_buf_ring(&ring, N_BUFS, BUF_BGID, 0, &err);
    if (!br) {
        fprintf(stderr, "Could not create buffer ring.\n");
    }
    io_uring_buf_ring_init(br);
    for (i = 0; i < N_BUFS; i++) {
        io_uring_buf_ring_add(br, bufs + i * BUFSIZE, BUFSIZE, i,
                              io_uring_buf_ring_mask(N_BUFS), i);

    }
    io_uring_buf_ring_advance(br, N_BUFS);

    /* Set up recv SQE */
    sqe = io_uring_get_sqe(&ring);
    io_uring_prep_recv_multishot(sqe, sock_fd, NULL, 0, 0); /* This doesn't work! */
    io_uring_sqe_set_flags(sqe, IOSQE_BUFFER_SELECT);
    sqe->buf_group = BUF_BGID;
    if (io_uring_submit(&ring) < 0) {
        fprintf(stderr, "Failed to submit SQE.\n");
        return 1;
    }

    do {
        if (io_uring_wait_cqe(&ring, &cqe) < 0) {
            fprintf(stderr, "Could not get CQE.\n");
            return 1;
        }

        if (cqe->res < 0) {
            fprintf(stderr, "Bad CQE result: %s\n", strerror(-cqe->res));
            return 1;
        }

        fprintf(stderr, "Buffer index: %d, res %d\n", cqe->flags >> IORING_CQE_BUFFER_SHIFT, cqe->res);

        if (!(cqe->flags & IORING_CQE_F_MORE)) {
	    fprintf(stderr, "No more CQEs\n");
	    break;
	}
	io_uring_cqe_seen(&ring, cqe);
    } while (1);

    /* Now do handling and cleanup. */

    return 0;
}

[mbr0wn]

Many thanks! Even on the old liburing, this works. Amazing. Thanks for the tip to prep before flagging.

(Note that I'm not attached to old software, or Ubuntu 24.04, but this is a commonly used distro so I'm happy if I can develop code that lots of people can run out of the box).

[axboe]

Yeah I know you or others don't run unsupported kernels out of love for them, it's more a jab at distros like Ubuntu! But they are moving towards doing the right thing, at least.

[axboe]

As another general note, the 6.8 kernel is some ubuntu special which isn't part of stable and doesn't receive updates from me. It may have bugs, I don't know, as they don't use the stable series of kernels that I support. Probably not your issue here (I suspect old liburing), but worth mentioning as a general caution that I do not support these types of kernels, as it's simply not possible for me to do.