Downloading data from InputStream

[andlinger]

Hello!

We provide an MP4 file for download via a Quarkus endpoint. This MP4 file comes from an S3 server that we use via the Minio extension. We receive an InputStream from the Minio library. We return this with a RestResponse.

However, when this GET endpoint is called, current browsers only read the headers and not the body, because the “Accept-Ranges: bytes” header is sent along with it. The browser then starts a new call with a “Range: bytes=x-y” header.

The problem is that with the current implementation, the entire InputStream is read and written using the transferTo method on the first call, even though the web server never reads it. And if the video file is 500MB, for example, this causes a relatively large amount of network and CPU overhead for nothing.

Is there any way to make this more efficient?

Example:

@RequestScoped
@Path("/download")
@PermitAll
public class ExampleResource {

    @Inject
    S3Service s3Service;

    @GET
    @Path("/{id}")
    public RestResponse<InputStream> downloadVideo(@RestHeader("Range") String range, @RestPath String id) {
        // Client doesn't send Range header. Read full file
        if(range == null) {
            InputStream stream = s3Service.download(id);
            // This entire input stream is read and written by Quarkus using transferTo, even though the browser never reads the data.
            return RestResponse.ResponseBuilder.ok(stream)
                    .type("video/mp4")
                    .header("Accept-Ranges", "bytes")
                    .build();
        } else { // Client sends Range header. Read provided range only
            Range range = Utils.parseRangeHeader(range);
            InputStream stream = s3Service.download(id, range);

            return RestResponse.ResponseBuilder.create(RestResponse.Status.PARTIAL_CONTENT, stream)
                    .type("video/mp4")
                    .header("Content-Range", Utils.toContentRangeHeader(range))
                    .build();
        }
    }
}

[dmlloyd]

So the client is definitely sending a GET with no range, and then discarding the resulting stream, before requesting a subrange? Could it possibly be sending e.g. a HEAD request first that is not being handled?

[andlinger]

Yes, that's right. I also implemented a HEAD endpoint. It was never called. I tested it with the latest versions of Chrome, Safari, and Firefox.

You can also recreate this with this test video.
https://test-videos.co.uk/vids/bigbuckbunny/mp4/h264/1080/Big_Buck_Bunny_1080_10s_1MB.mp4
The browsers react the same way here. First, a normal GET request. This is not read completely, and then a new GET request with a range header is created.

[dmlloyd]

I spent an unreasonable amount of time researching this and found no answers in the standards. But here are some things I would try:

• Try sending a 206 with 0 bytes if you don't get a Range (or maybe only do this for certain user-agents if some behave correctly)
• Maybe try a 416 as if they had requested an invalid range?
• Try a 206 but instead of 0 bytes give some content?

Also, compare the behavior of different browsers, because sometimes there are clues hidden there.

Good luck! If you find something that works, do share it.