Transport protocols to support live streaming HDR10 and HEVC in OBS

[khaller-youtube]

With release 27.1, the integration of the YouTube API in OBS studio landed. We would like to publicly thank Jim, Joel, Ben, Matt, Rodney and everyone that contributed to and tested that integration. The passion and engagement of the OBS community is amazing to see. For example, the community addition of chat support was a great improvement to the release.

In December of 2020, YouTube launched support for live streaming High Dynamic Range (HDR10) content and have since been working on making it easier for creators to live stream in HDR. Prior to that launch, we began discussions with the OBS maintainers about potentially supporting HDR10 in OBS. With the launch of the YouTube API, we have turned our focus into making this happen.

What we are working on (more details will be provided in an official RFC):

1. Adding an option in Preferences->Advanced->Video for HDR10 support.
2. The ability will only be enabled if your graphics system supports HDR10.
3. Content that is not HDR will be up mapped so the output can be encoded as HDR10. E.g., if you are capturing HDR content, but have a webcam that is SDR, the system will map the webcam content and produce HDR10 output for the video.
4. Recordings as well as streaming will be supported.

Following an approved RFC, we will initially submit enablement on Windows, with tested support on nVidia, AMD, and Intel graphics cards.

Next, we will release support for Linux, followed by MacOS (Intel only).

We have verified that we are able to generate HDR up to 1080p30 in h264 using software encoders. H264 encoding of HDR10 is not supported in graphics cards. An alternative would be to encode in hardware using a more modern codec: VP9, AV1 or HEVC. Only HEVC encoding is currently well supported by the major graphics cards.

Unfortunately, RTMP does not support any of these codecs. YouTube currently accepts HDR10 in HEVC using the HLS protocol. HLS is supported in OBS.

Facilitating use of HEVC streams in OBS will be a separate and independent RFC.

The change to support HDR10 is large. It will take support and testing from the community to make it happen. It will also take a lot of work from the OBS maintainers to review and qualify.

Some questions and discussion points for the community:

• H264 software encoding will not be great. If advanced codec support is completed after this work, would H264 support of HDR still be of value to you?
• Is using the HLS protocol a blocker for you? Would you be open to a QUIC based protocol if we could get adoption by other streaming platforms?

Additionally, there are a large number of external plugins for OBS. Those plugins may need to be updated to support HDR10.

Thank you!

[derrod]

Is using the HLS protocol a blocker for you? Would you be open to a QUIC based protocol if we could get adoption by other streaming platforms?

Are you suggesting to work with one of the existing efforts to build a protocol using QUIC for video transmission, such as Facebook's "RUSH", or do you mean a new development effort?

Personally I do not see HLS as a blocker, the latency disadvantage probably gets more than made up for once we're going HDR.

Facilitating use of HEVC streams in OBS will be a separate and independent RFC.

Is this purely technical or are you talking about licensing? You can already stream HEVC via HLS to YouTube if you compile OBS yourself and remove the restriction to H.264, so that itself probably does not require much work (aside from blocking the selection of HEVC if the current output protocol does not support it).

[khaller-youtube]

We want to make HEVC easier, more complete, and have some licensing protection for OBS.

[dodgepong]

It's also worth noting that WebRTC output is also on the OBS roadmap while we watch new protocol development.

[khaller-youtube]

In a recent "Kranky Geek" conference, our friends over at Google Meet showed that WebRTC with AV1 was viable for latency sensitive applications. We would need to test if that holds true with high quality live streaming (4K, HDR, etc). Definitely something to pay attention to...

[derrod]

We want to make HEVC easier, more complete, and have some licensing protection for OBS.

I'm definitely curious what the proposal is. Licensing has been the main reason we did not want to touch HEVC so far.

[khaller-youtube]

Not a lawyer, but lawyers are involved. Can't comment further on a public forum.

[jpark37]

The ability will only be enabled if your graphics system supports HDR10.

Isn't software AV1 supposed to be decently fast these days? Maybe (EDIT: the ability for) HDR should always be allowed if HDR over software is doable in real time?

there are a large number of external plugins for OBS. Those plugins may need to be updated to support HDR10.

As much as possible, plugins should be allowed to live in an SDR bubble without needing to be updated. You probably already meant this, but I'm just emphasizing.

I've done plenty of HDR work for my day job recently, so I have ideas/opinions how OBS should implement certain aspects. It would be nice if the RFC was made available ASAP, so we can get those details squared away.

[khaller-youtube]

Re: HDR if HDR can be done in real time.

I think that we can look at this. Getting HDR right, especially from a system that does not support HDR, is tricky. Despite that, we wanted to support HDR coming in from an external camera or from a gaming console, even on systems without an HDR monitor. Our choice was to look to see if at least one hardware video encoder was available that supports 10 bit color.

Re: Getting access to code.

There is a private github repository that is currently under internal review. We are open to giving early access to OBS contributors.

[jpark37]

Having an external factor like system hardware gate a feature may be problematic. What should OBS do if someone copies their profile with HDR enabled to a system that doesn't support have a 10-bit video encoder? (This is assuming there's no good software encoder, in which case, we should just always allow HDR to be enabled.) I guess we have similar challenges when they select a hardware encoder like NVENC, and swap in an AMD card. (I don't know what OBS does off-hand.) Regardless, I would anticipate less friction if we allow users to tweak their profile/scenes freely, and only stop them if there's a hard reason they can't stream/record at the time they attempt to stream/record.

I'll email you separately about private repo access.

[jpark37]

Adding an option in Preferences->Advanced->Video for HDR10 support

We may want to keep internal and output formats separate. Current OBS always composites to BGRA8 (sRGB), and can output to a variety of pixel formats (I420, NV12, etc.). I'm thinking we could eventually add support for a variety of internal formats via dropdown UI:

• BGRA8 (sRGB)
• RGBA16F (linear scRGB, 80 nits per unit) - probably fastest for HDR (least conversions?), but can't properly blend wide gamut colors since they are represented by negative values
• RGBA16F (linear Rec. 2020, paper white adjusted) - wide enough to handle game consoles
• RGBA16F (linear ACES AP1, paper white adjusted) - professional color work?

We could even use RGBA16F linear scRGB to composite for high-precision SDR someday even if we don't have plans to implement that in the near future. I think we want to leave that possibility open via enum & UI. For now, just restrict the output pixel formats to be mutually exclusive:

BGRA8 -> current formats
RGBA16F -> P010 (PQ, Rec. 2020), etc.

I would start with scRGB for the internal HDR format given that OBS has historically has favored speed over accuracy. And when I say we might eventually add more, that could mean "years" or "never."