libobs-opengl: Treat pixel format 0 in IOSurface as 32BGRA; mac-syphon: Use opaque effect on non-OpenGL renderer

[gxalpha]

Description

Per the Apple documentation, when 0 is returned from IOSurfaceGetPixelFormat, the IOSurface is invalid. Prior to 53ad05d however, any invalid surfaces were (likely accidentally) treated as 32BGRA.
And as it turns out, any IOSurface received from Syphon does not have the pixel format set, thus returning 0. Because the pixel format happens to always be 32BGRA, this was never caught. 53ad05d then slightly refactored the code, fixing the bug not realizing it ever existed. This lead to Syphon no longer working.

Treating 0 as 32BGRA appears to be the only way to fix this situation for now and restore prior behavior. Ideally in the future Syphon provides valid IOSurfaces. An upstream fix to Syphon-Framework is already submitted at Syphon/Syphon-Framework#103, however even if merged, servers would need to update to that fixed version before things work again. Therefore, this workaround is needed on the OBS side.

However, the plan for now is for the Metal renderer to not get this relaxation. It never had it (so Syphon never worked there, making this not a regression) and it's marked as experimental. Hopefully, by the time it's stable and/or default, Syphon will be fixed and servers have been updated.

There is a fix for the Metal renderer included in this PR however: Testing with the patched Syphon server showed that the DrawOpaque technique, used when "Allow Transparency" is off, does not exist in default.effect which is used with the Metal renderer. Therefore, I added that technique.

Motivation and Context

Fixes #12684.

How Has This Been Tested?

Using the OpenGL renderer: Used the Simple Server from https://github.com/Syphon/Simple and confirmed it now works with OBS (with no changes to Syphon other than updating the submodule to latest master).
Using the Metal renderer: Used the Simple Server from https://github.com/Syphon/Simple but with Syphon/Syphon-Framework#103 included in the submodule, and confirmed that that worked. Without the Syphon patch, confirmed that it still doesn't work, stopping at at the assertion complaining about the wrong pixel format.

Types of changes

• Bug fix (non-breaking change which fixes an issue)

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

[gxalpha]

Update: The upstream Syphon fix (Syphon/Syphon-Framework#103) has been merged. Apps can make themselves compatible with the currently released version of OBS (32.0.1, only OpenGL renderer) by updating to that version of Syphon Framework (current main branch, at least commit Syphon/Syphon-Framework@71351d4).

The relaxation made by this PR for the OpenGL renderer is still pretty much required because many apps likely won't update immediately. For the Metal renderer, both this PR (the shader fix) and an update by the developers of server apps to the new Syphon Framework are needed.

[PatTheMav]

Just a nitpick: The PR title doesn't entirely match the changes now that we moved the opaque rendering change into the Syphon module.

[RytoEX]

It would be nice if we could differentiate between "fixed" and "non-fixed" Syphon, but I don't see how we can do that since there is not yet a tag or version with the fix in it.

[RytoEX]

One comment, but seems fine otherwise.

[RytoEX]

Could be more specific that this is before Syphon/Syphon-Framework@71351d4 or Syphon/Syphon-Framework@2dc6d31, but it might not matter much.

[PatTheMav]

Not to put too fine a point on it, but this is not specific to Syphon: Any invalid IOSurface will be interpreted to use BGRA32 implicitly, regardless where that IOSurface might come from (e.g. CEF).