Relax config check when Discovery is enabled

[tsandall]

Currently OPA does not allow users to supply plugin config when Discovery is enabled. The reason for this was to prevent OPAs being deployed that could not be controlled through discovery. In some cases though we've found that the user deploying OPA must be able to make local overrides even when discovery is in place, e.g., because the system serving the discovery config is unaware of all options available in OPA.

To address this we can relax the configuration check when discovery is enabled so that the bootstrap configuration can contain plugin configs. In case of conflicts, the bootstrap configuration for plugins should win. These local configuration overrides from the bootstrap configuration could be included in the Status API messages so that management systems can at least visibility into the local overrides.

For example, to enable console decision logging when an opa-config.yaml relies on discovery, the user would just need to:

opa run -c opa-config.yaml --set decision_logs.console=true

[anderseknert]

Related: #5070

(not a duplicate, but if I'm reading things right, fixing this would likely fix that too, so adding here for tracking)

[mjungsbluth]

PR #5314 already did some work on this and left it to the plugins what can be overridden (my assumption was that some things should just not be overridable but that might be wrong). It uses the same merge values method that is used by the command line‘s —set args if I remember correctly

[mjungsbluth]

Since this is removed from planning and we are hit by this: Is there agreement on doing it as described?

• Overriding of any configuration is possible with local bootstrap config (or transitively —set)
• The overrides are added as a delta in the status API

The linked PR does no. 1 but is more specific, i.e some conditions would need to be removed.

What is missing is essentially the delta tracking and publishing this via status API

[tsandall]

@mjungsbluth we've mostly seen this request come from Styra DAS users, however, since Styra DAS now supports the ability to override the discovery config (via the API) we decided to deprioritize the work in OPA. Here is a link to the Styra DAS docs that show how to override the discovery config: https://docs.styra.com/das/policies/policy-organization/systems/opa-discovery#add-or-override-a-value-in-discovery-configuration.

[mjungsbluth]

@tsandall understood and I have seen that change which mitigates the problem for now.
There is however an aspect of split accountability in this as well as outlined in #5070 where the team that owns the config (SRE or some cloud infrastructure department) can be very different / far away from an IAM engineering department that owns Control Plane + central policies.