Add a priority to a backendRef

[arkodg]

What would you like to be added:
Add a field within backendRef allowing the user to specify the priority of a backend. The algo for traffic distribution could be the following

1. Pick all the healthy backendRefs with the highest priority level
2. Distribute traffic across them based on the weight ratio
3. If no healthy backendRef exists for the highest priority, perform above steps for the next priority

Why this is needed:

• Cross cluster failover (prioritize service within the cluster before routing outside)
• As a user I would like to specify a default / fallback / back up backend in case my main backend is no longer active
  also relates to https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#default-backend

[shaneutt]

To me this is reminiscent of kubernetes/kubernetes#56440 (also relatedly, kubernetes/kubernetes#115917) where we were discussing fallback positions for traffic when things go wrong and we were starting to discuss whether Gateway API might be a good place for such a thing (as opposed to Service which we're trying to avoid adding anything to). Was that intentional? If not give those a read and let me know what you think 🤔

[arkodg]

thanks for sharing the links @shaneutt, the intent is similar (sending traffic to a backup backend), but that intent needs to be defined at the backendRef level rather than the specific type of backend (Service) so traffic can be also prioritized to different backend Kinds

[howardjohn]

Similar use case is things like "give priority to local endpoints within a service" (where local could be node, region, zone, cluster, mesh, network, or arbitrary things). Perhaps that is a different case though

[arkodg]

sidenote - if priority as a noun is hard to comprehend, another option is to add a defaultBackendRef or backupBackendRef (which reduces optionality to 2 priorities instead of N priorities)

[howardjohn]

Another alternative is to make a non-Service backend that represents some "Composite" backend. Which could be implementation specific or not. Related: #1764

[shaneutt]

Seems like between us and the fact that the kubernetes/kubernetes has a relevant similar use case which @arkodg identifies with, we would just need someone to champion this issue, bring it up at the community meetings, and start the GEP. Are either of you particularly interested in leading on this one?

/help

It's also not clear to me whether this is something we need to resolve before GA, so for now I'm not marking it as such. If someone who wants to champion this feels strongly that it should be something we do for GA please make a case for that.

[k8s-ci-robot]

@shaneutt:
This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• Does this issue have zero to low barrier of entry?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

Seems like between us and the fact that the kubernetes/kubernetes has a relevant similar use case which @arkodg identifies with, we would just need someone to champion this issue, bring it up at the community meetings, and start the GEP. Are either of you particularly interested in leading on this one?

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.