Allow codeflare-operator-mcad-controller to create Ingress

[tedhtchang]

Name of Feature or Improvement

Allow codeflare-operator to create Ingress.

Description of Problem the Feature Should Solve

This is allow project-codeflare/codeflare-sdk#251 to create ingress

E1017 22:21:05.375392 1 queuejob_controller_ex.go:2064] [manageQueueJob] Error dispatching generic item for app wrapper='default/hfgputest-1' type=ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:openshift-operators:codeflare-operator-controller-manager" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope err=%!v(MISSING)

Describe the Solution You Would Like to See

Add codeflare-operator-mcad-controller-ray-clusterrole

- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch

Describe Alternatives You Have Considered

Description of any alternative solutions or features you have considered.

Additional Context

Add any other context, screenshots, console logs, etc. about the request here.

[astefanutti]

@tedhtchang could you confirm the approach taken by project-codeflare/codeflare-sdk#251 is to replace Routes creation with Ingresses entirely?

[Bobbins228]

@astefanutti Yes with project-codeflare/codeflare-sdk#251 we no longer create routes with the SDK but create ingresses instead.

[astefanutti]

@Bobbins228 thanks a lot for the feedback. So we'll remove the extra permissions on Routes that we had previously added in the KubeRay ClusterRole.

[Bobbins228]

@astefanutti Yes that sounds good to me 👍

[tedhtchang]

I think until this PR is merged into the next SDK release we will need both route and ingress; otherwise current SDK release cannot create route.