User reports for TCPRoute and UDPRoute

[youngnick]

This discussion is for folks who are actively using TCPRoute and UDPRoute. The community would like to know what you're using them for, and how you find the experience?

[pamelia]

I've been collaborating with a client on a project that focuses on secure video meetings. The service leverages WebRTC for real-time communication and incorporates the STUNner project as a WebRTC media gateway, running seamlessly in Kubernetes. STUNner uses Gateway API with UDPRoute for efficient traffic management, which has significantly enhanced scalability and reliability. It's been a fascinating experience working on such cutting-edge technologies to deliver secure and high-performing video communication solutions running natively in Kubernetes.

[LionelJouin]

I investigated Gateway API with TCP and UDP routes for a potential implementation of a Kubernetes Load-Balancer for Telco use-cases.

The first thing I noticed is the port and protocol that must be written in listener (Gateway), ParentReds (Route), BackendRef (Route) and Service. I have had a hard time to understand how the listeners field fits with the Routes on layer 4 (TCP/UDP Route) and potentially lower layers. Is it really required? Maybe a better solution could be to have the possibility to allow any port and any protocol? Here are some related discussion related to this: #130 ; #780 (comment) ; #818 ; #1061

For my use case on telco load-balancer, I would probably also need some kind of SCTP Route. But, in my opinion, a Layer 3 and 4 Route would probably make more sense. The route would match based on Src/Dst IPs and Ports + Protocol, and use some kind of service as a backend which would define the actual endpoints.

Few month ago, I made a PoC in this area, I would be happy to demo what I have to the community if there is an interest: https://github.com/LionelJouin/l-3-4-gateway-api-poc
It is about layer 3 and 4 Routes and (Kubernetes) service decomposition. In this PoC, the Gateway workload is collocated with the service implementation.
The 2 main experiments in this PoC are the following:

1. Service as a Route
   • The Kubernetes service object is considered and used as route + load-balancer (service.kubernetes.io/service-proxy-name indicates in which Gateway the service must be running).
   • It (Kubernetes Service) is considered as a route as we can specify the Port + Protocol + IPs.
   • It (Kubernetes Service) is considered as a load-balancer as we have a selector selecting the endpoints and load-balancing the traffic towards them.
2. Service decomposition: Layer 3/4 Route + Load-Balancer
   • New Route L34Route pointing to a service (backendRefs).
   • The service is a headless service containing only the Selector field (Maybe the service object could contain traffic manipulation fields. E.g. Load-Balancer algorithm, NAT being applied, Port forwarding, ...).

Probably what I said is also related to this discussion: #3351
We (I and some people from SIG-Network) also had some discussion around this topic in the hallway of Kubecon NA.

[kflynn]

Linkerd uses TCPRoute in managing egress. We added an EgressNetwork CRD to be a parentRef to define the class of "traffic leaving the mesh", gave the EgressNetwork the ability to specific default policy for that class, and then we allow Routes to express which egress traffic should be allowed. At present, we support GRPCRoute, HTTPRoute, TCPRoute, and TLSRoute in this role.

There's more detail in the Linkerd egress documentation.

[MeydanOzeri]

I worked on 2 use cases where UDPRoute was needed.

1. Deploying Janus inside Kubernetes to facilitate audio + video calls, requires UDP to work, and before it was supported we needed to deploy Janus on a VM introducing scaling difficulties.
2. Using WebTransport, the WebSockets successor which uses https3 and the QUIC protocol which requires UDP.

[mikemorris]

I'm curious if we may eventually want to add an explicit ProtocolType enum value for Gateway listeners to support handling http3/QUIC with an attached HTTPRoute (or are there capabilities for which we would actually want an entirely new route type?) rather than falling back to the rather basic functionality of UDPRoute.

[MeydanOzeri]

For WebTransport you currently need 2 routes, TLSRoute and UDPRoute, i didn't check but i assume even for regular http3 without WebTransport you still need both routes, or an HTTPRoute with TLS terminate instead of the TLSRoute with passthrough.

Since both TCP and UDP are needed behind the scene I'm not sure how it can be done using only HTTPRoute which attaches only to a TCP oriented gateway listener.
In order to be future proof and allow adopters to use WebTransport once it's not experimental anymore, i advise to support direct UDP capabilities, either with the existing UDPRoute or with a new route type.

WebTransport has 2 modes, reliable and unreliable, so either you use QUIC to facilitate traditional WebSocket like connections just on top of UDP and not TCP, or you use unreliable and send datagrams directly on UDP which is the preferred solution for media transfers, online gaming and other real time apps.

[mikemorris]

Since both TCP and UDP are needed behind the scene I'm not sure how it can be done using only HTTPRoute which attaches only to a TCP oriented gateway listener.

Yea, I'm basically questioning from a spec author and Gateway API implementing project perspective whether/what we might want to add to the spec to better serve this use case. It's really interesting what it sounds like you've been able to get working, but the UX of needing two routes feels a bit cumbersome and something we should maybe try to improve!

[MeydanOzeri]

Well, from this point of view it certainly would be better from a dev UX perspective to achieve it with a single route and listener, spares the need to go low level and deal with the underlying protocols of http3 which i assume a lot of devs might find cumbersome.

Perhaps adding a ProtocolType of type 'quic' and allowing normal HTTPRoutes to attach to it for majority of traditional use cases + a QUICRoute or the existing UDPRoute to allow for more flexibility and control in cases like datagrams on top of WebTransport.

[youngnick]

Great feedback, thanks @MeydanOzeri - we've talked about QUIC before, but I don't think any of us had enough experience with it to feel super comfortable designing something.

[christian-stephen]

At CircleCI, we make use of TCPRoute for routing SSH to specific Pods in order to support our SSH reruns feature in container runner, which runs CircleCI jobs in self-hosted Kubernetes environments. There are some more details here, and the Helm chart for container runner is open source.

Our customers have had success with both the Envoy and Istio implementations of this resource.

[shaneutt]

Does anyone here have an implementation using TCPRoute and/or UDPRoute today that can help us develop our conformance tests, and then report results so we can move these APIs out of experimental?

I have https://github.com/kubernetes-sigs/blixt which isn't far from being able to do this, but it would help if we had a few allies coming together to do this.

[howardjohn]

Istio can do TCPRoute but not UDPRoute

[arkodg]

Envoy Gateway's e2e has these TCPRoute and UDPRoute tests that can be easily moved into gateway-api conformance tests

[shaneutt]

Cool thanks guys! So that's great, just need more UDPRoute... @rg0now are you potentially interested?

[rg0now]

Our integration test suite is a mess so I'm afraid there's nothing I could contribute for now. We need to refactor the test suite anyway so once that's done I'll get back here and update this thread.

[rg0now]

As @pamelia mentioned, we use UDPRoutes in STUNner to ingest WebRTC traffic into Kubernetes clusters transparently. WebRTC is special though as media servers (the backends) typically expose tens of thousands of UDP ports and clients need to be able to reach any of these ports via a single UDPRoute. Thus STUNner ignores the Service/UDPRoute port all together, which has become terribly confusing to our users once the port in *Routes became mandatory. To minimize confusion we ended up forking the Gateway API to provide our own UDPRoute that tries to mimic the [port:endPort] semantics from this age-old issue.

Eventually I arrived to the conclusion that it is maybe better for us to provide our own UDPRoute rather than forcing WebRTC's braindead port handling semantics to all Gateway API users.

[youngnick]

Folks implementing TCPRoute and UDPRoute today, does your implementation do layer 4 forwarding or proxying?

Proxying is where the Layer 4 packets are received by the Gateway implementation, and then new connections or sessions are sent to the backend. For TCP sessions, this means that the TCP sessions are terminated by the Gateway implementation (the Gateway handles the SYN/ACK/FIN/RST etc), and a separate TCP connection to the backend is built. For UDP packets, the backend will see the Gateway implementation as the source IP of the packets.

Forwarding is where the packets are forwarded on to their recipients while only doing a maximum of some form of NAT. No termination of sessions (for TCP) or crafting new packets (for UDP) takes place.

This is super important in my mind, because proxying implementations can't easily do Client IP address visibility without networking in the cluster supporting some form of DSR.

This is not an issue for protocols like HTTP that have higher-layer mechanisms for passing the actual client IP address through proxies (like X-Forwarded-For), or if you are using an encapsulation protocol like Proxy protocol.

So, my questions for implementations that support TCPRoute and UDPRoute are this:

• Is your implementation a proxying one or a forwarding one?
• Do your use cases require Client IP visibility? If so, how do you achieve that?

I think at a minimum, any conformance tests we add for these features should include at least an Extended feature called ClientIPVisibility or similar, that allows users to know if that feature is available if they need it for their use case.

[rg0now]

STUNner: proxying with protocol conversion (TURN/UDP->UDP). ClientIPVisibility is handled by a separate out-of-band IETF protocol called ICE.

[arkodg]

@youngnick yes, proxy protocol needs to be enabled in downstream and upstream, and the upstream/backend will need to do some work to retrieve the value

[shaneutt]

Blixt is notably a bit different here, as it is effectively a Service implementation (I say effectively because we don't have official support for Service yet, we plan on that after kubernetes/enhancements#4771 is completed) which also supports Gateway + TCPRoute as an analog to Service, and therefore does the "forwarding" option here utilizing eBPF programs (in a way slightly reminiscent to Cilium).

[mikemorris]

By DSR you mean something like https://www.haproxy.com/glossary/what-is-direct-server-return-dsr I'm assuming?

@youngnick Curious what you're looking at re client IP address visibility, something like #1141 or NetworkPolicy implementations, or more like Cilium AuthZ?

[youngnick]

@mikemorris for DSR, yes, exactly. If a proxy does a transparent proxying of a packet, then the original source address is visible, and the backend will see it as the source address. That means that the return flow from the backend will go straight back to the original source, bypassing the transparent proxy. Note that this really only works if pods use publicly visible IP addresses, or you have a cloud LB or similar that's also the default gateway for the network, as many networks will drop RFC1918 traffic from the internet as bogons.

For Client IP visibility, it's just been my experience that most applications that use bare TCP or UDP, not HTTP or other higher-layer protocol really care about being able to see the actual source IP address. SSH connections really want to be able to log where the connections are coming from, game servers that use UDP really want the same for IP banlists, and services like syslog or dns depend on client IP visibility for similar in-application reasons.

So just having implementations that proxy TCP or UDP and then say "sure, we support TCP and UDP" may not end up delivering what users actually want.

[LarsStegman]

We have hundreds of IIoT devices that send UDP packets at around 50Hz. These packets are sent to our ingress ip address and then routed to the associated Service that is running within the cluster. Packets are routed based on the receiving port, but we would prefer to route them based on a header on the binary payload. This is kind of a stretch goal, though.

We currently do this with ingress-nginx.

We do not need to respond to a device for it to send packets, so it's purely passive listening.

[youngnick]

Hi @LarsStegman, thanks for the use case. In this case, is preserving the original client IP important? Is it important that the backend service sees the source of the traffic as original source (forwarding), or could a Gateway implementation terminate and restart connections, having it visible as the source (proxying)?

[LarsStegman]

Currently, we do not use the source IP, however I can imagine that at some point we do need it for identification/logging purposes.

[mohag]

We would like to move some Kafka TCP stuff to TCPRoute from ingress-nginx, but there are few implementations with TCPRoute support. (Proxying is fine for that use case) (currently strimzi, previously the Bitnami Kafka chart) (Ingress / Gateway API makes certificate management a LOT easier with cert-manager) Other possible use-cases include MQTT. (HTTP is used for Kafka-UI on the same hostname)

UDP is sometimes used as well, currently mostly proxied through a custom nginx-based pod. (Where UDPRoute would make a nice replacement) (Protocols used include CoAP and SNMP (trap reception mostly)) (proxying is used)

The main reason to use it is to share a hostname / IP with HTTP(S) services (and sometimes using those to automate certificates with cert-manager)

(A specific class for the PROXY protocol might be useful as well for preserving IP information if the destination service supports it) (It would probably need a standard way to enable it on LoadBalancer services though...)

[jonasbartho]

Non-"stable" gateway api CRDs like TCPRoute and UDPRoute are not allowed to use in OKD/OCP 4.19(it can bork OKD/OCP when upgrading), which makes gateway api unusable for me at the moment. (Note that I am not using istio operator in Openshift, and do not want to use it)

When using ingress nginx controller(which will be discontinued, hence why i am looking for alternatives), we enabled tcp/udp in the configmap for some customers where this was a use case. So basically that the ingress nginx controller could accept tcp/udp traffic and forward it on a non-traditional port(so not the default port 80/443) to a backend pod that would expect tcp or udp traffic. This worked perfectly.

I basically want to be able to do the same in gateway api, and it looks like this only is possible with TCPRoute /UDPRoute which is not allowed since it still is in alpha and not in stable.

I preferably do not use a Service and use a separate LB due to security/routing challenges.

[jonasbartho]

Background: We run multi-tenant Kubernetes/OKD(=free version of Openshift) on openstack in combination with dedicated High Available frontend servers for each customer with nginx/apache/varnish/... for complex network management/capabilities.

These frontend servers service as our loadbalancers to have full control (instead of the built in "Octavia"-LB that comes with Openstack). Not that these frontend servers also serve non-k8s traffic.

It makes more sense to us to use a shared nginx ingresscontroller OR shared EnvoyProxy/Gateway with TCPRoute/UDPRoute(where needed) per customer that for example runs on dedicated infra nodes that can accept extraordinary ports/protocols and route it to correct ingress/service/pod instead of opening up extra ingress ports on customer worker nodes where the backed application is running. We preferably keep these worker nodes as closed down as possible ingress wise.

Thank you for using time on this. :)

[mohag]

The ability to use something supported by cert-manager (like ingress or HTTPRoute) on the same IP / set of IPs as the TCP / UDP services helps a lot for HTTP-01 validation of certs for the services. (DNS validation might be an option, but with non-enterprise Cloudflare, that requires API creds that can take over your domain to be present on the cluster)

We often use this for certificates for e.g. Strimzi

(Often reducing the hostnames by providing HTTP(S) and and other services on the same hostname is also useful)

[youngnick]

Thanks for the use cases. In these use-cases, is it important that the original client IP is visible?

[mohag]

Original client IPs are always useful for logging / blocking.... (requiring that limits the implementation options though - it feels like a "SHOULD" rather than a "MUST" in RFC terminology) (It is more relevant on the non-HTTP routes - HTTP normally have options to add it to headers)

[youngnick]

There's so much difference in behavior between proxying and forwarding of Layer 4 traffic though, that's the reason I keep asking this question. SHOULD rather than MUST is a useful way to put it, thanks.

It really feels to me like the client IP distinction is crucial enough to require specificity in the API requesting it. That is, it feels to me like there should be a way to say "the original Client IP MUST be visible", because some use cases will have a hard requirement, and others will not. That suggests that we need to find an API that lets users specify if they care about client IP visibility, and allow implementations to say "oh, I can't do that", and reject Routes they can't handle.