How to make kwok controller handle pod faster

[jokerwyt]

I need to create 2000 pods per second for benchmarking and I found that kwok controller cannot assign pod ready status so fast. What can I do to make it catch up the speed I create pods?

[wzshiming]

I suspect your bottleneck is that the kube-scheduler isn't fast enough

You can try the following steps to speed things up

• Fill the nodeName of pod to bypass kube-scheduler
• Twisted kube-apiserver qps limits on client's connections

[jokerwyt]

It was made by the log of kube-controller-manager (i.e. the replicaset controller). That controller watches pod creation and pod ready status to calculate replicaset replicas number and ready replicas number.

Is it possible to compare different parallel numbers? Yeah I will do that comparison if I needed, and post the result here.

[wzshiming]

So that means kube-controller-manager creates pods faster than kwok updates them...

[wzshiming]

Is the kwok-controller on the same node as the kube-apiserver?

[jokerwyt]

Yes. I suspect that ETCD is the bottleneck. It cannot handle such high rate of POST and UPDATE mix workload

[jokerwyt]

I introduced two Prometheus metrics to monitor how fast kwok-controller preprocesses and playstage. Here is the result when creating 1k pods per second for 1 minute (4:38:45~4:39:45). I don't know why the kwok controller playstage speed rises after 4:39:45. It does not look like cpu pressure because overall machine CPU usage is about 60%.

One possible explanation is ETCD performs better after 4:39:45 because there is no pod creation anymore.

[wzshiming]

I plan to find the bottleneck and fix it. #1283

[ali-a-a]

@jokerwyt

I had the same issue with KWOK as I wanted to load my K8s cluster using CL2. My test was creating 2000 deployments with 30 replicas at a rate of 50 deployments per second. I faced many PendingScheduled pods waiting for KWOK to change their status to Running. I worked around this by running multiple KWOK processes, each managing some worker nodes using annotation selectors. In my case, having 4 KWOK processes could catch up with K8s components.

[jokerwyt]

Yeah, I think this proves that Kwok processes is a very possible bottleneck.

[wzshiming]

I've been busy lately, and I'll focus on improving performance in the next release.

[wzshiming]

Sorry for taking so long to reply.

Lately I have had some time to devote to this project, I have been debugging the root cause which is QPS limiting apiserver to kwok-controller.

I've tested that increasing concurrency doesn't improve much, adjust APF configuration for maximum improvement #1341