Slow pod readiness gate adding with many nodes and pods

[julius88]

Hello!

We are using AWS Load Balancer Controller (2.1.3) in our cluster. We are experiencing very random wait times for the pod readiness gate to be added.

From this state:

to this state:

Sometimes it takes 10 seconds and sometimes can be longer than 10 minutes. Our containers in the pod take usually about 30-60 seconds to start.

If there are 2-4 pods things go very smoothly. Problem occurs when there are 100-150 pods (70-100 nodes) running and a rollout restart or deployment update is executed. We are running t4g.medium instances as worker nodes with ALB in front of the service.

Why can it take a very long time to add the pod readiness gate to the pod conditions? I tried to dig in the source code, but couldn't find a reason. When the pod readiness gate has been added, transition to Status=True goes fast. I've tried changing targetgroupbinding-max-concurrent-reconciles to higher values, but it doesn't help.

[M00nF1sh]

@julius88
Is this a ALB or NLB? also, is there any logs in the controller that indicate throttle?

[julius88]

We are using ALB. I will try to get more verbose logs and see if there are anything related to throttling.

[julius88]

I didn't find any throttling related events in the controller's logs. I ran controller with --log-level=debug and here are my results. Testing contained following steps:

1. Run AWS Load Balancer Controller with --log-level=debug.
2. Change minimum pods for a deployment 2 -> 100.
3. Monitor AWS Load Balancer Controller logs.

Pods take 1-3 min to launch fully, depending whether a new worker node is needed and how fast that is launched.

Here is a portion of the logs and I can see something interesting there.
controller-logs.txt

You can find the row with "<95 targets>" in it and compare its timestamp to the previous one. The difference is almost 1000 seconds and that was the time pods were stuck in the state waiting for readiness gates to appear.

[julius88]

I noticed an interesting behaviour today.

We had a startup probe for our pod. It does some warmup tasks for uWSGI so full launch takes about 15-30 seconds longer with that. I removed that probe and basically this issue disappeared.

Could there be an issue when containers take longer to launch which then causes these delays in actions of the controller?

[M00nF1sh]

@julius88
Is there any chance that the health check to your pods is unhealthy after its enters containerReady condition?

We'll register your pods into ALB as soon as it enters container ready state. Then we keep monitor the target health.
If any pod in the target group is have health state of "initial", we'll re-query the state of target group after 15 second.
Otherwise, we'll re-query the state of target group in an exponential backoff manner from 10 ms to 16 minutes.

What could happen in your case is when your pods get registered to ALB, it's still not ready to take traffic, so ALB's health check will fail. Then our controller keep exponential backoff retry monitor the target group, which reaches the 1000 second.

Do you think whether it's what happening to your case(application takes a relevantly long time to start accept traffic after container ready)? If you are not sure, are you ok to run a patched controller with more logs to verify it?

BTW, if it's indeed this case, i think we can tune the exponential backoff behavior or change the logic deciding when to register the pod. we have the exponential backoff to avoid spam query target group if customer used a wrong application that never becomes healthy.

[julius88]

I believe our issue was caused by #1974. I can see that a PR #2029 has already been merged. I will test that fix and come back.

[julius88]

I've built the latest image for the controller and tested PR #2029 and that fixes our issue too. Thanks a lot!