Common tag prevents subnet discovery after upgrade

[ChristianHersevoort]

The old v1 ALBIngressController automatically added the "common tag" with the value "shared" to a subnet when it was discovered, unless it was "owned" by another cluster. This auto tagging behavior seems to have been removed from the v2 loadbalancer controller.

As expected; after upgrading to v2 the previously created "common tags" remain and are not created for new clusters.

However when I create a new cluster the aws-loadbalancer-controller isn't able discovery subnets
{"level":"error","ts":1623679233.7658975,"logger":"controller","msg":"Reconciler error","controller":"ingress","name":"app-2048","namespace":"default","error":"couldn't auto-discover subnets: unable to discover at least one subnet"}

The subnets have the following tags;

• kubernetes.io/cluster/existing-cluster-name: shared
• kubernetes.io/role/elb: 1
• kubernetes.io/role/internal-elb: 1

I expected this to work because the subnet is marked as "shared" however this logic might've been (accidentally) removed in pr;
#1773 so it looks like "shared" is treated the same way as "owned"

A workaround is to manually remove the existing "common tag".

Or am I doing something else wrong?

[kishorj]

@ChristianHersevoort, in addition to the cluster tagged (shared or owned) subnets, controller will also consider subnets without the cluster tag during auto discovery. I suspect your subnet is tagged only for cluster existing-cluster-name and not for the new cluster you created. You can either remove the cluster tag altogether, or tag the subnet for the new cluster as well.

[ChristianHersevoort]

@kishorj Yes you are correct, the subnet is only tagged for a cluster named "existing-cluster-name" and not for "new-cluster-name"

When creating a new cluster I expect the subnet to be discovered even when a "shared" tag exists for an existing cluster

I've tested the following scenario's:

Scenario	expected subnet to be discovered	actual result
"new cluster", subnet without tags	✔️	✔️
"new cluster", subnet with tag "shared" for "existing-cluster"	✔️	❌
"new cluster", subnet with tags "shared" for "existing-cluster" and "shared" for "new-cluster"	✔️	✔️
"new cluster", subnet with tag "owned" for "existing-cluster" tag and "shared" for "new-cluster"	❌ /❓	✔️
"new cluster", subnet with tag "owned" for "existing-cluster"	❌	❌
"existing cluster", subnet with tag "shared" for "existing-cluster"	✔️	✔️
"existing cluster", subnet with tag "owned" for "existing-cluster"	✔️	✔️

[M00nF1sh]

@ChristianHersevoort
We currently didn't defined any semantic around the "owned/shared" keyword.
Previously before k8s 1.19, subnets is required to have the k8s...cluster/new-cluster: owned/shared tag.
on k8s 1.19, we relaxed the restriction to also consider subnets without any other cluster Tags. we did this change to both in-tree code and this AWSLoadBalancerController.

we are still discussing whether to deprecate the common tag with owned/shared value, thus we currently don't have plans to further relax this restriction.

So you'll need to tag the subnet for new clusters accordingly.

[M00nF1sh]

closing this since the "owned/shared" semantic is not something we plan to support in near future.
but feel free to open a feature request to https://github.com/aws/containers-roadmap if you feel this is essential.
As this semantic need to be honored for AWS-cloudprovider as well.

[markussiebert]

Can we reopen this issue? We are running into the same issue and I think, the loadbalancer controller should respect the expected behaviour of the table @ChristianHersevoort posted above.

@M00nF1sh

I see no reason, why we should create a new feature request in the containers-roadmap. At least this is a bug in the existing documentation.