Skip to content

Commit 126b76d

Browse files
manojdhandam-kased
manojdhanda
and
m-kased
authored
Utho autoscaler (#8398)
* cloudprovider: init Utho cloud provider * add utho cloud provider logic * implement tests and mock client for Utho cloud provider functionality * enhance Utho cloud provider: improve logging, add ReadNodePool method, and restore toProviderID function * add unit tests for Utho cloud provider node group functionality * add deployment and secret configuration for Utho cloud provider * remove outdated Utho Go module versions from go.sum * add stress-test deployment example, improve error messages, and enhance node group tests * update utho autoscaler image version to 1.0.0 * refactor: reorder parameters in setupMockListNodePools * fix: correct license formatting in utho_cloud_provider_test.go and add license header to utils.go * fix: comment out approvers and reviewers in OWNERS file * Remove utho-go as dependency * add comments to clarify structures and services in Utho API * add comments to Utho SDK * add comments * remove unnecessary comments in cloud_instances.go and kubernetes.go * Revert changes to go.mod * Revert changes to go.mod * ensure newline at end of go.mod and go.sum files --------- Co-authored-by: hmada15 <[email protected]> Co-authored-by: m-kased <[email protected]>
1 parent 65be976 commit 126b76d

24 files changed

+4058
-3
lines changed

‎cluster-autoscaler/FAQ.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -990,7 +990,7 @@ The following startup parameters are supported for cluster autoscaler:
990990
| `check-capacity-provisioning-request-batch-timebox` | Maximum time to process a batch of provisioning requests. | 10s |
991991
| `check-capacity-provisioning-request-max-batch-size` | Maximum number of provisioning requests to process in a single batch. | 10 |
992992
| `cloud-config` | The path to the cloud provider configuration file. Empty string for no configuration file. | |
993-
| `cloud-provider` | Cloud provider type. Available values: [aws,azure,gce,alicloud,cherryservers,cloudstack,baiducloud,magnum,digitalocean,exoscale,externalgrpc,huaweicloud,hetzner,oci,ovhcloud,clusterapi,ionoscloud,kamatera,kwok,linode,bizflycloud,brightbox,equinixmetal,vultr,tencentcloud,civo,scaleway,rancher,volcengine] | "gce" |
993+
| `cloud-provider` | Cloud provider type. Available values: [aws,azure,gce,alicloud,cherryservers,cloudstack,baiducloud,magnum,digitalocean,exoscale,externalgrpc,huaweicloud,hetzner,oci,ovhcloud,clusterapi,ionoscloud,kamatera,kwok,linode,bizflycloud,brightbox,equinixmetal,vultr,tencentcloud,civo,scaleway,rancher,volcengine,utho] | "gce" |
994994
| `cloud-provider-gce-l7lb-src-cidrs` | CIDRs opened in GCE firewall for L7 LB traffic proxy & health checks | 130.211.0.0/22,35.191.0.0/16 |
995995
| `cloud-provider-gce-lb-src-cidrs` | CIDRs opened in GCE firewall for L4 LB traffic proxy & health checks | 130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16 |
996996
| `cluster-name` | Autoscaled cluster name, if available | |

‎cluster-autoscaler/README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ You should also take a look at the notes and "gotchas" for your specific cloud p
4040
* [Scaleway](./cloudprovider/scaleway/README.md)
4141
* [TencentCloud](./cloudprovider/tencentcloud/README.md)
4242
* [Vultr](./cloudprovider/vultr/README.md)
43+
* [Utho](./cloudprovider/utho/README.md)
4344

4445
# Releases
4546

@@ -225,3 +226,4 @@ Supported cloud providers:
225226
* Scaleway https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/scaleway/README.md
226227
* TencentCloud https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/tencentcloud/README.md
227228
* Vultr https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/vultr/README.md
229+
* Utho https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/utho/README.md

‎cluster-autoscaler/cloudprovider/builder/builder_all.go

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
1-
//go:build !gce && !aws && !azure && !kubemark && !alicloud && !magnum && !digitalocean && !clusterapi && !huaweicloud && !ionoscloud && !linode && !hetzner && !bizflycloud && !brightbox && !equinixmetal && !oci && !vultr && !tencentcloud && !scaleway && !externalgrpc && !civo && !rancher && !volcengine && !baiducloud && !cherry && !cloudstack && !exoscale && !kamatera && !ovhcloud && !kwok
2-
// +build !gce,!aws,!azure,!kubemark,!alicloud,!magnum,!digitalocean,!clusterapi,!huaweicloud,!ionoscloud,!linode,!hetzner,!bizflycloud,!brightbox,!equinixmetal,!oci,!vultr,!tencentcloud,!scaleway,!externalgrpc,!civo,!rancher,!volcengine,!baiducloud,!cherry,!cloudstack,!exoscale,!kamatera,!ovhcloud,!kwok
1+
//go:build !gce && !aws && !azure && !kubemark && !alicloud && !magnum && !digitalocean && !clusterapi && !huaweicloud && !ionoscloud && !linode && !hetzner && !bizflycloud && !brightbox && !equinixmetal && !oci && !vultr && !tencentcloud && !scaleway && !externalgrpc && !civo && !rancher && !volcengine && !baiducloud && !cherry && !cloudstack && !exoscale && !kamatera && !ovhcloud && !kwok && !utho
2+
// +build !gce,!aws,!azure,!kubemark,!alicloud,!magnum,!digitalocean,!clusterapi,!huaweicloud,!ionoscloud,!linode,!hetzner,!bizflycloud,!brightbox,!equinixmetal,!oci,!vultr,!tencentcloud,!scaleway,!externalgrpc,!civo,!rancher,!volcengine,!baiducloud,!cherry,!cloudstack,!exoscale,!kamatera,!ovhcloud,!kwok,!utho
33

44
/*
55
Copyright 2018 The Kubernetes Authors.
@@ -48,6 +48,7 @@ import (
4848
"k8s.io/autoscaler/cluster-autoscaler/cloudprovider/rancher"
4949
"k8s.io/autoscaler/cluster-autoscaler/cloudprovider/scaleway"
5050
"k8s.io/autoscaler/cluster-autoscaler/cloudprovider/tencentcloud"
51+
"k8s.io/autoscaler/cluster-autoscaler/cloudprovider/utho"
5152
"k8s.io/autoscaler/cluster-autoscaler/cloudprovider/volcengine"
5253
"k8s.io/autoscaler/cluster-autoscaler/cloudprovider/vultr"
5354
"k8s.io/autoscaler/cluster-autoscaler/config"
@@ -85,6 +86,7 @@ var AvailableCloudProviders = []string{
8586
cloudprovider.ScalewayProviderName,
8687
cloudprovider.RancherProviderName,
8788
cloudprovider.VolcengineProviderName,
89+
cloudprovider.UthoProviderName,
8890
}
8991

9092
// DefaultCloudProvider is GCE.
@@ -153,6 +155,8 @@ func buildCloudProvider(opts config.AutoscalingOptions,
153155
return rancher.BuildRancher(opts, do, rl)
154156
case cloudprovider.VolcengineProviderName:
155157
return volcengine.BuildVolcengine(opts, do, rl)
158+
case cloudprovider.UthoProviderName:
159+
return utho.BuildUtho(opts, do, rl)
156160
}
157161
return nil
158162
}

‎cluster-autoscaler/cloudprovider/builder/builder_utho.go

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
//go:build utho
2+
// +build utho
3+
4+
/*
5+
Copyright 2022 The Kubernetes Authors.
6+
7+
Licensed under the Apache License, Version 2.0 (the "License");
8+
you may not use this file except in compliance with the License.
9+
You may obtain a copy of the License at
10+
11+
http://www.apache.org/licenses/LICENSE-2.0
12+
13+
Unless required by applicable law or agreed to in writing, software
14+
distributed under the License is distributed on an "AS IS" BASIS,
15+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
16+
See the License for the specific language governing permissions and
17+
limitations under the License.
18+
*/
19+
20+
package builder
21+
22+
import (
23+
"k8s.io/autoscaler/cluster-autoscaler/cloudprovider"
24+
"k8s.io/autoscaler/cluster-autoscaler/cloudprovider/utho"
25+
"k8s.io/autoscaler/cluster-autoscaler/config"
26+
"k8s.io/client-go/informers"
27+
)
28+
29+
// AvailableCloudProviders supported by the cloud provider builder.
30+
var AvailableCloudProviders = []string{
31+
cloudprovider.UthoProviderName,
32+
}
33+
34+
// DefaultCloudProvider for utho-only build is utho.
35+
const DefaultCloudProvider = cloudprovider.UthoProviderName
36+
37+
func buildCloudProvider(opts config.AutoscalingOptions, do cloudprovider.NodeGroupDiscoveryOptions, rl *cloudprovider.ResourceLimiter, _ informers.SharedInformerFactory) cloudprovider.CloudProvider {
38+
switch opts.CloudProviderName {
39+
case cloudprovider.UthoProviderName:
40+
return utho.BuildUtho(opts, do, rl)
41+
}
42+
43+
return nil
44+
}

‎cluster-autoscaler/cloudprovider/cloud_provider.go

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,8 @@ const (
9090
CivoProviderName = "civo"
9191
// RancherProviderName gets the provider name of rancher
9292
RancherProviderName = "rancher"
93+
// UthoProviderName gets the provider name of utho
94+
UthoProviderName = "utho"
9395
)
9496

9597
// GpuConfig contains the label, type and the resource name for a GPU.

‎cluster-autoscaler/cloudprovider/utho/OWNERS

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
approvers:
2+
# - uthoplatforms
3+
# - m-kased
4+
reviewers:
5+
# - uthoplatforms
6+
# - m-kased
7+
labels:
8+
- area/provider/utho

‎cluster-autoscaler/cloudprovider/utho/README.md

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# Cluster Autoscaler for Utho
2+
3+
The cluster autoscaler for Utho scales nodes in a Utho Kubernetes cluster.
4+
5+
## Utho Kubernetes Engine
6+
7+
Utho Kubernetes Engine https://utho.com/kubernetes is the managed kubernetes solution provided by Utho.
8+
9+
Utho Kubernetes lets users create Node Pools, i.e. groups of nodes each of the same type.
10+
11+
The size of a Node Pool can be configured at any moment. The user cannot select specific nodes to be deleted when downsizing a Node Pool, rather, Utho Kubernetes will randomly select nodes to be deleted to reach the defined size, even if a node is not healthy or has been manually deleted.
12+
13+
Nodes in a Node Pool are considered disposable: they can be deleted and recreated at any moment, deleting a single node outside of Utho Kubernetes will be recreated by Utho after a small amount of time.
14+
15+
## Configuration
16+
17+
It is mandatory to define the cloud configuration file `cloud-config`. You can see an example of the cloud config file at [examples/cluster-autoscaler-secret.yaml](examples/cluster-autoscaler-secret.yaml), it is an INI file with the following fields:
18+
19+
The (JSON) configuration file of the Utho cloud provider supports the following values:
20+
21+
- `cluster_id`: the ID of the Utho Kubernetes cluster.
22+
- `token`: the Utho API key literally defined.
23+
24+
25+
Configuring the autoscaler such as if it should be monitoring node pools or what the minimum and maximum values. Should be configured through the [Utho API](https://utho.com/api-docs/#api-Kubernetes).
26+
The autoscaler will pick up any changes and adjust accordingly.
27+
28+
## Development
29+
30+
Make sure you are inside the `cluster-autoscaler` path of the [autoscaler repository](https://github.com/kubernetes/autoscaler).
31+
32+
Create the docker image:
33+
```
34+
make container
35+
```
36+
tag the generated docker image and push it to a registry.

‎cluster-autoscaler/cloudprovider/utho/examples/cluster-autoscaler-deployment.yaml

Lines changed: 165 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,165 @@
1+
---
2+
apiVersion: v1
3+
kind: ServiceAccount
4+
metadata:
5+
labels:
6+
k8s-addon: cluster-autoscaler.addons.k8s.io
7+
k8s-app: cluster-autoscaler
8+
name: cluster-autoscaler
9+
namespace: kube-system
10+
---
11+
apiVersion: rbac.authorization.k8s.io/v1
12+
kind: ClusterRole
13+
metadata:
14+
name: cluster-autoscaler
15+
labels:
16+
k8s-addon: cluster-autoscaler.addons.k8s.io
17+
k8s-app: cluster-autoscaler
18+
rules:
19+
- apiGroups: [""]
20+
resources: ["events", "endpoints"]
21+
verbs: ["create", "patch"]
22+
- apiGroups: [""]
23+
resources: ["pods/eviction"]
24+
verbs: ["create"]
25+
- apiGroups: [""]
26+
resources: ["pods/status"]
27+
verbs: ["update"]
28+
- apiGroups: [""]
29+
resources: ["endpoints"]
30+
resourceNames: ["cluster-autoscaler"]
31+
verbs: ["get", "update", "patch"]
32+
- apiGroups: [""]
33+
resources: ["nodes"]
34+
verbs: ["watch", "list", "get", "update", "patch"]
35+
- apiGroups: [""]
36+
resources:
37+
- "namespaces"
38+
- "pods"
39+
- "services"
40+
- "replicationcontrollers"
41+
- "persistentvolumeclaims"
42+
- "persistentvolumes"
43+
verbs: ["watch", "list", "get"]
44+
- apiGroups: ["extensions"]
45+
resources: ["replicasets", "daemonsets"]
46+
verbs: ["watch", "list", "get"]
47+
- apiGroups: ["policy"]
48+
resources: ["poddisruptionbudgets"]
49+
verbs: ["watch", "list"]
50+
- apiGroups: ["apps"]
51+
resources: ["statefulsets", "replicasets", "daemonsets"]
52+
verbs: ["watch", "list", "get"]
53+
- apiGroups: ["storage.k8s.io"]
54+
resources: ["storageclasses", "csinodes", "csistoragecapacities", "csidrivers", "volumeattachments"]
55+
verbs: ["watch", "list", "get"]
56+
- apiGroups: ["batch", "extensions"]
57+
resources: ["jobs"]
58+
verbs: ["get", "list", "watch", "patch"]
59+
- apiGroups: ["coordination.k8s.io"]
60+
resources: ["leases"]
61+
verbs: ["create", "patch"]
62+
- apiGroups: ["coordination.k8s.io"]
63+
resourceNames: ["cluster-autoscaler"]
64+
resources: ["leases"]
65+
verbs: ["get", "update", "patch"]
66+
---
67+
apiVersion: rbac.authorization.k8s.io/v1
68+
kind: Role
69+
metadata:
70+
name: cluster-autoscaler
71+
namespace: kube-system
72+
labels:
73+
k8s-addon: cluster-autoscaler.addons.k8s.io
74+
k8s-app: cluster-autoscaler
75+
rules:
76+
- apiGroups: [""]
77+
resources: ["configmaps"]
78+
verbs: ["create", "list", "watch", "patch"]
79+
- apiGroups: [""]
80+
resources: ["configmaps"]
81+
resourceNames: ["cluster-autoscaler-status", "cluster-autoscaler-priority-expander"]
82+
verbs: ["delete", "get", "update", "watch", "patch"]
83+
---
84+
apiVersion: rbac.authorization.k8s.io/v1
85+
kind: ClusterRoleBinding
86+
metadata:
87+
name: cluster-autoscaler
88+
labels:
89+
k8s-addon: cluster-autoscaler.addons.k8s.io
90+
k8s-app: cluster-autoscaler
91+
roleRef:
92+
apiGroup: rbac.authorization.k8s.io
93+
kind: ClusterRole
94+
name: cluster-autoscaler
95+
subjects:
96+
- kind: ServiceAccount
97+
name: cluster-autoscaler
98+
namespace: kube-system
99+
---
100+
apiVersion: rbac.authorization.k8s.io/v1
101+
kind: RoleBinding
102+
metadata:
103+
name: cluster-autoscaler
104+
namespace: kube-system
105+
labels:
106+
k8s-addon: cluster-autoscaler.addons.k8s.io
107+
k8s-app: cluster-autoscaler
108+
roleRef:
109+
apiGroup: rbac.authorization.k8s.io
110+
kind: Role
111+
name: cluster-autoscaler
112+
subjects:
113+
- kind: ServiceAccount
114+
name: cluster-autoscaler
115+
namespace: kube-system
116+
---
117+
apiVersion: apps/v1
118+
kind: Deployment
119+
metadata:
120+
name: utho-cluster-autoscaler
121+
namespace: kube-system
122+
labels:
123+
app: cluster-autoscaler
124+
spec:
125+
replicas: 1
126+
selector:
127+
matchLabels:
128+
app: cluster-autoscaler
129+
template:
130+
metadata:
131+
labels:
132+
app: cluster-autoscaler
133+
spec:
134+
serviceAccountName: cluster-autoscaler
135+
containers:
136+
- name: utho-cluster-autoscaler
137+
image: utho/autoscaler:1.0.0
138+
imagePullPolicy: IfNotPresent
139+
command:
140+
- ./cluster-autoscaler
141+
- --v=4
142+
- --cloud-provider=utho
143+
- --cloud-config=/config/cloud-config
144+
- --node-group-auto-discovery=utho:regexp=.*
145+
resources:
146+
limits:
147+
cpu: 100m
148+
memory: 300Mi
149+
requests:
150+
cpu: 100m
151+
memory: 300Mi
152+
volumeMounts:
153+
- name: ssl-certs
154+
mountPath: /etc/ssl/certs/ca-certificates.crt
155+
readOnly: true
156+
- name: cloud-config
157+
mountPath: /config
158+
readOnly: true
159+
volumes:
160+
- name: ssl-certs
161+
hostPath:
162+
path: /etc/ssl/certs/ca-certificates.crt
163+
- name: cloud-config
164+
secret:
165+
secretName: cluster-autoscaler-cloud-config

‎cluster-autoscaler/cloudprovider/utho/examples/cluster-autoscaler-secret.yaml

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
---
2+
apiVersion: v1
3+
kind: Secret
4+
metadata:
5+
name: cluster-autoscaler-cloud-config
6+
namespace: kube-system
7+
type: Opaque
8+
stringData:
9+
cloud-config: |-
10+
{
11+
"cluster_id": "<CLUSTER_ID>",
12+
"token": "<TOEKN>"
13+
}

‎cluster-autoscaler/cloudprovider/utho/examples/stress-test.yaml

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
name: stress-test
5+
spec:
6+
replicas: 10
7+
selector:
8+
matchLabels:
9+
app: stress
10+
template:
11+
metadata:
12+
labels:
13+
app: stress
14+
spec:
15+
containers:
16+
- name: stress-container
17+
image: nginx
18+
resources:
19+
requests:
20+
cpu: "750m" # Request .75 CPU cores

0 commit comments

Comments
 (0)