Bump Kubernetes dependencies to v1.27.3 #3730

RainbowMango · 2023-06-28T08:36:49Z

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

This PR updates Kubernetes dependencies to v1.27.3 as well as transitive dependencies and adoption:

Update Kubernetes dependencies to v1.27.3
Update controller-runtime to v0.15.0 (this is required during the Kubernetes dependencies update)
Update cluster-api to v1.5.0-rc.0 (this is required during the controller-runtime update)
Adopted Kubernetes changes Set singular names for core types to pass to discovery kubernetes/kubernetes#113542
Adopted controller-runtime breaking change: ⚠️ make *http.Client configurable and use/ share the same http.Client in the default configuration kubernetes-sigs/controller-runtime#2122
Adopted controller-runtime breaking change: ⚠️ Use and provide a single dynamic rest mapper kubernetes-sigs/controller-runtime#2296
Adopted controller-runtime breaking change: ⚠ Add context to EventHandler(s) kubernetes-sigs/controller-runtime#2139
Adopted controller-runtime breaking change: ⚠ Improve builder's capabilities and general UX kubernetes-sigs/controller-runtime#2135
Adopted Kubernetes breaking change: update lease controller kubernetes/kubernetes#116218
Adopted controller-runtime breaking change: ⚠️ Deprecate pkg/config/v1alpha1 types kubernetes-sigs/controller-runtime#2149
Adopted controller-runtime breaking change: ⚠ Remove DelegatedClient, move Options in client.New kubernetes-sigs/controller-runtime#2150
Adopted controller-runtime breaking change: ⚠️ Remove pkg/inject kubernetes-sigs/controller-runtime#2134
Adopted controller-runtime breaking change: ⚠️ Allow passing a custom webhook server kubernetes-sigs/controller-runtime#2293
Adopted controller-runtime breaking change: ⚠️ Add constructor for conversion webhooks, rm admission.GetDecoder() kubernetes-sigs/controller-runtime#2144
Adopted Kubernetes breaking change: Add API for watch list kubernetes/kubernetes#115402
Adopted Kubernetes breaking change: Move genericclioptions.IOStreams to genericiooptions.IOStreams kubernetes/kubernetes#117102
Adopted controller-runtime breaking change: ⚠ Update fake client deletionTimestamp behavior kubernetes-sigs/controller-runtime#2316
Adopted controller-runtime breaking change: ⚠️ Fakeclient: Fix status handling kubernetes-sigs/controller-runtime#2259

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Karmada is now built with Kubernetes v1.27.3 dependencies.

RainbowMango · 2023-06-28T08:38:54Z

Hi @lonelyCZ, need your help here.
Now I'm blocked by updating Swagger docs:

# hack/update-swagger-docs.sh 
W0628 16:27:02.787558 1826111 recommended.go:152] Neither kubeconfig is provided nor service-account is mounted, so APIPriorityAndFairness will be disabled
W0628 16:27:02.788751 1826111 authentication.go:49] Authentication is disabled
F0628 16:27:02.791130 1826111 render.go:113] unable to install api resources: unable to setup API &{[cluster.karmada.io/v1alpha1] map[v1alpha1:map[clusters:0xc000419380 clusters/proxy:0x3d66230 clusters/status:0xc000669e00]] v1 <nil>r
exit status 255

I'm looking into it, but don't have a clue yet.

lonelyCZ · 2023-06-28T08:53:32Z

Hi @lonelyCZ, need your help here. Now I'm blocked by updating Swagger docs:

# hack/update-swagger-docs.sh 
W0628 16:27:02.787558 1826111 recommended.go:152] Neither kubeconfig is provided nor service-account is mounted, so APIPriorityAndFairness will be disabled
W0628 16:27:02.788751 1826111 authentication.go:49] Authentication is disabled
F0628 16:27:02.791130 1826111 render.go:113] unable to install api resources: unable to setup API &{[cluster.karmada.io/v1alpha1] map[v1alpha1:map[clusters:0xc000419380 clusters/proxy:0x3d66230 clusters/status:0xc000669e00]] v1 <nil>r
exit status 255

I'm looking into it, but don't have a clue yet.

I will look it today!

lonelyCZ · 2023-06-28T18:21:34Z

I just inspected it, and I found this commit at kubernetes codebase .

We can resolve it by implementing this interface. Like here cc7494c

The testing result is that

RainbowMango · 2023-06-29T01:58:59Z

Thank you so much for working on this issue late into the night! Take care of yourself.
I followed your advice and it's working now. Thank you.

RainbowMango · 2023-07-15T10:22:28Z

The next breaking change we need to adopt is kubernetes-sigs/controller-runtime#2296 which removed the WithCustomMapper option that relied at

karmada/pkg/util/restmapper/restmapper.go

Line 92 in 9a03a10

option := apiutil.WithCustomMapper(func() (meta.RESTMapper, error) {

-bash-5.0# make karmada-controller-manager
BUILD_PLATFORMS=linux/amd64 hack/build.sh karmada-controller-manager
!!! Building karmada-controller-manager for linux/amd64:
+ CGO_ENABLED=0
+ GOOS=linux
+ GOARCH=amd64
+ go build -ldflags '-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.7.0-alpha.2-43-g3a07a1ed -X github.com/karmada-io/karmada/pkg/version.gitCommit=3a07a1edbb60728755afde7a55f835ae62d9324d -X github.com/karmada-io/karmadar
# github.com/karmada-io/karmada/pkg/util/restmapper
pkg/util/restmapper/restmapper.go:92:20: undefined: apiutil.WithCustomMapper
make: *** [Makefile:43: karmada-controller-manager] Error 1

References:

the reason why introduced CachedRESTMapper: performance improvement for karmada-controller-manager to use DeferredDiscoveryRESTMapper #2105

RainbowMango · 2023-07-26T06:55:25Z

Current blocker is that karmada-metrics-adapter failed to start:

-bash-5.0# kubectl logs -n karmada-system karmada-metrics-adapter-6474fb596-2ppkh 
I0726 06:46:58.638131       1 options.go:97] karmada-metrics-adapter version: version.Info{GitVersion:"v1.7.0-alpha.2-85-g4184b562", GitCommit:"4184b562b4b7f80e1484389608941f534a8c8f03", GitTreeState:"clean", BuildDate:"2023-07-26T06:41:19Z", GoVersion:"go1.20.4", Compiler:"gc", Platform:"linux/amd64"}
I0726 06:46:58.869444       1 serving.go:342] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
E0726 06:46:59.600734       1 options.go:88] Unable to install resource metrics adapter: unable to get openapi models: cannot find model definition for k8s.io/metrics/pkg/apis/metrics/v1beta1.NodeMetrics. If you added a new type, you may need to add +k8s:openapi-gen=true to the package or type and run code-gen again
E0726 06:46:59.600766       1 run.go:74] "command failed" err="unable to get openapi models: cannot find model definition for k8s.io/metrics/pkg/apis/metrics/v1beta1.NodeMetrics. If you added a new type, you may need to add +k8s:openapi-gen=true to the package or type and run code-gen again"

edit: Fixed.

RainbowMango · 2023-07-28T01:41:01Z

Hi @ikaven1024 Need your help here.

The TestMultiClusterCache_Watch and TestMultiClusterCache_Watch_Namespaced is failing after updating the Kubernetes dependencies, note that, the E2E tests work fine.

# go test ./pkg/search/proxy/store/... -run=TestMultiClusterCache_Watch 
I0728 09:35:19.163117 1701082 multi_cluster_cache.go:84] Add cache for cluster cluster2
I0728 09:35:19.163160 1701082 cluster_cache.go:70] Add cache for cluster2 /v1, Resource=pods
I0728 09:35:19.163478 1701082 multi_cluster_cache.go:84] Add cache for cluster cluster1
I0728 09:35:19.163504 1701082 cluster_cache.go:70] Add cache for cluster1 /v1, Resource=pods
I0728 09:35:27.165983 1701082 resource_cache.go:36] Stop store for cluster2 /v1, Resource=pods
I0728 09:35:27.166022 1701082 resource_cache.go:36] Stop store for cluster1 /v1, Resource=pods
--- FAIL: TestMultiClusterCache_Watch (8.00s)
    --- FAIL: TestMultiClusterCache_Watch/resource_version_is_empty (5.00s)
        multi_cluster_cache_test.go:994: timeout
    --- FAIL: TestMultiClusterCache_Watch/resource_version_of_cluster2_is_empty (1.00s)
        multi_cluster_cache_test.go:1000: Watch() got = map[pod13:{}], but want = map[pod13:{} pod21:{} pod22:{} pod23:{}]
I0728 09:35:27.166242 1701082 multi_cluster_cache.go:84] Add cache for cluster cluster1
I0728 09:35:27.166269 1701082 cluster_cache.go:70] Add cache for cluster1 /v1, Resource=pods
I0728 09:35:27.166344 1701082 multi_cluster_cache.go:84] Add cache for cluster cluster2
I0728 09:35:27.166375 1701082 cluster_cache.go:70] Add cache for cluster2 /v1, Resource=pods
I0728 09:35:38.169094 1701082 resource_cache.go:36] Stop store for cluster1 /v1, Resource=pods
I0728 09:35:38.169118 1701082 resource_cache.go:36] Stop store for cluster2 /v1, Resource=pods
--- FAIL: TestMultiClusterCache_Watch_Namespaced (11.00s)
    --- FAIL: TestMultiClusterCache_Watch_Namespaced/request_all_namespaces (5.00s)
        multi_cluster_cache_test.go:1127: timeout
    --- FAIL: TestMultiClusterCache_Watch_Namespaced/request_ns1_namespace (5.00s)
        multi_cluster_cache_test.go:1127: timeout
    --- FAIL: TestMultiClusterCache_Watch_Namespaced/request_ns2_namespace (0.00s)
        multi_cluster_cache_test.go:1132: Watch() got = map[], but want = map[pod22:{}]
FAIL
FAIL	github.com/karmada-io/karmada/pkg/search/proxy/store	19.026s
FAIL

Update controller-runtime to v0.15.0 Update cluster-api to v1.5.0-rc.0 Update sigs.k8s.io/metrics-server to v0.6.1-0.20230706083104-796fc0f832c1 Update sigs.k8s.io/custom-metrics-apiserver to v1.27.0 Signed-off-by: RainbowMango <[email protected]>

…ames for core types Adopt breaking change of contriller-runtime PR 2122 Adopt breaking change of contriller-runtime PR 2296 Adopt breaking change of controller-runtime PR 2139 Adopt breaking change of controller-runtime PR 2135 Adopt breaking change of Kubernetes PR 116218 Adopt breaking change of controller-runtime PR 2149 Adopt breaking change of controller-runtime PR 2150 Adopt breaking change of controller-runtime PR 2134 Adopt breaking change of controller-runtime PR 2293 Adopt breaking change of controller-runtime PR 2144 Adopt k8s feature: watch list. PR 115402 Adopt breaking change of Kubernetes PR 117102 Adopt Kubernetes change: disable staticcheck Fix unit tests that failure due to fake-client behavior changes. Signed-off-by: RainbowMango <[email protected]>

Update auto-generated files Signed-off-by: RainbowMango <[email protected]>

codecov-commenter · 2023-07-29T09:06:39Z

Codecov Report

Merging #3730 (3f5c907) into master (04d2ef1) will decrease coverage by 0.26%.
Report is 2 commits behind head on master.
The diff coverage is 15.00%.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

@@            Coverage Diff             @@
##           master    #3730      +/-   ##
==========================================
- Coverage   55.18%   54.93%   -0.26%     
==========================================
  Files         227      227              
  Lines       21671    21676       +5     
==========================================
- Hits        11960    11907      -53     
- Misses       9074     9130      +56     
- Partials      637      639       +2

Flag	Coverage Δ
unittests	`54.93% <15.00%> (-0.26%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
...ers/binding/cluster_resource_binding_controller.go	`0.00% <0.00%> (ø)`
pkg/controllers/status/crb_status_controller.go	`48.33% <0.00%> (-5.00%)`	⬇️
pkg/controllers/status/rb_status_controller.go	`46.77% <0.00%> (-4.84%)`	⬇️
...controllers/unifiedauth/unified_auth_controller.go	`25.11% <0.00%> (ø)`
...ependenciesdistributor/dependencies_distributor.go	`3.02% <ø> (ø)`
pkg/registry/cluster/storage/storage.go	`0.00% <0.00%> (ø)`
pkg/registry/search/storage/cache.go	`0.00% <0.00%> (ø)`
pkg/registry/search/storage/proxy.go	`0.00% <0.00%> (ø)`
pkg/registry/search/storage/resourceregistry.go	`0.00% <0.00%> (ø)`
pkg/registry/search/storage/search.go	`0.00% <0.00%> (ø)`
... and 9 more

... and 3 files with indirect coverage changes

RainbowMango · 2023-07-29T09:57:46Z

Great thanks to @lonelyCZ @ikaven1024 @XiShanYongYe-Chang @jwcesign @zishen @Poor12 !
I couldn't deal with so many breaking changes without your help!

XiShanYongYe-Chang

/lgtm

karmada-bot · 2023-07-31T04:29:22Z

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by:

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

karmada-bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. labels Jun 28, 2023

karmada-bot requested review from Garrybest, lonelyCZ and rv64m June 28, 2023 08:37

karmada-bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Jun 28, 2023

RainbowMango force-pushed the pr_update_k8s_v127 branch from c016ba3 to afb1a69 Compare June 29, 2023 01:50

RainbowMango force-pushed the pr_update_k8s_v127 branch from afb1a69 to 6f4f3b4 Compare July 15, 2023 09:09

RainbowMango force-pushed the pr_update_k8s_v127 branch from 3a07a1e to 2d7e2f9 Compare July 17, 2023 08:34

RainbowMango mentioned this pull request Jul 20, 2023

build cached restmapper based on Kubernetes restmapper #3819

Merged

RainbowMango force-pushed the pr_update_k8s_v127 branch 2 times, most recently from 9977684 to 453fdab Compare July 24, 2023 08:23

RainbowMango mentioned this pull request Jul 24, 2023

Make some cleanup in dependencies distributor #3827

Merged

RainbowMango force-pushed the pr_update_k8s_v127 branch 3 times, most recently from 139af7a to 2ff237c Compare July 25, 2023 09:02

RainbowMango mentioned this pull request Jul 25, 2023

Cleanup deprecated methods in wait package #3835

Closed

4 tasks

RainbowMango force-pushed the pr_update_k8s_v127 branch from 6dfab1f to 4184b56 Compare July 25, 2023 11:10

RainbowMango force-pushed the pr_update_k8s_v127 branch from 4184b56 to 1150e36 Compare July 27, 2023 02:45

ikaven1024 mentioned this pull request Jul 29, 2023

fix ut in TestMultiClusterCache watch #3862

Merged

RainbowMango force-pushed the pr_update_k8s_v127 branch from 1150e36 to 01e788c Compare July 29, 2023 07:15

Update vendor

3f5c907

Update auto-generated files Signed-off-by: RainbowMango <[email protected]>

RainbowMango force-pushed the pr_update_k8s_v127 branch from 01e788c to 3f5c907 Compare July 29, 2023 08:48

RainbowMango marked this pull request as ready for review July 29, 2023 09:08

karmada-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 29, 2023

XiShanYongYe-Chang reviewed Jul 31, 2023

View reviewed changes

karmada-bot assigned XiShanYongYe-Chang Jul 31, 2023

karmada-bot added the lgtm Indicates that a PR is ready to be merged. label Jul 31, 2023

RainbowMango added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 31, 2023

karmada-bot merged commit 7381933 into karmada-io:master Jul 31, 2023

RainbowMango mentioned this pull request Aug 8, 2023

Fix SSA being silently disabled by lack of OpenAPI config kubernetes/kubernetes#114998

Merged

This was referenced Aug 16, 2023

repair the cluster_status_controller_test #3948

Merged

Test ci zishen/karmada#1

Closed

RainbowMango added this to the v1.7 milestone Oct 21, 2023

RainbowMango mentioned this pull request Oct 21, 2023

Cherrypick #4144: fix cannot access ResourceRegistry with its singularName #4152

Closed

3 tasks

RainbowMango deleted the pr_update_k8s_v127 branch March 30, 2024 08:20

zhzhuang-zju mentioned this pull request May 16, 2024

migrate wait.PollUntil to wait.PollUntilContextCancel #4947

Merged

RainbowMango mentioned this pull request May 28, 2024

Enable deprecation check and bump golangci-lint to v1.59.0 #4992

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Bump Kubernetes dependencies to v1.27.3 #3730

Bump Kubernetes dependencies to v1.27.3 #3730

Uh oh!

RainbowMango commented Jun 28, 2023 •

edited

Loading

Uh oh!

RainbowMango commented Jun 28, 2023

Uh oh!

lonelyCZ commented Jun 28, 2023

Uh oh!

lonelyCZ commented Jun 28, 2023

Uh oh!

RainbowMango commented Jun 29, 2023

Uh oh!

RainbowMango commented Jul 15, 2023 •

edited

Loading

Uh oh!

RainbowMango commented Jul 26, 2023 •

edited

Loading

Uh oh!

RainbowMango commented Jul 28, 2023

Uh oh!

codecov-commenter commented Jul 29, 2023

Uh oh!

RainbowMango commented Jul 29, 2023

Uh oh!

XiShanYongYe-Chang left a comment

Uh oh!

karmada-bot commented Jul 31, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Bump Kubernetes dependencies to v1.27.3 #3730

Bump Kubernetes dependencies to v1.27.3 #3730

Uh oh!

Conversation

RainbowMango commented Jun 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RainbowMango commented Jun 28, 2023

Uh oh!

lonelyCZ commented Jun 28, 2023

Uh oh!

lonelyCZ commented Jun 28, 2023

Uh oh!

RainbowMango commented Jun 29, 2023

Uh oh!

RainbowMango commented Jul 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RainbowMango commented Jul 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RainbowMango commented Jul 28, 2023

Uh oh!

codecov-commenter commented Jul 29, 2023

Codecov Report

Uh oh!

RainbowMango commented Jul 29, 2023

Uh oh!

XiShanYongYe-Chang left a comment

Choose a reason for hiding this comment

Uh oh!

karmada-bot commented Jul 31, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

RainbowMango commented Jun 28, 2023 •

edited

Loading

RainbowMango commented Jul 15, 2023 •

edited

Loading

RainbowMango commented Jul 26, 2023 •

edited

Loading