Streaming Encoding for LIST Responses

[serathius]

Enhancement Description

• One-line enhancement description (can be used as a release note): Streaming Response Encoding
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/5116-streaming-response-encoding/README.md
• Previous discussion: Don't copy whole response during response marshalling  kubernetes#129304, Streaming json list encoder kubernetes#129334
• Primary contact (assignee): serathius@
• Responsible SIGs: api-machinery
• Enhancement target (which target equals to which milestone):
  • Beta release target (x.y): 1.33
  • Stable release target (x.y): 1.34

Milestones:

Beta

• KEP (k/enhancements) update PR(s): KEP-5116: Add KEP (Streaming response encoding) #5119
• Code (k/k) update PR(s):
  • Implement tests for deferredResponseWriter to validate behavior on multiple writer calls. kubernetes#130168
  • Implement chunking for gzip encoder in deferredResponseWriter kubernetes#130264
  • Implement JSON and Proto list benchmarks
  • Implement static analysis to validate json and proto tags of builtin List types kubernetes#130216
  • Add tests for encoding collections in JSON for KEP-5116 kubernetes#130220
  • Implement tests for encoding collections in Proto kubernetes#130395
  • Streaming json list encoder kubernetes#129334
  • Implement streaming proto list encoder kubernetes#129407
• Docs (k/website) update(s):
  • No planning doc update as feature is not user facing.

• Stable

  • KEP (k/enhancements) update PR(s): Move KEP-5116 to GA #5324
  • Code (k/k) update PR(s):
    • Graduate streaming list encoding feature gates kubernetes#132648
  • Docs (k/website) update(s): Document KEP-5116 website#51457

• Optional improvements being worked on that are not blocking for graduation:

  • [optional] Stabilize benchmark list by limiting QPS kubernetes#130869
  • [optional] Implement checking apiserver memory limit below threshold in list benchmarks kubernetes#131885
  • [optional] Implement missing LIST benchmarks kubernetes#130169

[serathius]

/sig api-machinery

[chenk008]

I'm glad to see this proposal. We have also implemented similar capabilities in our inner repo and are preparing to push this part to upstream. We have submitted a CFP for the upcoming KubeCon China conference.

In our implementation, we use sync.Pool to efficiently manage memory allocation and cache the serialized results of each item. When the buffer reaches a certain size, we execute a flush operation to parallelize the serialization processing and write to http2.

Additionally, we have added support for gzip compression, which is only enabled when the first batch of cached data reaches 128 * 1024.

For json serialization, we have customized the StreamMarshal method for unstructuredList.

As for protobuf, we generate code through a generator to ensure reverse protobuf marshalling compatibility.

type StreamMarshaller interface {
	// return the object size and the item size slice
	StreamSize() (uint64, []int)

	StreamMarshal(w stream.Writer, itemSize []int) error
}

And it has conducted extensive testing with large datasets and have obtained comparative results. @yulongfang Can you share some benchmark results?

[yulongfang]

Thank @chenk008 for your introduction. We have many large-scale clusters in Alibaba Cloud. When the controllers of these large-scale clusters are restarted, they will initiate a full list request to the apiserver, which will have a certain impact on the stability of the cluster. We have to use larger machines to run the apiserver, resulting in a waste of resources.

In this context, we adopted the method to carry out relevant optimization and achieved the following results.

list json format return data stress test scenario description:

• apiserver version: 1.30
• apiserver specification: 32c 128GB
• apiserver replica number: 1 replica
• number of stock resources: build 10,000 100kb cr information
• stress test scenario: increase pressure according to the gradient of qps 0.1 / 0.5

list json format return data related stress test data:

qps 0.05

• before optimization: cpu 35.7 c mem 89Gb
• stream json after optimization: cpu 6.22 c mem 60 Gb

qps 0.1

• before optimization: cpu 11 c mem 146Gb
• stream json after optimization: cpu 7.45 c mem 97 Gb

list protobuf Format Returned data Stress test scenario description:

• apiserver version: 1.30
• apiserver specification: 32c 128GB
• apiserver replica number: 1 replica
• Number of existing resources: Build 10,000 configmaps information of size 100kb
• Stress test scenario: Increase pressure according to the gradient of qps 0.1 / 0.5

list configmaps format Returned data Related stress test data:

qps 0.05

• Before optimization: cpu 16.8 c mem 54.3Gb
• After stream json optimization: cpu 16.8 c mem 16.1 Gb

qps 0.1

• Before optimization: cpu 42 c mem 122Gb
• After stream json optimization: cpu 42 c mem 18 Gb

[BenTheElder]

FYI: Technical details are usually discussed in KEP PRs or elsewhere, with the KEP issue serving as a place to link back work.

@chenk008 @yulongfang you might consider reviewing #5119

[serathius]

Hey @chenk008 @yulongfang please see the previous discussion in kubernetes/kubernetes#129304 and kubernetes/kubernetes#129334.
We also have already done a performance analysis of our changes in kubernetes/kubernetes#129304 (comment).

We also added running a automatic benchmark of list requests. You can see the results in https://perf-dash.k8s.io/#/?jobname=benchmark%20list&metriccategoryname=E2E&metricname=Resources&Resource=memory&PodName=kube-apiserver-benchmark-list-master%2Fkube-apiserver

We currently run it in JSON + configmap with RV="" configuration, hope to expand it to include Proto, Pods, CustomResources and other types of LIST request. Would be awesome if you can contribute.

[serathius]

/milestone v1.33

[pacoxu]

@jpbetz @dipesh-rawat this is target to v1.33 and the KEP was merged.
Should the lead-opted-in and tracked label be added and tracked by release team?