KEP-961: Bump Statefulset maxUnavailable to beta #3997

kerthcet · 2023-05-11T07:28:50Z

One-line PR description: Bump statefulset maxUnavailable to beta

Issue link: maxUnavailable for StatefulSets #961

Other comments:

No. 1 commit: adapt to the latest KEP template
No. 2 commit: what extra needed when bumping to beta.

keps/sig-apps/961-maxunavailable-for-statefulset/README.md

keps/sig-apps/961-maxunavailable-for-statefulset/kep.yaml

keps/sig-apps/961-maxunavailable-for-statefulset/README.md

soltysh · 2023-05-18T11:23:37Z

/assign

kerthcet · 2023-05-30T12:16:49Z

Sorry for the delay, updates are on the way.

k8s-ci-robot · 2023-05-31T11:50:37Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: kerthcet
Once this PR has been reviewed and has the lgtm label, please ask for approval from soltysh, wojtek-t. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

keps/prod-readiness/OWNERS
keps/sig-apps/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

kerthcet

Thanks @soltysh for your detailed reviews, most of the part I just copy-paste from the original documents to make them up-to-date, I will pay more attention to this next time.

keps/sig-apps/961-maxunavailable-for-statefulset/README.md

soltysh

There are still several things to address here.

keps/sig-apps/961-maxunavailable-for-statefulset/kep.yaml

keps/sig-apps/961-maxunavailable-for-statefulset/README.md

wojtek-t

Please address @soltysh 's comments

keps/sig-apps/961-maxunavailable-for-statefulset/README.md

wojtek-t · 2023-06-12T07:55:27Z

keps/sig-apps/961-maxunavailable-for-statefulset/README.md


-Yes, it can impact already running workloads.
+This could happen when we downgrade the kube-controller-manager from maxUnavailable-enabled release to a disabled one,
+but it will not impact running workloads, because `maxUnavailable` only works in rolling update.


+1

Also the other question that I have is what happens if there is an in-progress rollout (with maxUnavailable>1) and we disable the feature....

keps/sig-apps/961-maxunavailable-for-statefulset/README.md

Signed-off-by: kerthcet <[email protected]>

kerthcet · 2023-06-15T12:42:25Z

PTAL @soltysh @wojtek-t @logicalhan
Besides addressing your comments, added the logics about the new metric rolling-update-duration-seconds.

keps/sig-apps/961-maxunavailable-for-statefulset/README.md

wojtek-t · 2023-06-15T13:26:44Z

keps/sig-apps/961-maxunavailable-for-statefulset/README.md

+will rollout across nodes.
+-->

+It depends, like


I'm not sure this really answers the question.

Please correct me if I'm wrong, but in general:

this doesn't affect running workload on rollout [the only expection is if the feature was already enabled in the past and there are StatefulSets with the new field set and now they are going through a rolling upgrade when we enable the feature-gate-again]

On rollback may understanding is unclear - what happens if my StatsefulSet is going through a rolling upgrade now?

If you mean what will happen if we disable the feature during a statefulset's rolling-update, then it will wait for all the pods ready and rolling-udpate a pod one time.

wojtek-t · 2023-06-15T13:27:44Z

keps/sig-apps/961-maxunavailable-for-statefulset/README.md

+that might indicate a serious problem?
+-->
+
+- As a normal user, with `maxUnavailable` enabled, and we're rolling update a statefulset, the number of pods brought down should


This isn't actionable at all if I'm managing e.g. 1000 clusters - I can't monitor number of pods brought down for individual statefulsets.

wojtek-t · 2023-06-15T13:30:44Z

keps/sig-apps/961-maxunavailable-for-statefulset/README.md

+- As a administrator(as well for normal user), I can check the new metric `rolling-update-duration-seconds`, if we enabled the feature with
+greater than 1 `maxUnavailable` set for statefulsets, but didn't see the duration fall down, then we may have some problems here.
+This is not precise because rolling-update duration can be impacted by several factors, e.g. pulling down a big, new image, but
+can be some indicators refer to.


So can we come up with some better metric for it?

We're reporting ReadyReplicas/AvailableReplicas in Status - can we expose a metric if a difference between these and spec.Replicas is more than maxUnavailable as an example?

@soltysh - for thoughts

I like the proposed metric a lot more than the rolling time.

Sorry, what messages can we get from ReadyReplicas != AvailableReplicas, in my understanding, this only relates to another filed minReadySeconds.

Not ReadyReplicas != AvailableReplicas.

What I was saying is spec.Replicas - status.readyReplicas > spec.maxUnavailable

We can play with the idea what exactly to expose (maybe a counter, maybe a histogram of replicas - readyReplica - maxUnavailable. Maybe instead of readyReplicas use avalableReplicas.
But something along those lines.

spec.Replicas - status.readyReplicas should always be less than or equal to spec.maxUnavailable, or this is a bug.
The metric makes little sense I think ...

So for the rolling time, it values not only for the maxUnavailable, it can help to measure how long the rolling update takes, providing data for decisions whether to rolling-update in parallel if possible. It can be independent of the KEP actually.

How should I compare my 3-replica hello-world with 1000-replica gigantitc java app with 10m startup

Yes this also makes me doubtable about the metric, ok, then I'll drop this idea and it looks more simple now. Thanks.

@wojtek-t these information are for the most part exposed in kube-state-metrics today: https://github.com/kubernetes/kube-state-metrics/blob/main/docs/statefulset-metrics.md and we could add maxUnavailable with this effort. Would that be acceptable from a PRR standpoint or is it limited to native metrics?

That's a good question. Technically not everyone is running kube-state-metrics, so I would really prefer native one.
But I added it to next PRR meeting as an agenda item so that we have a common answer across approvers.

Just for info, kube-state-metrics doesn't follow Kubernetes metrics stability framework just yet, so there are less guarantees towards its metrics than the ones we have in k/k.

But, although that's the case today, it would still be interesting to know if having API related metrics defined in kube-state-metrics would be acceptable from a KEP standpoint since in most cases they are unfit to the Kubernetes codebase.

Just to follow-up on it.
We didn't come up with a very concrete conclusion, but the hand-wavy one was:

for the very simple metrics that effectively monitor a trivial logic on k8s side, we're fine with ksm metrics [like number of objects with a given field etc.]

for things where the logic of the feature is non-trivial, we actually want a native metric

Now, with this feature we get somewhere on the fence. Let's say we add "maxUnavailable" as a metric to ksm.
IIUC, we can then check the value that I suggested above (replicas-read > maxUnavailable) for a given statefulset purely based on ksm.
Can we easily check how many statefulsets are in a bad state in a given time?

Now thinking further about it, maybe the metric that I would actually want to see is something like
stateful_set_in_bad_state [we obviously need a better name] with a label reason.

Now, for this feature we can set reason=exceededMaxUnavailable and increase the value of that metric at any time when we process a statefulset and we observe that replicase-ready >maxUnavailable".

That actually seems (a) simple enough to implement (b) generic enough for future extension (c) useful enough as it shows if something goes wrong.

Thoughts?

wojtek-t · 2023-06-15T13:31:38Z

keps/sig-apps/961-maxunavailable-for-statefulset/README.md

 kubectl get statefulsets -o yaml | grep maxUnavailable
 ```

+Or refer to the new metric `rolling-update-duration-seconds`, it should exist.


This isn't true - the metric will exist even if none of the workloads is using the feature.

wojtek-t · 2023-06-15T13:34:04Z

keps/sig-apps/961-maxunavailable-for-statefulset/README.md

+    - Testing: Are there any tests for failure mode? If not, describe why.
+-->
+
+Disable the feature-gate when performing rolling-update.


I don't think I understood the message of this answer - can you clarify this?

soltysh

There are still quite a few comments to address

soltysh · 2023-06-15T13:23:06Z

keps/sig-apps/961-maxunavailable-for-statefulset/README.md

+-->
+
+With `maxUnavailable` feature enabled, we'll bring down more than one pod at a time, if your application can't tolerate
+this behavior, you should absolutely disable this feature or leave the field unconfigured, which will behave as the default


Suggested change

this behavior, you should absolutely disable this feature or leave the field unconfigured, which will behave as the default

this behavior, you should absolutely leave the field unset or use 1, which will match default behavior

soltysh · 2023-06-15T13:24:11Z

keps/sig-apps/961-maxunavailable-for-statefulset/README.md

-tried this feature in Alpha, we would have time to fix issues.
+#### Metrics
+
+We'll add a new metric named `rolling-update-duration-seconds`, it tracks how long a statefulset takes to finish a rolling-update,


Nit: all the metrics have underscores, so rolling_update_duration_seconds will be more appropriate.

soltysh · 2023-06-15T13:27:29Z

keps/sig-apps/961-maxunavailable-for-statefulset/README.md

+
+    // <Newly Added>
+    // rollingUpdateStartTimes records when each statefulset starts to perform rolling update.
+    // key is the statefulset name, value is the start time.


name is not unique across namespaces, I'd suggest types.UID, since that's more unique.

soltysh · 2023-06-15T13:28:52Z

keps/sig-apps/961-maxunavailable-for-statefulset/README.md

+  The logic general looks the same, we'll update the time together with status update, so there's no
+  extra api calls.

+  Prefer Opt-2 as we already record the rolling update messages in the status.


I don't agree with that approach, I don't see a reason we want to internal information in the API. First option is much more preferable.

soltysh · 2023-06-15T13:44:37Z

keps/sig-apps/961-maxunavailable-for-statefulset/README.md

+-->
+
+- `test/e2e/apps: we'll add a test to verify that the number of pods brought down each
+time is equal to configured maxUnavailable.


Hmmm... that kind of test actually might be easier as integration, because like you wrote above "pod statuses are faked".

soltysh · 2023-06-15T14:08:33Z

keps/sig-apps/961-maxunavailable-for-statefulset/README.md

+
+Rolling update duration is impacted by a lot of factors, like the container hooks, image size, network, container logics,
+so what we can roughly know here is for each individual statefulset. So generally, if we set the `maxUnavailable` to X, then
+the rolling update duration should reduce X times. Keep in mind that, this is not precise.


I wouldn't put it as X times, a lot will depend how you implement the metric, and I don't expect you'll expose this as per statefulset, since that wouldn't be reasonable, but rather some kind of histogram, which will represent all statefulset rollouts, so in that case a lot will depend how many statefulset are using this feature. I'd probably write that when using this feature you should at least achieve similar rollout times, and with increased adoption of this feature the overall drop of the metric will be more significant.

soltysh · 2023-06-15T14:08:49Z

keps/sig-apps/961-maxunavailable-for-statefulset/README.md

+-->
+
+- [x] Metrics
+  - Metric name: rolling-update-duration-seconds


rolling_update_duration_seconds

soltysh · 2023-06-15T14:09:03Z

keps/sig-apps/961-maxunavailable-for-statefulset/README.md

 The struct in question is IntOrString.

+If we introduce the new metric with Opt-2, then we'll have a new field `rollingUpdateStartTime` with `*Time` type
+in the `StatefulSetStatus`.


Nope, see my other comment.

soltysh · 2023-06-15T14:09:42Z

keps/sig-apps/961-maxunavailable-for-statefulset/README.md

+Disable the feature-gate when performing rolling-update.
+
+- [Failure mode brief description]
+  - Detection: The `maxUnavailabel` is not respected.


Suggested change

- Detection: The `maxUnavailabel` is not respected.

- Detection: The `maxUnavailable` is not respected.

soltysh · 2023-06-15T14:09:58Z

keps/sig-apps/961-maxunavailable-for-statefulset/README.md

+- [Failure mode brief description]
+  - Detection: The `maxUnavailabel` is not respected.
+  - Mitigations: If rolling update in parallel is tolerated for application, set the `podManagementPolicy=Parallel`.
+  - Diagnostics: Set the logger level great than 4.


Suggested change

- Diagnostics: Set the logger level great than 4.

- Diagnostics: Set the logger level greater than 4.

wojtek-t · 2023-07-20T10:51:53Z

@kerthcet - will you have time to update the KEP - it's now a good time to start thinking about 1.29 and bigger chances for review capacity :)

jeremyrickard · 2023-10-02T00:06:53Z

This looks like it has a number of unresolved comments from June, it doesn't seem likely that it's being targeted for 1.29?

kerthcet · 2023-11-07T02:39:42Z

This looks like it has a number of unresolved comments from June, it doesn't seem likely that it's being targeted for 1.29?

Yes, I'll make some efforts next release. Thanks.

atiratree · 2024-01-17T18:11:39Z

@kerthcet are you able to work on pushing this to beta in 1.30 timeframe?

soltysh · 2024-02-05T12:45:23Z

Replaced with #4474
/close

k8s-ci-robot · 2024-02-05T12:45:29Z

@soltysh: Closed this PR.

In response to this:

Replaced with #4474
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

…) (#5228) * Adapt to the latest template Signed-off-by: kerthcet <[email protected]> * Bump to beta Signed-off-by: kerthcet <[email protected]> * address comments Signed-off-by: kerthcet <[email protected]> * Address commetns Signed-off-by: kerthcet <[email protected]> * Add new metric rollingUpdateDurationSeconds to statefulset Signed-off-by: kerthcet <[email protected]> * go over review from #3997, change metric * review notes, still need approval on metrics * running and available instead of running and ready * review notes and UNRESOLVED blocks Signed-off-by: Lucas Severo Alves <[email protected]> * added metric information * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Janet Kuo <[email protected]> * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Janet Kuo <[email protected]> * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Janet Kuo <[email protected]> * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Janet Kuo <[email protected]> * addressed comments * added metric for detection * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Maciej Szulik <[email protected]> * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Maciej Szulik <[email protected]> * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Maciej Szulik <[email protected]> * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Maciej Szulik <[email protected]> * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Maciej Szulik <[email protected]> * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Maciej Szulik <[email protected]> * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Maciej Szulik <[email protected]> * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Maciej Szulik <[email protected]> * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Maciej Szulik <[email protected]> * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Maciej Szulik <[email protected]> * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Maciej Szulik <[email protected]> * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Maciej Szulik <[email protected]> * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Maciej Szulik <[email protected]> * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Maciej Szulik <[email protected]> * addressing comments * updated test coverage numbers * reworded sentences * added explanation on mid-rollout flag disable --------- Signed-off-by: kerthcet <[email protected]> Signed-off-by: Lucas Severo Alves <[email protected]> Co-authored-by: kerthcet <[email protected]> Co-authored-by: Lucas Severo Alves <[email protected]> Co-authored-by: Janet Kuo <[email protected]> Co-authored-by: Maciej Szulik <[email protected]>

…ernetes#4474) (kubernetes#5228) * Adapt to the latest template Signed-off-by: kerthcet <[email protected]> * Bump to beta Signed-off-by: kerthcet <[email protected]> * address comments Signed-off-by: kerthcet <[email protected]> * Address commetns Signed-off-by: kerthcet <[email protected]> * Add new metric rollingUpdateDurationSeconds to statefulset Signed-off-by: kerthcet <[email protected]> * go over review from kubernetes#3997, change metric * review notes, still need approval on metrics * running and available instead of running and ready * review notes and UNRESOLVED blocks Signed-off-by: Lucas Severo Alves <[email protected]> * added metric information * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Janet Kuo <[email protected]> * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Janet Kuo <[email protected]> * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Janet Kuo <[email protected]> * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Janet Kuo <[email protected]> * addressed comments * added metric for detection * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Maciej Szulik <[email protected]> * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Maciej Szulik <[email protected]> * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Maciej Szulik <[email protected]> * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Maciej Szulik <[email protected]> * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Maciej Szulik <[email protected]> * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Maciej Szulik <[email protected]> * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Maciej Szulik <[email protected]> * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Maciej Szulik <[email protected]> * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Maciej Szulik <[email protected]> * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Maciej Szulik <[email protected]> * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Maciej Szulik <[email protected]> * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Maciej Szulik <[email protected]> * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Maciej Szulik <[email protected]> * Update keps/sig-apps/961-maxunavailable-for-statefulset/README.md Co-authored-by: Maciej Szulik <[email protected]> * addressing comments * updated test coverage numbers * reworded sentences * added explanation on mid-rollout flag disable --------- Signed-off-by: kerthcet <[email protected]> Signed-off-by: Lucas Severo Alves <[email protected]> Co-authored-by: kerthcet <[email protected]> Co-authored-by: Lucas Severo Alves <[email protected]> Co-authored-by: Janet Kuo <[email protected]> Co-authored-by: Maciej Szulik <[email protected]>

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory labels May 11, 2023

k8s-ci-robot requested review from kow3ns and soltysh May 11, 2023 07:29

k8s-ci-robot added sig/apps Categorizes an issue or PR as relevant to SIG Apps. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels May 11, 2023

wojtek-t self-assigned this May 11, 2023

kerthcet force-pushed the feat/upgrade-maxUnavailable-to-beta branch from a28043c to e767866 Compare May 11, 2023 07:32

peng2x reviewed May 14, 2023

View reviewed changes

keps/sig-apps/961-maxunavailable-for-statefulset/README.md Show resolved Hide resolved

kerthcet changed the title ~~[WIP] KEP-961: Bump Statefulset maxUnavailable to beta~~ KEP-961: Bump Statefulset maxUnavailable to beta May 16, 2023

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 16, 2023

soltysh requested changes May 18, 2023

View reviewed changes

k8s-ci-robot assigned soltysh May 18, 2023

soltysh mentioned this pull request May 31, 2023

maxUnavailable for StatefulSets #961

Open

8 tasks

kerthcet commented May 31, 2023

View reviewed changes

soltysh requested changes Jun 6, 2023

View reviewed changes

wojtek-t reviewed Jun 12, 2023

View reviewed changes

logicalhan reviewed Jun 14, 2023

View reviewed changes

kerthcet added 5 commits June 15, 2023 20:40

Adapt to the latest template

f58c046

Signed-off-by: kerthcet <[email protected]>

Bump to beta

21a765b

Signed-off-by: kerthcet <[email protected]>

address comments

301e2d8

Signed-off-by: kerthcet <[email protected]>

Address commetns

ddd8c0c

Signed-off-by: kerthcet <[email protected]>

Add new metric rollingUpdateDurationSeconds to statefulset

c9a8b37

Signed-off-by: kerthcet <[email protected]>

kerthcet force-pushed the feat/upgrade-maxUnavailable-to-beta branch from e9eae01 to c9a8b37 Compare June 15, 2023 12:41

wojtek-t reviewed Jun 15, 2023

View reviewed changes

soltysh reviewed Jun 15, 2023

View reviewed changes

kerthcet mentioned this pull request May 19, 2023

Add a new metric rollingUpdateDuration to statefulset for measuring kubernetes/kubernetes#117928

Closed

dgrisonnet mentioned this pull request Aug 28, 2023

add metrics about statefulset rolling update durations kubernetes/kubernetes#120074

Closed

knelasevero added a commit to knelasevero/enhancements that referenced this pull request Feb 1, 2024

go over review from kubernetes#3997, change metric

8d52fc9

knelasevero mentioned this pull request Feb 1, 2024

KEP-961: Bump Statefulset maxUnavailable to beta (built on top of #3997) #4474

Closed

k8s-ci-robot closed this Feb 5, 2024

YuryLysov mentioned this pull request Feb 20, 2025

[statefulset] Adding option to enable feature "MaxUnavailableStatefulSet" Kuberenetes deckhouse/deckhouse#12025

Closed

2 tasks

Edwinhr716 pushed a commit to Edwinhr716/enhancements that referenced this pull request Apr 3, 2025

go over review from kubernetes#3997, change metric

35c55fa

	this behavior, you should absolutely disable this feature or leave the field unconfigured, which will behave as the default
	this behavior, you should absolutely leave the field unset or use 1, which will match default behavior

	- Detection: The `maxUnavailabel` is not respected.
	- Detection: The `maxUnavailable` is not respected.

	- Diagnostics: Set the logger level great than 4.
	- Diagnostics: Set the logger level greater than 4.

KEP-961: Bump Statefulset maxUnavailable to beta #3997

KEP-961: Bump Statefulset maxUnavailable to beta #3997

Uh oh!

Conversation

kerthcet commented May 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

soltysh commented May 18, 2023

Uh oh!

kerthcet commented May 30, 2023

Uh oh!

k8s-ci-robot commented May 31, 2023

Uh oh!

kerthcet left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

soltysh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wojtek-t left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kerthcet commented Jun 15, 2023

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

kerthcet commented May 11, 2023 •

edited

Loading

kerthcet Jun 16, 2023 •

edited

Loading

dgrisonnet Aug 31, 2023 •

edited

Loading