How can I simulate different lifecycles for multiple pods and jobs

[restrelas]

I want to make a simulation in which there will be multiple pods and jobs with different behavior for different timestamps, and then mesure the resource allocation of the nodes at multiple timestamps.

For the pod and job lifecycle I've tried to experiment using the stage CR, however after some days trying I still can't figure it out how to make the pods change state in the simulator.

Is it possible to change the state of pods differently for various timestamps using the simulator? And if so, how can I do it?

[wzshiming]

Is it possible to change the state of pods differently for various timestamps using the simulator? And if so, how can I do it?

You can refer to this docs, that is pod-create.yaml -> pod-init-container-running.yaml -> pod-init-container-completed.yaml -> pod-ready.yaml, I think it may be the effect you want if I understand correctly

So basically, the selector matches the state, and then how long does it delay, and then applies the next action, that's a stage, and then you need to connect each stage step by step. the selector matches the next of the last stage action as connect.

[restrelas]

@wzshiming Thanks for explaining the stage configuration, following what you've shown, I think I can simulate the desired behavior using this stage template and specifying the delay of each pod for each stage using a annotation such as '.metadata.annotations["pod-create.stage.kwok.x-k8s.io/delay"]'

However, trying to replicate the stage demo, the stage configuration seem to not be altering the behavior of pods like the demo shows

Is there anything I am doing wrong?

[wzshiming]

I think you might be confused with kwokctl and kwok in a exist cluster.

In your attempt, you can just replace kwokctl with kind, and replace kwokctl scale node --replicas 2 with create node.

If you want to use kwokctl, that's a fake cluster and all CR is config by default, so you need put together all config with kwokctl create cluster --config

or you can use --enable-crds Stage to enable stage CRD and ignore stage config, then you need manage all the stage CR by youself.

[restrelas]

Thanks for clarifying the CRs configuration of kwok.

I was able to make the the jobs and pods run with the time delay configured in the stage.

Now I'm trying to set up the delay of each stage using a annotation on the job, because I would like that each job/pod
changed state at a specific time to simulate a dataset.

Here is the YAML file for creating both jobs, one of the jobs has the annotations for a delay of 20 seconds and the other don't.

apiVersion: batch/v1
kind: Job
metadata:
  name: test-pod-for-job
spec:
  template:
    spec:
      tolerations:
      - key: "kwok-controller/provider"
        operator: "Exists"
        effect: "NoSchedule"
      containers:
      - name: job
        image: fake
      restartPolicy: Never
  backoffLimit: 4
---
apiVersion: batch/v1
kind: Job
metadata:
  name: test-pod-for-job-2
  annotations:
    pod-create.stage.kwok.x-k8s.io/delay: "20000"
    pod-complete.stage.kwok.x-k8s.io/delay: "20000"
    pod-init-container-completed.stage.kwok.x-k8s.io/delay: "20000"
    pod-init-container-running.stage.kwok.x-k8s.io/delay: "20000"
    pod-delete.stage.kwok.x-k8s.io/delay: "20000"
spec:
  template:
    spec:
      tolerations:
      - key: "kwok-controller/provider"
        operator: "Exists"
        effect: "NoSchedule"
      containers:
      - name: job
        image: fake
      restartPolicy: Never
  backoffLimit: 4

On each stage I have the fields setting the default delay for 10 seconds

delay:
    durationMilliseconds: 10000
    durationFrom:
    expressionFrom: '.metadata.annotations["{STAGE-NAME}.stage.kwok.x-k8s.io/delay"]'
    jitterDurationMilliseconds: 10000
    jitterDurationFrom:
      expressionFrom: '.metadata.annotations["{STAGE-NAME}.stage.kwok.x-k8s.io/jitter-delay"]'

When I create the jobs, both of them progress trough their status simultaneously, however the second one should take double the time according the priority order for .durationFrom. With this in mind, I would like to ask you if the way I'm using these annotations for configuring a delay individually for each pod is correct. And if not, is it possible to configure a individual delay for each pod?

Status of the jobs running

[wzshiming]

The annotations needs to be in the pod (or pod template), not the job

apiVersion: batch/v1
kind: Job
metadata:
  name: test-pod-for-job
spec:
  template:
    spec:
      tolerations:
      - key: "kwok-controller/provider"
        operator: "Exists"
        effect: "NoSchedule"
      containers:
      - name: job
        image: fake
      restartPolicy: Never
  backoffLimit: 4
---
apiVersion: batch/v1
kind: Job
metadata:
  name: test-pod-for-job-2
spec:
  template:
    metadata:
      annotations:
        pod-create.stage.kwok.x-k8s.io/delay: "20s"
        pod-complete.stage.kwok.x-k8s.io/delay: "20s"
        pod-init-container-completed.stage.kwok.x-k8s.io/delay: "20s"
        pod-init-container-running.stage.kwok.x-k8s.io/delay: "20s"
        pod-delete.stage.kwok.x-k8s.io/delay: "20s"
    spec:
      tolerations:
      - key: "kwok-controller/provider"
        operator: "Exists"
        effect: "NoSchedule"
      containers:
      - name: job
        image: fake
      restartPolicy: Never
  backoffLimit: 4

[restrelas]

That's it @wzshiming thanks you are my hero :D