Reference¶

All of the magic that kube-burner does is described in its configuration file. As previously mentioned, the location of this configuration file is provided by the flag -c. This flag points to a YAML-formatted file that consists of several sections.

It is possible to use go-template semantics within this configuration file. It is also important to note that every environment variable is passed to this template, so we can reference them using the syntax {{.MY_ENV_VAR}}. For example, you could define the indexers section of your own configuration file, such as:

metricsEndpoints:
{{ if .OS_INDEXING }}
  - prometheusURL: http://localhost:9090
    indexer:
      type: opensearch
      esServers: ["{{ .ES_SERVER }}"]
      defaultIndex: {{ .ES_INDEX }}
{{ end }}
{{ if .LOCAL_INDEXING }}
  - prometheusURL: http://localhost:9090
    indexer:
      type: local
      metricsDirectory: {{ .METRICS_FOLDER }}
{{ end }}

This feature can be very useful at the time of defining secrets, such as the user and password of our indexer, or a token to use in pprof collection.

Global¶

In this section is described global job configuration, it holds the following parameters:

Option	Description	Type	Default
`measurements`	List of measurements. Detailed in the measurements section	List	[]
`requestTimeout`	Client-go request timeout	Duration	60s
`gc`	Garbage collect created namespaces	Boolean	false
`gcMetrics`	Flag to collect metrics during garbage collection	Boolean	false
`gcTimeout`	Garbage collection timeout	Duration	1h
`waitWhenFinished`	Wait for all pods/jobs (including probes) to be running/completed when all jobs are completed	Boolean	false
`clusterHealth`	Checks if all the nodes are in "Ready" state	Boolean	false

Note

The precedence order to wait on resources is Global.waitWhenFinished > Job.waitWhenFinished > Job.podWait

kube-burner connects k8s clusters using the following methods in this order:

KUBECONFIG environment variable
$HOME/.kube/config
In-cluster config (Used when kube-burner runs inside a pod)

Jobs¶

This section contains the list of jobs kube-burner will execute. Each job can hold the following parameters.

Option	Description	Type	Default
`name`	Job name	String	""
`jobType`	Type of job to execute. More details at job types	String	create
`jobIterations`	How many times to execute the job	Integer	0
`namespace`	Namespace base name to use	String	""
`namespacedIterations`	Whether to create a namespace per job iteration	Boolean	true
`iterationsPerNamespace`	The maximum number of `jobIterations` to create in a single namespace. Important for node-density workloads that create Services.	Integer	1
`cleanup`	Cleanup clean up old namespaces	Boolean	true
`podWait`	Wait for all pods/jobs (including probes) to be running/completed before moving forward to the next job iteration	Boolean	false
`waitWhenFinished`	Wait for all pods/jobs (including probes) to be running/completed when all job iterations are completed	Boolean	true
`maxWaitTimeout`	Maximum wait timeout per namespace	Duration	4h
`jobIterationDelay`	How long to wait between each job iteration. This is also the wait interval between each delete operation	Duration	0s
`jobPause`	How long to pause after finishing the job	Duration	0s
`beforeCleanup`	Allows to run a bash script before the workload is deleted	String	""
`qps`	Limit object creation queries per second	Integer	0
`burst`	Maximum burst for throttle	Integer	0
`objects`	List of objects the job will create. Detailed on the objects section	List	[]
`verifyObjects`	Verify object count after running each job	Boolean	true
`errorOnVerify`	Set RC to 1 when objects verification fails	Boolean	true
`skipIndexing`	Skip metric indexing on this job	Boolean	false
`preLoadImages`	Kube-burner will create a DS before triggering the job to pull all the images of the job	Boolean
`preLoadPeriod`	How long to wait for the preload DaemonSet	Duration	1m
`preloadNodeLabels`	Add node selector labels for the resources created in preload stage	Object	{}
`namespaceLabels`	Add custom labels to the namespaces created by kube-burner	Object	{}
`namespaceAnnotations`	Add custom annotations to the namespaces created by kube-burner	Object	{}
`churn`	Churn the workload. Only supports namespace based workloads	Boolean	false
`churnCycles`	Number of churn cycles to execute	Integer	100
`churnPercent`	Percentage of the jobIterations to churn each period	Integer	10
`churnDuration`	Length of time that the job is churned for	Duration	1h
`churnDelay`	Length of time to wait between each churn period	Duration	5m
`churnDeletionStrategy`	Churn deletion strategy to apply, `default` or `gvr` (where `default` churns namespaces and `gvr` churns objects within namespaces)	String	default
`defaultMissingKeysWithZero`	Stops templates from exiting with an error when a missing key is found, meaning users will have to ensure templates hand missing keys	Boolean	false

Note

Both churnCycles and churnDuration serve as termination conditions, with the churn process halting when either condition is met first. If someone wishes to exclusively utilize churnDuration to control churn, they can achieve this by setting churnCycles to 0. Conversely, to prioritize churnCycles, one should set a longer churnDuration accordingly.

Our configuration files strictly follow YAML syntax. To clarify on List and Object types usage, they are nothing but the Lists and Dictionaries in YAML syntax.

Examples of valid configuration files can be found in the examples folder.

Objects¶

The objects created by kube-burner are rendered using the default golang's template library. Each object element supports the following parameters:

Option	Description	Type	Default
`objectTemplate`	Object template file path or URL	String	""
`replicas`	How replicas of this object to create per job iteration	Integer	-
`inputVars`	Map of arbitrary input variables to inject to the object template	Object	-
`wait`	Wait for object to be ready	Boolean	true
`waitOptions`	Customize how to wait for object to be ready	Object	{}
`runOnce`	Create or delete this object only once during the entire job	Boolean	false

Warning

Kube-burner is only able to wait for a subset of resources, unless waitOptions are specified.

Info

Find more info about the waiters implementation in the pkg/burner/waiters.go file

Object wait Options¶

If you want to override the default waiter behaviors, you can specify wait options for your objects.

Option	Description	Type	Default
`kind`	Object kind to consider for wait	String	""
`labelSelector`	Objects with these labels will be considered for wait	Object	{}
`forCondition`	Wait for the object condition with this name to be true	String	""
`customStatusPath`	A jq path to the status field of the object	String	""

For example, the snippet below can be used to make kube-burner wait for all containers from the pod defined at pod.yml to be ready.

objects:
- objectTemplate: deployment.yml
  replicas: 3
  waitOptions:
    kind: Pod
    labelSelector: {kube-burner-label : abcd}
    forCondition: Ready

Additionally, you can use customStatusPath to specify a custom path to check the condition of the object, for example, to wait for a deployment to be available.

objects:
  - kind: Deployment
    objectTemplate: deployment.yml
    replicas: 1
    waitOptions:
      forCondition: "Available"
      customStatusPath: ".conditions[].type"

This allows kube-burner to check the status of the specified path and wait for the condition you specify in forCondition.

Note

waitOptions.kind, waitOptions.customStatusPath and waitOptions.labelSelector are fully optional. waitOptions.kind is used when an application has child objects to be waited & waitOptions.labelSelector is used when we want to wait on objects with specific labels.

Default labels¶

All objects created by kube-burner are labeled with kube-burner-uuid=<UUID>,kube-burner-job=<jobName>,kube-burner-index=<objectIndex>. They are used for internal purposes, but they can also be used by the users.

Job types¶

Configured by the parameter jobType, kube-burner supports four types of jobs with different parameters each:

Create
Delete
Read
Patch

Create¶

The default jobType is create. Creates objects listed in the objects list as described in the objects section. The amount of objects created is configured by jobIterations, replicas. If the object is namespaced and has an empty .metadata.namespace field, kube-burner creates a new namespace with the name namespace-<iteration>, and creates the defined amount of objects in it.

Delete¶

This type of job deletes objects described in the objects list. Using delete as job type the objects list would have the following structure:

objects:
- kind: Deployment
  labelSelector: {kube-burner-job: cluster-density}
  apiVersion: apps/v1

- kind: Secret
  labelSelector: {kube-burner-job: cluster-density}

Where:

kind: Object kind of the k8s object to delete.
labelSelector: Deletes the objects with the given labels.
apiVersion: API version from the k8s object.

This type of job supports the following parameters. Described in the jobs section:

waitForDeletion: Wait for objects to be deleted before finishing the job. Defaults to true.
name
qps
burst
jobPause
jobIterationDelay

Read¶

This type of job reads objects described in the objects list. Using read as job type the objects list would have the following structure:

objects:
- kind: Deployment
  labelSelector: {kube-burner-job: cluster-density}
  apiVersion: apps/v1

- kind: Secret
  labelSelector: {kube-burner-job: cluster-density}

Where:

kind: Object kind of the k8s object to read.
labelSelector: Reads the objects with the given labels.
apiVersion: API version from the k8s object.

This type of job supports the following parameters. Described in the jobs section:

name
qps
burst
jobPause
jobIterationDelay
jobIterations

Patch¶

This type of job can be used to patch objects with the template described in the object list. This object list has the following structure:

objects:
- kind: Deployment
  labelSelector: {kube-burner-job: cluster-density}
  objectTemplate: templates/deployment_patch_add_label.json
  patchType: "application/strategic-merge-patch+json"
  apiVersion: apps/v1

Where:

kind: Object kind of the k8s object to patch.
labelSelector: Map with the labelSelector.
objectTemplate: The YAML template or JSON file to patch.
apiVersion: API version from the k8s object.
patchType: The Kubernetes request patch type (see below).

Valid patch types:

application/json-patch+json
application/merge-patch+json
application/strategic-merge-patch+json
application/apply-patch+yaml (requires YAML)

As mentioned previously, all objects created by kube-burner are labeled with kube-burner-uuid=<UUID>,kube-burner-job=<jobName>,kube-burner-index=<objectIndex>. Therefore, you can design a workload with one job to create objects and another one to patch or remove the objects created by the previous.

jobs:
- name: create-objects
  namespace: job-namespace
  jobIterations: 100
  objects:
  - objectTemplate: deployment.yml
    replicas: 10

  - objectTemplate: service.yml
    replicas: 10

- name: remove-objects
  jobType: delete
  objects:
  - kind: Deployment
    labelSelector: {kube-burner-job: create-objects}
    apiVersion: apps/v1

  - kind: Secret
    labelSelector: {kube-burner-job: create-objects}

Churning Jobs¶

Churn is the deletion and re-creation of objects, and is supported for namespace-based jobs only. This occurs after the job has completed but prior to uploading metrics, if applicable. It deletes a percentage of contiguous namespaces randomly chosen and re-creates them with all of the appropriate objects. It will then wait for a specified delay (or none if set to 0) before deleting and recreating the next randomly chosen set. This cycle continues until the churn duration has passed.

An example implementation that would churn 20% of the 100 job iterations for 2 hours with no delay between sets:

jobs:
- name: cluster-density
  jobIterations: 100
  namespacedIterations: true
  namespace: churning
  churn: true
  churnPercent: 20
  churnDuration: 2h
  churnDelay: 0s
  objects:
  - objectTemplate: deployment.yml
    replicas: 10

  - objectTemplate: service.yml
    replicas: 10

Injected variables¶

All object templates are injected with the variables below by default:

Iteration: Job iteration number.
Replica: Object replica number. Keep in mind that this number is reset to 1 with each job iteration.
JobName: Job name.
UUID: Benchmark UUID.

In addition, you can also inject arbitrary variables with the option inputVars of the object:

- objectTemplate: service.yml
  replicas: 2
  inputVars:
    port: 80
    targetPort: 8080

The following code snippet shows an example of a k8s service using these variables:

apiVersion: v1
kind: Service
metadata:
  name: sleep-app-{{.Iteration}}-{{.Replica}}
  labels:
    name: my-app-{{.Iteration}}-{{.Replica}}
spec:
  selector:
    app: sleep-app-{{.Iteration}}-{{.Replica}}
  ports:
  - name: serviceport
    protocol: TCP
    port: "{{.port}}"
    targetPort: "{{.targetPort}}"
  type: ClusterIP

You can also use golang template semantics in your objectTemplate definitions

kind: ImageStream
apiVersion: image.openshift.io/v1
metadata:
  name: {{.prefix}}-{{.Replica}}
spec:
{{ if .image }}
  dockerImageRepository: {{.image}}
{{ end }}

Template functions¶

In addition to the default golang template semantics, kube-burner is compiled with the sprig library, which adds over 70 template functions for Go’s template language.

RunOnce¶

All objects within the job will iteratively run based on the JobIteration number, but there may be a situation if an object need to be created only once (ex. clusterrole), in such cases we can add an optional field as runOnce for that particular object to execute only once in the entire job.

An example scenario as below template, a job iteration of 100 but create the clusterrole only once.

jobs:
- name: cluster-density
  jobIterations: 100
  namespacedIterations: true
  namespace: cluster-density
  objects:
  - objectTemplate: clusterrole.yml
    replicas: 1
    runOnce: true

  - objectTemplate: clusterrolebinding.yml
    replicas: 1
    runOnce: true

  - objectTemplate: deployment.yml
    replicas: 10