# Runbook: Upgrade KUDO Kafka

This runbook explains how to upgrade a running KUDO Kafka to a newer version of KUDO Kafka.

# Pre-conditions

  • Kubernetes cluster with KUDO version >= 0.11.0 installed
  • Have a KUDO Kafka cluster version 1.2.0 up and running in the namespace kudo-kafka
  • Have binaries of jq and grep installed in the $PATH

# Steps

# Verifying if KUDO Kafka is ready for the upgrade

# 1. Get the KUDO Kafka Instance object name

Verify the KUDO Kafka instance object is present in the expected namespace

kubectl get instances -n kudo-kafka

expected output are the KUDO Instance objects present in the namespace kudo-kafka:

NAME    AGE
kafka   71m
zk      130m

# 2. Verify the KUDO Kafka instance plans are COMPLETE

kubectl kudo plan status --instance=kafka -n kudo-kafka

expected output is the plan status for instance kafka:

Plan(s) for "kafka" in namespace "kudo-kafka":
.
└── kafka (Operator-Version: "kafka-1.2.0" Active-Plan: "deploy")
    ├── Plan deploy (serial strategy) [COMPLETE], last updated 2020-04-23 13:13:30
    │   └── Phase deploy-kafka (serial strategy) [COMPLETE]
    │       └── Step deploy [COMPLETE]
    ├── Plan mirrormaker (serial strategy) [NOT ACTIVE]
    │   └── Phase deploy-mirror-maker (serial strategy) [NOT ACTIVE]
    │       └── Step deploy [NOT ACTIVE]
    └── Plan not-allowed (serial strategy) [NOT ACTIVE]
        └── Phase not-allowed (serial strategy) [NOT ACTIVE]
            └── Step not-allowed [NOT ACTIVE]

# 3. Get the Operator Version of KUDO Kafka Instance

From step 1 we know the name of KUDO Kafka instance is kafka. We can get the Operator Version of the KUDO Kafka Instance. kubectl get instance kafka -n kudo-kafka -o json | jq -r '.spec.operatorVersion.name'

expected output is the operator version

kafka-1.2.0

# 4. Get the installed operator versions of KUDO Kafka

kubectl get operatorversions -n kudo-kafka

expected output is the list the operator versions installed in the namespace kudo-kafka

NAME              AGE
kafka-1.2.0       4m51s
zookeeper-0.3.0   5m39s

# Preparing for the upgrade

# 5. Verify the inter.broker.protocol.version is matching the current OperatorVersion

Get the inter.broker.protocol.version version in the instance.

kubectl exec -ti kafka-kafka-2 -n kudo-kafka -c k8skafka -- cat /opt/kafka/server.properties | grep inter.broker.protocol.version

expected output is the inter.broker.protocol.version

inter.broker.protocol.version=2.1

The inter.broker.protocol.version should match with the app minor version

kubectl get operatorversion -n kudo-kafka kafka-1.2.0 -o json | jq -r '.spec.appVersion' in this case the expected output is the Kafka version

2.4.0

If inter.broker.protocol.version already matches the app version, skip to step 7.

# 6. Update the inter.broker.protocol.version to match the current version

kubectl kudo update -n kudo-kafka --instance=kafka -p INTER_BROKER_PROTOCOL_VERSION=2.4

expected output is confirmation that instance has been updated.

Instance kafka was updated.

Repeat the step 5 to confirm that now the inter.broker.protocol.version is matching app version.

# 7. Install the new OperatorVersion using --skip-instance

Install the OperatorVersion to what we are upgrading using the skip-instance flag. If the new OperatorVersion is already installed you can skip to the step 7.

kubectl kudo install kafka -n kudo-kafka --operator-version=1.2.1 --skip-instance

expected output is the CRDs installed for the OperatorVersion

operatorversion.kudo.dev/v1beta1/kafka-1.2.1 created

# 8. Verify the new OperatorVersion is installed correctly

kubectl get operatorversions -n kudo-kafka

expected output is the list the operator versions installed in the namespace kudo-kafka

NAME              AGE
kafka-1.2.0       13m
kafka-1.2.1       42s
zookeeper-0.3.0   13m

# Upgrade the KUDO Kafka Instance

# 9. Upgrade the KUDO Kafka Instance

kubectl kudo upgrade kafka --instance=kafka --operator-version=1.2.1 -n kudo-kafka

expected output is the confirmation that instance kafka has been updated

instance.kudo.dev/v1beta1/kafka updated

check the plan status

kubectl kudo plan status --instance=kafka -n kudo-kafka

expected output should show deploy in progress and the Operator-Version to be kafka-1.2.1

Plan(s) for "kafka" in namespace "kudo-kafka":
.
└── kafka (Operator-Version: "kafka-1.2.1" Active-Plan: "deploy")
    ├── Plan cruise-control (serial strategy) [NOT ACTIVE]
    │   └── Phase cruise-addon (serial strategy) [NOT ACTIVE]
    │       └── Step deploy-cruise-control [NOT ACTIVE]
    ├── Plan deploy (serial strategy) [IN_PROGRESS], last updated 2020-04-23 13:26:47
    │   ├── Phase deploy-kafka (serial strategy) [IN_PROGRESS]
    │   │   ├── Step generate-tls-certificates [COMPLETE]
    │   │   ├── Step configuration [COMPLETE]
    │   │   ├── Step service [COMPLETE]
    │   │   └── Step app [IN_PROGRESS]
    │   └── Phase addons (parallel strategy) [PENDING]
    │       ├── Step monitoring [PENDING]
    │       ├── Step access [PENDING]
    │       ├── Step mirror [PENDING]
    │       └── Step load [PENDING]
    ├── Plan external-access (serial strategy) [NOT ACTIVE]
    │   └── Phase resources (serial strategy) [NOT ACTIVE]
    │       └── Step deploy [NOT ACTIVE]
    ├── Plan kafka-connect (serial strategy) [NOT ACTIVE]
    │   └── Phase deploy-kafka-connect (serial strategy) [NOT ACTIVE]
    │       ├── Step deploy [NOT ACTIVE]
    │       └── Step setup [NOT ACTIVE]
    ├── Plan mirrormaker (serial strategy) [NOT ACTIVE]
    │   └── Phase app (serial strategy) [NOT ACTIVE]
    │       └── Step deploy [NOT ACTIVE]
    ├── Plan not-allowed (serial strategy) [NOT ACTIVE]
    │   └── Phase not-allowed (serial strategy) [NOT ACTIVE]
    │       └── Step not-allowed [NOT ACTIVE]
    ├── Plan service-monitor (serial strategy) [NOT ACTIVE]
    │   └── Phase enable-service-monitor (serial strategy) [NOT ACTIVE]
    │       └── Step deploy [NOT ACTIVE]
    ├── Plan update-instance (serial strategy) [NOT ACTIVE]
    │   └── Phase app (serial strategy) [NOT ACTIVE]
    │       ├── Step conf [NOT ACTIVE]
    │       ├── Step svc [NOT ACTIVE]
    │       └── Step sts [NOT ACTIVE]
    └── Plan user-workload (serial strategy) [NOT ACTIVE]
        └── Phase workload (serial strategy) [NOT ACTIVE]
            └── Step toggle-workload [NOT ACTIVE]

Once the pods are ready, which can be checked using next command kubectl get pods -n kudo-kafka the expected output is the list of the pods

NAME                    READY   STATUS    RESTARTS   AGE
kafka-kafka-0    2/2     Running   0          2m47s
kafka-kafka-1    2/2     Running   0          3m28s
kafka-kafka-2    2/2     Running   0          4m14s

Once the are passing the both readiness and liveness checks the plan should be complete kubectl kudo plan status --instance=kafka -n kudo-kafka

expected output should show the deploy plan complete and the Operator-Version to be kafka-1.2.0

Plan(s) for "kafka" in namespace "kudo-kafka":
.
└── kafka (Operator-Version: "kafka-1.2.1" Active-Plan: "deploy")
    ├── Plan cruise-control (serial strategy) [NOT ACTIVE]
    │   └── Phase cruise-addon (serial strategy) [NOT ACTIVE]
    │       └── Step deploy-cruise-control [NOT ACTIVE]
    ├── Plan deploy (serial strategy) [COMPLETE], last updated 2020-04-23 13:28:12
    │   ├── Phase deploy-kafka (serial strategy) [COMPLETE]
    │   │   ├── Step generate-tls-certificates [COMPLETE]
    │   │   ├── Step configuration [COMPLETE]
    │   │   ├── Step service [COMPLETE]
    │   │   └── Step app [COMPLETE]
    │   └── Phase addons (parallel strategy) [COMPLETE]
    │       ├── Step monitoring [COMPLETE]
    │       ├── Step access [COMPLETE]
    │       ├── Step mirror [COMPLETE]
    │       └── Step load [COMPLETE]
    ├── Plan external-access (serial strategy) [NOT ACTIVE]
    │   └── Phase resources (serial strategy) [NOT ACTIVE]
    │       └── Step deploy [NOT ACTIVE]
    ├── Plan kafka-connect (serial strategy) [NOT ACTIVE]
    │   └── Phase deploy-kafka-connect (serial strategy) [NOT ACTIVE]
    │       ├── Step deploy [NOT ACTIVE]
    │       └── Step setup [NOT ACTIVE]
    ├── Plan mirrormaker (serial strategy) [NOT ACTIVE]
    │   └── Phase app (serial strategy) [NOT ACTIVE]
    │       └── Step deploy [NOT ACTIVE]
    ├── Plan not-allowed (serial strategy) [NOT ACTIVE]
    │   └── Phase not-allowed (serial strategy) [NOT ACTIVE]
    │       └── Step not-allowed [NOT ACTIVE]
    ├── Plan service-monitor (serial strategy) [NOT ACTIVE]
    │   └── Phase enable-service-monitor (serial strategy) [NOT ACTIVE]
    │       └── Step deploy [NOT ACTIVE]
    ├── Plan update-instance (serial strategy) [NOT ACTIVE]
    │   └── Phase app (serial strategy) [NOT ACTIVE]
    │       ├── Step conf [NOT ACTIVE]
    │       ├── Step svc [NOT ACTIVE]
    │       └── Step sts [NOT ACTIVE]
    └── Plan user-workload (serial strategy) [NOT ACTIVE]
        └── Phase workload (serial strategy) [NOT ACTIVE]
            └── Step toggle-workload [NOT ACTIVE]

# 10. Verify the Kafka version through the pod logs

kubectl logs kafka-kafka-0 -c k8skafka -n kudo-kafka | grep "Kafka version:"

The expected output is the Kafka version and that should be the upgraded version 2.4.1

[2020-04-23 07:58:00,246] INFO Kafka version: 2.4.1 (org.apache.kafka.common.utils.AppInfoParser)

# 11. Verify the Kafka version through the container version

kubectl get pods kafka-kafka-0 -n kudo-kafka -o json | jq -r '.spec.containers[].image'

The expected output will show the container images used by the KUDO Kafka pods. The mesosphere/kafka image should be the upgraded version 2.4.1-1.2.1

quay.io/prometheus/node-exporter:v0.18.1
mesosphere/kafka:2.4.1-1.2.1

# 12. Verify the Kafka version through the app version of the installed KUDO Instance

kubectl get instances.kudo.dev kafka -n kudo-kafka -o json | jq -r .spec.operatorVersion.name | xargs kubectl get operatorversion -n kudo-kafka -o json | jq -r .spec.appVersion

The expected output should be the app version specified in the operator version of the instance

2.4.1

# 13. (Optional) Bump the inter.broker.protocol.version to upgraded version

kubectl kudo update --instance=kafka -n kudo-kafka -p INTER_BROKER_PROTOCOL_VERSION=2.4

# Future improvements

Custom upgrade plans by KUDO would help to automate steps of verifying the inter.broker.protocol.version and bumping it to match the current version.