# Building your first KUDO operator - Part 3

In part 1 and 2 of this blog series, I showed how we bootstrapped a Galera cluster using KUDO, by building up our operator in a series of steps, testing each one as we went.

In this third part, we’ll finish our production-ready deployment phase, adding connectivity and ensuring our cluster can scale up and down without interrupting service.

In part 2, we brought up the remaining nodes of our cluster, but we still have the bootstrap node running, which is no longer required. Let’s extend our operator to get rid of that.

Firstly we’ll want to delete the resources associated with the bootstrap node. In our operator.yaml, let’s create a step to do that :

          - name: cleanup
            tasks:
              - bootstrap_cleanup

  - name: bootstrap_cleanup
      kind: Delete
      spec:
        resources:
          - bootstrap_deploy.yaml
          - bootstrap_service.yaml
          - bootstrap_config.yaml

As we can see, this time we’ve added a Delete task, and this task’s resources are all of the yaml files we developed in Part 1. Once this task runs, it will delete all of the bootstrap resources defined in those yaml files from our cluster - the configmap, the service, and the deployment itself.

However, we still have the bootstrap node configured in the configmap we are using on our actual Galera nodes. This isn’t necessarily an issue for a running Galera instance, since it will work around missing nodes, but in the interests of completeness let’s see how we can remove that. Firstly, to the cleanup step we just defined, let’s add another task :

       - name: cleanup
            tasks:
              - bootstrap_cleanup
              - config

    - name: config
      kind: Apply
      spec:
        resources:
          - galera_config.yaml

We’ve added two tasks to a single step here. These will be executed serially, since in our deploy phase we have configured the execution strategy as serial.

    phases:
      - name: deploy
        strategy: serial

Notice that also we’re referring to the pre-existing galera_config.yaml that we used in Part 2 to configure our nodes. In our templates, we also have access to a few other system variables, one of which is .StepName

We can use that variable to insert conditionals into our templates depending on which step they were called in, so let’s modify that file :

# galera_config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Name }}-nodeconfig
  namespace: {{ .Namespace }}
data:
  galera.cnf: |
    [galera]
    wsrep_on = ON
    wsrep_provider = /usr/lib/galera/libgalera_smm.so
    wsrep_sst_method = mariabackup
    {{ if eq .StepName "firstboot_config" -}}
    wsrep_cluster_address = gcomm://{{ .Name }}-bootstrap-svc,{{ $.Name }}-{{ $.OperatorName }}-0.{{ $.Name }}-hs{{ range $i, $v := untilStep 1 (int .Params.NODE_COUNT) 1 }},{{ $.Name }}-{{ $.OperatorName }}-{{ $v }}.{{ $.Name }}-hs{{end}}
    {{ else -}}
    wsrep_cluster_address = gcomm://{{ $.Name }}-galera-0.{{ $.Name }}-hs{{ range $i, $v := untilStep 1 (int .Params.NODE_COUNT) 1 }},{{ $.Name }}-galera-{{ $v }}.{{ $.Name }}-hs{{end}}
    {{ end -}}
    wsrep_sst_auth = "root:{{ .Params.MYSQL_ROOT_PASSWORD }}"
    binlog_format = ROW
  innodb.cnf: |
    [innodb]
    innodb_autoinc_lock_mode = 2
    innodb_flush_log_at_trx_commit = 0
    innodb_buffer_pool_size = 122M

You can see we’ve added the conditional step around the wsrep_cluster_address, which means it gets the bootstrap configuration if called in the firstboot_config step, otherwise it just has the nodes in the cluster.

{{ if eq .StepName "firstboot_config" -}}
    wsrep_cluster_address = gcomm://{{ .Name }}-bootstrap-svc,{{ $.Name }}-{{ $.OperatorName }}-0.{{ $.Name }}-hs{{ range $i, $v := untilStep 1 (int .Params.NODE_COUNT) 1 }},{{ $.Name }}-{{ $.OperatorName }}-{{ $v }}.{{ $.Name }}-hs{{end}}
    {{ else -}}
    wsrep_cluster_address = gcomm://{{ $.Name }}-galera-0.{{ $.Name }}-hs{{ range $i, $v := untilStep 1 (int .Params.NODE_COUNT) 1 }},{{ $.Name }}-galera-{{ $v }}.{{ $.Name }}-hs{{end}}
    {{ end -}}

Note that we terminate both the if, else and end statements with a hyphen. This tells the templating engine not to leave an empty line where those statements were in the template. For a full explanation of the templating code, have a look at Part 2 of this blog.

By using these kinds of conditionals inside our templates, we can re-use code across different tasks, meaning we have less code to write and maintain.

Now if any of our pods restart, they will start back up again with the correct configuration for the post-bootstrap cluster.

Let’s go ahead and test this piece of our operator :

operator $ kubectl kudo install .
operator.kudo.dev/v1beta1/galera created
operatorversion.kudo.dev/v1beta1/galera-0.1.0 created
instance.kudo.dev/v1beta1/galera-instance created

operator $ kubectl kudo plan status --instance galera-instance
Plan(s) for "galera-instance" in namespace "default":
.
└── galera-instance (Operator-Version: "galera-0.1.0" Active-Plan: "deploy")
    └── Plan deploy (serial strategy) [COMPLETE], last updated 2020-07-03 15:09:40
        └── Phase deploy (serial strategy) [COMPLETE]
            ├── Step bootstrap_config [COMPLETE]
            ├── Step bootstrap_service [COMPLETE]
            ├── Step bootstrap_deploy [COMPLETE]
            ├── Step firstboot_config [COMPLETE]
            ├── Step cluster_services [COMPLETE]
            ├── Step statefulset [COMPLETE]
            └── Step cleanup [COMPLETE]

operator $ kubectl describe configmap galera-instance-nodeconfig
Name:         galera-instance-nodeconfig
Namespace:    default
Labels:       heritage=kudo
              kudo.dev/instance=galera-instance
              kudo.dev/operator=galera
Annotations:  kudo.dev/last-applied-configuration:
                {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"galera-instance-nodeconfig","namespace":"default","creationTimestamp":null,"labe...
              kudo.dev/last-plan-execution-uid: b59912c2-9a52-4130-95cd-bd76d215c10e
              kudo.dev/operator-version: 0.1.0
              kudo.dev/phase: deploy
              kudo.dev/plan: deploy
              kudo.dev/step: cleanup

Data
====
galera.cnf:
----
[galera]
wsrep_on = ON
wsrep_provider = /usr/lib/galera/libgalera_smm.so
wsrep_sst_method = mariabackup
wsrep_cluster_address = gcomm://galera-instance-galera-0.galera-instance-hs,galera-instance-galera-1.galera-instance-hs,galera-instance-galera-2.galera-instance-hs
wsrep_sst_auth = "root:admin"
binlog_format = ROW

innodb.cnf:
----
[innodb]
innodb_autoinc_lock_mode = 2
innodb_flush_log_at_trx_commit = 0
innodb_buffer_pool_size = 122M

Events:  <none>

operator $ kubectl get pods
NAME                       READY   STATUS    RESTARTS   AGE
galera-instance-galera-0   1/1     Running   0          3m20s
galera-instance-galera-1   1/1     Running   0          3m
galera-instance-galera-2   1/1     Running   0          2m40s

So now when our operator deploys, we end up with a working cluster minus our bootstrap node, and all with the correct working configuration !

We next need a way for clients to connect to our Galera cluster. Since the cluster is multi-master, we can read or write from any of them safely, so let’s define a Service for that. Services have no dependencies on anything else, so we can create this at any stage in our deploy plan. We already had a step defined to create our internal cluster services, so let’s use that and add to it :

    - name: cluster_services
      kind: Apply
      spec:
        resources:
          - hs-service.yaml
          - cs-service.yaml

So the cluster_services step will now add both the original hs-service.yaml and a new file cs-service.yaml :

apiVersion: v1
kind: Service
metadata:
  name: {{ .Name }}-cs
  namespace: {{ .Namespace }}
  labels:
    app: galera
    galera: {{ .Name }} 
spec:
  ports:
    - port: {{ .Params.MYSQL_PORT }}
      name: mysql
  selector:
    app: galera
    instance: {{ .Name }}

For this service, we only need the MySQL port, and by default this will give us a load balanced ClusterIP, which our clients can use to connect to our cluster.

Let’s go ahead and test this part :

operator $ kubectl kudo plan status --instance galera-instance
Plan(s) for "galera-instance" in namespace "default":
.
└── galera-instance (Operator-Version: "galera-0.1.0" Active-Plan: "deploy")
    └── Plan deploy (serial strategy) [COMPLETE], last updated 2020-07-03 15:28:28
        └── Phase deploy (serial strategy) [COMPLETE]
            ├── Step bootstrap_config [COMPLETE]
            ├── Step bootstrap_service [COMPLETE]
            ├── Step bootstrap_deploy [COMPLETE]
            ├── Step firstboot_config [COMPLETE]
            ├── Step cluster_services [COMPLETE]
            ├── Step statefulset [COMPLETE]
            └── Step cleanup [COMPLETE]

operator $ kubectl get services
NAME                 TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                               AGE
galera-instance-cs   ClusterIP   10.99.32.75   <none>        3306/TCP                              17m
galera-instance-hs   ClusterIP   None          <none>        3306/TCP,4444/TCP,4567/TCP,4568/TCP   17m
kubernetes           ClusterIP   10.96.0.1     <none>        443/TCP                               54m

operator $ kubectl describe service galera-instance-cs
Name:              galera-instance-cs
Namespace:         default
Labels:            app=galera
                   galera=galera-instance
                   heritage=kudo
                   kudo.dev/instance=galera-instance
                   kudo.dev/operator=galera
Annotations:       kudo.dev/last-applied-configuration:
                     {"kind":"Service","apiVersion":"v1","metadata":{"name":"galera-instance-cs","namespace":"default","creationTimestamp":null,"labels":{"app"...
                   kudo.dev/last-plan-execution-uid: a4fdb1f6-b00d-48f7-9364-e39e2e892aaa
                   kudo.dev/operator-version: 0.1.0
                   kudo.dev/phase: deploy
                   kudo.dev/plan: deploy
                   kudo.dev/step: cluster_services
Selector:          app=galera,instance=galera-instance
Type:              ClusterIP
IP:                10.99.32.75
Port:              mysql  3306/TCP
TargetPort:        3306/TCP
Endpoints:         10.244.1.6:3306,10.244.2.8:3306,10.244.3.7:3306
Session Affinity:  None
Events:            <none>

operator $ kubectl run mysql-client --image=mysql:5.7 -it --rm --restart=Never -- mysql -u root -h galera-instance-cs -p
If you don't see a command prompt, try pressing enter.

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 583
Server version: 5.5.5-10.5.4-MariaDB-1:10.5.4+maria~focal mariadb.org binary distribution

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> 

You’ll need to enter the password defined in params.yaml when you see the line:

If you don't see a command prompt, try pressing enter.

So at this point we can definitely connect to our cluster using the Service address we have just defined.

The next thing we need to configure is the behaviour of our cluster when we change the NODE_COUNT value, ie. when we scale up or down. By default in KUDO, a change to a parameter will trigger the deploy plan to run again, which is not what we want. What needs to happen in our cluster is that the ConfigMap needs to change, to reflect the new number of nodes, and then our StatefulSet will need to restart to pick up the new number of nodes in the cluster. To do all of this we’re going to create a new plan in our operator.yaml

  node_update:
    strategy: serial
    phases:
      - name: deploy
        strategy: serial
        steps:
          - name: config
            tasks:
              - config
          - name: statefulset
            tasks:
              - statefulset

We don’t need any new tasks for this change, we’re just defining two steps which will run in this new plan when it is triggered. We then need to tell KUDO to run this plan when our NODE_COUNT variable is changed by changing the configuration in params.yaml :

  - name: NODE_COUNT
    description: "Number of nodes to create in the cluster"
    default: "3"
    trigger: node_update

Let’s reinstall our operator, and see what happens when we change this parameter once the operator is running

operator $ kubectl kudo install .
operator.kudo.dev/v1beta1/galera created
operatorversion.kudo.dev/v1beta1/galera-0.1.0 created
instance.kudo.dev/v1beta1/galera-instance created

operator $ kubectl kudo update --instance galera-instance -p NODE_COUNT=3
Instance galera-instance was updated.

operator $ kubectl kudo plan status --instance galera-instance
Plan(s) for "galera-instance" in namespace "default":
.
└── galera-instance (Operator-Version: "galera-0.1.0" Active-Plan: "node_update")
    ├── Plan deploy (serial strategy) [NOT ACTIVE]
    │   └── Phase deploy (serial strategy) [NOT ACTIVE]
    │       ├── Step bootstrap_config [NOT ACTIVE]
    │       ├── Step bootstrap_service [NOT ACTIVE]
    │       ├── Step bootstrap_deploy [NOT ACTIVE]
    │       ├── Step firstboot_config [NOT ACTIVE]
    │       ├── Step cluster_services [NOT ACTIVE]
    │       ├── Step statefulset [NOT ACTIVE]
    │       └── Step cleanup [NOT ACTIVE]
    └── Plan node_update (serial strategy) [IN_PROGRESS], last updated 2020-07-03 16:14:43
        └── Phase deploy (serial strategy) [IN_PROGRESS]
            ├── Step config [COMPLETE]
            └── Step statefulset [IN_PROGRESS]

If you watch the output of kubectl get pods during this operation, we first see the two additional nodes get deployed, and then the remaining original nodes get restarted one at a time, which is exactly the behaviour we are looking for. You can test the reverse by doing :

operator $ kubectl kudo update --instance galera-instance -p NODE_COUNT=3
Instance galera-instance was updated.

At this point our operator can scale up and down, but we may have a problem when scaling down. If a node in a Galera cluster isn’t synchronised when it’s removed from the cluster, this can cause issues with the cluster state, so we need to ensure that our nodes are synchronized before we terminate them.

To do this, we’ll add a script that checks the status, and then we’ll modify our StatefulSet spec to add a preStop check. Firstly let’s add the script to our original deploy plan:

          - name: node_scripts
            tasks:
              - node_scripts

    - name: node_scripts
      kind: Apply
      spec:
        resources:
          - node_scripts.yaml

And now lets create that script :

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Name }}-node-scripts
  namespace: {{ .Namespace }}
  wait-for-sync.sh: |
    until mysql -u root -p{{ .Params.MYSQL_ROOT_PASSWORD }} -e "SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';" | grep -q Synced 
    do
        echo "Waiting for sync"
        sleep 5
    done

And we'll also configure our stateful set to mount it :

        volumeMounts:
        - name: {{ .Name }}-config
          mountPath: /etc/mysql/conf.d
        - name: {{ .Name }}-datadir
          mountPath: /var/lib/mysql
        - name: {{ .Name }}-node-scripts
          mountPath: /etc/galera/wait-for-sync.sh
          subPath: wait-for-sync.sh
      volumes:
      - name: {{ .Name }}-config
        configMap:
          name: {{ .Name }}-nodeconfig
          items:
            - key: galera.cnf
              path: galera.cnf
            - key: innodb.cnf
              path: innodb.cnf
      - name: {{ .Name }}-node-scripts
        configMap:
          name: {{ .Name }}-node-scripts
          defaultMode: 0755

We are mounting the script directly as a sub path from the ConfigMap, although it’s worth noting that there are some limitations when using subPaths.

We also need to ensure that our StatefulSet pods are scheduled on to separate nodes in our Kubernetes cluster, to protect us from node failures. We can do this with AntiAffinity rules, and we want to make this configurable so we can turn it off for testing. In our statefulset.yaml, let’s create a conditional section :

spec:
      {{ if eq .Params.ANTI_AFFINITY "true" }}
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                    - galera
                  - key: "instance"
                    operator: In
                    values:
                    - {{ .Name }}
              topologyKey: "kubernetes.io/hostname"
     {{ end }}

And in params.yaml let’s make a parameter to control that :

  - name: ANTI_AFFINITY
    description: "Enforce pod anti-affinity"
    default: False

So now our operator can manage node affinity, safely scale up and down, and has connectivity for external clients !

At this point, we can try some benchmarks, and scale our cluster up and down whilst doing so to make sure that everything works correctly. I used https://www.percona.com/blog/2020/02/07/how-to-measure-mysql-performance-in-kubernetes-with-sysbench/ to do this.

So in the three parts of this blog, we’ve seen how to build KUDO operators up piece by piece and how to test them while we are doing so. We’ve also seen various examples of how to use templating - including conditionals, using the built in variables provided by KUDO, and Sprig functions.

In future blogs in this series, I will continue to develop the Galera operator, adding additional functionality and custom plans to expand its feature set for more day 2 operations.

Matt About the author
Matt runs the Open Source Program Office at D2iQ. He's a regular speaker at conferences and meetups all over the world, and has been building stuff with open source software for longer than he cares to remember. Find Matt on GitHub Twitter LinkedIn