GitHub Twitter Slack

# Developing and Testing Kubernetes Webhooks

Kubernetes development, testing and debugging can be challenging by itself, but fewer things are more challenging than debugging webhooks or technologies like them. Specifically components which require a callback when that callback requires a valid certificate to succeed. Seeing requests for Come up with webhook developer workflow to test it locally (opens new window), I thought it would be valuable to the community to post a solution.

Working on KUDO we have this concern as well and thought it would be worth showing a technique we use to debug locally. While this post uses KUDO as an example, it could be any Kubernetes webhook. It is worth pointing out that KUDO uses controller-runtime (opens new window) under the hood which is the component that is actually listening on port 443. The current optimized solution requires ngrok (opens new window) which requires an account registration for working with https port 443.

# Local Development Env and Assumptions

We commonly work with kind (opens new window) for local development and will assume that through the article. For the rest of the article we will assume that kind create cluster was used to create a new local cluster and that you have access to that cluster with kubectl

For KUDO, the manager is normally deployed into a cluster as a pod with a container from kudobuilder/controller:v0.15.0. This manager uses controller-runtime and registers a kudo controller and a MutatingWebhookConfiguration webhook. The kubernetes configuration for this can be viewed in KUDO by dumping the prerequisite manifests generated by the CLI's init command. go run cmd/kubectl-kudo/main.go init --unsafe-self-signed-webhook-ca --version dev --dry-run -o yaml will show all manifests including kind: MutatingWebhookConfiguration.

# Running Locally and Working with Logs

One option is to just work with logs 😦. For KUDO that means deploying to the cluster. This solution is a lot of overhead but is provided for completeness. Steps for working with the latest code:

  1. build in docker (for KUDO: make docker-build)
  2. tag and push
  3. deploy go run cmd/kubectl-kudo/main.go init --unsafe-self-signed-webhook-ca --kudo-image kudobuilder/controller:v0.15.0

note: replacing the --kudo-image with the appropriate image

Then debug against logs with kubectl logs kudo-controller-manager-0 -n kudo-system -f

This approach has a long dev cycle and is challenging when needing to make rapid changes.

# Running Locally, Debugging in an Editor

In order to run the webhook process outside the cluster, we need to solve the challenge of kube making a callback which requires a valid certificate for https. For this we currently use ngrok. Using ngrok this way requires a registration (which is free and so far does not lead to nagging solitications). ngrok was OSS for v1, but the latest version is not. ngrok was originally developed for this callback usecase and has some pretty cool features via the UI at localhost:4040 for replaying requests which is a great feature. It is worth pointing out that we've had success running with localtunnel (opens new window) in a similar way. Once ngrok is installed and registered (needed for 443), you should be able to run ngrok http 443

This results in a running process that shows forwarding details such as:

Web Interface                 http://127.0.0.1:4040                                                                                                                           
Forwarding                    http://1a3430057023.ngrok.io -> https://localhost:443                                                                                           
Forwarding                    https://1a3430057023.ngrok.io -> https://localhost:443

It also has web socket at 4040 as indicated. Which means we can get the tunnel urls in an automated fashion via: curl -s localhost:4040/api/tunnels. The order of return of the end points changes. In order to reliably get the https url, it requires some jq magic. curl -s localhost:4040/api/tunnels | jq '.tunnels[] | select(.proto == "https") | .public_url' -r

# Working with MutatingWebhookConfiguration

The default configuration for KUDO looks like this (abbreviated by removing "caBundle"):

apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
  creationTimestamp: null
  name: kudo-manager-instance-admission-webhook-config
webhooks:
- clientConfig:
    caBundle: abbreviated
    service:
      name: kudo-controller-manager-service
      namespace: kudo-system
      path: /admit-kudo-dev-v1beta1-instance
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: instance-admission.kudo.dev
  rules:
  - apiGroups:
    - kudo.dev
    apiVersions:
    - v1beta1
    operations:
    - CREATE
    - UPDATE
    resources:
    - instances
    scope: Namespaced
  sideEffects: None

For a webhook, there are 2 ways to configure the callback. The default for KUDO, which is likely the most common in-cluster configuration, is to configure the clientConfig.service configuration. The alternative option is via the clientConfig.url. It is NOT valid to have both of these configured at the same time. Which also means it is challenging to patch update an existing configuration which currently has .service with a .url. It is best to delete the configuration in this case: kubectl delete MutatingWebhookConfiguration kudo-manager-instance-admission-webhook-config.

We need to switch from the clientConfig.service approach above to the clientConfig.url approach using the current ngrok configuration by replacing:

- clientConfig:
    caBundle: abbreviated
    service:
      name: kudo-controller-manager-service
      namespace: kudo-system
      path: /admit-kudo-dev-v1beta1-instance

with:

- clientConfig:
    url: https://1a3430057023.ngrok.io/admit-kudo-dev-v1beta1-instance

Resulting in a full example of:

apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
  creationTimestamp: null
  name: kudo-manager-instance-admission-webhook-config
webhooks:
- clientConfig:
    url: https://1a3430057023.ngrok.io/admit-kudo-dev-v1beta1-instance
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: instance-admission.kudo.dev
  rules:
  - apiGroups:
    - kudo.dev
    apiVersions:
    - v1beta1
    operations:
    - CREATE
    - UPDATE
    resources:
    - instances
    scope: Namespaced
  sideEffects: None

Once this is deployed in the cluster, you can run your webhook locally. For us this is simply make run... but more importantly we can run in Goland or VSCode with break points. We can even use the ngrok UI to replace kube calls via the replay feature. With this as a solution, we can repeatedly restart the webhook process and just develop and debug without any extra overhead.

# Advanced Automation of MutatingWebhookConfiguration

It is worth reviewing more details in the KUDO repo (opens new window). We generated and cached the code's manifests via the update-manifest.sh (opens new window). While processing the webhook-configuration, we leverage yq to modify the manifest to the local dev configuration (essentially, automating the changes above):

yq w -i "$MANCACHE/kudo-manager-instance-admission-webhook-config.yaml" webhooks[0].clientConfig.url https://replace-url.com
yq d -i "$MANCACHE/kudo-manager-instance-admission-webhook-config.yaml" webhooks[0].clientConfig.caBundle
yq d -i "$MANCACHE/kudo-manager-instance-admission-webhook-config.yaml" webhooks[0].clientConfig.service

Now we have a local manifest file which is "ready" to be used for local development (if we replace the url with the latest ngrok url).

To automate the replacement of the url we use the following:

yq  w hack/manifest-gen/kudo-manager-instance-admission-webhook-config.yaml webhooks[0].clientConfig.url "$(curl -s localhost:4040/api/tunnels | jq '.tunnels[] | select(.proto == "https") | .public_url' -r)/admit-kudo-dev-v1beta1-instance" | kubectl apply -f -

There is a lot here but we've documented most of it already. We are rewriting the manifest and replacing the clientConfig.url with the output from the ngrok https tunnel url, then pass that to the kubectl apply -f -. and your READY!

# Summary

Having been unsuccessful in getting a local only running environment... ngrok provides a good solution for working with callbacks that require valid certificates to succeed. Hopefully the details above helps other teams looking to simplifies their webhook development. If you have alternative approaches we would love to discuss!

Ken About the author
Ken Sipe is a Distributed Application Engineer at D2iQ working on the Orchestration team. In addition to being a committer on the KUDO project, Ken is an author and award winning international speaker on the practices of software architecture and engineering and has been honored with the JavaOne Rockstar Award. Ken is also a contributor to Apache Mesos and a committer on a number of OSS projects including Marathon and Metronome. When not coding or talking about code, Ken is an IFR private pilot, a SCUBA dive master and has recently taken up the art of glass blowing. He is based in St. Charles, MO, USA. Find Ken on GitHub (opens new window) Twitter (opens new window)

← KUDO Dependencies →