# Developing and Testing Kubernetes Webhooks
Kubernetes development, testing and debugging can be challenging by itself, but fewer things are more challenging than debugging webhooks or technologies like them. Specifically components which require a callback when that callback requires a valid certificate to succeed. Seeing requests for Come up with webhook developer workflow to test it locally (opens new window), I thought it would be valuable to the community to post a solution.
Working on KUDO we have this concern as well and thought it would be worth showing a technique we use to debug locally. While this post uses KUDO as an example, it could be any Kubernetes webhook. It is worth pointing out that KUDO uses controller-runtime (opens new window) under the hood which is the component that is actually listening on port 443. The current optimized solution requires ngrok (opens new window) which requires an account registration for working with https port 443.
# Local Development Env and Assumptions
We commonly work with kind (opens new window) for local development and will assume that through the article. For the rest of the article we will assume that
kind create cluster was used to create a new local cluster and that you have access to that cluster with
For KUDO, the manager is normally deployed into a cluster as a pod with a container from
kudobuilder/controller:v0.15.0. This manager uses controller-runtime and registers a kudo controller and a
MutatingWebhookConfiguration webhook. The kubernetes configuration for this can be viewed in KUDO by dumping the prerequisite manifests generated by the CLI's
go run cmd/kubectl-kudo/main.go init --unsafe-self-signed-webhook-ca --version dev --dry-run -o yaml will show all manifests including
# Running Locally and Working with Logs
One option is to just work with logs 😦. For KUDO that means deploying to the cluster. This solution is a lot of overhead but is provided for completeness. Steps for working with the latest code:
- build in docker (for KUDO:
- tag and push
go run cmd/kubectl-kudo/main.go init --unsafe-self-signed-webhook-ca --kudo-image kudobuilder/controller:v0.15.0
note: replacing the
--kudo-image with the appropriate image
Then debug against logs with
kubectl logs kudo-controller-manager-0 -n kudo-system -f
This approach has a long dev cycle and is challenging when needing to make rapid changes.
# Running Locally, Debugging in an Editor
In order to run the webhook process outside the cluster, we need to solve the challenge of kube making a callback which requires a valid certificate for https. For this we currently use
ngrok. Using ngrok this way requires a registration (which is free and so far does not lead to nagging solitications). ngrok was OSS for v1, but the latest version is not. ngrok was originally developed for this callback usecase and has some pretty cool features via the UI at
localhost:4040 for replaying requests which is a great feature. It is worth pointing out that we've had success running with localtunnel (opens new window) in a similar way.
Once ngrok is installed and registered (needed for 443), you should be able to run
ngrok http 443
This results in a running process that shows forwarding details such as:
Web Interface http://127.0.0.1:4040 Forwarding http://1a3430057023.ngrok.io -> https://localhost:443 Forwarding https://1a3430057023.ngrok.io -> https://localhost:443
It also has web socket at 4040 as indicated. Which means we can get the tunnel urls in an automated fashion via:
curl -s localhost:4040/api/tunnels. The order of return of the end points changes. In order to reliably get the https url, it requires some jq magic.
curl -s localhost:4040/api/tunnels | jq '.tunnels | select(.proto == "https") | .public_url' -r
# Working with MutatingWebhookConfiguration
The default configuration for KUDO looks like this (abbreviated by removing "caBundle"):
apiVersion: admissionregistration.k8s.io/v1beta1 kind: MutatingWebhookConfiguration metadata: creationTimestamp: null name: kudo-manager-instance-admission-webhook-config webhooks: - clientConfig: caBundle: abbreviated service: name: kudo-controller-manager-service namespace: kudo-system path: /admit-kudo-dev-v1beta1-instance failurePolicy: Fail matchPolicy: Equivalent name: instance-admission.kudo.dev rules: - apiGroups: - kudo.dev apiVersions: - v1beta1 operations: - CREATE - UPDATE resources: - instances scope: Namespaced sideEffects: None
For a webhook, there are 2 ways to configure the callback. The default for KUDO, which is likely the most common in-cluster configuration, is to configure the
clientConfig.service configuration. The alternative option is via the
clientConfig.url. It is NOT valid to have both of these configured at the same time. Which also means it is challenging to patch update an existing configuration which currently has
.service with a
.url. It is best to delete the configuration in this case:
kubectl delete MutatingWebhookConfiguration kudo-manager-instance-admission-webhook-config.
We need to switch from the
clientConfig.service approach above to the
clientConfig.url approach using the current ngrok configuration by replacing:
- clientConfig: caBundle: abbreviated service: name: kudo-controller-manager-service namespace: kudo-system path: /admit-kudo-dev-v1beta1-instance
- clientConfig: url: https://1a3430057023.ngrok.io/admit-kudo-dev-v1beta1-instance
Resulting in a full example of:
apiVersion: admissionregistration.k8s.io/v1beta1 kind: MutatingWebhookConfiguration metadata: creationTimestamp: null name: kudo-manager-instance-admission-webhook-config webhooks: - clientConfig: url: https://1a3430057023.ngrok.io/admit-kudo-dev-v1beta1-instance failurePolicy: Fail matchPolicy: Equivalent name: instance-admission.kudo.dev rules: - apiGroups: - kudo.dev apiVersions: - v1beta1 operations: - CREATE - UPDATE resources: - instances scope: Namespaced sideEffects: None
Once this is deployed in the cluster, you can run your webhook locally. For us this is simply
make run... but more importantly we can run in Goland or VSCode with break points. We can even use the ngrok UI to replace kube calls via the replay feature. With this as a solution, we can repeatedly restart the webhook process and just develop and debug without any extra overhead.
# Advanced Automation of MutatingWebhookConfiguration
It is worth reviewing more details in the KUDO repo (opens new window). We generated and cached the code's manifests via the update-manifest.sh (opens new window). While processing the webhook-configuration, we leverage
yq to modify the manifest to the local dev configuration (essentially, automating the changes above):
yq w -i "$MANCACHE/kudo-manager-instance-admission-webhook-config.yaml" webhooks.clientConfig.url https://replace-url.com yq d -i "$MANCACHE/kudo-manager-instance-admission-webhook-config.yaml" webhooks.clientConfig.caBundle yq d -i "$MANCACHE/kudo-manager-instance-admission-webhook-config.yaml" webhooks.clientConfig.service
Now we have a local manifest file which is "ready" to be used for local development (if we replace the url with the latest ngrok url).
To automate the replacement of the url we use the following:
yq w hack/manifest-gen/kudo-manager-instance-admission-webhook-config.yaml webhooks.clientConfig.url "$(curl -s localhost:4040/api/tunnels | jq '.tunnels | select(.proto == "https") | .public_url' -r)/admit-kudo-dev-v1beta1-instance" | kubectl apply -f -
There is a lot here but we've documented most of it already. We are rewriting the manifest and replacing the
clientConfig.url with the output from the ngrok https tunnel url, then pass that to the
kubectl apply -f -. and your READY!
Having been unsuccessful in getting a local only running environment... ngrok provides a good solution for working with callbacks that require valid certificates to succeed. Hopefully the details above helps other teams looking to simplifies their webhook development. If you have alternative approaches we would love to discuss!