cert-manager, Webhooks, and CA Bundles: The Stuff That Always Bites You

Kubernetes admission webhooks are simple until TLS gets involved.

The API server calls a service inside the cluster. That service presents a certificate. The API server verifies the certificate using the caBundle in the webhook configuration.

If any part of that chain is wrong, the webhook fails.

The moving parts

A typical webhook setup includes:

• a Deployment running the webhook server
• a Service pointing to the webhook pods
• a Secret containing tls.crt and tls.key
• a Certificate resource if cert-manager is used
• an Issuer or ClusterIssuer
• a ValidatingWebhookConfiguration or MutatingWebhookConfiguration
• a caBundle that lets the API server trust the webhook certificate

The certificate must match the service DNS name

If the webhook service is named:

example-webhook.example-namespace.svc

The certificate should be valid for that DNS name.

A useful cert-manager Certificate shape is:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: example-webhook-tls
  namespace: example-namespace
spec:
  secretName: example-webhook-tls
  dnsNames:
    - example-webhook.example-namespace.svc
    - example-webhook.example-namespace.svc.cluster.local
  issuerRef:
    name: example-issuer
    kind: ClusterIssuer

The CA bundle must be valid base64

A common mistake is putting raw text, a placeholder, or malformed content into caBundle.

Check it:

kubectl get validatingwebhookconfiguration <name> -o yaml

The caBundle value should be base64-encoded certificate authority data.

Debug with the API server error

Webhook failures often show up as errors when applying a resource.

The exact message matters:

• certificate signed by unknown authority
• service not found
• no endpoints available for service
• context deadline exceeded
• invalid webhook response
• response UID does not match request UID

Each points to a different failure layer.

The bigger lesson

For webhooks, certificate management is not a side quest. It is part of the application.

A good webhook deployment should include certificate issuance, CA injection, health checks, logging, and a renewal strategy from the beginning.