Getting SSL working for Traefik Dashboard with k3s and cert-manager

k3s ships with traefik that works out of the box, but for certificate generation while running on Kubernetes you should consider using cert-manager which is apparently a go-to tool for certs management in the k8s world.

For the simplicity of this example let’s assume you have a single node k3s cluster running which doesn’t have any external load balance in front of it (otherwise you might need a DNS challenge and here we will use HTTP-01).

The official way of installing cert-manager seems to be just kubectl apply -f, which reminds me of piping response from curl directly to bash with a sudo on top. Anyway, I shall talk about ArgoCD in some other blog post. So:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.1/cert-manager.yaml

Should spin up 3 new pods in cert-manager namespace and create a bunch of other resources.

Now let’s configure ACME HTTP-01 using Terraform.

First, configure let’s encrypt ClusterIssuer - we’re going to use ClusterIssuer and not Issuer so we can issue certificates in any namespace. Is that somehow less secure? I have no idea, but for the sake of let’s just go with it.

resource "kubernetes_manifest" "letsencrypt_issuer" {
  manifest = {
    "apiVersion" = "cert-manager.io/v1"
    "kind"       = "ClusterIssuer"
    "metadata"   = {
      "name" = "letsencrypt-prod"
    }
    "spec" = {
      "acme" = {
        # NOTE: remember to update email here
        "email"               = "your@email"
        "server"              = "https://acme-v02.api.letsencrypt.org/directory"
        "privateKeySecretRef" = {
          "name" = "letsencrypt-prod"
        }
        "solvers" = [
          {
            "http01" = {
              "ingress" = {
                "class" = "traefik"
              }
            }
          }
        ]
      }
    }
  }
}

Applying the plan should create a new resource, but nothing interesting will happen yet. Now let’s try to generate a new certificate; at this point, you should have a domain (see traefik.mydomain.com part) pointing to your k3s node.

resource "kubernetes_manifest" "traefik_cert" {
  manifest = {
    "apiVersion" = "cert-manager.io/v1"
    "kind"       = "Certificate"
    "metadata"   = {
      "name" = "traefik-cert"
      "namespace" = "kube-system"
    }
    "spec" = {
      "secretName" = "traefik-cert"
      "issuerRef"  = {
        "name" = "letsencrypt-prod"
        "kind" = "ClusterIssuer"
      }
      "dnsNames" = [
        # NOTE: remember to change this
        "traefik.mydomain.com"
      ]
    }
  }
}

When you apply the plan, the cert-manager should create a new resource - cert-manager.io/v1/certificaterequests - and will spin a new temporary pod that will serve an HTTP-01 request. Everything should work automagically with the default k3s traefik configuration (as we specified ingress class in the ClusterIssuer definition).

The new certificate should be stored in secrets as kubernetes.io/tls. Now we can expose traefik dashboard to the world.

resource "kubernetes_manifest" "traefik_route" {
  manifest = {
    "apiVersion" = "traefik.containo.us/v1alpha1"
    "kind"       = "IngressRoute"
    "metadata"   = {
      "name"        = "dashboard-ingressroute"
      "namespace"   = "kube-system"
    }
    "spec" = {
      "entryPoints" = [
        "websecure"
      ]
      "routes" = [
        {
          "kind"        = "Rule"
          # NOTE: remember to change the domain
          "match"       = "Host(`traefik.mydomain.com`) && (PathPrefix(`/dashboard`) || PathPrefix(`/api`))"
          "services" = [
            {
              "name" = "api@internal"
              "kind" = "TraefikService"
            }
          ]
        }
      ]
      "tls" = {
        "secretName" = "traefik-cert"
      }
    }
  }
}

After applying plan once you go to traefik.mydomain.com/dashboard you should see good, old traefik web ui dashboard.

Is there a simpler way?

If prefer achieving same result using more automagical approach you can try using annotations - look for cert-manager.io/cluster-issuer.

What about auto https redirect?

You might be tempted to redirect all non-http traffic to https automatically by configuring a traefik helm chart shipped with k3s by setting chart values to something like:

ports:
  web:
    redirectTo: websecure

That seems to be breaking the cert-manager HTTP-01 challenge so I would recommend doing http->https redirect explicitly by specifying redirect middleware. That said allowCrossNamespace might interest you in case you don’t want to define same middleware in every namespace.

…or you can try using DNS01 challenge instead (you can configure two or more separate ClusterIssuers, which what I ended up eventually due to my DNS entries scattered across few providers because some historical reasons).