Running Private Certificate Authorities in Clusters

December 1, 2019. 2307 words.

When operating web-based systems, one of the foremost concerns is how to protect users’ web traffic. Often, now, the first suggestion is to use Let’s Encrypt’s certificates, which can be obtained free of cost at a public certificate authority, automated utilizing the (soon to be standardized?) (ACME RFC 8555) protocol. However, some may either desire to free themselves from external dependencies under third-party control ⊕Recently, it actually occurred to some that cloud operations come with their own set of unique and exciting problems, cf. Kleppmann et al. and Adrian Colyer’s entertaining commentary for further inspiration. or operate in environments where ACME HTTP-01 is not feasible and exposing DNS zones (ACME DNS-01) is undesirable.

Then, Hashicorp’s Vault lends itself naturally to implement an in-house certificate authority and while certificates may be obtained and rotated by some lines of bash with curl, leveraging Jetstack’s cert-manager utility in a system run by the Kubernetes container orchestrator leads to tangible results surprisingly fast.

Provisioning the Vault

Provided with a sufficiently recent Kubernetes cluster of v1.9 or higher, the vault software may be installed using the Helm Kubernetes Package manager and Hashicorp’s Helm Chart.

server:
  authDelegator:
    enabled: true
ui:
  enabled: true

I propose to enable the authDelegator to authenticate with a Kubernetes serviceAccount and to enable the user interface as some settings cannot be properly tuned with the vault CLI and curling JSONs is a bad substitute. The other settings can be left for the time being, most notably, we should not enable TLS from the start because we are bootstrapping and do not have certificates to secure the endpoint yet.

Vault can now be accessed kubectl port-forwarding 8200:8200 to the vault-0 pod and setting VAULT_ADDR="http://localhost:8200". The binary vault, which also acts as CLI, may be conveniently be optained calling kubectl cp vault-0:/bin/vault $HOME/bin/vault. I propose to immediately enable the audit log, which helps with debugging. ⊕Note that vault audit-logs certain parameters hashed, which keeps spilling certain information from a system administrator policy domain to a log reader policy domain or a storage reader policy domain. When no such separation exists, it may be handled with more slack. When learning and/or debugging, it actually helps.

export VAULT_ADDR="http://localhost:8200"
vault login
vault audit \
  enable \
  file \
    file_path=/vault/audit/audit.log \
    log_raw=true

The vault software exposes so called “secret” and “authentication endpoints”. From secrets endpoints, secrets such as credentials or certificates may be generated and obtained, authentication endpoints establish proof of identity using various methods. Generally, the upstream Hashicorp documentation is excellent, a walk-through may be consulted to suit the practitioner’s needs.

Obtaining the CA’s Root Certificate

The pki secret engine allows to manage certificate generation, storage, distribution and, if needed, revocation of X509 certificates.

vault secrets \
  enable \
    --path=vault-some-domain \
    pki

vault secrets \
  tune \
    -max-lease-ttl=24000h \
  vault-some-domain

The engine is enabled and to be able to distinguish between different CAs (root and intermediate here) exposed under the path given as --path=. The maximum lease time is, as customary for root CAs, high. It is beneficiary to customize issuing and revocation endpoints before generating any certificates.

vault write \
  vault-some-domain/config/urls \
    issuing_certificates="https://vault.some.domain.tld:8200/v1/vault-some-domain/ca" \
    crl_distribution_points="https://vault.some.domain.tld:8200/v1/vault-some-domain/crl"

Other attributes may be easier customized using the UI, no other are strictly speaking, necessary at this point.

The root certificate is generated calling the root/generate/internal endpoint and is self-signed.

vault write \
  -field=certificate \
  vault-some-domain/root/generate/internal \
    common_name="vault.some.domain.tld ROOT Authority" \
    ttl=24000h \
> vault-some.domain.tld-ROOT-CA_cert.pem

Deriving an Intermediate CA

As with the root CA, the engine for the intermediate CA is enabled, albeit under different path and with shorter TTL.

vault secrets \
  enable \
    --path=vault-some-domain-int \
    pki

vault secrets \
  tune \
    -max-lease-ttl=12000h \
    vault-some-domain-int
    
vault write \
  vault-some-domain-int/config/urls \
    issuing_certificates="https://vault.some.domain.tld:8200/v1/vault-some-domain-int/ca" \
    crl_distribution_points="https://vault.some.domain.tld:8200/v1/vault-some-domain-int/crl"

A certificate is generated, unsigned this time, signed by the root CA and uploaded back to the CA endpoint.

vault write \
  -format=json \
  vault-some-domain-int/intermediate/generate/internal \
    common_name="vault.some.domain.tld Intermediate Authority" \
> vault-some-domain-intermediate-csr.json

vault write \
  -format=json \
  vault-some-domain/root/sign-intermediate \
    csr="$(jq -r '.data.csr' vault-some-domain-intermediate-csr.json)" \
    format=pem_bundle \
    ttl="12000h" \
> vault-some-domain-intermediate-crt.json

vault write \
  vault-some-domain-int/intermediate/set-signed \
  certificate="$(jq -r '.data.certificate' vault-some-domain-intermediate-crt.json)"

A set of defaults is then set on a named sub-path, which is called a role in vault lingo.

vault write \
  vault-some-domain-int/roles/cert-manager-some.domain.tld \
    allowed_domains="some.domain.tld" \
    allow_subdomains=true \
    max_ttl="2160h"

⊕Note that for compatibility with the cert-manager utility, the TTL should be set ≥ 90 d, because certificates specified in kubernetes ingress resources may not configure a custom lifetime and by default require said 90 days.

Settings to Allow cert-manager to Provision Certificates

To provide for later automation, any role meant for cert-manager needs it’s settings set so that they are not in contradiction with cert-manager’s defaults. In particular, cert-manager considers CN to be deprecated – I do not know why – and vault requires a common name by default. Switching off Require Common Name fixes the issue, so that cert-manager’s Subject Alternative Name CSRs are considered legit.

In addition, I opted to allow namespaced kubernetes domains to be eligible for certificate issuance, so I added e.g. management-infra.svc.cluster.local to the set of allowed domains.

I was simply too lazy to figure out how to do that via the vault CLI, so I set options in the “Edit PKI Role” dialogue.

Allow Kubernetes Service Accounts to Authenticate Against vault

vault allows to declare trust in a Kubernetes cluster with the kubernetes engine. That trust is per cluster and, in essence, will allow tokens signed with the cluster’s self-signed CA certificate to be used to identify technical users, so called service accounts. An again excellent walk-through the Kubernetes authentication engine is provided by Hashicorp.

vault auth \
  enable \
    -path=some-domain-k8s \
  kubernetes

⊕Note that because here, vault is run on the same cluster it will be configured to trust, the URL of kubernetes_hosts will and only needs to resolve in-cluster.

vault write \
  auth/some-domain-k8s/config \
    kubernetes_host=https://kubernetes.default:443 \
    kubernetes_ca_cert="$( \
      cat "$KUBECONFIG" \
      | "$HOME"/.local/bin/yq -r \
        '.clusters[]
         | select(.name=="some-domain")
         | .cluster."certificate-authority-data" ' \
       | base64 -d \
    )"

Permissions to allow identified users (which here will be service accounts from Kubernetes) to operate on secret engines are given in hcl polciy files, concisely described by the upstream documentation.

path "vault-some-domain-int/*" { 
  capabilities = ["create", "read", "update", "delete", "list"]
}

vault policy write my-policy /tmp/policy.hcl

Give Kubernetes Service Accounts Permission to Obtain Certificates

Individual service accounts are then represented by roles on the authentication engine, their permissions are defined by an attached policy.

vault write \
  auth/some-domain-k8s/role/cert-manager-role \
    bound_service_account_names=vault \
    bound_service_account_namespaces=management-infra \
    policies=cert-manager-policy \
    ttl=1h

I have opted to permit the vault service account to obtain certificates because it is necessary to protect the vault’s endpoints with TLS. To do so, vault now needs to provide it’s own certificate (which I hinted at earlier). This certificate’s provisioning will be controlled by cert-manager, which, spelunking forward, will require an issuer, which will require a service account with permissions to obtain certificates.

Provisioning Jetstack’s cert-manager

cert-manager is a utility developed by the company Jetstack, which offers a set of mechanisms to automatically obtain certificate from various certificate authorities in an automated fashion.

Installation from the released kubernetes manifests is most easy calling

https://github.com/jetstack/cert-manager/releases/download/v0.11.1/cert-manager.yaml \
| kubectl apply -f -

Interested readers may opt to download as an intermediate step, which then will also allow to use the resulting file, which also contains the necessary Custom Resource Definitions, as a validation source for modern IDEs. ⊕Intellij’s IDEA can configure these from file as well as from URL sources under Settings → Languages & Frameworks → Kubernetes → CRDs. The set of Kubernetes Resources will install into the namespace cert-manager, and because the instances of cert-manager strings referring to the namespace alone are difficult to isolate and thus difficult to search & replace. I advise against customization at this point.

Provisioning a Certificate Issuer

With cert-manager in place and operating, certificates may be obtained from Issuers or ClusterIssuers

apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  name: vault-issuer
  namespace: management-infra
spec:
  vault:
    # caBundle: <base64 encoded caBundle PEM file>
    path: vault-some-domain-int/sign/cert-manager-some.domain.tld
    server: http://vault.management-infra.svc.cluster.local
    auth:
      kubernetes:
        mountPath: some-domain-k8s
        role: cert-manager-role
        secretRef:
          name: vault-token-4r9j5
          key: token

⊕Note that the vault’s endpoints are not protected by TLS yet and no caBundle is given accordingly. The spec.vault.path property defines the secret engine’s endpoint, the method (to sing a CSR) and the role containing defaults for the certificate. The spec.vault.auth.kubernetes.mountPath property defines the autentication engine with corresponding role at the same level. The spec.vault.auth.kubernetes.secretRef sepcifies where to get the authentication credentials, these must correspond to the authentication roles’ bound_service_account_names and bound_service_account_namespaces properties from the role’s creation.

Provisioning Certificates Directly

With that issuer in place, a certificate may be declared (and obtained) with a Kubernetes Custom Resource, which the cert-manager will pick up and collect a certificate for at vault’s. In this case, the issuer is used to provide certificates to protect the vault’s endpoints with. ⊕Note that after vault has been upon reconfigured to expose TLS endpoints only, upon certificate re-issuing, the vault process needs to be SIGHUPed, which is out of the scope of cert-manager.

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: vault-endpoint
  namespace: management-infra
spec:
  secretName: vault-endpoint-tls
  duration: 2160h # 90d
  renewBefore: 360h # 15d
  isCA: false
  keySize: 2048
  keyAlgorithm: rsa
  keyEncoding: pkcs1
  usages:
    - server auth
    - client auth
  dnsNames:
    - vault.some.domain.tld
    - vault.management-infra.svc.cluster.local
  issuerRef:
    name: vault-issuer
    kind: Issuer

⊕Unless the intermediate certificate which the vault endpoint’s certificate has been signed with is packaged into the local CA-chains, vault needs to be called -tls-skip-verify from this point onwards. Then, the vault Kubernetes manifests may (should!) be reconfigured to use the X509 certificates stored in the secret named spec.secretName, with the VAULT_ADDR and the VAULT_API_ADDR environment variables set to https://.

Provisioning Certificates Indirectly From ingress-Resources

When configuring certificates directly may seem undesirable, Kubernetes ingress resources may be annotated with an issuer cert-manager.io/issuer: some-name, so that cert-manager obtains certificates from the configured issuer. ingress-controllers will pick these up from the secret specified in spec.tls.hosts[].

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/issuer: mail-issuer
spec:
  tls:
  - hosts:
    - "just.a.name.tld"
    secretName: mail-some-domain-certificate

Closing Considerations

With Kubernetes acting as a service registry and authentication source, vault acting as a source of certificates and with cert-manager being the cluster household’s butler, it is surprisingly easy to obtain and distribute certificates from a private CA, free from the pains usually associated with operating a CA from and with human labour. Only the issuer’s need manual intervention when intermediate CAs and thus the corresponding CA bundle specs will need to be changed. Operations cut so are as close to a one-time job as I can realisitically imagine.