Running Private Certificate Authorities in Clusters
December 1, 2019. 2324 words.
When operating web-based systems, one of the foremost concerns is how to protect users’ web traffic.
Often, now, the first suggestion is to use Let’s Encrypt’s certificates, which can be obtained free of cost at a public certificate authority, automated utilizing the (soon to be standardized?) (ACME RFC 8555) protocol.
However, some may desire to free themselves from external dependencies they do not control and which they do not control Recently, it actually occurred to some that cloud operations come with their own set of unique and exciting problems, cf. Kleppmann et al. and Adrian Colyer’s entertaining commentary for further inspiration. or operate in environments where the
ACME HTTP-01 method is not feasible and where putting entire DNS zones with
ACME DNS-01 under the control of third-party software is undesirable.
Then, Hashicorp’s Vault lends itself naturally to implement an in-house certificate authority and while certificates may be obtained and rotated by some lines of
curl, leveraging Jetstack’s
cert-manager utility in a system run by the Kubernetes container orchestrator leads to tangible results surprisingly fast.
Provisioning the Vault
Provided with a sufficiently recent Kubernetes cluster of v1.9 or higher, the
vault software may be installed using the Helm Kubernetes Package manager and Hashicorp’s Helm Chart.
I propose to enable the
authDelegator to authenticate with a Kubernetes
serviceAccount and to enable the user interface as some settings cannot be properly tuned with the
vault CLI and
JSONs is a bad substitute.
The other settings can be left for the time being, most notably, we should not enable TLS from the start because we are bootstrapping and do not have certificates to secure the endpoint yet.
Vault can now be accessed
8200:8200 to the
vault-0 pod and setting
vault, which also acts as CLI, may be conveniently be optained calling
kubectl cp vault-0:/bin/vault $HOME/bin/vault.
I propose to immediately enable the audit log, which helps with debugging. Note that vault audit-logs certain parameters hashed, which keeps spilling certain information from a system administrator policy domain to a log reader policy domain or a storage reader policy domain. When no such separation exists, it may be handled with more slack. When learning and/or debugging, it actually helps.
vault software exposes so called “secret” and “authentication endpoints”.
From secrets endpoints, secrets such as credentials or certificates may be generated and obtained, authentication endpoints establish proof of identity using various methods.
Generally, the upstream Hashicorp documentation is excellent, a walk-through may be consulted to suit the practitioner’s needs.
Obtaining the CA’s Root Certificate
pki secret engine allows to manage certificate generation, storage, distribution and, if needed, revocation of X509 certificates.
The engine is enabled and to be able to distinguish between different CAs (root and intermediate here) exposed under the path given as
The maximum lease time is, as customary for root CAs, high.
It is beneficiary to customize issuing and revocation endpoints before generating any certificates.
Other attributes may be easier customized using the UI, no other are strictly speaking, necessary at this point.
The root certificate is generated calling the
root/generate/internal endpoint and is self-signed.
Deriving an Intermediate CA
As with the root CA, the engine for the intermediate CA is enabled, albeit under different path and with shorter TTL.
A certificate is generated, unsigned this time, signed by the root CA and uploaded back to the CA endpoint.
A set of defaults is then set on a named sub-path, which is called a role in vault lingo.
Note that for compatibility with the
cert-manager utility, the
TTL should be set ≥ 90 d, because certificates specified in kubernetes ingress resources may not configure a custom lifetime and by default require said 90 days.
Settings to Allow
cert-manager to Provision Certificates
To provide for later automation, any role meant for
cert-manager needs it’s settings set so that they are not in contradiction with
CN to be deprecated – I do not know why – and
vault requires a common name by default.
Switching off Require Common Name fixes the issue, so that
cert-manager’s Subject Alternative Name CSRs are considered legit.
In addition, I opted to allow namespaced kubernetes domains to be eligible for certificate issuance, so I added e.g.
management-infra.svc.cluster.local to the set of allowed domains.
I was simply too lazy to figure out how to do that via the
vault CLI, so I set options in the “Edit PKI Role” dialogue.
Allow Kubernetes Service Accounts to Authenticate Against
vault allows to declare trust in a Kubernetes cluster with the
That trust is per cluster and, in essence, will allow tokens signed with the cluster’s self-signed CA certificate to be used to identify technical users, so called service accounts.
An again excellent walk-through the Kubernetes authentication engine is provided by Hashicorp.
Note that because here,
vault is run on the same cluster it will be configured to trust, the
kubernetes_hosts will and only needs to resolve in-cluster.
Permissions to allow identified users (which here will be service accounts from Kubernetes) to operate on secret engines are given in
hcl polciy files, concisely described by the upstream documentation.
Give Kubernetes Service Accounts Permission to Obtain Certificates
Individual service accounts are then represented by roles on the authentication engine, their permissions are defined by an attached policy.
I have opted to permit the
vault service account to obtain certificates because it is necessary to protect the
vault’s endpoints with TLS.
To do so,
vault now needs to provide it’s own certificate (which I hinted at earlier).
This certificate’s provisioning will be controlled by
cert-manager, which, spelunking forward, will require an issuer, which will require a service account with permissions to obtain certificates.
cert-manager is a utility developed by the company Jetstack, which offers a set of mechanisms to automatically obtain certificate from various certificate authorities in an automated fashion.
Installation from the released kubernetes manifests is most easy calling
Interested readers may opt to download as an intermediate step, which then will also allow to use the resulting file, which also contains the necessary Custom Resource Definitions, as a validation source for modern IDEs.
Intellij’s IDEA can configure these from file as well as from
URL sources under Settings → Languages & Frameworks → Kubernetes → CRDs.
The set of Kubernetes Resources will install into the namespace
cert-manager, and because the instances of
cert-manager strings referring to the namespace alone are difficult to isolate and thus difficult to search & replace.
I advise against customization at this point.
Provisioning a Certificate Issuer
cert-manager in place and operating, certificates may be obtained from
Note that the vault’s endpoints are not protected by TLS yet and no
caBundle is given accordingly.
spec.vault.path property defines the secret engine’s endpoint, the method (to sing a CSR) and the role containing defaults for the certificate.
spec.vault.auth.kubernetes.mountPath property defines the autentication engine with corresponding role at the same level.
spec.vault.auth.kubernetes.secretRef sepcifies where to get the authentication credentials, these must correspond to the authentication roles’
bound_service_account_namespaces properties from the role’s creation.
Provisioning Certificates Directly
With that issuer in place, a certificate may be declared (and obtained) with a Kubernetes Custom Resource, which the
cert-manager will pick up and collect a certificate for at
In this case, the issuer is used to provide certificates to protect the
vault’s endpoints with.
Note that after
vault has been upon reconfigured to expose TLS endpoints only, upon certificate re-issuing, the vault process needs to be SIGHUPed, which is out of the scope of
Unless the intermediate certificate which the
vault endpoint’s certificate has been signed with is packaged into the local CA-chains,
vault needs to be called
-tls-skip-verify from this point onwards.
vault Kubernetes manifests may (should!) be reconfigured to use the X509 certificates stored in the secret named
spec.secretName, with the
VAULT_ADDR and the
VAULT_API_ADDR environment variables set to
Provisioning Certificates Indirectly From ingress-Resources
When configuring certificates directly may seem undesirable, Kubernetes ingress resources may be annotated with an issuer
cert-manager.io/issuer: some-name, so that
cert-manager obtains certificates from the configured issuer.
ingress-controllers will pick these up from the secret specified in
With Kubernetes acting as a service registry and authentication source,
vault acting as a source of certificates and with
cert-manager being the cluster household’s butler, it is surprisingly easy to obtain and distribute certificates from a private CA, free from the pains usually associated with operating a CA from and with human labour.
Only the issuer’s need manual intervention when intermediate CAs and thus the corresponding CA bundle specs will need to be changed.
Operations cut so are as close to a one-time job as I can realisitically imagine.