Creating Reusable Kuma Installation YAML
Published on 21 Oct 2021 · Filed in Tutorial · 1458 words (estimated 7 minutes to read)Using CLI tools—instead of a “wall of YAML”—to install things onto Kubernetes is a growing trend, it seems. Istio and Cilium, for example, each have a CLI tool for installing their respective project. I get the reasons why; you can build logic into a CLI tool that you can’t build into a YAML file. Kuma, the open source service mesh maintained largely by Kong and a CNCF Sandbox project, takes a similar approach with its kumactl
tool. In this post, however, I’d like to take a look at creating reusable YAML to install Kuma, instead of using the CLI tool every time you install.
You might be wondering, “Why?” That’s a fair question. Currently, the kumactl
tool, unless configured otherwise, will generate a set of TLS assets to be used by Kuma (and embeds some of those assets in the YAML regardless of the configuration). Every time you run kumactl
, it will generate a new set of TLS assets. This means that the command is not declarative, even if the output is. Unfortunately, you can’t reuse the output, as that would result in duplicate TLS assets across installations. That brings me to the point of this post: how can one create reusable YAML to install Kuma?
Fortunately, this is definitely possible. There are two parts to this process:
- Define replacement TLS assets using cert-manager.
- Modify the output of
kumactl
to reference the replacement TLS assets.
Defining TLS Assets
Instead of allowing kumactl
to generate TLS assets every time the command is run, you need a way to be able to declaratively define what TLS assets are needed and what the properties of those assets should be. Fortunately, that’s exactly what the cert-manager project does!
Relying on cert-manager to handle TLS assets does mean that cert-manager becomes a dependency (or a prerequisite) for Kuma—it will have to be installed before Kuma can be installed.
To define the necessary TLS assets, you’ll use cert-manager to:
- Create a self-signed ClusterIssuer.
- Use the self-signed ClusterIssuer to issue a CA root certificate (and a corresponding Secret to store the private key).
- Configure the root CA certificate as an Issuer.
- Issue a TLS certificate and key that will be used by Kuma.
The root CA certificate definition could look something like this:
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: kuma-root-ca
namespace: kuma-system
spec:
isCA: true
commonName: kuma-root-ca
secretName: kuma-root-ca
duration: 43800h # 5 years
renewBefore: 720h # 30d
privateKey:
algorithm: RSA
encoding: PKCS1
size: 2048
usages:
- digital signature
- key encipherment
- cert sign
issuerRef:
name: selfsigned-issuer # References self-signed ClusterIssuer
kind: ClusterIssuer
group: cert-manager.io
Here’s an example of a cert-manager Certificate resource for the TLS certificate that Kuma would use:
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: kuma-tls-general
namespace: kuma-system
spec:
commonName: kuma-tls-general
secretName: kuma-tls-general
duration: 8760h # 1 year
renewBefore: 360h # 15d
subject:
organizations:
- kuma
isCA: false
privateKey:
algorithm: RSA
encoding: PKCS1
size: 2048
usages:
- digital signature
- key encipherment
- server auth
- client auth
dnsNames:
- kuma-control-plane.kuma-system
- kuma-control-plane.kuma-system.svc
issuerRef:
name: kuma-ca-issuer # References Issuer based on kuma-root-ca
kind: Issuer
group: cert-manager.io
Make note of the name
and secretName
values for both certificates; they will be needed later.
After all the TLS assets have been defined—they don’t need to be actually applied against the cluster, just defined—then you’re ready to modify the installation YAML to make it reusable.
Creating Reusable Installation YAML
Once you’ve defined the TLS assets, then you can make the necessary changes to the YAML output of kumactl
to make it reusable. Keep in mind, as described in the previous section, using cert-manager to manage TLS assets means that cert-manager becomes a dependency for Kuma (in other words, you’ll need to install cert-manager before you can install Kuma).
Begin by creating a starting point with kumactl
and piping the output to a file:
kumactl install control-plane --tls-general-secret=kuma-tls-general \
--tls-general-ca-bundle=$(echo "blah") > kuma.yaml
For the --tls-general-secret
parameter, you’re specifying the name of the Secret created by the general TLS certificate you defined earlier with cert-manager.
The file created by this command needs four changes made to it:
- The
caBundle
value supplied for all webhooks needs to be deleted (hence, the value you specify on the command line doesn’t matter). - All webhooks need to be annotated for the cert-manager CA Injector to automatically inject the correct
caBundle
value. - The “kuma-control-plane” Deployment needs to be modified to mount the root CA certificate’s Secret (created by cert-manager) as a volume.
- The “kuma-control-plane” Deployment needs to be changed to pass in a different value for the
KUMA_RUNTIME_KUBERNETES_INJECTOR_CA_CERT_FILE
environment variable (it should point to theca.crt
file on the volume added in step 3).
You could make these changes manually, but since we’re going for declarative why not use something like Kustomize?
To make the first change—removing the caBundle
value embedded by kumactl
—you could use this JSON 6902 patch:
[
{ "op": "remove", "path": "/webhooks/0/clientConfig/caBundle" },
{ "op": "remove", "path": "/webhooks/1/clientConfig/caBundle" },
{ "op": "remove", "path": "/webhooks/2/clientConfig/caBundle" }
]
To make the second change, you could use a JSON 6902 patch like this (the use of “kuma-root-ca” in the patch below refers to the name of the root CA Certificate resource defined earlier with cert-manager):
[
{ "op": "add",
"path": "/metadata/annotations",
"value":
{ "cert-manager.io/inject-ca-from": "kuma-system/kuma-root-ca" }
}
]
These two changes enable you to remove the Base64-encoded copy of the CA certificate—referenced by Kuma’s webhooks—and instead have cert-manager’s CA Injector insert the correct value instead.
This JSON 6902 patch would handle the third change:
[
{
"op": "add",
"path": "/spec/template/spec/volumes/0",
"value": {
"name": "general-ca-crt",
"secret": {
"secretName": "kuma-root-ca"
}
}
},
{
"op": "add",
"path": "/spec/template/spec/containers/0/volumeMounts/0",
"value": {
"name": "general-ca-crt",
"mountPath": "/var/run/secrets/kuma.io/ca-cert",
"readOnly": true
}
}
]
The Secret referenced in the first part of the patch above references the Secret for the root CA Certificate resource, as noted in the secretName
field on the Certificate’s manifest.
And, finally, the fourth change can be handled using this JSON 6902 patch:
[
{ "op": "replace",
"path": "/spec/template/spec/containers/0/env/11/value",
"value": "/var/run/secrets/kuma.io/ca-cert/ca.crt" }
]
I won’t walk through all of these changes in great detail, but I do want to take a moment to dive a bit deeper into the Secret mounted as an additional volume. Using the JSON 6902 patch above against the base YAML created using kumactl
will result in a configuration that looks like this (focused only on volumes and volumeMounts in the Deployment, everything else is stripped away):
spec:
template:
spec:
containers:
- volumeMounts:
- mountPath: /var/run/secrets/kuma.io/ca-cert
name: general-ca-crt
readOnly: true
- mountPath: /var/run/secrets/kuma.io/tls-cert
name: general-tls-cert
readOnly: true
- mountPath: /etc/kuma.io/kuma-control-plane
name: kuma-control-plane-config
readOnly: true
volumes:
- name: general-ca-crt
secret:
secretName: kuma-root-ca
- name: general-tls-cert
secret:
secretName: kuma-tls-general
- configMap:
name: kuma-control-plane-config
name: kuma-control-plane-config
The “general-ca-cert” Secret and volume are what’s been added by the Kustomize patch. Why? When generating the YAML output, kumactl
combines three resources—the TLS certificate, the TLS key, and the CA certificate—into a single Secret. However, cert-manager won’t create a Secret like that. So, to avoid an additional manual step, you mount the Secret created by cert-manager for CA root certificate as a separate volume. This allows you modify the environment variable passed to the control plane specifying where the CA certificate is located to the new path where the volume is mounted.
After the four changes are complete, the resulting modified YAML is completely reusable.
Using the Reusable YAML
To actually use the reusable YAML you’ve just created with the above steps:
- Apply the cert-manager TLS definitions to the cluster where you want to install Kuma. This will create all the necessary Certificate resources and Secrets. (Obviously cert-manager will need to be installed already.)
- Apply the Kuma YAML created by the steps above. It’s configured to reference the cert-manager assets created in step 1.
That’s it.
Caveats/Limitations
The process and changes outlined in this post only apply to a standalone (single zone) installation. It’s absolutely possible to do this for multizone deployments, too, but I’ll leave that as an exercise for the readers.
Additional Resources
I’ve recently published two GitHub repositories with content that supports what’s described in this post:
- The “kuma-cert-manager” repository outlines how to replace
kumactl
-generated TLS assets with resources from cert-manager. This supports the “Defining TLS Assets” section above. Full examples of all the related cert-manager resources are found in this repository. - The “kuma-declarative-install” repository builds on the previous repository by showing the additional changes that must be made to the YAML output generated by
kumactl
. This supports the “Creating Reusable Installation YAML” section above. This includes a Kustomize configuration that will automate all the changes necessary to make the YAML reusable.
I hope this information is useful. If you have questions, feel free to find me on the Kuma community Slack or contact me on Twitter (my DMs are open).