EKS, Bottlerocket, and Cilium with Pulumi

In late 2023, I added some Go code for use with Pulumi to stand up an Amazon Elastic Kubernetes Service (EKS) cluster “from scratch,” meaning without using any prebuilt Pulumi components (like the AWSX VPC component or the EKS component). The code is largely illustrative for newer users, written to show how to stitch together all the components needed for an EKS cluster. In this post, I’ll show you how to modify that code to use Bottlerocket OS as the node OS for your EKS cluster—and share some information on installing Cilium into (onto?) the cluster.

The example code can be found in the pulumi/eks-from-scratch folder in my “learning-tools” GitHub repository. As I mentioned, it’s written in Go, and the associated README file has full instructions for how to use that code in your own environment. Since the code was intended to be illustrative, I have tried to provide enough comments in the code for readers to be able to decode what’s happening without too much difficulty.

To use Bottlerocket OS on the EKS nodes in your cluster, you’ll have to modify the main.go file. Specifically, changes are needed in the section of code that creates a node group (starting on line 62):

// Create a node group for the EKS cluster
_, err := eks.NewNodeGroup(ctx, "node-group", &eks.NodeGroupArgs{
    ClusterName: testCluster.Name,
    // Additional code omitted for brevity
})

Amazon EKS node groups support specifying an AMI type. You’ll leverage this functionality to provide a value (found on this page in the Amazon EKS documentation) that instructs EKS to use Bottlerocket OS. You’ll supply this value via the amiType argument to Pulumi’s node group resource (described in more detail on this page in the Pulumi documentation).

If you modify main.go to add the amiType to the node group definition, then it should look something like this:

// Create a node group for the EKS cluster
_, err := eks.NewNodeGroup(ctx, "node-group", &eks.NodeGroupArgs{
    amiType: pulumi.String("BOTTLEROCKET_x86_64") // Or BOTTLEROCKET_ARM_64
    ClusterName: testCluster.Name,
    // Additional code omitted for brevity
})

Optionally, you could also include the instanceTypes argument to the node group definition, which would allow you to control the specific instance types that Amazon EKS would use in the node group.

Once you make that change, just run pulumi up and watch Pulumi do its magic. When it’s all said and done (it’ll take a little bit of time, so go grab coffee or tea while you wait), you’ll have an Amazon EKS cluster with nodes running Bottlerocket OS. If you’re at all unsure why that’s a (generally) good thing, then I encourage you to check out the Bottlerocket OS web site for more details on Bottlerocket OS and its advantages over a traditional general purpose OS.

(I’m also working on how to use Flatcar Container Linux on EKS nodes, but that’s proving to be a tad more difficult.)

Once the cluster is up and running, then installing Cilium is a matter of following the instructions, found on this page in the Cilium documentation. However, if I know in advance that I’m planning to deploy Cilium on the cluster, then there are a few additional changes I make to my Pulumi code.

Put these changes in your main.go file, starting on line 47 where the EKS cluster itself is defined:

// Create an EKS cluster
testCluster, err := eks.NewCluster(ctx, "test-cluster", &eks.ClusterArgs{
    DefaultAddonsToRemoves: pulumi.StringArray{ // This line is new
        pulumi.String("vpc-cni") // Prevents default CNI from being installed
        pulumi.String("kube-proxy") // If using Cilium kube-proxy replacement
    },
    Name: pulumi.String("testcluster"),
    // Additional code omitted for brevity
})

These additional lines (the DefaultAddonsToRemoves argument and its parameters) prevent Amazon EKS from installing certain default add-ons. In this example, the AWS VPC CNI and kube-proxy are not installed. This allows you to more easily install Cilium in ENI mode with kube-proxy replacement functionality (you can skip a couple of steps related to cleaning up configurations from these components), but the drawback is that the Pulumi program takes far longer to run since the EKS nodes never go into a ready state. For me, the tradeoff is worth it; however, you’ll need to decide for yourself which approach works best in your environment and for your situation.

That’s all I have to share this time around! While I didn’t share anything revolutionary or incredibly insightful, I hope that sharing this information in the context of example code has been useful. If you have any questions, I’d certainly love to hear from you; feel free to reach out to me on Twitter, on the Fediverse, in the Cilium Slack instance, or via e-mail. Thanks for reading!

Metadata and Navigation