Scott's Weblog The weblog of an IT pro focusing on cloud computing, Kubernetes, Linux, containers, and networking

Technology Short Take 105

Welcome to Technology Short Take #105! Here’s another collection of articles and blog posts about some of the common technologies that modern IT professionals will typically encounter. I hope that something I’ve included here proves to be useful for you.




Cloud Computing/Cloud Management

  • I talk about Terraform a fair amount (and I use it a fair amount). Many times that’s in the context of a public cloud (since that’s where I spend most of my time), but here’s an example of using it for vSphere and OpenStack.
  • Chris Herrera tackles a little bit of the “why” in this article on Kubernetes cluster design.
  • Bob Killen discusses exposing StatefulSets in Kubernetes.
  • Lots of folks are super-bullish on Helm, but Bartlomiej Antoniak suggests users think twice before using Helm.
  • Robert Verdam has a two-part series on deploying an application to AWS with Terraform and Ansible (part 1, part 2). Of particular interest—to me, anyway—was Robert’s use of the Terraform provider+inventory script, which I’m exploring for use in some of my own projects.
  • The kubespy tool, released by the Pulumi folks, looks interesting. Have a look at part 1 and part 2 of their blog posts about the CLI tool.
  • Grant Orchard has a three-part series (so far) on VMware Cloud Assembly (part 1, part 2, part 3).
  • The folks at Platform9 recently open-sourced a tool called etcdadm (inspired by kubeadm). The GitHub repository is here, and the blog post with the announcement is here.

Operating Systems/Applications


I don’t have anything to share this time, but I’ll stay alert for content or links to include next time.


Career/Soft Skills

  • I found a couple of resources for folks interested in learning Golang. First up is this Go study group, which has both US and India meeting times. Next up is Awesome Go, which is—in the author’s words—“a curated list of awesome Go frameworks, libraries, and software.”

That’s all for this Technology Short Take. Thanks for reading! If you have questions, comments, or suggestions for improvement, feel free to contact me on Twitter. Have a great weekend!

VMworld EMEA 2018 and Spousetivities

Registration is now open for Spousetivities at VMworld EMEA 2108 in Barcelona! Crystal just opened registration in the last day or so, and I wanted to help get the message out about these activities.

Here’s a quick peek at what Crystal has lined up for Spousetivities participants:

  • A visit to the coastal village of Calella de Palafrugell, the village of Llafranc, and Pals (one of the most well-preserved medieval villages in all of Catalunya), along with wine in the Empordá region
  • Tour of the Dali Museum
  • Lunch and tour of Girona
  • A lunch-time food tour
  • A visit to the Collsera Natural Park and Mount Tibidabo, along with lunch at a 16th century stone farmhouse

For even more details, visit the Spousetivities site.

These activities look amazing. Even if you’ve been to Barcelona before, these unique activities and tours are not available to the public—they’re specially crafted specifically for Spousetivities participants.

Prices for all these activities are reduced thanks to Veeam’s sponsorship, and to help make things even more affordable there is a Full Week Pass that gives you access to all the activities at an additional discount.

These activities will almost certainly sell out, so register today!

Side note: Helping bring a more sane work-life balance to the demanding and often draining schedules of IT conferences like VMworld is something about which I’m very passionate. I believe that a lack of work-life balance is a strong contributing factor to burnout. Supporting Spousetivities is one small thing that I can do. That’s why I post these announcements, and why I do my best to help spread word of the activities. I’d ask you to please consider how you might be able to help.

Setting up the Kubernetes AWS Cloud Provider

The AWS cloud provider for Kubernetes enables a couple of key integration points for Kubernetes running on AWS; namely, dynamic provisioning of Elastic Block Store (EBS) volumes and dynamic provisioning/configuration of Elastic Load Balancers (ELBs) for exposing Kubernetes Service objects. Unfortunately, the documentation surrounding how to set up the AWS cloud provider with Kubernetes is woefully inadequate. This article is an attempt to help address that shortcoming.

More details are provided below, but at a high-level here’s what you’ll need to make the AWS cloud provider in Kubernetes work:

  • The hostname of each node must match EC2’s private DNS entry for that node
  • An IAM role and policy that EC2 instances can assume as an instance profile
  • Kubernetes-specific tags applied to the AWS resources used by the cluster
  • Particular command-line flags added to the Kubernetes API server, Kubernetes controller manager, and the Kubelet

Let’s dig into these requirements in a bit more detail.

Node Hostname

It’s important that the name of the Node object in Kubernetes matches the private DNS entry for the instance in EC2. You can use hostnamectl or a confiugration management tool (take your pick) to set the instance’s hostname to the FQDN that matches the EC2 private DNS entry. This typically looks something like, where 10-15-30-45 is the private IP address and us-west-1 is the region where the instance was launched.

If you’re unsure what it is, or if you’re looking for a programmatic way to retrieve the FQDN, just curl the AWS metadata server:


Make sure you set the hostname before attempting to bootstrap the Kubernetes cluster, or you’ll end up with nodes whose name in Kubernetes doesn’t match up, and you’ll see various “permission denied”/“unable to enumerate” errors in the logs. For what it’s worth, preliminary testing indicates that this step—setting the hostname to the FQDN—is necessary for Ubuntu but may not be needed for CentOS/RHEL.

IAM Role and Policy

Because the AWS cloud provider performs some tasks on behalf of the operator—like creating an ELB or an EBS volume—the instances need IAM permissions to perform these tasks. Thus, you need to have an IAM instance profile assigned to the instances that gives them permissions.

The exact permissions that are needed haven’t been documented anywhere that I’ve found (please contact me if you have more information), but here’s a summary of permissions for control plane nodes that works:

  • Allow permissions on ec2:*
  • Allow permissions on elasticloadbalancing:*
  • Allow permissions on ecr:GetAuthorizationToken, ecr:BatchCheckLayerAvailability, ecr:GetDownloadUrlForLayer, ecr:GetRepositoryPolicy, ecr:DescribeRepositories, ecr:ListImages, and ecr:BatchGetImage
  • Allow permissions on autoscaling:DescribeAutoScalingGroup and autoscaling:UpdateAutoScalingGroup

It’s probably possible to whittle down the ec2:* permission for control plane nodes. Some preliminary thoughts that my colleague Joe Beda shared with me indicates that the following permissions may be enough: ec2:DescribeInstances, ec2:DescribeSecurityGroups, ec2:AttachVolume, ec2:DetachVolume, ec2:DescribeVolumes, ec2:CreateVolume, ec2:DeleteVolume, ec2:DescribeSubnets, ec2:CreateSecurityGroup, ec2:DeleteSecurityGroup, ec2:AuthorizeSecurityGroupIngress, ec2:RevokeSecurityGroupIngress, ec2:CreateTags, ec2:DescribeRouteTables, ec2:CreateRoute, ec2:DeleteRoute, and ec2:ModifyInstanceAttribute. Note, however, that this list has not been tested or validated.

For worker nodes, the permissions are not as far-reaching:

  • Allow permission on ec2:Describe*
  • Allow permissions on ecr:GetAuthorizationToken, ecr:BatchCheckLayerAvailability, ecr:GetDownloadUrlForLayer, ecr:GetRepositoryPolicy, ecr:DescribeRepositories, ecr:ListImages, and ecr:BatchGetImage

You’ll want to capture these permissions in a policy statement, associate that policy statement with a role, and then associate that role with an IAM instance profile. You’ll end up with two IAM instance profiles: one for the control plane nodes with a broader set of permissions, and one for the worker nodes with a more restrictive set of permissions.

AWS Tags

The AWS cloud provider needs a specific tag to be present on almost all the AWS resources that a Kubernetes cluster needs. The tag key is<cluster-name>; the value of the tag is immaterial (this tag replaces an older KubernetesCluster tag). Note that Kubernetes itself will also use this tag on things that it creates, and it will use a value of “owned”. This value does not need to be used on resources that Kubernetes itself did not create. Most of the documentation I’ve seen indicates that the tag is needed on all instances and on exactly one security group (this is the security group that will be modified to allow ELBs to access the nodes, so the worker nodes should be a part of this security group). However, I’ve also found it necessary to make sure the<cluster-name> tag is present on subnets and route tables in order for the integration to work as expected.

Kubernetes Configuration

On the Kubernetes side of the house, you’ll need to make sure that the --cloud-provider=aws command-line flag is present for the API server, controller manager, and every Kubelet in the cluster.

If you’re using kubeadm to set up your cluster, you can have kubeadm add the flags to the API server and controller manager by using the “apiServerExtraArgs” and “controllerManagerExtraArgs” sections in a configuration file, like this:

  cloud-provider: aws
  cloud-provider: aws

Likewise, you can use the “nodeRegistration” section of a kubeadm configuration file to pass extra arguments to the Kubelet, like this:

    cloud-provider: aws

I’d probably also recommend setting the name of the Kubelet to the node’s private DNS entry in EC2 (this ensures it matches the hostname, as described earlier in this article). Thus, the full “nodeRegistration” section might look like this:

    cloud-provider: aws

You would need to substitute the correct fully-qualified domain name for each instance, of course.

Finally, for dynamic provisioning of Persistent Volumes you’ll need to create a default Storage Class. The AWS cloud provider has one, but it doesn’t get created automatically. Use this command to define the default Storage Class:

kubectl apply -f

This will create a Storage Class named “gp2” that has the necessary annotation to make it the default Storage Class (see here). Once this Storage Class is defined, dynamic provisioning of Persistent Volumes should work as expected.


Troubleshooting is notoriously difficult, as most errors seem to be “transparently swallowed” instead of exposed to the user/operator. Here are a few notes that may be helpful:

  • You must have the --cloud-provider=aws flag added to the Kubelet before adding the node to the cluster. Key to the AWS integration is a particular field on the Node object—the .spec.providerID field—and that field will only get populated if the flag is present when the node is added to the cluster. If you add a node to the cluster and then add the command-line flag afterward, this field/value won’t get populated and the integration won’t work as expected. No error is surfaced in this situation (at least, not that I’ve been able to find).
  • If you do find yourself with a missing .spec.providerID field on the Node object, you can add it with a kubectl edit node command. The format of the value for this field is aws:///<az-of-instance>/<instance-id>.
  • Missing AWS tags on resources will cause odd behaviors, like failing to create an ELB for a LoadBalancer-type Service. I haven’t had time to test all the different failure scenarios, but if the cloud provider integration isn’t working as expected I’d double-check that the Kubernetes-specific tags are present on all the AWS resources.

Hopefully the information in this article helps remove some of the confusion and lack of clarity around getting the AWS cloud provider working with your Kubernetes cluster. I intend to keep this document updated as I discover additional failure scenarios or find more detailed documentation. If you have questions, feel free to hit me on Twitter or find me in the Kubernetes Slack community. (If you’re an expert in the AWS cloud provider code and can help flesh out the details of this post, please contact me as well!) Have fun out there, fellow Kubernauts!

A Markdown-to-PDF Workflow on Linux

In May of last year I wrote about using a Makefile with Markdown documents, in which I described how I use make and a Makefile along with CLI tools like multimarkdown (the binary, not the format) and Pandoc. At that time, I’d figured out how to use combinations of the various CLI tools to create various formats from the source Markdown document. The one format I hadn’t gotten right at that time was PDF. Pandoc can create PDFs, but only if LaTeX is installed. This article describes a method I found that allows me to create PDFs from my Markdown documents without using LaTeX.

Two tools are involved in this new conversion process: Pandoc, which I’ve discussed on this site before; and wkhtmltopdf, a new tool I just recently discovered. Basically, I use Pandoc to go from Markdown (MultiMarkdown, specifically) to HTML, and then use wkhtmltopdf to generate a PDF file from the HTML.

The first step in the process is to use Pandoc to convert from Markdown to HTML, including the use of CSS to include custom formatting. The command looks something like this:

pandoc --from=markdown_mmd+yaml_metadata_block+smart --standalone \
--to=html -V css=/home/slowe/Documents/std-styles.css \
--output=<destination-html-filename> <source-md-filename>

This generates an HTML document and links it to the specified CSS document for formatting. I could embed the CSS specification in the YAML metadata block at the top of the Markdown document, but then I’d lose the ability to create multiple versions of the document with different formatting. The -V css= parameter lets me specify formatting instructions at the time of creating the HTML document.

Once I have the HTML document, then I can create a paginated PDF with this command:

wkhtmltopdf -B 25mm -T 25mm -L 25mm -R 25mm \
-q -s Letter <source-html-filename> <destination-pdf-filename>

This will create a paginated PDF, based on US Letter paper size, with 25mm margins all the way around.

Putting these together into a make workflow, I can do something like this:

PD = /usr/local/bin/pandoc
PDFLAGS = --from=markdown_mmd+yaml_metadata_block+smart --standalone
WK = /usr/bin/wkhtmltopdf
WKFLAGS = -B 25mm -T 25mm -L 25mm -R 25mm -q -s Letter
CSS ?= /home/slowe/Documents/std-styles.css
        $(PD) $(PDFLAGS) --to=html -V css=$(CSS) $< | $(WK) $(WKFLAGS) - $@

With this in place, all I have to do is type make Make will use pandoc to convert from Markdown to HTML, then pipe those results to wkhtmltopdf to create a paginated PDF. The beauty of defining the stylesheet to use as a variable is that I can override it on the command line if desired:

make CSS=/path/to/alternate/styles.css

This makes it super-easy to create PDFs from source Markdown documents.

Have suggestions for improvement? I’m always open to learn more—feel free to hit me on Twitter.

Running the gcloud CLI in a Docker Container

A few times over the last week or two I’ve had a need to use the gcloud command-line tool to access or interact with Google Cloud Platform (GCP). Because working with GCP is something I don’t do very often, I prefer to not install the Google Cloud SDK; instead, I run it in a Docker container. However, there is a trick to doing this, and so to make it easier for others I’m documenting it here.

The gcloud tool stores some authentication data that it needs every time it runs. As a result, when you run it in a Docker container, you must take care to store this authentication data outside the container. Most of the tutorials I’ve seen, like this one, suggest the use of a named Docker container. For future invocations after the first, you would then use the --volumes-from parameter to access this named container.

There’s only one small problem with this approach: what if you’re using another tool that also needs access to these GCP credentials? In my case, I needed to be able to run Packer against GCP as well. If the authentication information is stored inside a named Docker container (and then accessed using the --volumes-from parameter), that information won’t be accessible to commands not running in a Docker container.

The fix for this is to bind mount a host path into the container instead of using a named volume. First, create the ~/.config/gcloud directory on your system. Then you’ll initialize and authenticate with this command:

docker run --rm -ti -v $HOME/.config/gcloud:/root/.config/gcloud \
google/cloud-sdk gcloud init

This will take you through the initialization/authentication process, and will store the authentication information outside the container (so that tools like Packer can still access them). From there, just include the bind mount for future invocations of the Docker image. For example, to see a list of your GKE clusters:

docker run --rm -ti -v $HOME/.config/gcloud:/root/.config/gcloud \
google/cloud-sdk gcloud container clusters list

You could then make this easier for yourself with a Bash alias:

alias gcloud="docker run --rm -ti \
-v $HOME/.config/gcloud:/root/.config/gcloud \
google/cloud-sdk gcloud"

Nothing terribly new or revolutionary here, but I hope it’s useful to someone nevertheless.

Recent Posts

Technology Short Take 104

Welcome to Technology Short Take 104! For many of my readers, VMworld 2018 in Las Vegas was “front and center” for them since the last Tech Short Take. Since I wasn’t attending the conference, I won’t try to aggregate information from the event; instead, I’ll focus on including some nuggets you may have missed amidst all the noise.


Kubernetes with Cilium and Containerd using Kubeadm

Now, if that isn’t a title jam-packed with buzzwords, I don’t know what is! In seriousness, though, I wanted to share how to use kubeadm to turn up a Kubernetes cluster using containerd (instead of Docker) and Cilium as the CNI plugin. I’m posting this because I wasn’t able to find a reasonable article that combined all the different threads—some posts talked about using containerd, others talked about using Cilium, and the official Kubernetes docs have examples for using kubeadm. The purpose of this post is to try to pull those threads together.


Book Review: REST API Design Rulebook

REST API Design Rulebook (written by Mark Masse and published by O’Reilly Media; more details here) is an older book, published in late 2011. However, having never attempted to design a REST API before, I found lots of useful information inside that really helped shape my understanding of REST APIs and REST API design.


Better XMind-GNOME Integration

In December of 2017 I wrote about how to install XMind 8 on Fedora 27, and at the time of that writing I hadn’t quite figured out how to define a MIME type for XMind files that would allow users to double-click on an XMind file in Nautilus and open that file in XMind. After doing a bit of additional research and testing, I’ve found a solution and would like to share it here.


Populating New Namespaces Using Heptio Ark

Heptio Ark is a tool designed to backup and restore Kubernetes cluster resources and persistent volumes. As such, it enables users to do a bunch of very useful things like copy cluster resources across cloud providers or replicate environments for development, staging, testing, QA, etc. In this post, I’ll share a slightly different use case for Ark: populating resources into new Kubernetes namespaces.


A Simple Kubernetes Context Switcher

I recently needed to find a simple way of switching between Kubernetes contexts. I already use powerline-go (here’s the GitHub repo), which allows me to display the Kubernetes context in the prompt so I always know which context is the active (current) context. However, switching between contexts using kubectl config set-context <name> isn’t the easiest approach; not to mention it requires merging multiple config files into a single file (which is itself a bit of a task). So, I set out to create a simple Kubernetes context switcher—and here’s the initial results of my efforts.


Bootstrapping an etcd Cluster with TLS using Kubeadm

The etcd distributed key-value store is an integral part of Kubernetes. I first wrote about etcd back in 2014 in this post, but haven’t really discussed it in any great detail since then. However, as part of my recent efforts to dive much deeper into Kubernetes, I needed to revisit etcd. In this post, I wanted to share how to boostrap a new etcd cluster with TLS certificates using kubeadm.


Troubleshooting TLS Certificates

I was recently working on a blog post involving the use of TLS certificates for encryption and authentication, and was running into errors. I’d checked all the “usual suspects”—AWS security groups, host-level firewall rules (via iptables), and the application configuration itself—but still couldn’t get it to work. When I did finally find the error, I figured it was probably worth sharing the commands I used in the event others might find it helpful.


Technology Short Take 103

Welcome to Technology Short Take 103, where I’m back yet again with a collection of links and articles from around the World Wide Web (Ha! Bet you haven’t seen that term used in a while!) on various technology areas. Here’s hoping I’ve managed to include something useful to you!


VMworld 2018 Prayer Time

For the last several years, I’ve organized a brief morning prayer time at VMworld. This year, I won’t be at the conference, but I’d like to help coordinate a time for believers to meet nevertheless. So, if you’re a Christian interested in gathering together with other Christians for a brief time of prayer, here are the details.


Bolstering my Software Development Skills

I recently tweeted that I was about to undertake a new pet project where I was, in my words, “probably going to fall flat on my face”. Later, I asked on Twitter if I should share some of the learning that will occur (is ocurring) as a result of this new project, and a number of folks indicated that I should. So, with that in mind, I’m announcing this project I’ve undertaken is a software development project aimed at helping me bolster my software development skills, and that I’ll be blogging about it along the way so that others can benefit from my mistakes…er, learning.


Cloning All Repositories in a GitHub Organization

I’ve recently started playing around with Ballerina, and upon the suggestion of some folks on Twitter wanted to clone down some of the “official” Ballerina GitHub repositories to provide code examples and guides that would assist in my learning. Upon attempting to do so, however, I found myself needing to clone down 39 different repositories (all under a single organization), and so I asked on Twitter if there was an easy way to do this. Here’s what I found.


Spousevitivities at VMworld 2018

In case there was any question whether Spousetivities would be present at VMworld 2018, let this settle it for you: Spousetivities will be there! In fact, registration for Spousetivities at VMworld 2018 is already open. If previous years are any indication, there’s a really good possibility these activities will sell out. Better get your tickets sooner rather than later!


Additive Loops with Ansible and Jinja2

I don’t know if “additive” is the right word, but it was the best word I could come up with to describe the sort of configuration I recently needed to address in Ansible. In retrospect, the solution seems pretty straightforward, but I’ll include it here just in case it proves useful to someone else. If nothing else, it will at least show some interesting things that can be done with Ansible and Jinja2 templates.


Technology Short Take 102

Welcome to Technology Short Take 102! I normally try to get these things published biweekly (every other Friday), but this one has taken quite a bit longer to get published. It’s no one’s fault but my own! In any event, I hope that you’re able to find something useful among the links below.


Older Posts

Find more posts by browsing the post categories, content tags, or site archives pages. Thanks for visiting!