Scott's Weblog The weblog of an IT pro focusing on cloud computing, Kubernetes, Linux, containers, and networking

Technology Short Take 106

Welcome to Technology Short Take #106! It’s been quite a while (over a month) since the last Tech Short Take, as this one kept getting pushed back. Sorry about that, folks! Hopefully I’ve still managed to find useful and helpful links to include below. Enjoy!



  • The Intel Management Engine (ME) has received a bit of attention as a potential security vulnerability; in this article, authors Maxim Goryachy and Mark Ermolov expose some new concerns around the Intel ME and its undocumented Manufacturing Mode.
  • Serve The Home takes a critical look at the Bloomberg Supermicro stories, debunking or at least calling into question many details of the alleged hardware hack as reported by Bloomberg.


Cloud Computing/Cloud Management

Operating Systems/Applications


Nothing this time around, but I’ll stay alert for items I can include in the next Technology Short Take.


Career/Soft Skills

OK, that’s all I have for now. Question for the readers (we’ll see how many of you make it this far)—which is better for you, regular Tech Short Takes that might be shorter or Tech Short Takes about this length (~30-ish links) but less frequently? Hit me on Twitter and let me know. Thanks!

Spousetivities at DockerCon EU 18

DockerCon EU 18 is set to kick off in early December (December 3-5, to be precise!) in Barcelona, Spain. Thanks to Docker’s commitment to attendee families—something for which I have and will continue to commend them—DockerCon will offer both childcare (as they have in years past) and spouse/partner activities via Spousetivities. Let me just say: Spousetivities in Barcelona rocks. Crystal lines up a great set of activities that really cannot be beat.

Here’s some details on what’s available in Barcelona for DockerCon EU 18:

  • On Monday, December 3, there will be a private tour of Costa Brava and Girona. Girona is an extremely well-preserved medieval walled city dating back to the first century! If you’re a fan of history, this is one not to miss. The tour will, of course, include an amazing lunch in a traditional local restaurant.
  • Tuesday, December 4, participants will do a combined Barcelona city tour along with a visit to the famous La Sagrada Familia. The city tour will include stops to sample a wide variety of tapas in local venues. You’ll also get to visit La Sagrada Familia, which is a definite must-see if you’ve never visited before.
  • Wrapping up the events on Wednesday, December 5, Spousetivities will go to Montserrat. This impressive monastery is a fantastic place to visit, full of history and culture. Also found at Montserrat is “Moreneta de Montserrat,” one of the famous black Madonnas of Europe. The tour will also include a fantastic lunch of local Catalan cuisine, followed by a visit to a nearby medieval village for more historic architecture.

For this event, Docker is stepping up their support of Spousetivities by integrating Spousetivities registration directly into their conference registration. So, if you’re a DockerCon attendee and you’d like to add activities for your spouse or significant other, you can do so directly from within your DockerCon registration. This page has more details on the activities and links to update your DockerCon registration with one or more activities.

(BTW, if you need information on childcare, see here.)

The pricing for Spousetivities is also pretty impressive: only 150 euros a day, and if you pay for 2 days you get a third day at no additional cost! That’s right—150 euros for any one day of activities, or 300 euros for all three days of activities! Visit the DockerCon Spousetivities landing page for a link to add activities to a DockerCon registration—and hurry before the events sell out!

More on Setting up etcd with Kubeadm

A while ago I wrote about using kubeadm to bootstrap an etcd cluster with TLS. In that post, I talked about one way to establish a secure etcd cluster using kubeadm and running etcd as systemd units. In this post, I want to focus on a slightly different approach: running etcd as static pods. The information on this post is intended to build upon the information already available in the Kubernetes official documentation, not serve as a replacement.

For reference, the Kubernetes official documentation has a write-up on using kubeadm to establish an etcd cluster with etcd running as static pods. For Kubernetes 1.12.x (the current version as of this writing), that information is here; for Kubernetes 1.11.x, that same information is here.

When using these instructions for use with Kubernetes 1.11.x, the official guide leaves something out that is very important: reconfiguring the kubelet to operate in a standalone fashion (without the Kubernetes control plane). This information is present in the 1.12.x documentation, but it applies to both versions.

Now, lest you think you can just follow the 1.12.x documentation for a 1.11.x cluster, you need to know that the kubeadm API changed from 1.11.x (where it is v1alpha2) to 1.12.x (where it is v1alpha3), and this means a number of changes elsewhere as well. This means that the kubeadm config files in the 1.12.x guide won’t work for 1.11.x. Additionally, I found that the 1.12.x guide leaves out one additional piece of information that might be necessary in order to get a working kubelet.

The extra step that is needed for 1.11.x (but that is included in the 1.12.x version of the documentation) is to provide an additional systemd drop-in file that configures the kubelet as a standalone kubelet. What the 1.12.x instructions are potentially missing—depending on your Linux distribution and/or the configuration of the Docker container runtime—is matching the kubelet’s cgroup driver to the runtime’s cgroup driver. If these two are mismatched, the kubelet will fail to start.

In my testing on Ubuntu 16.04 images prepared using Wardroom, the Docker container runtime was using the systemd cgroup driver. The kubelet, however, was not, and this caused the kubelet to fail even after adding the drop-in from the 1.12.x documentation.

If this affects you, you can amend the 1.12.x systemd drop-in to add a --cgroup-driver=systemd flag, so that the complete drop-in now looks like this:

ExecStart=/usr/bin/kubelet --address= --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --cgroup-driver=systemd

Put this file in /etc/systemd/system/kubelet.service.d with a higher priority than any other drop-ins found there. I found that, per the 1.12.x instructions, using the filename 20-etcd-service-manager.conf was sufficient. Once this file is in place, then doing a systemctl daemon-reload and systemctl restart kubelet should be enough to get a running kubelet that can run etcd as a static pod.

The official docs—for both 1.11.x and 1.12.x—also assume that you’ll run all the kubeadm alpha phase commands from one specific host, then copy files around to the other hosts as appropriate. This will work, but if you do it this way the certificates generated by kubeadm will have an extra hostname attached to them: the hostname from the system generating the certificates. To avoid having this extra hostname attached to the certificates generated by kubeadm, you’ll need to instead copy the kubeadm configuration files to each node, then run the kubeadm alpha phase commands on each node. This also means that you’ll need to copy both the ca.crt and the ca.key generated on the first node to the second and third nodes. The instructions only say to copy the ca.crt file, but this is because it isn’t needed on the other nodes when you’re generating all the certificates on one node.

Aside from these two (very) minor issues, the procedures in the official docs will get you a working etcd cluster, properly secured with TLS certificates, and running as static pods under the kubelet (instead of as systemd units). Between the official docs plus these notes (which I’ll see about getting merged into the official docs) or my earlier article about running etcd as systemd units, you now have a couple of different approaches to setting up a properly-secured etcd cluster that can back your Kubernetes infrastructure.

Feel free to hit me up on Twitter if you have any questions, or if you feel I’ve made a mistake or misrepresented anything. Thanks!

Validating RAML Files Using Docker

Back in July of this year I introduced Polyglot, a project whose only purpose is to provide a means for me to learn more about software development and programming (areas where am I sorely lacking real knowledge). In the limited spare time I’ve had to work on Polyglot in the ensuing months, I’ve been building out an API specification using RAML, and in this post I’ll share how I use Docker and a Docker image to validate my RAML files.

Since I was (am) using Visual Studio Code as my primary text editor/development environment these days, I started out by looking for a RAML extension that would provide some sort of linting/validation functionality. I found an extension to do RAML syntax highlighting, which seemed like a reasonable first step.

After a bit more research, I found that there was a raml-cli NPM package that one could use to validate RAML files from the command line. I was a bit leery of installing an NPM package on my system, so I thought, “Why not use a Docker container for this?” It will keep my system clean of excess/unnecessary packages and dependencies, and it will provide some practice with building Docker containers.

A bit of work later I had this Dockerfile (which you can find in my “dockerfiles” GitHub repository):

FROM node:6.14.3-alpine

LABEL maintainer "Scott Lowe <>"

RUN apk add --no-cache git && \
    rm -rf /var/cache/apk/*

RUN npm install -qg raml-cli && \
    npm cache clean

RUN mkdir -p /data
VOLUME /data


CMD ["--help"]

Feeding this to docker build results in an image of about 196MB in size, which isn’t too shabby (I’d like to continue to try to shave this down even further). With this image, I can now easily validate RAML files:

docker run --rm -v $(pwd):/data raml-cli:0.1 validate orders-api.raml

And, of course, I could make this even easier with a Bash alias:

alias raml-cli="docker run --rm -v $(pwd):/data raml-cli:0.1"

Using the alias means I can now simply run raml-cli validate <filename.raml>. This makes it very easy for me to validate the RAML specifications I’m writing.

In the event others may find this helpful, the Dockerfile is on GitHub in my “dockerfiles” repository, and I’ve pushed the Docker image to Docker Hub for anyone to pull down.

Technology Short Take 105

Welcome to Technology Short Take #105! Here’s another collection of articles and blog posts about some of the common technologies that modern IT professionals will typically encounter. I hope that something I’ve included here proves to be useful for you.




Cloud Computing/Cloud Management

  • I talk about Terraform a fair amount (and I use it a fair amount). Many times that’s in the context of a public cloud (since that’s where I spend most of my time), but here’s an example of using it for vSphere and OpenStack.
  • Chris Herrera tackles a little bit of the “why” in this article on Kubernetes cluster design.
  • Bob Killen discusses exposing StatefulSets in Kubernetes.
  • Lots of folks are super-bullish on Helm, but Bartlomiej Antoniak suggests users think twice before using Helm.
  • Robert Verdam has a two-part series on deploying an application to AWS with Terraform and Ansible (part 1, part 2). Of particular interest—to me, anyway—was Robert’s use of the Terraform provider+inventory script, which I’m exploring for use in some of my own projects.
  • The kubespy tool, released by the Pulumi folks, looks interesting. Have a look at part 1 and part 2 of their blog posts about the CLI tool.
  • Grant Orchard has a three-part series (so far) on VMware Cloud Assembly (part 1, part 2, part 3).
  • The folks at Platform9 recently open-sourced a tool called etcdadm (inspired by kubeadm). The GitHub repository is here, and the blog post with the announcement is here.

Operating Systems/Applications


I don’t have anything to share this time, but I’ll stay alert for content or links to include next time.


Career/Soft Skills

  • I found a couple of resources for folks interested in learning Golang. First up is this Go study group, which has both US and India meeting times. Next up is Awesome Go, which is—in the author’s words—“a curated list of awesome Go frameworks, libraries, and software.”

That’s all for this Technology Short Take. Thanks for reading! If you have questions, comments, or suggestions for improvement, feel free to contact me on Twitter. Have a great weekend!

Recent Posts

VMworld EMEA 2018 and Spousetivities

Registration is now open for Spousetivities at VMworld EMEA 2108 in Barcelona! Crystal just opened registration in the last day or so, and I wanted to help get the message out about these activities.


Setting up the Kubernetes AWS Cloud Provider

The AWS cloud provider for Kubernetes enables a couple of key integration points for Kubernetes running on AWS; namely, dynamic provisioning of Elastic Block Store (EBS) volumes and dynamic provisioning/configuration of Elastic Load Balancers (ELBs) for exposing Kubernetes Service objects. Unfortunately, the documentation surrounding how to set up the AWS cloud provider with Kubernetes is woefully inadequate. This article is an attempt to help address that shortcoming.


A Markdown-to-PDF Workflow on Linux

In May of last year I wrote about using a Makefile with Markdown documents, in which I described how I use make and a Makefile along with CLI tools like multimarkdown (the binary, not the format) and Pandoc. At that time, I’d figured out how to use combinations of the various CLI tools to create various formats from the source Markdown document. The one format I hadn’t gotten right at that time was PDF. Pandoc can create PDFs, but only if LaTeX is installed. This article describes a method I found that allows me to create PDFs from my Markdown documents without using LaTeX.


Running the gcloud CLI in a Docker Container

A few times over the last week or two I’ve had a need to use the gcloud command-line tool to access or interact with Google Cloud Platform (GCP). Because working with GCP is something I don’t do very often, I prefer to not install the Google Cloud SDK; instead, I run it in a Docker container. However, there is a trick to doing this, and so to make it easier for others I’m documenting it here.


Technology Short Take 104

Welcome to Technology Short Take 104! For many of my readers, VMworld 2018 in Las Vegas was “front and center” for them since the last Tech Short Take. Since I wasn’t attending the conference, I won’t try to aggregate information from the event; instead, I’ll focus on including some nuggets you may have missed amidst all the noise.


Kubernetes with Cilium and Containerd using Kubeadm

Now, if that isn’t a title jam-packed with buzzwords, I don’t know what is! In seriousness, though, I wanted to share how to use kubeadm to turn up a Kubernetes cluster using containerd (instead of Docker) and Cilium as the CNI plugin. I’m posting this because I wasn’t able to find a reasonable article that combined all the different threads—some posts talked about using containerd, others talked about using Cilium, and the official Kubernetes docs have examples for using kubeadm. The purpose of this post is to try to pull those threads together.


Book Review: REST API Design Rulebook

REST API Design Rulebook (written by Mark Masse and published by O’Reilly Media; more details here) is an older book, published in late 2011. However, having never attempted to design a REST API before, I found lots of useful information inside that really helped shape my understanding of REST APIs and REST API design.


Better XMind-GNOME Integration

In December of 2017 I wrote about how to install XMind 8 on Fedora 27, and at the time of that writing I hadn’t quite figured out how to define a MIME type for XMind files that would allow users to double-click on an XMind file in Nautilus and open that file in XMind. After doing a bit of additional research and testing, I’ve found a solution and would like to share it here.


Populating New Namespaces Using Heptio Ark

Heptio Ark is a tool designed to backup and restore Kubernetes cluster resources and persistent volumes. As such, it enables users to do a bunch of very useful things like copy cluster resources across cloud providers or replicate environments for development, staging, testing, QA, etc. In this post, I’ll share a slightly different use case for Ark: populating resources into new Kubernetes namespaces.


A Simple Kubernetes Context Switcher

I recently needed to find a simple way of switching between Kubernetes contexts. I already use powerline-go (here’s the GitHub repo), which allows me to display the Kubernetes context in the prompt so I always know which context is the active (current) context. However, switching between contexts using kubectl config set-context <name> isn’t the easiest approach; not to mention it requires merging multiple config files into a single file (which is itself a bit of a task). So, I set out to create a simple Kubernetes context switcher—and here’s the initial results of my efforts.


Bootstrapping an etcd Cluster with TLS using Kubeadm

The etcd distributed key-value store is an integral part of Kubernetes. I first wrote about etcd back in 2014 in this post, but haven’t really discussed it in any great detail since then. However, as part of my recent efforts to dive much deeper into Kubernetes, I needed to revisit etcd. In this post, I wanted to share how to boostrap a new etcd cluster with TLS certificates using kubeadm.


Troubleshooting TLS Certificates

I was recently working on a blog post involving the use of TLS certificates for encryption and authentication, and was running into errors. I’d checked all the “usual suspects”—AWS security groups, host-level firewall rules (via iptables), and the application configuration itself—but still couldn’t get it to work. When I did finally find the error, I figured it was probably worth sharing the commands I used in the event others might find it helpful.


Technology Short Take 103

Welcome to Technology Short Take 103, where I’m back yet again with a collection of links and articles from around the World Wide Web (Ha! Bet you haven’t seen that term used in a while!) on various technology areas. Here’s hoping I’ve managed to include something useful to you!


VMworld 2018 Prayer Time

For the last several years, I’ve organized a brief morning prayer time at VMworld. This year, I won’t be at the conference, but I’d like to help coordinate a time for believers to meet nevertheless. So, if you’re a Christian interested in gathering together with other Christians for a brief time of prayer, here are the details.


Bolstering my Software Development Skills

I recently tweeted that I was about to undertake a new pet project where I was, in my words, “probably going to fall flat on my face”. Later, I asked on Twitter if I should share some of the learning that will occur (is ocurring) as a result of this new project, and a number of folks indicated that I should. So, with that in mind, I’m announcing this project I’ve undertaken is a software development project aimed at helping me bolster my software development skills, and that I’ll be blogging about it along the way so that others can benefit from my mistakes…er, learning.


Older Posts

Find more posts by browsing the post categories, content tags, or site archives pages. Thanks for visiting!