Scott's Weblog The weblog of an IT pro focusing on cloud computing, Kubernetes, Linux, containers, and networking

Spousevitivities at VMworld 2019

This year VMworld—VMware’s annual user conference—moves back to San Francisco from Las Vegas. Returning to the Bay Area with VMworld is Spousetivities, which is happening again this year for the 11th year at VMworld. Better get your tickets sooner rather than later, there’s quite a good chance these activities will sell out!

Registration is open right now.

This year’s activities are funded in part by the generous and community-minded support of Veeam and VMUG, who are “putting their money where their mouth is” when it comes to promoting strong work/life balance at events like VMworld.

Here’s a quick look at what’s planned for VMworld 2019 in San Francisco:

Monday, August 26: Spousetivities kicks off the week with a walking food tour. This tour, like all the others, will depart from the Marriott Marquis.

Tuesday, August 27: This full-day event will take participants up to Wine Country for several wine tastings. Transportion is provided, of course, and participants will enjoy lunch on the tour as well.

Wednesday, August 28: Nature, shopping, tranquility, and quaint towns—this tour has it all! Participants will visit the Golden Gate Bridge, the Marin headlands, Muir Woods, and Sausalito. Transportion and lunch are provided, of course.

Thursday, August 29: The week’s activities will wrap up with a visit to the famous spots and sights of Silicon Valley, including the Computer History Museum and the Google campus. Lunch and transportation are included.

Participants are welcome to sign up for any of these activities individually, but there’s also a “full week pass” that offers a discount when compared to signing up for all activities separately.

Head on over to the registration page to get more information about any of the activities, or to sign up. Remember that although it’s called “Spousetivities,” anyone is welcome! You don’t have to be a spouse to attend.

What are you waiting for? Go sign up now!

Calculating the CA Certificate Hash for Kubeadm

When using kubeadm to set up a new Kubernetes cluster, the output of the kubeadm init command that sets up the control plane for the first time contains some important information on joining additional nodes to the cluster. One piece of information in there that (until now) I hadn’t figured out how to replicate was the CA certificate hash. (Primarily I hadn’t figured it out because I hadn’t tried.) In this post, I’ll share how to calculate the CA certificate hash for kubeadm to use when joining additional nodes to an existing cluster.

When looking to figure this out, I first started with the kubeadm documentation. My searches led me here, which states:

The hash is calculated over the bytes of the Subject Public Key Info (SPKI) object (as in RFC7469). This value is available in the output of “kubeadm init” or can be calculated using standard tools.

That’s useful information, but what are the “standard tools” being referenced? I knew that a lot of work had been put into kubeadm init phase (for breaking down the kubeadm init workflow), but a quick review of that documentation didn’t reveal anything. Reviewing the referenced RFC also didn’t provide any suggestions on how the hash might be calculated. I’d messed around with openssl x509 with Kubernetes before (see this post or this post), so how about some trial-and-error?

Starting with what I knew, I started with decoding the certificate using openssl. After reading some openssl man pages, I arrived at this command to extract only the public key of the certificate:

openssl x509 -in /etc/kubernetes/pki/ca.crt -pubkey -noout

(Side note: the use of -noout threw me for a bit; it says “No output, just status”, but what it really means is don’t output anything more than what I’ve already told you to output with -pubkey or other flags.)

From there it was experimenting with openssl dgst to see if I could get results that matched against some known CA certificate hashes I’d captured for clusters I’d built. I wasn’t having much success until I stumbled on this Stack Overflow post, which pointed me in the direction of openssl pkey. That was the missing link I needed.

So, the final command is this:

openssl x509 -in /etc/kubernetes/pki/ca.crt -pubkey -noout |
openssl pkey -pubin -outform DER |
openssl dgst -sha256

I shared this with my team members (sharing is caring!), and my teammate Naadir Jeewan promptly responded with an Ansible filter to perform the same task. This is helpful when openssl isn’t present on the system where you need to calculate the hash. Naadir’s Ansible filter is found here (great work, Naadir!).

There you have it. Next time you find yourself needing to calculate the CA certificate hash to use with kubeadm, you now have two ways of getting there (either using openssl or using Naadir’s Ansible filter).

If you have any questions, don’t hesitate to reach out to me on Twitter. Thanks!

Building Jsonnet from Source

I recently decided to start working with jsonnet, a data templating language and associated command-line interface (CLI) tool for manipulating and/or generating various data formats (like JSON, YAML, or other formats; see the Jsonnet web site for more information). However, I found that there are no prebuilt binaries for jsonnet (at least, not that I could find), and so I thought I’d share here the process for building jsonnet from source. It’s not hard or complicated, but hopefully sharing this information will streamline the process for others.

As some readers may already know, my primary OS is Fedora. Thus, the process I share here will be specific to Fedora (and/or CentOS and possibly RHEL).

To keep my Fedora installation clean of any unnecessary packages, I decided to use a CentOS 7 VM—instantiated and managed by Vagrant—for the build process. If you don’t want to use a build VM, you can omit the steps involving Vagrant. You’ll also need to modify the commands used to install the necessary packages (on Fedora, you’d use dnf instead of yum, for example). Different distributions may also use different package names for some of the dependencies, so keep that in mind.

  1. Run vagrant up in a directory with a Vagrantfile configured to instantiate a CentOS 7 VM. The CentOS 7 box I used was the Libvirt-formatted “centos/7” box, version 1902.01.

  2. Log into the VM using vagrant ssh.

  3. Install the necessary prerequisites with sudo yum install gcc gcc-c++ git.

  4. Clone the GitHub repository for jsonnet with git clone This clones the repository into a directory named “jsonnet” in the current directory.

  5. Switch into the directory for the cloned repository with cd jsonnet.

  6. Run make to build Jsonnet. Two binaries will result: jsonnet and jsonnetfmt.

At this point, you should have functioning binaries, but they’re inside the CentOS build VM. To get them copied outside the VM, it is a simple matter of just a few quick commands:

  1. First, create an SSH configuration file with vagrant ssh-config > config. To know the hostname that Vagrant uses for the VM, you can cat config afterward and look at the Host line.

  2. Copy the jsonnet binary with scp -F config centos-7:/home/vagrant/jsonnet/jsonnet . (replace centos-7 with whatever hostname Vagrant is using for the VM, and adjust the path as needed).

  3. Copy the jsonnetfmt binary with scp -F config centos-7:/home/vagrant/jsonnet/jsonnetfmt . (replace centos-7 with whatever hostname Vagrant is using for the VM, and adjust the path as needed).

At this point, you can destroy the VM with vagrant destroy and then move the binaries into a directory in the PATH.

And that’s it! As I said, the process isn’t hard or difficult, but I did want to share the information nevertheless. Although it didn’t take me long to figure out what dependencies were needed to build the Jsonnet binaries, having them spelled out here may still save someone else some precious time.

Hit me up on Twitter if you find that I missed something in the instructions above, or if you have any questions.

Technology Short Take 116

Welcome to Technology Short Take #116! This one is a bit shorter than usual, due to holidays in the US and my life being busy. Nevertheless, I hope that I managed to capture something you find useful or helpful. As always, your feedback is welcome, so if you have suggestions, corrections, or comments, you’re welcome to contact me via Twitter.


  • David Gee discusses jSNAPy and how it can be used to enable unit tests in infrastructure-as-code scenarios.
  • Jon Langemak tackles understanding RTs (Route Targets) and RDs (Route Distinguishers) are in MPLS VPNs. I also appreciate that Jon included a “Lab time” section in his article that encourages readers to try out the concepts he’s explaining.


  • Although I’ve by and large moved away from Apple hardware (I still have a MacBook Pro running macOS that sees very little use, and a Mac Pro running Fedora), I did see this article regarding a new keyboard for the MacBook Air and MacBook Pro. That’s good—the butterfly keyboards are awful (in my opinion).


  • If you’re unfamiliar with public key infrastructure (PKI), digital certificates, or encryption, you may find this Linux Journal article helpful. It provides the basics behind X.509v3 digital certificates, how they help enable asymmetric (public/private key) encryption, and the connection to Transport Layer Security (TLS). Plus, there are some handy openssl commands!
  • As would be expected with any maturing open source project that is starting to see increased adoption, Kubernetes has seen its share of security vulnerabilities over the last couple of months. This article talks about a recent vulnerability in the kubectl command, which is typically used to interact with Kubernetes clusters.
  • Lennart Koopmann provides a guide to Yubikey authentication in the real world.

Cloud Computing/Cloud Management

Operating Systems/Applications


  • Kubernetes 1.15 introduces alpha support for volume cloning. John Griffith of Red Hat provides more details in this blog post on the Kubernetes web site. There are some notable caveats for this alpha support (CSI drivers only, same storage class, same namespace), but all these are laid out in Griffith’s article.
  • Vito Botta provides a few tips for OpenEBS.


Career/Soft Skills

  • Working effectively as a remote employee or as part of a distributed team is increasingly important. Via Chris Short, I saw this CircleCI blog talking about some best practices they’ve discovered/created for their distributed team. There’s a few good ideas here that may be worth exploring in your situation or team as well.
  • I liked David Varnum’s article on applying essentialism to certifications and skills development. In other words, you can’t know/learn everything, so be smart about where you choose to apply your time, focus, and attention.

That’s all for now. Enjoy your weekend!

Technology Short Take 115

Welcome to Technology Short Take #115! I’m back from my much-needed vacation in Bali, and getting settled back into work and my daily routine (which, for the last few weeks, was mostly swimming in the pool and sitting on the beach). Here’s a fresh new collection of links and articles from the around the web to propel myself back into blogging. I hope you find something useful here!



Nothing this time around, sorry!


  • Software company Agile Bits recently announced support for U2F-compatible hardware security keys in their 1Password product. Currently, the support is limited to the web interface of 1Password and only in specific browsers, but it would not be unreasonable to see the support expand in the future.
  • Vivek Gite shows to how use oathtool to generate time-based one-time passwords for use with 2FA systems (in the article Google is the example service being secured, but instructions are provided near the end for working with other online services as well).
  • Although this article is titled “How to use OpenSSL,” it’s really more of an educational article on hashes, digital signatures, and digital certificates, with some openssl commands thrown in along the way. It’s a misleading title, but the content differs from the title in a good way.

Cloud Computing/Cloud Management

Operating Systems/Applications


  • Cormac Hogan has recently published three good articles on storage in Kubernetes (the articles are all part of a larger “Kubernetes Storage on vSphere” series). The first article covers StatefulSets with a focus on PersistentVolumes (PVs) and PersistentVolumeClaims (PVCs). The second article covers failure scenarios with a focus on node failure/removal, and the third article discusses ReadWriteMany PVs using NFS. Good stuff!
  • Eli Finkelshteyn explains why data moats are not just about the data (they also about creating a data-driven culture).


Career/Soft Skills

  • This doesn’t really fit anywhere else, but it’s such a good article on network effect that I felt like it would be useful. Ali Yahya’s article on robot hiveminds and network effects does a great job, I think, of explaining network effect and defensibility through a real-world example.
  • Although this article is focused on teaching a kid to build a game in Python using Pygame Zero, I think some of the takeaways listed near the end could be applicable to anyone learning a complex new skill.

I have plenty more material I could include, but I’ll stop here so as to not overwhelm the readers (this is a lot of material to digest!). If you have any questions about any of these links, or comments about this or other articles on my site, you’re always welcome to interact with me via Twitter. Have a great weekend, all!

Recent Posts

Blogging Break

I wanted to let readers know that there will be a break in my blogging over the next few weeks. Crystal and I are celebrating our 20th wedding anniversary and have decided to take a very long trip to someplace very far away from civilization so that we can relax, unplug, and simply enjoy each other’s company.


Technology Short Take 114

Welcome to Technology Short Take #114! There will be a longer gap than usual before the next Tech Short Take (more details to come on Monday), but in the meantime here’s some articles and links to feed your technical appetite. Enjoy!


The Linux Migration: Preparing for the Migration

As far back as 2012, I was already thinking about migrating away from Mac OS X (now known as macOS). While the migration didn’t start in earnest until late 2016, a fair amount of work happened in advance of the migration. Since I’ve had a number of folks ask me about migrating to Linux, I thought I’d supplement my Linux migration series with a “prequel” about some of the work that happened to prepare for the migration.


A Sandbox for Learning Pulumi

I recently started using Pulumi, a way of using a general purpose programming language for infrastructure-as-code projects. I’ve been using Pulumi with JavaScript (I know, some folks would say I should question my life decisions), and while installing Pulumi itself is pretty low-impact (a small group of binaries) there are a number of dependencies that need to be installed when using Pulumi with JavaScript. As I’m a stickler for keeping my primary system very “clean” with regard to installed packages and software, I thought I’d create a means whereby I can easily spin up a “sandbox environment” for learning Pulumi.


Technology Short Take 113

Welcome to Technology Short Take #113! I hope the collection of links and articles I’ve gathered for you contains something useful for you. I think I have a pretty balanced collection this time around; there’s a little bit of something for almost everyone. Who says you can’t please everyone all the time?


Technology Short Take 112

Welcome to Technology Short Take #112! It’s been quite a while since the last one, as life and work have been keeping me busy. I have, however, finally managed to pull together this list of links and articles from around the Internet, and I hope that something I’ve included here proves useful to readers.


Using Kubeadm to Add New Control Plane Nodes with AWS Integration

In my recent post on using kubeadm to set up a Kubernetes 1.13 cluster with AWS integration, I mentioned that I was still working out the details on enabling AWS integration (via the AWS cloud provider) while also using new functionality in kubeadm (specifically, the --experimental-control-plane flag) to make it easier to join new control plane nodes to the cluster. In this post, I’ll share with you what I’ve found to make this work.


My Team's Blogs

I’m thankful to have the opportunity to work with an amazing team. Many of my teammates also produce some very useful content via their own sites, and so I thought it might be useful to my readers to share a list of links to my teammates’ blogs.


Spousetivities at Oktane 2019

It should come as no surprise to anyone that I’m a huge supporter of Spousetivities, and not just because it was my wife, Crystal Lowe, who launched this movement. What started as the gathering of a few folks at VMworld 2008 has grown over the last 11 years, and this year marks the appearance of Spousetivities at an entirely new conference: Oktane 2019!


Looking Ahead: My 2019 Projects

It’s been a little while now since I published my 2018 project report card, which assessed my progress against my 2018 project goals. I’ve been giving a fair amount of thought to the areas where I’d like to focus my professional (technical) development this coming year, and I think I’ve come up with some project goals that align both with where I am professionally right now and where I want to be technically as I grow and evolve. This is a really difficult balance to strike, and we’ll see at the end of the year how well I did.


Split Tunneling with vpnc

vpnc is a fairly well-known VPN connectivity package available for most Linux distributions. Although the vpnc web site describes it as a client for the Cisco VPN Concentrator, it works with a wide variety of IPSec VPN solutions. I’m using it to connect to a Palo Alto Networks-based solution, for example. In this post, I’d like to share how to set up split tunneling for vpnc.


Advanced AMI Filtering with JMESPath

I recently had a need to do some “advanced” filtering of AMIs returned by the AWS CLI. I’d already mastered the use of the --filters parameter, which let me greatly reduce the number of AMIs returned by aws ec2 describe-images. In many cases, using filters alone got me what I needed. In one case, however, I needed to be even more selective in returning results, and this lead me to some (slightly more) complex JMESPath queries than I’d used before. I wanted to share them here for the benefit of my readers.


Technology Short Take 111

Welcome to Technology Short Take #111! I’m a couple weeks late on this one; wanted to publish it earlier but work has been keeping me busy (lots and lots of interest in Kubernetes and cloud-native technologies out there!). In any event, here you are—I hope you find something useful for you!


Thoughts on VPNs for Road Warriors

A few days ago I was talking with a few folks on Twitter and the topic of using VPNs while traveling came up. For those that travel regularly, using a VPN to bypass traffic restrictions is not uncommon. Prompted by my former manager Martin Casado, I thought I might share a few thoughts on VPN options for road warriors. This is by no means a comprehensive list, but hopefully something I share here will be helpful.


Kubernetes, Kubeadm, and the AWS Cloud Provider

Over the last few weeks, I’ve noticed quite a few questions appearing in the Kubernetes Slack channels about how to use kubeadm to configure Kubernetes with the AWS cloud provider. You may recall that I wrote a post about setting up Kubernetes with the AWS cloud provider last September, and that post included a few snippets of YAML for kubeadm config files. Since I wrote that post, the kubeadm API has gone from v1alpha2 (Kubernetes 1.11) to v1alpha3 (Kubernetes 1.12) and now v1beta1 (Kubernetes 1.13). The changes in the kubeadm API result in changes in the configuration files, and so I wanted to write this post to explain how to use kubeadm 1.13 to set up a Kubernetes cluster with the AWS cloud provider.


Older Posts

Find more posts by browsing the post categories, content tags, or site archives pages. Thanks for visiting!