Scott's Weblog The weblog of an IT pro focusing on cloud computing, Kubernetes, Linux, containers, and networking

VMworld 2019 Prayer Time

For the last several years, I’ve organized a brief morning prayer time at VMworld. I didn’t attend the conference last year, but organized a prayer time nevertheless (and was able to join one morning for prayer). This year, now that I’m back at VMware (via the Heptio acquisition) and speaking at the conference, I’d once again like to coordinate a time for believers to meet. So, if you’re a Christian interested in gathering together with other Christians for a brief time of prayer, here are the details.

What: A brief time of prayer

Where: Yerba Buena Gardens behind Moscone North (near the waterfall)

When: Monday 8/26 through Thursday 8/29 at 7:45am (this should give everyone enough time to grab breakfast before keynotes/sessions start at 9am)

Who: All courteous attendees are welcome, but please note this will be a distinctly Christian-focused and Christ-centric activity (note that I encourage believers of other faiths/religions to organize equivalent activities)

Why: To spend a few minutes in prayer over the day, the conference, the attendees, and each other

As in previous years, you don’t need to RSVP or anything like that, although you’re welcome to if you’d like (just hit me up on Twitter).

There’s no need to bring anything other than an open heart, your faith, and a willingness to display that faith in front of others. The gathering is very casual—we’ll gather together, share a few prayer requests and needs, and then give folks the opportunity to pray as they feel led. If you don’t like praying out loud in public, no problem; we’re not going to force anyone. You’re welcome to pray silently. We just want to give believers the opportunity to strengthen one another in the faith.

Here’s hoping I see you there! If you’re a Christian and planning to attend, please reach out to other Christians you may know and tell them about morning prayer at VMworld!

Spousevitivities at VMworld 2019

This year VMworld—VMware’s annual user conference—moves back to San Francisco from Las Vegas. Returning to the Bay Area with VMworld is Spousetivities, which is happening again this year for the 11th year at VMworld. Better get your tickets sooner rather than later, there’s quite a good chance these activities will sell out!

Registration is open right now.

This year’s activities are funded in part by the generous and community-minded support of Veeam and VMUG, who are “putting their money where their mouth is” when it comes to promoting strong work/life balance at events like VMworld.

Here’s a quick look at what’s planned for VMworld 2019 in San Francisco:

Monday, August 26: Spousetivities kicks off the week with a walking food tour. This tour, like all the others, will depart from the Marriott Marquis.

Tuesday, August 27: This full-day event will take participants up to Wine Country for several wine tastings. Transportion is provided, of course, and participants will enjoy lunch on the tour as well.

Wednesday, August 28: Nature, shopping, tranquility, and quaint towns—this tour has it all! Participants will visit the Golden Gate Bridge, the Marin headlands, Muir Woods, and Sausalito. Transportion and lunch are provided, of course.

Thursday, August 29: The week’s activities will wrap up with a visit to the famous spots and sights of Silicon Valley, including the Computer History Museum and the Google campus. Lunch and transportation are included.

Participants are welcome to sign up for any of these activities individually, but there’s also a “full week pass” that offers a discount when compared to signing up for all activities separately.

Head on over to the registration page to get more information about any of the activities, or to sign up. Remember that although it’s called “Spousetivities,” anyone is welcome! You don’t have to be a spouse to attend.

What are you waiting for? Go sign up now!

Calculating the CA Certificate Hash for Kubeadm

When using kubeadm to set up a new Kubernetes cluster, the output of the kubeadm init command that sets up the control plane for the first time contains some important information on joining additional nodes to the cluster. One piece of information in there that (until now) I hadn’t figured out how to replicate was the CA certificate hash. (Primarily I hadn’t figured it out because I hadn’t tried.) In this post, I’ll share how to calculate the CA certificate hash for kubeadm to use when joining additional nodes to an existing cluster.

When looking to figure this out, I first started with the kubeadm documentation. My searches led me here, which states:

The hash is calculated over the bytes of the Subject Public Key Info (SPKI) object (as in RFC7469). This value is available in the output of “kubeadm init” or can be calculated using standard tools.

That’s useful information, but what are the “standard tools” being referenced? I knew that a lot of work had been put into kubeadm init phase (for breaking down the kubeadm init workflow), but a quick review of that documentation didn’t reveal anything. Reviewing the referenced RFC also didn’t provide any suggestions on how the hash might be calculated. I’d messed around with openssl x509 with Kubernetes before (see this post or this post), so how about some trial-and-error?

Starting with what I knew, I started with decoding the certificate using openssl. After reading some openssl man pages, I arrived at this command to extract only the public key of the certificate:

openssl x509 -in /etc/kubernetes/pki/ca.crt -pubkey -noout

(Side note: the use of -noout threw me for a bit; it says “No output, just status”, but what it really means is don’t output anything more than what I’ve already told you to output with -pubkey or other flags.)

From there it was experimenting with openssl dgst to see if I could get results that matched against some known CA certificate hashes I’d captured for clusters I’d built. I wasn’t having much success until I stumbled on this Stack Overflow post, which pointed me in the direction of openssl pkey. That was the missing link I needed.

So, the final command is this:

openssl x509 -in /etc/kubernetes/pki/ca.crt -pubkey -noout |
openssl pkey -pubin -outform DER |
openssl dgst -sha256

I shared this with my team members (sharing is caring!), and my teammate Naadir Jeewan promptly responded with an Ansible filter to perform the same task. This is helpful when openssl isn’t present on the system where you need to calculate the hash. Naadir’s Ansible filter is found here (great work, Naadir!).

There you have it. Next time you find yourself needing to calculate the CA certificate hash to use with kubeadm, you now have two ways of getting there (either using openssl or using Naadir’s Ansible filter).

If you have any questions, don’t hesitate to reach out to me on Twitter. Thanks!

Building Jsonnet from Source

I recently decided to start working with jsonnet, a data templating language and associated command-line interface (CLI) tool for manipulating and/or generating various data formats (like JSON, YAML, or other formats; see the Jsonnet web site for more information). However, I found that there are no prebuilt binaries for jsonnet (at least, not that I could find), and so I thought I’d share here the process for building jsonnet from source. It’s not hard or complicated, but hopefully sharing this information will streamline the process for others.

As some readers may already know, my primary OS is Fedora. Thus, the process I share here will be specific to Fedora (and/or CentOS and possibly RHEL).

To keep my Fedora installation clean of any unnecessary packages, I decided to use a CentOS 7 VM—instantiated and managed by Vagrant—for the build process. If you don’t want to use a build VM, you can omit the steps involving Vagrant. You’ll also need to modify the commands used to install the necessary packages (on Fedora, you’d use dnf instead of yum, for example). Different distributions may also use different package names for some of the dependencies, so keep that in mind.

  1. Run vagrant up in a directory with a Vagrantfile configured to instantiate a CentOS 7 VM. The CentOS 7 box I used was the Libvirt-formatted “centos/7” box, version 1902.01.

  2. Log into the VM using vagrant ssh.

  3. Install the necessary prerequisites with sudo yum install gcc gcc-c++ git.

  4. Clone the GitHub repository for jsonnet with git clone This clones the repository into a directory named “jsonnet” in the current directory.

  5. Switch into the directory for the cloned repository with cd jsonnet.

  6. Run make to build Jsonnet. Two binaries will result: jsonnet and jsonnetfmt.

At this point, you should have functioning binaries, but they’re inside the CentOS build VM. To get them copied outside the VM, it is a simple matter of just a few quick commands:

  1. First, create an SSH configuration file with vagrant ssh-config > config. To know the hostname that Vagrant uses for the VM, you can cat config afterward and look at the Host line.

  2. Copy the jsonnet binary with scp -F config centos-7:/home/vagrant/jsonnet/jsonnet . (replace centos-7 with whatever hostname Vagrant is using for the VM, and adjust the path as needed).

  3. Copy the jsonnetfmt binary with scp -F config centos-7:/home/vagrant/jsonnet/jsonnetfmt . (replace centos-7 with whatever hostname Vagrant is using for the VM, and adjust the path as needed).

At this point, you can destroy the VM with vagrant destroy and then move the binaries into a directory in the PATH.

And that’s it! As I said, the process isn’t hard or difficult, but I did want to share the information nevertheless. Although it didn’t take me long to figure out what dependencies were needed to build the Jsonnet binaries, having them spelled out here may still save someone else some precious time.

Hit me up on Twitter if you find that I missed something in the instructions above, or if you have any questions.

Technology Short Take 116

Welcome to Technology Short Take #116! This one is a bit shorter than usual, due to holidays in the US and my life being busy. Nevertheless, I hope that I managed to capture something you find useful or helpful. As always, your feedback is welcome, so if you have suggestions, corrections, or comments, you’re welcome to contact me via Twitter.


  • David Gee discusses jSNAPy and how it can be used to enable unit tests in infrastructure-as-code scenarios.
  • Jon Langemak tackles understanding RTs (Route Targets) and RDs (Route Distinguishers) are in MPLS VPNs. I also appreciate that Jon included a “Lab time” section in his article that encourages readers to try out the concepts he’s explaining.


  • Although I’ve by and large moved away from Apple hardware (I still have a MacBook Pro running macOS that sees very little use, and a Mac Pro running Fedora), I did see this article regarding a new keyboard for the MacBook Air and MacBook Pro. That’s good—the butterfly keyboards are awful (in my opinion).


  • If you’re unfamiliar with public key infrastructure (PKI), digital certificates, or encryption, you may find this Linux Journal article helpful. It provides the basics behind X.509v3 digital certificates, how they help enable asymmetric (public/private key) encryption, and the connection to Transport Layer Security (TLS). Plus, there are some handy openssl commands!
  • As would be expected with any maturing open source project that is starting to see increased adoption, Kubernetes has seen its share of security vulnerabilities over the last couple of months. This article talks about a recent vulnerability in the kubectl command, which is typically used to interact with Kubernetes clusters.
  • Lennart Koopmann provides a guide to Yubikey authentication in the real world.

Cloud Computing/Cloud Management

Operating Systems/Applications


  • Kubernetes 1.15 introduces alpha support for volume cloning. John Griffith of Red Hat provides more details in this blog post on the Kubernetes web site. There are some notable caveats for this alpha support (CSI drivers only, same storage class, same namespace), but all these are laid out in Griffith’s article.
  • Vito Botta provides a few tips for OpenEBS.


Career/Soft Skills

  • Working effectively as a remote employee or as part of a distributed team is increasingly important. Via Chris Short, I saw this CircleCI blog talking about some best practices they’ve discovered/created for their distributed team. There’s a few good ideas here that may be worth exploring in your situation or team as well.
  • I liked David Varnum’s article on applying essentialism to certifications and skills development. In other words, you can’t know/learn everything, so be smart about where you choose to apply your time, focus, and attention.

That’s all for now. Enjoy your weekend!

Recent Posts

Technology Short Take 115

Welcome to Technology Short Take #115! I’m back from my much-needed vacation in Bali, and getting settled back into work and my daily routine (which, for the last few weeks, was mostly swimming in the pool and sitting on the beach). Here’s a fresh new collection of links and articles from the around the web to propel myself back into blogging. I hope you find something useful here!


Blogging Break

I wanted to let readers know that there will be a break in my blogging over the next few weeks. Crystal and I are celebrating our 20th wedding anniversary and have decided to take a very long trip to someplace very far away from civilization so that we can relax, unplug, and simply enjoy each other’s company.


Technology Short Take 114

Welcome to Technology Short Take #114! There will be a longer gap than usual before the next Tech Short Take (more details to come on Monday), but in the meantime here’s some articles and links to feed your technical appetite. Enjoy!


The Linux Migration: Preparing for the Migration

As far back as 2012, I was already thinking about migrating away from Mac OS X (now known as macOS). While the migration didn’t start in earnest until late 2016, a fair amount of work happened in advance of the migration. Since I’ve had a number of folks ask me about migrating to Linux, I thought I’d supplement my Linux migration series with a “prequel” about some of the work that happened to prepare for the migration.


A Sandbox for Learning Pulumi

I recently started using Pulumi, a way of using a general purpose programming language for infrastructure-as-code projects. I’ve been using Pulumi with JavaScript (I know, some folks would say I should question my life decisions), and while installing Pulumi itself is pretty low-impact (a small group of binaries) there are a number of dependencies that need to be installed when using Pulumi with JavaScript. As I’m a stickler for keeping my primary system very “clean” with regard to installed packages and software, I thought I’d create a means whereby I can easily spin up a “sandbox environment” for learning Pulumi.


Technology Short Take 113

Welcome to Technology Short Take #113! I hope the collection of links and articles I’ve gathered for you contains something useful for you. I think I have a pretty balanced collection this time around; there’s a little bit of something for almost everyone. Who says you can’t please everyone all the time?


Technology Short Take 112

Welcome to Technology Short Take #112! It’s been quite a while since the last one, as life and work have been keeping me busy. I have, however, finally managed to pull together this list of links and articles from around the Internet, and I hope that something I’ve included here proves useful to readers.


Using Kubeadm to Add New Control Plane Nodes with AWS Integration

In my recent post on using kubeadm to set up a Kubernetes 1.13 cluster with AWS integration, I mentioned that I was still working out the details on enabling AWS integration (via the AWS cloud provider) while also using new functionality in kubeadm (specifically, the --experimental-control-plane flag) to make it easier to join new control plane nodes to the cluster. In this post, I’ll share with you what I’ve found to make this work.


My Team's Blogs

I’m thankful to have the opportunity to work with an amazing team. Many of my teammates also produce some very useful content via their own sites, and so I thought it might be useful to my readers to share a list of links to my teammates’ blogs.


Spousetivities at Oktane 2019

It should come as no surprise to anyone that I’m a huge supporter of Spousetivities, and not just because it was my wife, Crystal Lowe, who launched this movement. What started as the gathering of a few folks at VMworld 2008 has grown over the last 11 years, and this year marks the appearance of Spousetivities at an entirely new conference: Oktane 2019!


Looking Ahead: My 2019 Projects

It’s been a little while now since I published my 2018 project report card, which assessed my progress against my 2018 project goals. I’ve been giving a fair amount of thought to the areas where I’d like to focus my professional (technical) development this coming year, and I think I’ve come up with some project goals that align both with where I am professionally right now and where I want to be technically as I grow and evolve. This is a really difficult balance to strike, and we’ll see at the end of the year how well I did.


Split Tunneling with vpnc

vpnc is a fairly well-known VPN connectivity package available for most Linux distributions. Although the vpnc web site describes it as a client for the Cisco VPN Concentrator, it works with a wide variety of IPSec VPN solutions. I’m using it to connect to a Palo Alto Networks-based solution, for example. In this post, I’d like to share how to set up split tunneling for vpnc.


Advanced AMI Filtering with JMESPath

I recently had a need to do some “advanced” filtering of AMIs returned by the AWS CLI. I’d already mastered the use of the --filters parameter, which let me greatly reduce the number of AMIs returned by aws ec2 describe-images. In many cases, using filters alone got me what I needed. In one case, however, I needed to be even more selective in returning results, and this lead me to some (slightly more) complex JMESPath queries than I’d used before. I wanted to share them here for the benefit of my readers.


Technology Short Take 111

Welcome to Technology Short Take #111! I’m a couple weeks late on this one; wanted to publish it earlier but work has been keeping me busy (lots and lots of interest in Kubernetes and cloud-native technologies out there!). In any event, here you are—I hope you find something useful for you!


Thoughts on VPNs for Road Warriors

A few days ago I was talking with a few folks on Twitter and the topic of using VPNs while traveling came up. For those that travel regularly, using a VPN to bypass traffic restrictions is not uncommon. Prompted by my former manager Martin Casado, I thought I might share a few thoughts on VPN options for road warriors. This is by no means a comprehensive list, but hopefully something I share here will be helpful.


Older Posts

Find more posts by browsing the post categories, content tags, or site archives pages. Thanks for visiting!