Scott's Weblog The weblog of an IT pro focusing on cloud computing, Kubernetes, Linux, containers, and networking

Technology Short Take 142

Welcome to Technology Short Take #142! This time around, the Networking section is a bit light, but I’ve got plenty of cloud computing links and articles for you to enjoy, along with some stuff on OSes and applications, programming, and soft skills. Hopefully there’s something useful here for you!


  • Bill Doerrfeld discusses the use of WebAssembly to extend the Envoy proxy here after a discussion with Idit Levine, CEO and founder of Envoy is increasingly becoming an important component in many cloud-native networking solutions, so the ability to extend or customize Envoy’s behavior is quite important in my opinion.



Cloud Computing/Cloud Management

  • Etcd—the key-value store behind Kubernetes, among other things—saw it’s 3.5 release recently; here’s the announcement blog post. I was particularly impressed by some of the improvements in this release, such as the 50% reduction in memory consumption during peak usage and the reduction of the etcd code base by half. Great work!
  • I found this two-part series on Kinesis data streams (part 1, part 2) by Anahit Pogosova to be very informative. I was not familiar with AWS Kinesis Data Streams, but at least now I have a rough idea of what’s involved. There’s a ton of information in these two posts, so if this is a key technology for you I highly recommend giving this series a read.
  • This is a good whitepaper on organizing your AWS environment using multiple accounts.
  • Sarah Wang and Martin Casado sparked quite the discussion with this post on the cost of cloud computing. I didn’t find the conclusions of the article to be all that “shocking,” but many folks did (the “TL;DR” is that you should consider/evaluate cloud/cost optimizations as you grow, which may include repatriation).
  • It’s neat to see the Kubernetes Ingress APIs evolving; you can get more information on the Kubernetes Gateway API here. TGIK 148 with the inimitable Josh Rosso may also be useful.
  • Chip Zoller looks at using Kyverno for Kubernetes Custom Resources.

Operating Systems/Applications



Career/Soft Skills

  • Here’s a list of 21 hands-on AWS builders you may want to consider following.
  • Julia Evans has a rule to help with blogging: blog about what you’ve struggled with (hat tip to Ivan Pepelnjak for linking to Julia’s post). It’s a rule I’ve used in the past, and it’s helpful. If you’re thinking about blogging (it’s not for everyone, but it can be quite fulfilling), you might consider following this guideline.
  • This article on the cultural implications of silence was a great read (hat tip to the Learning How to Learn community, which referred me to this article).

And that’s all, folks! If you found this post useful, please feel free to share it via your social media accounts. If you have feedback—and I’m always open to constructive feedback—I’d love to hear from you! Feel free to contact me on Twitter, find me on one of the Slack communities I frequent, or send me an e-mail (my address is not hard to find). Thanks for reading!

Adding Multiple Items Using Kustomize JSON 6902 Patches

Recently, I needed to deploy a Kubernetes cluster via Cluster API (CAPI) into a pre-existing AWS VPC. As I outlined in this post from September 2019, this entails modifying the CAPI manifest to include the VPC ID and any associated subnet IDs, as well as referencing existing security groups where needed. I knew that I could use the kustomize tool to make these changes in a declarative way, as I’d explored using kustomize with Cluster API manifests some time ago. This time, though, I needed to add a list of items, not just modify an existing value. In this post, I’ll show you how I used a JSON 6902 patch with kustomize to add a list of items to a CAPI manifest.

By the way, if you’re not familiar with kustomize, you may find my introduction to kustomize post to be helpful. Also, for those readers who are unfamiliar with JSON 6902 patches, the associated RFC is useful, as is this site.

In this particular case, the addition of the VPC ID and the subnet IDs were easily handled with a strategic merge patch that referenced the AWSCluster object. More challenging, though, was the reference to the existing security groups that I needed to add (necessary in order to use my existing SSH bastion infrastructure to reach CAPI-instantiated instances). In hindsight, I suppose I could have written a single strategic merge patch to do the same thing, but I went down the JSON 6902 patch route (and learned something along the way).

Here’s a snippet of what the final (rendered) YAML needed to look like (taken from an AWSMachineTemplate object in the CAPI manifest, the security group IDs are obviously fake):

        - id: sg-01234567890123456
        - id: sg-abcdef0123456789a

The knowledge I was missing to make this work was how to correctly format the JSON 6902 patch such that it would add a list of items when rendered using kustomize. The site gave me the hint I needed—it just needed to be formatted like a JSON list. (It’s so obvious to me now, but I guess lots of things seem obvious after we learn them, right?)

Here’s a JSON 6902 patch that would create the YAML I was seeking:

  { "op": "add",
    "path": "/spec/template/spec/additionalSecurityGroups",
    "value": [
        { "id": "sg-01234567890123456" },
        { "id": "sg-abcdef0123456789a" }

The second useful piece to glean from this is something that I also mentioned in the post on using kustomize with CAPI, and that’s the use of a regular expression (regex) in the kustomization.yaml file to control where this JSON 6902 patch is applied. As outlined in the previous post on using existing AWS security groups with CAPI, the additionalSecurityGroups field can be added to an AWSMachineTemplate—this is exactly what I wanted so that both my MachineDeployments and my KubeadmControlPlane would pick up the change. So how do I get one patch to apply to multiple objects? With a regex, of course!

  - path: securitygroups.json
      kind: AWSMachineTemplate
      name: ".*"
      version: v1alpha3

The use of the .* regex for the name field means that kustomize will apply the referenced JSON 6902 patch to all objects that match the specified API group, kind, and API version. This is an extremely useful way to apply patches, and it’s not limited to JSON 6902 patches (which is why I said earlier I could have written a strategic merge patch instead).

Once I had the JSON 6902 patch file in place and the necessary changes to the kustomization.yaml file made, running kustomize build . created exactly the YAML I needed. Once I’d verified the correctness of the kustomize build ouptut, a quick kustomize build . | kubectl apply -f - set the creation of my workload cluster in action.

I hope this information proves useful to someone. If you have questions, or if you spot a mistake I’ve made in this post, feel free to reach out. I’m always open to constructive feedback. You can contact me on Twitter (DMs are open) or hit me up on the Kubernetes Slack. I’d love to hear from you.

Using WireGuard on macOS via the CLI

I’ve written a few different posts on WireGuard, the “simple yet fast and modern VPN” (as described by the WireGuard web site) that aims to supplant tools like IPSec and OpenVPN. My first post on WireGuard showed how to configure WireGuard on Linux, both on the client side as well as on the server side. After that, I followed it up with posts on using the GUI WireGuard app to configure WireGuard on macOS and—most recently—making WireGuard from Homebrew work on an M1-based Mac. In this post, I’m going to take a look at using WireGuard on macOS again, but this time via the CLI.

Some of this information is also found in this WireGuard quick start. Here I’ll focus only on using macOS as a WireGuard client, not as a server; refer to the WireGuard docs (or to my earlier post) for information on setting up a WireGuard server. I’ll also assume that you’ve installed WireGuard via Homebrew.

Generating Keys

The first step is to generate the public/private keys you’ll need. If the /usr/local/etc/wireguard (or the /opt/homebrew/etc/wireguard for users on an M1-based Mac) directory doesn’t exist, you’ll need to first create that directory. (It didn’t exist on my system.) Then, from that directory, run these commands:

umask 077
wg genkey | tee privatekey | wg pubkey > publickey

This generates files named privatekey and publickey, the contents of which you’ll use in the configuration of WireGuard. Some tutorials indicate the need to become root or use sudo, but my configuration seems to work fine without either of those.

Setting up a WireGuard Interface

Once you have public/private keys, you’re ready to set up the interface configuration. Note that you’ll also need the public key of the server (peer) system to which you’re connecting.

Create a file named wg0.conf in /usr/local/etc/wireguard (that would be in /opt/homebrew/etc/wireguard for M1-based Macs). The contents of the file should look like this:

PrivateKey = <private-key-generated-earlier>
Address = <vpn-ip-address-for-this-interface>

PublicKey = <public-key-for-peer-system>
Endpoint = <public-ip-address>:51280
AllowedIPs = <vpn-ip-address-for-peer-interface>, <additional-cidrs>
PersistentKeepalive = 25

Most of this configuration file is pretty straightforward:

  • You’ll need to specify the private key for the local system plus the public key of the peer system.
  • Each system will need “VPN IP addresses” that are routable/reachable from each other (I use a /29 CIDR from which I assign peer VPN IP addresses). The VPN IP addresses do not have to be publicly routable (they can be RFC 1918 ranges), but they should not overlap other IP address ranges—otherwise you’ll run into routing table issues.
  • For the AllowedIPs setting, you need to specify the VPN IP address of the peer plus any additional networks that are reachable via the peer. In my case, I use WireGuard to access private EC2 instances in an AWS VPC, so I specify the VPC CIDR here.

The peer system will have/need a similar configuration. As described here, my primary use case is enabling connectivity to EC2 instances with private IP addresses inside a VPC, so the peer system for me is a Linux instance with WireGuard installed and configured.

Activating the VPN

Once you have the WireGuard interface configuration finished, you can activate the VPN connection using wg-quick up wg0 (if you called your configuration file from the previous section something other than wg0.conf, then change the command accordingly). If the peer system is already configured and its interface is up, then the VPN connection should establish automatically and you should be able to start routing traffic through the peer (assuming you specified some additional networks to be routed through the peer).

Use wg-quick down wg0 to take the VPN connection down.

I did see some references to being able to use launchd on macOS to automatically start the WireGuard interface, but I haven’t explored that configuration yet.

Troubleshooting the Connection

If you run into problems getting the connection up and working, here are a few things you can check:

  • When specifying AllowedIPs, make sure to include the VPN interface IP address as well as any additional networks. Failing to include the VPN interface IP address can cause the connection to fail.
  • Double-check that you’ve specified public and private keys correctly. It’s easy to get these mixed up. Each system’s configuration should reference its own private key and the peer’s public key.
  • Make sure you’ve specified the peer’s endpoint correctly.
  • Make sure that the traffic is being allowed to reach each system. Double-check firewalls, network access control lists, host-based firewalls, and security groups.
  • Make sure the WireGuard interface is up and active on both systems. When troubleshooting connections, it can be easy to forget whether the interface is active or not.

Aside from the location of the configuration files, configuring WireGuard via the CLI using configuration files is very, very similar across systems. The interface configuration shown above is—as far as I’ve been able to tell so far—identical across both macOS and Linux.

I hope this article is helpful. If you have any questions, comments, or corrections, please feel free to contact me. You can reach me on Twitter (DMs are open), or hit me up on one of a variety of Slack communities. I’d be happy to chat with you.

Installing Older Versions of Kumactl on an M1 Mac

The Kuma community recently released version 1.2.0 of the open source Kuma service mesh, and along with it a corresponding version of kumactl, the command-line utility for interacting with Kuma. To make it easy for macOS users to get kumactl, the Kuma community maintains a Homebrew formula for the CLI utility. That includes providing M1-native (ARM64) macOS binaries for kumactl. Unfortunately, installing an earlier version of kumactl on an M1-based Mac using Homebrew is somewhat less than ideal. Here’s one way—probably not the only way—to work around some of the challenges.

Note that this post really only applies to users of M1-based Macs; users of Intel-based Macs can extract the kumactl binary from the release archive available linked from the Kuma install docs. (The same goes for users of Linux distributions running on Intel-based hardware.) On the Kuma website, simply select the desired version of Kuma from the drop-down in the upper left, check the page for the direct download link, and off you go. This doesn’t work for M1-based Macs because at the time this post was written, the Kuma community was not providing ARM64 binaries. This leaves Homebrew as the only way (aside from building it yourself from source) to get a native ARM64 binary of kumactl for an M1-based Mac.

There are a couple of things to point out—caveats, if you will—about this workaround before I get into the details.

  1. It seems (based what I’ve observed regarding Homebrew’s behavior) that the ability to have multiple versions installed via Homebrew relies on what I’m calling “versioned formulas,” like terraform@0.13 (where the version number is embedded in the name of the formula).
  2. Although you can have multiple versions of a formula present on the disk in the correct locations, that doesn’t necessarily mean that Homebrew will recognize them as “installed.”
  3. Without Homebrew recognizing the packages as “installed,” then using commands like brew unlink or brew link will be severely limited. This is particularly impactful in this situation.

With these limitations in mind, let’s look at the workaround I’ve found. It’s based on information from a variety of sources; this one and this one are good examples. It was this GitHub gist, however, that finally got me to success. The approaches that describe using a remote URL failed with a Homebrew error for me (this is using Homebrew 3.2.0).

Here’s the steps that worked for me, based very largely on the referenced GitHub gist:

  1. Navigate to the directory where the Homebrew formulas are stored. On my M1-based system, that location was /opt/homebrew/Library/Taps/homebrew/homebrew-core/Formula. You can use brew --prefix to determine the base location where Homebrew is installed; the rest of the path should be the same.
  2. As it turns out, the directory where Homebrew is installed is a giant Git repository. Thus, we can “roll back” to the state of the repository when the previous version of the formula you want is available. To figure that out for kumactl, use git log --follow kumactl.rb. Examine the output to determine the correct Git commit that will get you the desired version. I needed version 1.1.6 of kumactl, so the Git commit I needed was 89b6866.
  3. Run git checkout -b kumactl-1.1.6 89b6866 to create a new branch at that commit hash.
  4. Reinstall the older version with brew reinstall ./kumactl.rb.
  5. If you have a newer version installed, this command will remove the newer version.

If you only needed to roll back to a previous kumactl version, then you’re good to go. If you need multiple versions, read on. The following steps will help you get multiple copies installed (with some limitations):

  1. Copy the /opt/homebrew/Cellar/kumactl/<version> directory to a backup location on disk somewhere. You can replace /opt/homebrew with $(brew --prefix) if you’re not sure where Homebrew is installed.
  2. Assuming you’re still inside the /opt/homebrew/Library/Taps/homebrew/homebrew-core/Formula directory, run git checkout master to switch out of the branch you created earlier in order to install the older version.
  3. Run brew install kumactl to install the latest version. This will remove the earlier version.
  4. Copy the <version> directory from step 1 from the backup location to /opt/homebrew/Cellar/kumactl. In my example, this meant I had both a 1.1.6 and a 1.2.0 directory in /opt/homebrew/Cellar/kumactl.

You now have multiple versions installed. However, due to the caveats/limitations listed earlier, Homebrew doesn’t recognize both versions as “installed.” This is in spite of the fact that brew info kumactl shows that it recognizes both versions are present on the disk. Since both versions aren’t recognized as “installed,” this means you cannot use brew unlink or brew link to switch between versions. (Some of the articles I found also referenced a brew switch command, which apparently does the same function but appears to have been removed from the latest versions of Homebrew.)

So what to do? Probably the easiest thing to do is this:

ln -s $HOME/bin/kumactl-1.1.6 /opt/homebrew/Cellar/kumactl/1.1.6/bin/kumactl

Replace $HOME/bin with the path of your choice, preferably something already in $PATH. This allows you to run the older version by specifying the binary (via symbolic link) with the version in the name, and running the latest version by just using the binary’s name. It’s not a great solution, but it works.

I hope this (somewhat limited) workaround is useful. There are discussions happening in the Kuma community about what, if anything, the community can do to help solve this problem. Join the Kuma community Slack and have your opinions heard—or, even better, offer to help! In the meantime, feel free to hit me on Twitter if you have any questions, or if you have a better solution to this problem.

Making WireGuard from Homebrew Work on an M1 Mac

After writing the post on using WireGuard on macOS (using the official WireGuard GUI app from the Mac App Store), I found the GUI app’s behavior to be less than ideal. For example, tunnels marked as on-demand would later show up as no longer configured as an on-demand tunnel. When I decided to set up WireGuard on my M1-based MacBook Pro (see my review of the M1 MacBook Pro), I didn’t want to use the GUI app. Fortunately, Homebrew has formulas for WireGuard. Unfortunately, the WireGuard tools as installed by Homebrew on an M1-based Mac won’t work. Here’s how to fix that.

The key issues with WireGuard as installed by Homebrew on an M1-based Mac are:

  • On an M1-based Mac, Homebrew installs (by default) to the /opt/homebrew prefix. By comparison, Homebrew uses /usr/local on Intel-based Macs. Some of the WireGuard-related scripts are hard-coded to use /usr/local as the Homebrew prefix. Because the prefix has changed, though, these scripts now don’t work on an M1-based Mac.
  • WireGuard has a dependency on Bash. Unfortunately, the version of Bash supplied by macOS isn’t supported by WireGuard (it’s too old). Without a very specific PATH configuration, even installing the Homebrew version of Bash—which is supported by WireGuard—won’t help. (You’ll have to install it anyway; Homebrew requires it to satisfy a dependency.)

Fortunately, there is a way to work around these issues. The following steps should enable you to use Homebrew’s WireGuard on your M1-based Mac:

  1. Go ahead and use brew install to install the “wireguard-go”, “wireguard-tools”, and “bash” packages.
  2. Edit your shell startup file(s) to modify your PATH variable in such a way so that a personal directory—like $HOME/bin or similar—is listed first, followed by the /opt/homebrew/bin directory. (I think the order of the /opt/homebrew/bin directory in the PATH may not matter, but I need to do more testing before I’ll know for certain.)
  3. Copy the wg-quick script installed by Homebrew into the personal directory specified in the previous step. This is necessary because the script is read-only in the location where Homebrew installs it.
  4. Edit the shebang line of the wg-quick script to say #!/usr/bin/env -P/opt/homebrew/bin bash. Basically, you’re adding -P/opt/homebrew/bin to the shebang line, right before bash. (You can check man env to understand what the -P parameter does, but the “TL;DR” is that you’re limiting where env will search to find a bash binary.)
  5. On or about line 44, the script defines a variable named CONFIG_SEARCH_PATHS. Edit this line to add /opt/homebrew/etc/wireguard to the existing list of directories that the script will search for the WireGuard configuration files.

Once you’ve done these steps, you can proceed with configuring WireGuard as outlined in a variety of articles and posts (such as this one I wrote on configuring WireGuard on macOS via the CLI). Replace any references to /usr/local/etc/wireguard in these posts with /opt/homebrew/etc/wireguard.

When you run sudo wg-quick up wg0 to activate the VPN, your shell will find your modified version of this script first (because of step 2 above). This means your changes to the script will be in effect. Your modified version of this script will find the Homebrew-installed version of Bash (because of the change made in step 4, limiting where env will look to find the bash binary), which is compatible with WireGuard and will allow the script to complete properly. Finally, because you added the /opt/homebrew/etc/wireguard directory to the list of paths that wg-quick will search for WireGuard configuration files, it will find your configuration files and appropriately configure the interface(s).

The steps described above should get you a working WireGuard installation on an M1-based Mac. Hopefully, I can contribute these changes (or improved versions of these changes) back to the Homebrew project. If you have questions, or if I have made a mistake in my recommendations, please contact me on Twitter. I hope you find this useful!

Recent Posts

Kubernetes Port Names and Terminating HTTPS Traffic on AWS

I recently came across something that wasn’t immediately intuitive with regard to terminating HTTPS traffic on an AWS Elastic Load Balancer (ELB) when using Kubernetes on AWS. At least, it wasn’t intuitive to me, and I’m guessing that it may not be intuitive to some other readers as well. Kudos to my teammates Hart Hoover and Brent Yarger for identifying the resolution, which I’m going to call out in this post.


Technology Short Take 141

Welcome to Technology Short Take #141! This is the first Technology Short Take compiled, written, and published entirely on my M1-based MacBook Pro (see my review here). The collection of links shared below covers a fairly wide range of topics, from old Sun hardware to working with serverless frameworks in the public cloud. I hope that you find something useful here. Enjoy!


Review: Logitech Ergo K860 Ergonomic Keyboard

As part of an ongoing effort to refine my work environment, several months ago I switched to a Logitech Ergo K860 ergonomic keyboard. While I’m not a “keyboard snob,” I am somewhat particular about the feel of my keyboard, so I wasn’t sure how I would like the K860. In this post, I’ll provide my feedback, and provide some information on how well the keyboard works with both Linux and macOS.


Review: 2020 M1-Based MacBook Pro

I hadn’t done a personal hardware refresh in a while; my laptop was a 2017-era MacBook Pro (with the much-disliked butterfly keyboard) and my tablet was a 2014-era iPad Air 2. Both were serviceable but starting to show their age, especially with regard to battery life. So, a little under a month ago, I placed an order for some new Apple equipment. Included in that order was a new 2020 13" MacBook Pro with the Apple-designed M1 CPU. In this post, I’d like to provide a brief review of the 2020 M1-based MacBook Pro based on the past month of usage.


The Next Step

The Greek philosopher Heraclitus is typically attributed as the creator of the well-known phrase “Change is the only constant.” Since I left VMware in 2018 to join Heptio, change has been my companion. First, there was the change of focus, moving to a focus on Kubernetes and related technologies. Then there was the acquisition of Heptio by VMware, and all the change that comes with an acquisition. Just when things were starting to settle down, along came the acquisition of Pivotal by VMware and several more rounds of changes as a result. Today, I mark the start of another change, as I begin a new role and take the next step in my career journey.


Technology Short Take 140

Welcome to Technology Short Take #140! It’s hard to believe it’s already the start of May 2021—my how time flies! In this Technology Short Take, I’ve gathered some links for you covering topics like Azure and AWS networking, moving from macOS to Linux (and back again), and more. Let’s jump right into the content!


Making Firefox on Linux use Private Browsing by Default

While there are a couple different methods to make Firefox use private browsing by default (see this page for a couple methods), these methods essentially force private browsing and disable the ability to use “regular” (non-private) browsing. In this post, I’ll describe what I consider to be a better way of achieving this, at least on Linux.


Technology Short Take 139

Welcome to Technology Short Take #139! This Technology Short Take is a bit heavy on cloud, OS, and programming topics, but there should be enough other interesting links to be useful to plenty of folks. (At least, I hope that’s the case!) Now, let’s get on to the content!


Using WireGuard on macOS

A short while ago I published a post on setting up WireGuard for AWS VPC access. In that post, I focused on the use of Linux on both the server side (on an EC2 instance in your AWS VPC) as well as on the client side (using the GNOME Network Manager interface). However, WireGuard is not limited to Linux, and I recently configured one of my macOS systems to take advantage of this WireGuard infrastructure for access to the private subnets in my AWS VPC. In this post, I’ll walk readers through configuring macOS to use WireGuard.


Adding a MachineHealthCheck using Kustomize

MachineHealthChecks are a powerful feature in the Kubernetes Cluster API (CAPI), and something I played around with not too long ago on TGIK 143. Recently, I was helping to document the use of kustomize with Cluster API for inclusion in the upstream CAPI documentation, and I learned a simple trick with kustomize that I’d apparently overlooked in the past. If you’ve used kustomize for any great length of time you probably already know and have used the functionality I’ll describe in this post, but if you’re new to kustomize or, like me, a user of kustomize that hasn’t had time to dig into all of its functionality, then read on and see how you can use kustomize to add a MachineHealthCheck to a CAPI workload cluster.


Technology Short Take 138

Welcome to Technology Short Take #138. I have what I hope is an interesting and useful set of links to share with everyone this time around. I didn’t do so well on storage links; apologies to my storage-focused friends! However, there should be something for most everyone else. Enjoy!


Deploying a CNI Automatically with a ClusterResourceSet

Not too long ago I hosted an episode of TGIK8s, where I explored some features of Cluster API. One of the features I explored on the show was ClusterResourceSet, an experimental feature that allows users to automatically install additional components onto workload clusters when the workload clusters are provisioned. In this post, I’ll show how to deploy a CNI plugin automatically using a ClusterResourceSet.


Setting up WireGuard for AWS VPC Access

Seeking more streamlined access to AWS EC2 instances on private subnets, I recently implemented WireGuard for VPN access. WireGuard, if you’re not familiar, is a relatively new solution that is baked into recent Linux kernels. (There is also support for other OSes.) In this post, I’ll share what I learned in setting up WireGuard for VPN access to my AWS environments.


Closing out the Tokyo Assignment

In late 2019, I announced that I would be temporarily relocating to Tokyo for a six-month assignment to build out a team focused on cloud-native services and offerings. A few months later, I was still in Colorado, and I explained what was happening in a status update on the Tokyo assignment. I’ve had a few folks ask me about it, so I thought I’d go ahead and share that the Tokyo assignment did not happen and will not happen.


Technology Short Take 137

Welcome to Technology Short Take #137! I’ve got a wide range of topics for you this time around—eBPF, Falco, Snort, Kyverno, etcd, VMware Code Stream, and more. Hopefully one of these links will prove useful to you. Enjoy!


Older Posts

Find more posts by browsing the post categories, content tags, or site archives pages. Thanks for visiting!