Scott's Weblog The weblog of an IT pro focusing on cloud computing, Kubernetes, Linux, containers, and networking

Cloning All Repositories in a GitHub Organization

I’ve recently started playing around with Ballerina, and upon the suggestion of some folks on Twitter wanted to clone down some of the “official” Ballerina GitHub repositories to provide code examples and guides that would assist in my learning. Upon attempting to do so, however, I found myself needing to clone down 39 different repositories (all under a single organization), and so I asked on Twitter if there was an easy way to do this. Here’s what I found.

Fairly quickly after I posted my tweet asking about a solution, a follower responded indicating that I should be able to get the list of repositories via the GitHub API. He was, of course, correct:

curl -s https://api.github.com/orgs/ballerina-guides/repos

This returns a list of the repositories in JSON format. Now, if you’ve been paying attention to my site, you know there’s a really handy way of parsing JSON data at the CLI (namely, the jq utility). However, to use jq, you need to know the overall structure of the data. What if you don’t know the structure?

No worries, this post outlines another tool—jid—that allows us to interactively explore the data. So, I ran:

curl -s https://api.github.com/orgs/ballerina-guides/repos | jid

This let me explore the data being returned by the GitHub API call, and I quickly determined that I needed the clone_url property for each repository. With this information in hand, I can now construct this command:

curl -s https://api.github.com/orgs/ballerina-guides/repos |
jq -r '.[].clone_url'

Now I have a list of all the clone URLs for all the repositories, right? Not quite—the GitHub API paginates results, so a minor adjustment is needed:

curl -s https://api.github.com/orgs/ballerina-guides/repos?per_page=200 | jq -r '.[].clone_url'

From here it’s a simple matter of piping the results to xargs, like this:

curl -s https://api.github.com/orgs/ballerina-guides/repos?per_page=200 | jq -r '.[].clone_url' | xargs -n 1 git clone

Boom! Problem solved. As fate would have it, I’m not the only one thinking along these lines; here’s another example. Several others also suggested solutions involving Ruby; here’s one such example (this is written for GitHub Enterprise but should work for “ordinary” GitHub).

Naturally, further tweaks to the API URL may be necessary; if you needed private repos, for example, then you’ll have to add &type=private on the URL. Of course, that also means you’ll need to supply authentication details…but you get the idea.

I hope others find this useful! (And thanks to those who took the time to respond on Twitter, I appreciate it!)

Spousevitivities at VMworld 2018

In case there was any question whether Spousetivities would be present at VMworld 2018, let this settle it for you: Spousetivities will be there! In fact, registration for Spousetivities at VMworld 2018 is already open. If previous years are any indication, there’s a really good possibility these activities will sell out. Better get your tickets sooner rather than later!

This year’s activities are funded in part by the generous and community-minded support of Veeam, ActualTech Media, Datrium, and VMUG.

Here’s a brief peek at what’s planned for VMworld in Las Vegas this August:

Monday, August 27

  • It’s a tradition to kick the week off with a Welcome/“Getting to Know You” breakfast, and this year—year 11 for Spousetivities at VMworld—is no different! There will be great food, great company, and the opportunity to win some cool prizes.
  • On Monday afternoon you can secure a spot in a private pod on the High Roller, including an open bar. Nice!
  • Monday is also the first of three (yes, three) cabana days by the pool. This is the perfect way to relax by the Mandalay Bay pool! Monday’s cabana day is sponsored by ActualTech Media.

Tuesday, August 28

  • On Tuesday, Spousetivities participants will head to Zion National Park. I went to Zion earlier this year, and let me tell you—it is absolutely amazing. Don’t miss this trip if you enjoy spectacular natural views.
  • A cabana day is available for those who would prefer to relax by the pool and catch some sun. Tuesday’s cabana day is also brought to you by ActualTech Media.

Wednesday, August 29

  • The third and final cabana day is available on Wednesday. Haven’t had your chance in the sun yet, or can’t get enough time by the pool? No worries, you still have another chance! Wednesday’s cabana day is sponsored by Datrium.
  • For those seeking a different kind of adventure, how about an ATV adventure through the Valley of Fire? This sounds like so much fun, and the Spousetivities price is nearly 50% off the retail price.

Thursday, August 30

Thursday’s sole activity is a walking food tour, available with or without specialty cocktails. Don’t worry: if you partied hard at the official event party the previous evening, this event doesn’t start until noon, so you have time to recover.

Head on over to the registration page to get more information about any of the activities, or to sign up. Remember that although it’s called “Spousetivities,” anyone is welcome! You don’t have to be a spouse to attend.

What are you waiting for? Go sign up now!

Additive Loops with Ansible and Jinja2

I don’t know if “additive” is the right word, but it was the best word I could come up with to describe the sort of configuration I recently needed to address in Ansible. In retrospect, the solution seems pretty straightforward, but I’ll include it here just in case it proves useful to someone else. If nothing else, it will at least show some interesting things that can be done with Ansible and Jinja2 templates.

First, allow me to explain the problem I was trying to solve. As you may know, Kubernetes 1.11 was recently released, and along with it a new version of kubeadm, the tool for bootstrapping Kubernetes clusters. As part of the new release, the Kubernetes community released a new setup guide for using kubeadm to create a highly available cluster. This setup guide uses new functionality in kubeadm to allow you to create “stacked masters” (control plane nodes running both the Kubernetes components as well as the etcd key-value store). Because of the way etcd clusters work, and because of the way you create HA control plane members, the process requires that you start with a single etcd node, then add the second node, and finally add the third node. If you start out with all three, then the cluster won’t establish quorum and establishing a functional Kubernetes control plane node will fail.

As a result, this means the kubeadm configuration file used on the first Kubernetes control plane node must be written to boot a new single-node cluster. However, the kubeadm configuration file on the second control plane node specifies the first node and adds the second node. Likewise, the kubeadm configuration file for the third control plane node has the first and second nodes and adds the third node (hence my use of “additive loops” in the blog post title).

So, when I set out to automate this process using Ansible, I knew this wasn’t going to be your standard “run-of-the-mill” inventory loop in a Jinja2 template—at least, that wasn’t all it was going to be.

I arrived at a solution with three templates (one for each of the stacked masters). In the first template, the section for configuring etcd specifies the “initial-cluster” in the following way:

initial-cluster: "{%- for host in groups['masters'] -%}
{%- if loop.first -%}
{{ hostvars[host]['ansible_fqdn'] }}=https://{{ hostvars[host]['ansible_' + primary_interface]['ipv4']['address'] }}:2380{%- endif -%}
{%- endfor -%}"

The result, as you can probably determine, is that the first item in the loop—the first server in that inventory group—is the only item rendered in the template.

The second server was a bit more challenging, mostly because of a stupid error on my part. After accounting for my stupid error (if you must know, I was using hostvars[inventory_hostname] instead of hostvars[host]), this is where I ended:

initial-cluster: "{%- for host in groups['masters'] -%}
{%- if loop.first -%}
{{ hostvars[host]['ansible_fqdn'] }}=https://{{ hostvars[host]['ansible_' + primary_interface]['ipv4']['address'] }}:2380,{%- endif -%}
{%- if (not loop.last and not loop.first) -%}
{{ hostvars[host]['ansible_fqdn'] }}=https://{{ hostvars[host]['ansible_' + primary_interface]['ipv4']['address'] }}:2380{%- endif %}
{%- endfor -%}"

This adds to the previous template by including the if (not loop.last and not loop.first) conditional, which—for a group of three—ends up meaning the second host in the group. Great; now we have a template for the first host which has only the first host, and a template for the second host which has both the first and second hosts listed.

The third and final template is, after all, a standard “run-of-the-mill” inventory loop:

initial-cluster: "{%- for host in groups['masters'] -%}
{%- if loop.last -%}
{{ hostvars[host]['ansible_fqdn'] }}=https://{{ hostvars[host]['ansible_' + primary_interface]['ipv4']['address'] }}:2380
{%- else -%}
{{ hostvars[host]['ansible_fqdn'] }}=https://{{ hostvars[host]['ansible_' + primary_interface]['ipv4']['address'] }}:2380,
{%- endif -%}{%- endfor -%}"

This one probably requires no explanation; it simply iterates through the inventory group. The only “special” thing about it is knowing whether it should include a comma after the rendered text or not by testing for loop.last.

I should probably explain, though, the use of the ['ansible_' + primary_interface] syntax. I needed a way to get the IP address of an interface on the target system, but the names of interfaces change depending on platform. For example, on one platform the first Ethernet interface may be called “eth0”; on another, it may be called “ens0p0” or similar. Using the “primary_interface” variable allows me to use the same Ansible playbooks across platforms, only needing to adjust the variable when interface names change between platforms. (This was an Ansible trick I picked up from my colleague Craig.) Ansible substitutes the value of the variable when pulling the host variable to render the template. If I specify “eth0” as the value for “primary_interface”, Ansible will treat that as ansible_eth0 and thus pull the IPv4 address for eth0 when rendering the template. Handy! (Thanks Craig!)

As I said, looking back on the solution now it seems simple and straightforward. There’s probably a more elegant solution out there somewhere, but for now this will suffice. Oh, and if you’re interested in seeing the full template or the Ansible playbook used to render the template, have a look at the ansible/kubeadm-etcd-template directory in my GitHub “learning-tools” repository. Enjoy!

Technology Short Take 102

Welcome to Technology Short Take 102! I normally try to get these things published biweekly (every other Friday), but this one has taken quite a bit longer to get published. It’s no one’s fault but my own! In any event, I hope that you’re able to find something useful among the links below.

Networking

Security

Cloud Computing/Cloud Management

  • Joe Duffy takes the wraps off Pulumi, which looks very interesting. This is an area where I’d love to spend some time really digging in.
  • Interesting article here on Chick-Fil-A’s use of Kubernetes in their restaurants, and the tools/process they follow for establishing those clusters.
  • Jeff Geerling—well-known in the Ansible and Drupal communities—discusses the (perceived) complexity of Kubernetes. He spent some time actually working with Kubernetes to solve a problem, and he came away from that time recognizing (in his words) “most of the complexity is necessary,” and seeing value in using Kubernetes for appropriate use cases. This article is not (in my opinion) a knee-jerk reaction to some of the debate around Kubernetes’ complexity, but rather a measured and honest evaluation of a tool to solve problems.
  • Maish Saidel-Keesing has a two-part (so far?) series on comparing CloudFormation, Terraform, and Ansible (part 1, part 2). This is worth a read if you’re trying to determine which of these tools may be right for your use case.
  • Here’s a three-part series on Helm (part 1, part 2, and part 3).

Operating Systems/Applications

  • This is probably well-known, but I found this little tidbit about read-only containers handy.
  • This article on another reason your Docker containers may be slow was an excellent reminder that containerization is not equal to virtualization (that doesn’t make it better or worse, just different), and therefore can’t be treated the same. Different design and architecture considerations apply in each instance.
  • Lightroom is one of only a few applications that I keep around for macOS; this article gives me some alternatives.
  • Here’s an article on getting started with Buildah, part of a suite of tools being built as alternatives to Docker.
  • Tom Sweeney also talks about Buildah and how it can be used to create small container images. My takeaway from this article is that building really small container images requires either a) arcane knowledge of Docker layers and the Dockerfile, or b) arcane knowledge of little-used yum parameters.
  • Shahidh Muhammad has a review/comparison of various tools aimed at helping developers build and deploy their apps on Kubernetes.
  • William Lam shares his list of Visual Studio Code extensions.
  • Finally, a reasonably usable CLI client for Slack—written in Bash, no less!

Storage

Virtualization

  • Mike Foley walks you through configuring TPM 2.0 on a vSphere 6.7 host. (This must be the “future TPM content” that Mike mentioned.)
  • Ed Haletky documents the approach he uses to produce segregated virtual-in-virtual test environments that live within his production environment.
  • It seems as if there’s quite a bit of complexity in this article on using KubeVirt witih GlusterFS, but I can’t tell if that is a byproduct of KubeVirt, GlusterFS, or the combination of the two.

Career/Soft Skills

A couple of useful career-related articles popped up on my radar over the last couple of weeks:

  • First, there’s this article by Eric Lee on IT burnout. If you find yourself experiencing some of the same issues that Eric describes, I’d encourage you to take a deeper look and see if changes are necessary.
  • Next, Cody De Arkland tackles the topic of imposter syndrome. This is an area where I personally have struggled in the past, and I had to learn to humbly accept praise and stop negative self-talk. Talk to someone if you’re wrestling with this.

That’s all for now. Look for the next Technology Short Take in 2-3 weeks, and feel free to contact me via Twitter if you have any links or articles you think I should share. Thanks!

More Handy CLI Tools for JSON

In late 2015 I wrote a post about a command-line tool named jq, which is used for parsing JSON data. Since that time I’ve referenced jq in a number of different blog posts (like this one). However, jq is not the only game in town for parsing JSON data at the command line. In this post, I’ll share a couple more handy CLI tools for working with JSON data.

(By the way, if you’re new to JSON, check out this post for a gentle introduction.)

JMESPath and jp

JMESPath is used by both Amazon Web Services (AWS) in their AWS CLI as well as by Microsoft in the Azure CLI. For examples of JMESPath in action, see the AWS CLI documentation on the --query functionality, which makes use of server-side JMESPath queries to reduce the amount of data returned by an AWS CLI command (as opposed to filtering on the client side).

However, you can also use JMESPath on the client-side through the jp command-line utility. As a client-side parsing tool, jp is similar in behavior to jq, but I find the JMESPath query language to be a bit easier to use than jq in some situations.

Let’s assume we are working with the output of the command aws ec2 describe-security-groups, which returns—as JSON—a list of security groups and their properties. Naturally, parsing this data down to find only the specific information you need is a prime use case for jp. Perhaps you know that a security group named “internal-only” exists, but you don’t know anything else—only the name. Using jp, we could get more details on that specific group with this command:

aws ec2 describe-security-groups | jp "SecurityGroups[?GroupName == 'internal-only']"

The AWSCLI command will return all the security groups, and jp will filter through the data to return only the properties of the security group whose name (as defined in the “GroupName” field/property) is equal to “internal-only.” Compare that syntax to the equivalent jq syntax:

aws ec2 describe-security-groups | jq '.SecurityGroups[] | select (.GroupName == "internal-only")'

That’s handy, but what if we needed only a particular property of that security group? No problem, we’d just append the property name to the end of the query:

aws ec2 describe-security-groups | jp "SecurityGroups[?GroupName == 'internal-only'].GroupId"

This is fundamentally equivalent to this jq command:

aws ec2 describe-security-groups | jq '.SecurityGroups[] | select (.GroupName == "internal-only").GroupId'

However, if you try both jp and jq as described above, you’ll note a significant difference in the output. First, here’s the output of a jq command like the one above:

"sg-7f7efe02"

Now compare that to the output of the equivalent jp command:

[
  "sg-7f7efe02"
]

As you can see, jq returns a specific value, whereas jp returns an array of values (with only a single item in the array). In order to get all the way down to a single value with jp, you have to extend the query:

aws ec2 describe-security-groups | jp "SecurityGroups[?GroupName == 'internal-only'].GroupId | [0]"

This command will return only the value of the “GroupId” field. jp does support a -u command-line option that is equivalent to the -r option to jq in order to return unquoted (raw) strings. This is helpful when storing the command output into a variable for use later.

If you need more than a single property, you can build a JSON object from multiple properties by listing them in curly braces at the end of the query, like this:

aws ec2 describe-security-groups | jp "SecurityGroups[?GroupName == 'internal-only'].{ Name: GroupName, ID: GroupId, VPC: VpcId }"

This will return a JSON object with the properties listed.

I may explore writing a more in-depth article on jp in the future; if that’s something in which you’d be interested, please hit me up on Twitter and let me know.

JSON Incremental Digger (jid)

What if you’re not all that well-versed in the JMESPath syntax? Well, there are online simulators and parsers. Another approach would be to use jid, the JSON incremental digger. It doesn’t necessarily follow the JMESPath syntax, but it does allow you to interactively query some JSON data to find what you’re seeking.

To use jid, simply pipe in some JSON data. For example, you could direct the output of the AWS CLI into jid:

aws ec2 describe-instances | jid

This puts you into an interactive screen where you can explore and parse the data to find exactly what you need. jid will supply “suggested” queries at the top of the screen in green; just press Tab to accept the suggestion. Adjust the query at the top until you have the data you need, then press Enter. The specific data you’d selected in jid will be output to the shell. This is a super-handy way, in my opinion, of exploring JSON data structures with which you aren’t already familiar.

Have any other handy CLI tools for working with JSON? Hit me on Twitter with other tool suggestions, and I’ll update the post with feedback from readers. Thanks!

Recent Posts

A Quick Intro to the AWS CLI

This post provides a (very) basic introduction to the AWS CLI (command-line interface) tool. It’s not intended to be a deep dive, nor is it intended to serve as a comprehensive reference guide (the AWS CLI docs nicely fill that need). I also assume that you already have a basic understanding of the key AWS concepts and terminology, so I won’t bore you with defining an instance, VPC, subnet, or security group.

Read more...

Examining X.509 Certificates Embedded in Kubeconfig Files

While exploring some of the intricacies around the use of X.509v3 certificates in Kubernetes, I found myself wanting to be able to view the details of a certificate embedded in a kubeconfig file. (See this page if you’re unfamiliar with what a kubeconfig file is.) In this post, I’ll share with you the commands I used to accomplish this task.

Read more...

Using Variables in AWS Tags with Terraform

I’ve been working to deepen my Terraform skills recently, and one avenue I’ve been using to help in this area is expanding my use of Terraform modules. If you’re unfamiliar with the idea of Terraform modules, you can liken them to Ansible roles: a re-usable abstraction/function that is heavily parameterized and can be called/invoked as needed. Recently I wanted to add support for tagging AWS instances in a module I was building, and I found out that you can’t use variable interpolation in the normal way for AWS tags. Here’s a workaround I found in my research and testing.

Read more...

A Quadruple-Provider Vagrant Environment

In October 2016 I wrote about a triple-provider Vagrant environment I’d created that worked with VirtualBox, AWS, and the VMware provider (tested with VMware Fusion). Since that time, I’ve incorporated Linux (Fedora, specifically) into my computing landscape, and I started using the Libvirt provider for Vagrant (see my write-up here). With that in mind, I updated the triple-provider environment to add support for Libvirt and make it a quadruple-provider environment.

Read more...

Technology Short Take 101

Welcome to Technology Short Take #101! I have (hopefully) crafted an interesting and varied collection of links for you today, spanning all the major areas of modern data center technology. Now you have some reading material for this weekend!

Read more...

Exploring Kubernetes with Kubeadm, Part 1: Introduction

I recently started using kubeadm more extensively than I had in the past to serve as the primary tool by which I stand up Kubernetes clusters. As part of this process, I also discovered the kubeadm alpha phase subcommand, which exposes different sections (phases) of the process that kubeadm init follows when bootstrapping a cluster. In this blog post, I’d like to kick off a series of posts that explore how one could use the kubeadm alpha phase command to better understand the different components within Kubernetes, the relationships between components, and some of the configuration items involved.

Read more...

Book Review: Infrastructure as Code

As part of my 2018 projects, I committed to reading and reviewing more technical books this year. As part of that effort, I recently finished reading Infrastructure as Code, authored by Kief Morris and published in September 2015 by O’Reilly (more details here). Infrastructure as code is very relevant to my current job function and is an area of great personal interest, and I’d been half-heartedly working my way through the book for some time. Now that I’ve completed it, here are my thoughts.

Read more...

Technology Short Take 100

Wow! This marks 100 posts in the Technology Short Take series! For almost eight years (Technology Short Take #1 was published in August 2010), I’ve been collecting and sharing links and articles from around the web related to major data center technologies. Time really flies when you’re having fun! Anyway, here is Technology Short Take 100…I hope you enjoy!

Read more...

Quick Post: Parsing AWS Instance Data with JQ

I recently had a need to get a specific subset of information about some AWS instances. Naturally, I turned to the CLI and some CLI tools to help. In this post, I’ll share the command I used to parse the AWS instance data down using the ever-so-handy jq tool.

Read more...

Posts from the Past, May 2018

This month—May 2018—marks thirteen years that I’ve been generating content here on this site. It’s been a phenomenal 13 years, and I’ve enjoyed the opportunity to share information with readers around the world. To celebrate, I thought I’d do a quick “Posts from the Past” and highlight some content from previous years. Enjoy!

Read more...

DockerCon SF 18 and Spousetivities

DockerCon SF 18 is set to kick off in San Francisco at the Moscone Center from June 12 to June 15. This marks the return of DockerCon to San Francisco after being held in other venues for the last couple of years. Also returning to San Francisco is Spousetivities, which has organized activities for spouses, significant others/domestic partners, friends, and family members traveling with conference attendees!

Read more...

Manually Installing Firefox 60 on Fedora 27

Mozilla recently released version 60 of Firefox, which contains a number of pretty important enhancements (as outlined here). However, the Fedora repositories don’t (yet) contain Firefox 60 (at least not for Fedora 27), so you can’t just do a dnf update to get the latest release. With that in mind, here are some instructions for manually installing Firefox 60 on Fedora 27.

Read more...

One Week Until Spousetivities in Vancouver

Only one week remains until Spousetivities kicks off in Vancouver at the OpenStack Summit! If you are traveling to the Summit with a spouse, significant other, family member, or friend, I’d encourage you to take a look at the great activities Crystal has arranged during the Summit.

Read more...

Technology Short Take 99

Welcome to Technology Short Take 99! What follows below is a collection of various links and articles about (mostly) data center-related technologies. Hopefully something I’ve included will be useful. Here goes!

Read more...

Installing GitKraken on Fedora 27

GitKraken is a full-featured graphical Git client with support for multiple platforms. Given that I’m trying to live a multi-platform life, it made sense for me to give this a try and see whether it is worth making part of my (evolving and updated) multi-platform toolbelt. Along the way, though, I found that GitKraken doesn’t provide an RPM package for Fedora, and that the installation isn’t as straightforward as one might hope. I’m documenting the procedure here in the hope of helping others.

Read more...

Older Posts

Find more posts by browsing the post categories, content tags, or site archives pages. Thanks for visiting!