Virtualization

You are currently browsing articles tagged Virtualization.

In part 7 of the Learning NVP series, I mentioned that I was planning to transition this series from NVP to NSX through an upgrade. I had an existing NVP installation running (all virtually) inside an OpenStack cloud, and I would just upgrade that to NSX 4.0.0. Here’s a quick update on that plan and the NVP-to-NSX transition.

As I mentioned, I have an installation of NVP 3.1.1 running successfully in a nested (virtualized) environment. (Yes, it is possible to run all of NVP completely virtualized, though we don’t support that for production environments.) Starting with NVP 3.1.x, NVP offered an “Update Coordinator” that coordinated and orchestrated the upgrade of the various components within an NVP domain. Since I was running NVP 3.1.1, I could just use the Update Coordinator to upgrade my installation and walk you (the readers) through the process along the way.

Using the Update Coordinator (which is built into NVP Manager), an NVP upgrade would typically look something like this:

  • You’d log into NVP Manager and go to the Update Coordinator screen.
  • If you hadn’t already, you’d upload the update files (appliance update files and OVS update files) to NVP Manager.
  • Once all the update files were uploaded, you’d select the version to which you’re upgrading and kick it off.
  • NVP Manager itself is upgraded first.
  • Next, the Update Coordinator pushes out the appliance update files (sometimes called NUB files because of their .nub extension) out to all the appliances (service node, gateways, and controllers).
  • Next, the non-hypervisor transport nodes are upgraded (this is the service nodes and gateways).
  • Following that, the hypervisors need to be upgraded, though this isn’t handled by the Update Coordinator. (You could, of course, leverage a tool like Puppet or Chef or similar to help automate this process.)
  • After you’ve verified that the hypervisors have been updated, then the Update Coordinator upgrades the controller nodes.
  • Following the successful upgrade of the controller nodes, there is a cleanup phase and then you’re all set.

This is really high-level and I’m glossing over some details, naturally. Because an NVP upgrade is a pretty big deal—it could have an effect on the network connectivity of all the VMs and hypervisors within the NVP domain—it typically involves lots of planning, lots of testing, proper backups of all the components, and so on. However, since this was a lab environment and not a real production environment, just running through the Update Coordinator should have been fine.

As it turns out, though, I ran into a few problems—not problems with NVP, but problems with how I had deployed it. Basically, I didn’t do my due diligence and read the documentation.

When I first deployed the virtualized NVP appliances, I selected VMs that had a 10GB root disk. While this was enough to get NVP up and running, it turns out that it is not enough space to perform an upgrade. Specifically, it’s not enough space to do an upgrade on the controllers; the transport nodes upgraded successfully. After the installation of the controllers, I was left with only a couple gigabytes of free space remaining. A fair portion of that is taken up then by the appliance update file, and this did not leave enough to actually perform the controller software upgrade.

Unfortunately, there was no easy workaround. Because the NVP controller cluster is scale out and highly available, I could have taken the controllers out (one at a time), rebuilt them with more disk space, and then re-joined the cluster—a rolling upgrade, if you will. However, because NVP 3.1.1 is a much older build of NVP, it wasn’t possible to rebuild the controllers with a matching software version (not easily, anyway).

So, long story short: instead of wasting cycles trying to fix a deployment issue that is completely my fault (and, by the way, completely documented—had I paid closer attention to the documentation I wouldn’t find myself in this position), I’m simply going to rebuild my lab environment from scratch using NSX 4.0.0. I had really hoped to be able to walk you through the upgrade process, but sadly it just doesn’t make sense to do so.

This will be the last post titled “Learning NVP”; moving forward, all future posts will be titled “Learning NSX.” The next post will discuss adding a gateway service to a logical network; this builds on information from part 5 (creating a logical network) and part 6 (adding a gateway appliance).

As always, your feedback is welcome and encouraged, so feel free to speak up in the comments below.

Tags: , , , ,

In this post I’m going to expand a little bit on using libvirt to connect Linux containers (created using LXC) to Open vSwitch (OVS). I made brief mention of this in my post on using LXC with libvirt, but did not provide any details. This post aims to provide those details.

I’m assuming that you’re already familiar with LXC, OVS, and libvirt. If you aren’t familiar with these projects, I suggest you have a look back at other articles I’ve written about them in the past. One of the easiest ways to do that is to browse articles tagged LXC, tagged OVS, and/or tagged Libvirt. Further, I’m using Ubuntu 12.04 LTS in my environment, so if you’re using another Linux distribution please note that some commands and/or package names might be different.

The basic process for connecting a Linux container to OVS using libvirt looks something like this:

  1. Create one or more virtual networks in libvirt to “front-end” OVS.
  2. Create your container(s) using standard LXC user-space tools.
  3. Create libvirt XML definitions for your container(s).
  4. Start the container(s) using virsh.

Steps 2, 3, and 4 were covered in my previous post on using LXC and libvirt, so I won’t repeat them here. Step 1 is the focus here. (If you are a long-time reader and/or well-versed with libvirt and OVS, there isn’t a great deal of new information here; I just wanted to present it in the context of LXC for the sake of completeness.)

To create a libvirt virtual network to front-end OVS, you need to create an XML definition that you can use with virsh to define the virtual network. Here’s an example XML definition:

(If you can’t see the code block above, please click here.)

A few notes about this XML definition:

  • You normally wouldn’t include the UUID, as that is generated automatically by libvirt. If you were using this XML to create the virtual network from scratch, I would recommend just deleting the UUID line.
  • The network is named “bridged”, and points to the OVS bridge named br-ex. In this particular case, br-ex is a simple OVS bridge that contains a single physical interface.
  • This particular virtual network only has a single portgroup configured for untagged traffic. If you wanted to provide a virtual network that supported multiple VLANs, you could add more portgroups with the VLAN tags as I describe in my post on using VLANs with OVS and libvirt. You’d then modify the container’s XML definition to point to the appropriate portgroup, and in this way you could easily support running multiple containers across multiple VLANs on a single host.
  • A libvirt virtual network can only point to a single bridge, so if you wanted to support both bridged (as shown here) as well as tunneled connectivity (perhaps as described in my post on LXC, OVS, and GRE tunnels), you would need to create a second XML definition that creates a separate virtual network. You could then modify the container’s XML definition to point to the new network you just created.

In the bullets above, I mentioned modifying the container’s XML definition. In particular, I’m referring to the <interface type='network‘> portion of the container’s XML definition. To use a libvirt network for a container’s network connectivity, you’d specify <source network='bridged'/> (replacing “bridged” with whatever the name of your virtual network is; I’m using the name provided in the sample XML code above). For multiple interfaces in the container, simply supply multiple <interface type='network'> entries in the container’s XML definition, and configure the source network for each of them appropriately.

Hopefully this post provides some additional details and information on using libvirt to connect Linux containers to OVS. If you have any questions, or if you have more information to share on this topic, please feel free to speak up in the comments below. I encourage and welcome all courteous feedback!

Tags: , , , , ,

In this post, I’ll discuss how you could use Open vSwitch (OVS) and GRE tunnels to connect bare metal workloads. While OVS is typically used in conjunction with a hypervisor such as KVM or Xen, you’re certainly not restricted to only using it on hypervisors. Similarly, while GRE tunnels are commonly used to connect VMs or containers, you’re definitely not restricted from using them with bare metal workloads as well. In this post, I’ll explore how you would go about connecting bare metal workloads over GRE tunnels managed by OVS.

This post, by the way, was sparked in part by a comment on my article on using GRE tunnels with OVS, in which the reader asked: “Is there a way to configure bare Linux (Ubuntu)…with OVS installed…to serve as a tunnel endpoint…?” Hopefully this post helps answer that question. (By the way, the key to understanding how this works is in understanding OVS traffic patterns. If you haven’t yet read my post on examining OVS traffic patterns, I highly recommend you go have a look right now. Seriously.)

Once you have OVS installed (maybe this is helpful?), then you need to create the right OVS configuration. That configuration can be described, at a high level, like this:

  • Assign an IP address to a physical interface. This interface will be considered the “tunnel endpoint,” and therefore should have an IP address that is correct for use on the transport network.
  • Create an OVS bridge that has no physical interfaces assigned.
  • Create an OVS internal interface on this OVS bridge, and assign it an IP address for use inside the GRE tunnel(s). This interface will be considered the primary interface for the OS instance.
  • Create the GRE tunnel for connecting to other tunnel endpoints.

Each of these areas is described in a bit more detail in the following sections.

Setting Up the Transport Interface

When setting up the physical interface—which I’ll refer to as the transport interface moving forward, since it is responsible for transporting the GRE tunnel across to the other endpoints—you’ll just need to use an IP address and routing entries that enable it to communicate with other tunnel endpoints.

Let’s assume that we are going to have tunnel endpoints on the 192.168.1.0/24 subnet. On the bare metal OS instance, you’d configure a physical interface (I’ll assume eth0, but it could be any physical interface) to have an IP address on the 192.168.1.0/24 subnet. You could do this automatically via DHCP or manually; the choice is yours. Other than ensuring that the bare metal OS instance can communicate with other tunnel endpoints, no additional configuration is required. (I’m using “required” as in “necessary to make it work.” You may want to increase the MTU on your physical interface and network equipment in order to accommodate the GRE headers in order to optimize performance, but that isn’t required in order to make it work.)

Once you have the transport interface configured and operational, you can move on to configuring OVS.

Configuring OVS

If you’ve been following along at home with all of my OVS-related posts (you can browse all posts using the OVS tag), you can probably guess what this will look like (hint: it will look a little bit like the configuration I described in my post on running host management through OVS). Nevertheless, I’ll walk through the configuration for the benefit of those who are new to OVS.

First, you’ll need to create an OVS bridge that has no physical interfaces—the so-called “isolated bridge” because it is isolated from the physical network. You can call this bridge whatever you want. I’ll use the name br-int (the “integration bridge”) because it’s commonly used in other environments like OpenStack and NVP/NSX.

To create the isolated bridge, use ovs-vsctl:

ovs-vsctl add-br br-int

Naturally, you would substitute whatever name you’d like to use in the above command. Once you’ve created the bridge, then add an OVS internal interface; this internal interface will become the bare metal workload’s primary network interface:

ovs-vsctl add-port br-int mgmt0 -- set interface mgmt0 type=internal

You can use a name other than mgmt0 if you so desire. Next, configure this new OVS internal interface at the operating system level, assigning it an IP address. This IP address should be taken from a subnet “inside” the GRE tunnel, because it is only via the GRE tunnel that you’ll want the workload to communicate.

The following commands will take care of this part for you:

ip addr add 10.10.10.30/24 dev mgmt0
ip link set mgmt0 up

The process of ensuring that the mgmt0 interface comes up automatically when the system boots is left as an exercise for the reader (hint: use /etc/network/interfaces).

At this point, the bare metal OS instance will have two network interfaces:

  • A physical interface (we’re assuming eth0) that is configured for use on the transport network. In other words, it has an IP address and routes necessary for communication with other tunnel endpoints.
  • An OVS internal interface (I’m using mgmt0) that is configured for use inside the GRE tunnel. In other words, it has an IP address and routes necessary to communicate with other workloads (bare metal, containers, VMs) via the OVS-hosted GRE tunnel(s).

Because the bare metal OS instance sees two interfaces (and therefore has visibility into the routes both “inside” and “outside” the tunnel), you may need to apply some policy routing configuration. See my introductory post on Linux policy routing if you need more information.

The final step is establishing the GRE tunnel.

Establishing the GRE Tunnel

The commands for establishing the GRE tunnel have been described numerous times, but once again I’ll walk through the process just for the sake of completeness. I’m assuming that you’ve already completed the steps in the previous section, and that you are using an OVS bridge named br-int.

First, add the GRE port to the bridge:

ovs-vsctl add-port br-int gre0

Next, configure the GRE interface on that port:

ovs-vsctl set interface gre0 type=gre options:remote_ip=<IP address of remote tunnel endpoint>

Let’s say that you’ve assigned 192.168.1.10 to the transport interface on this system (the bare metal OS instance), and that the remote tunnel endpoint (which could be a host with multiple containers, or a hypervisor running VMs) has an IP address of 192.168.1.15. On the bare metal system, you’d configure the GRE interface like this:

ovs-vsctl set interface gre0 type=gre options:remote_ip=192.168.1.15

On the remote tunnel endpoint, you’d configure the GRE interface like this:

ovs-vsctl set interface gre0 type=gre options:remote_ip=192.168.1.10

In other words, each GRE interface points to the transport IP address on the opposite end of the tunnel.

Once the configuration on both ends is done, then you should be able to go into the bare metal OS instance and ping an IP address inside the GRE tunnel. For example, I used this configuration to connect a bare metal Ubuntu 12.04 instance, a container running on an Ubuntu host, and a KVM VM running on an Ubuntu host (I had a full mesh topology with STP enabled, as described here). I was able to successfully ping between the bare metal OS instance, the container, and the VM, all inside the GRE tunnel.

Summary, Caveats, and Other Thoughts

While this configuration is interesting as a “proof of concept” that OVS and GRE tunnels can be used to connect bare metal OS instances and workloads, there are a number of considerations and/or caveats that you’ll want to think about before trying something like this in a production environment:

  • The bare metal OS instance has visibility both “inside” and “outside” the tunnel, so there isn’t an easy way to prevent the bare metal OS instance from communicating outside the tunnel to other entities. This might be OK—or it might not. It all depends on your requirements, and what you are trying to achieve. (In theory, you might be able to provide some isolation using network namespaces, but I haven’t tested this at all.)
  • If you want to create a full mesh topology of GRE tunnels, you’ll need to enable STP on OVS.
  • There’s nothing preventing you from attaching an OpenFlow controller to the OVS instances (including the OVS instance on the bare metal OS) and pushing flow rules down. This would eliminate the need for STP, since OVS won’t be in MAC learning mode. This means you could easily incorporate bare metal OS instances into a network virtualization-type environment. However…
  • There’s no easy way to provide a separation of OVS and the bare metal OS instance. This means that users who are legitimately allowed to make administrative changes to the bare metal OS instance could also make changes to OVS, which could easily “break” the configuration and cause problems. My personal view is that this is why you rarely see this sort of OVS configuration used in conjunction with bare metal workloads.

I still see value in explaining how this works because it provides yet another example of how to configure OVS and how to use OVS to help provide advanced networking capabilities in a variety of environments and situations.

If you have any questions, I encourage you to add them in the comments below. Likewise, if I have overlooked something, made any mistakes, or if I’m just plain wrong, please speak up below (courteously, of course!). I welcome all useful/pertinent feedback and interaction.

Tags: , , , , , ,

One of the cool things about libvirt is the ability to work with multiple hypervisors and virtualization technologies, including Linux containers using LXC. In this post, I’m going to show you how to use libvirt with LXC, including leveraging libvirt to help automate attaching containers to Open vSwitch (OVS).

If you aren’t familiar with Linux containers and LXC, I invite you to have a look at my introductory post on Linux containers and LXC. It should give you enough background to make this post make sense.

To use libvirt with an LXC container, there are a couple of basic steps:

  1. Create the container using standard LXC user-space tools.
  2. Create a libvirt XML definition for the container.
  3. Define the libvirt container domain.
  4. Start the libvirt container domain.

The first part, creating the container, is pretty straightforward:

lxc-create -t ubuntu -n cn-02

This creates a container using the Ubuntu template and calls it cn–01. As you may recall from my introductory LXC post, this creates the container’s configuration and root filesystem in /var/lib/lxc by default. (I’m assuming you are using Ubuntu 12.04 LTS, as I am.)

Once you have the container created, you next need to get it into libvirt. Libvirt uses a standard XML-based format for defining VMs, containers, networks, etc. At first, I thought this might be the most difficult section, but thanks to this page I was able to create a template XML configuration.

Here’s the template I was able to create:

(If you can’t see the embedded code above, please click here.)

Simply take this XML template and save it as something like lxc-template.xml or similar. Then, after you’ve created your container using lxc-create as above, you can easily take this template and turn it into a specific container configuration with only one command. For example, suppose you created a container named “cn–02″ (as I did with the command I showed earlier). If you wanted to customize the XML template, just use this simple Unix/Linux command:

sed 's/REPLACE/cn-02/g' lxc-template.xml > cn-02.xml

Once you have a container-specific libvirt XML configuration, then defining it in libvirt is super-easy:

virsh -c lxc:// define cn-02.xml

Then start the container:

virsh -c lxc:// start cn-02

And connect to the container’s console:

virsh -c lxc:// console cn-02

When you’re done with the container’s console, press Ctrl-] (that’s Control and right bracket at the same time); that will return you to your host.

Pretty handy, eh? Further, since you’re now controlling your containers via libvirt, you can leverage libvirt’s networking functionality as well—which means that you can easily create libvirt virtual networks backed by OVS and automatically attach containers to OVS for advanced networking configurations. You only need to create an OVS-backed virtual network like I describe in this post on VLANs with OVS and libvirt.

I still need to do some additional investigation and testing to see how the networking configuration in the container’s config file interacts with the networking configuration in the libvirt XML file. For example, how do you define multiple network interfaces? Can you control the name of the veth pairs that show up in the host? I don’t have any answers for these questions (yet). If you know the answers, feel free to speak up in the comments!

All courteous feedback and interaction is welcome, so I invite you to start (or join) the discussion via the comments below.

Tags: , , , , , ,

This post will show you how to use Open vSwitch (OVS) to connect LXC containers on different hosts together using GRE tunnels. This process is quite similar to using OVS and GRE tunnels to connect VMs, and even has some similarity to connecting network namespaces over a GRE tunnel.

If you’re not familiar with LXC, my introductory post on LXC might be useful. I also found this post on networking with LXC to be extremely helpful, as was this overview of LXC on Ubuntu 12.04. As you might guess, I used Ubuntu 12.04.3 LTS for my testing; note that if you are using a different distribution, the specific commands and/or file locations may differ.

At a high level, the process for creating this configuration looks like this:

  • Create your containers using virtual Ethernet (veth) networking.
  • Create a OVS bridge that does not contain any physical interfaces and attach a GRE interface configured to connect to a remote container host.
  • Attach the containers’ veth interfaces to this new OVS bridge.

Let’s take a look at each of these steps in a bit more detail.

Create the Containers

My LXC introduction post covers this in a fair amount of detail, but I’ll walk you through the steps again just for the sake of completeness. Additionally, I wanted to use containers with two separate interfaces: a “public” interface that could reach the outside world, and a “private” interface for inter-container communications.

To create the containers, you can simply use lxc-create to create the basic container. For example, if you are also using Ubuntu 12.04 64-bit for your testing, then creating a container with the same release and architecture would require only this command (note you might need to prepend sudo to this command, depending on your specific configuration):

lxc-create -t ubuntu -n <container name>

Once the base container is created, you can then edit the container configuration (which is found at /var/lib/lxc/container name/config by default) to add the second interface. You’d add this text to the configuration:

lxc.network.type = veth
lxc.network.flags = up
lxc.network.veth.pair = c1eth1
lxc.network.ipv4 = 10.10.10.10/24

In my specific testing, I left the container’s first network interface attached to the default lxcbr0 Linux bridge, which provides connectivity to external networks via NAT. If you wanted bridged connectivity, you’d need to set that up separately. Also, feel free to substitute a different name for c1eth1 above; this just provides a user-recognizable name for the host-facing side of the veth pair that provides connectivity for the container.

After you’ve created and configured the container(s) appropriately, start the container with lxc-start -n <container name> to ensure that it boots successfully and without any unexpected errors. Then shut it down and start it again with the console detached, like so:

lxc-start -d -n <container name>

Now that the container is up and running, you’re ready to configure OVS.

Create and Configure the OVS Bridge

If you’re not familiar with connecting things with GRE tunnels using OVS, I suggest you take a look here and here. You will probably also find this post on examining OVS traffic patterns to be helpful in understanding why you will configure OVS in this particular way.

Create the OVS bridge you’ll use for inter-container connectivity using ovs-vsctl, like this (feel free to substitute whatever name you want for “br-int” in the command below):

ovs-vsctl add-br br-int

Add a GRE interface to this bridge:

ovs-vsctl add-port br-int gre0

Then configure the GRE interface appopriately:

set interface gre0 type=gre options:remote_ip=192.168.1.100

The IP address specified above should point to a reachable interface on the remote host where the other container(s) is/are running. Also, you are welcome to use a different name for the port and interface than what I used (I used gre0), but be sure the port and interface match (as far as I know you can’t use one name for the port and a different name for the interface).

If you are connecting only two hosts with containers, then you’ll repeat this configuration on the second host, except that its GRE tunnel will point back to the first host (the first host specifies the IP address of the second host, the second host specifies the IP address of the first host).

If you are connecting more than two hosts, you’ll either want to a) manually build a non-looping topology of GRE tunnels, or b) build a full mesh of GRE tunnels and enable STP, as outlined here.

Once you’ve created the OVS bridge and the GRE interface, you’re ready for the final step: connecting the containers to OVS.

Attach the Containers to OVS

The final step is to connect the container’s veth interface to OVS. If you’re using an early enough version of OVS that still contains the Linux bridge compatibility code, then you can just specify the name of the OVS bridge with the lxc.network.link directive in the container’s configuration file. Newer versions of OVS, however, lack the Linux bridge compatibility layer, so this doesn’t work. In this case, you’ll need to manually attach the veth interfaces to OVS.

To manually attach the container’s veth interface to OVS, you must first identify the veth interface. This is why I used the lxc.network.veth.pair configuration directive in the container to assign a user-recognizable name. If you didn’t use that directive, you can use the ip link list command to list the interfaces.

Once you’ve identified the correct veth interface, simply add it to OVS with this command:

ovs-vsctl add-port br-int c1eth1

I used the c1eth1 name in the command above; you’d substitute the correct name for the appropriate veth interface for your container(s).

Once that is done, you should be able to ping over the GRE tunnel using the IP addresses assigned to the inter-container communication interfaces (which should be recognized as eth1 in the containers). Congratulations—you’ve just connected containers on two different hosts together over a GRE tunnel!

There are a couple of things to note about this configuration:

  • Note that we only used a single interface on the host. Obviously, we could have used more (for example, we could have use eth0 for host management and eth1 for host-to-host GRE tunnel connections), but using only a single host interface allows us to use this method in public cloud environments that only provide a single interface.
  • While the GRE tunnel provides encapsulation, it does not provide encryption.
  • The process of attaching the containers’ veth interface to OVS is manual right now, so that won’t scale in an environment of any size. I’m exploring ways to help automate that process. One possible avenue is the use of libvirt, so stay tuned for posts describing any progress I make in that area.

If you have any questions, corrections, suggestions, or clarifications, please feel free to post them in the comments below. All feedback is welcome and appreciated.

Tags: , , , , ,

In this post, I’m going to provide a brief introduction to working with Linux containers via LXC. Linux containers are getting a fair amount of attention these days (perhaps due to Docker, which leverages LXC on the back-end) as a lightweight alternative to full machine virtualization such as that provided by “traditional” hypervisors like KVM, Xen, or ESXi.

Both full machine virtualization and containers have their advantages and disadvantages. Full machine virtualization offers greater isolation at the cost of greater overhead, as each virtual machine runs its own full kernel and operating system instance. Containers, on the other hand, generally offer less isolation but lower overhead through sharing certain portions of the host kernel and operating system instance. In my opinion full machine virtualization and containers are complementary; each offers certain advantages that might be useful in specific situations.

Now that you have a rough idea of what containers are, let’s take a closer look at using containers with LXC. I’m using Ubuntu 12.04.3 LTS for my testing; if you’re using something different, keep in mind that certain commands may differ from what I show you here.

Installing LXC is pretty straightforward, at least on Ubuntu. To install LXC, simply use apt-get:

apt-get install lxc

Once you have LXC installed, your next step is creating a container. To create a container, you’ll use the lxc-create command and supply the name of the container template as well as the name you want to assign to the new container:

lxc-create -t <template> -n <container name>

You’ll need Internet access to run this command, as it will download (via your configured repositories) the necessary files to build a container according to the template you specified on the command line. For example, to use the “ubuntu” template and create a new container called “cn–01″, the command would look like this:

lxc-create -t ubuntu -n cn-01

Note that the “ubuntu” template specified in this command has some additional options supported; for example, you can opt to create a container with a different release of Ubuntu (it defaults to the latest LTS) or a different architecture (it defaults to the host’s architecture).

Once you have at least one container created, you can list the containers that exist on your host system:

lxc-list

This will show you all the containers that have been created, grouped according to whether the container is stopped, frozen (paused), or running.

To start a container, use the lxc-start command:

lxc-start -n <container name>

Using the lxc-start command as shown above is fine for initial testing of your container, to ensure that it boots up as you expect. However, you won’t want to run your containers long-term like this, as the container “takes over” your console with this command. Instead, you want the container to run in the background, detached from the console. To do that, you’ll add the “-d” parameter to the command:

lxc-start -d -n <container name>

This launches your container in the background. To attach to the console of the container, you can use the lxc-console command:

lxc-console -n <container name>

To escape out of the container’s console back to the host’s console, use the “Ctrl-a q” key sequence (press Ctrl-a, release, then press q).

You can freeze (pause) a container using the lxc-freeze command:

lxc-freeze -n <container name>

Once frozen, you can unfreeze (resume) a container just as easily with the lxc-unfreeze command:

lxc-unfreeze -n <container name>

You can also make a clone (a copy) of a container:

lxc-clone -o <existing container> -n <new container name>

On Ubuntu, LXC is configured by default to start containers in /var/lib/lxc. Each container will have a directory there. In a container’s directory, that container’s configuration will be stored in a file named config. I’m not going to provide a comprehensive breakdown of the settings available in the container’s configuration (this is a brief introduction), but I will call out a few that are worth noting in my opinion:

  • The lxc.network.type option controls what kind of networking the container will use. The default is “veth”; this uses virtual Ethernet pairs. (If you aren’t familiar with veth pairs, see my post on Linux network namespaces.)
  • The lxc.network.veth.pair configuration option controls the name of the veth interface created in the host. By default, a container sees one side of the veth pair as eth0 (as would be expected), and the host sees the other side as either a random name (default) or whatever you specify here. Personally, I find it useful to rename the host interface so that it’s easier to tell which veth interface goes to which container, but YMMV.
  • lxc.network.link specifies a bridge to which the host side of the veth pair should be attached. If you leave this blank, the host veth interface is unattached.
  • The configuration option lxc.rootfs specifies where the container’s root file system is stored. By default it is /var/lib/lxc/<container name>/rootfs.

There are a great deal of other configuration options, naturally; check out man 5 lxc.conf for more information. You may also find this Ubuntu page on LXC to be helpful; I certainly did.

I’ll have more posts on Linux containers in the future, but this should suffice to at least help you get started. If you have any questions, any suggestions for additional resources other readers should consider, or any feedback on the post, please add your comment below. I’d love to hear from you (courteous comments are always welcome).

Tags: , , , ,

In this post, I’m going to provide an update on using GRE tunnels with Open vSwitch (OVS) to include more than 2 hosts. I previously showed you how to use GRE tunnels with OVS to connect VMs on different hypervisor hosts, but in my testing I didn’t use this technique with more than two hypervisors. A few readers posted comments to that article asking how to extend the solution to more than 2 hypervisors, but I hadn’t had the time to test anything more.

Now, as a result of some related work I’ve been doing, I have an update on using this technique for more than two hosts. If you didn’t read the post on using GRE tunnels with OVS, go back and read that now. Also, be sure to read my post on examining OVS traffic patterns, as this is also useful information. Finally, note that this information applies to any use of GRE tunnels with OVS, not just GRE tunnels with OVS on hypervisors.

Let’s say you have three hosts:

  1. HostA, with an IP address of 10.1.1.1
  2. HostB, with an IP address of 10.1.1.2
  3. HostC, with an IP address of 10.1.1.3

To connect entities (VMs, containers, etc.) on these hosts using GRE tunnels, you’d need to manually configure OVS on each of hosts:

  • On HostA, you’d need a GRE tunnel to HostB (10.1.1.2) and a GRE tunnel to HostC (10.1.1.3)
  • On HostB, you’d need a GRE tunnel to HostA (10.1.1.1) and one to HostC (10.1.1.3)
  • On HostC, you’d need two GRE tunnels, one to HostA (10.1.1.1) and one to HostB (10.1.1.2)

I won’t repeat the specific commands to create those tunnels here, as it is well explained in my earlier article. What this creates is a virtual topology like this:

GRE tunnel full mesh

What you’ll find when you try this yourself is that everything works fine when there are just two hosts; this is what I also found when I first wrote the article. When you add the third host, though, you’ll find—assuming you created a full mesh of GRE tunnels—is that everything stops working.

Here’s how to fix that. Run this command on each of the hosts running OVS:

ovs-vsctl set bridge <bridge name> stp_enable=true

Yes, that’s right: looping is the culprit here. Look back at the topology figure above. In the physical world, a topology like that using switches (without STP) would take down your network because of a bridging loop. The same applies here as well. In both cases (physical or virtual) you have two choices: you can either not create a full mesh topology (you could use a star topology or something if you wanted) or you run STP. It’s up to you.

Assuming you turn on STP, then you’ll find after a few minutes that you’ll be able to happily ping between VMs on these hypervisors.

I do want to share one final note before I wrap up. STP is needed in this instance because we are relying on OVS in MAC learning mode (just like a physical switch). If we were to add an OpenFlow controller to this mix and push flow rules down to OVS, OVS would stop using MAC learning, and we would no longer need STP in order to build full-mesh topologies of tunnels.

Feel free to post any questions or comments below. All courteous comments are welcome!

Tags: , , , ,

Welcome to part 7 of the Learning NVP blog series, in which I will discuss transitioning from a focus on NVP to looking at NSX.

If you’re just now joining me for this series, here’s what’s transpired thus far:

When I first started this series back in May of this year, I said this:

Before continuing, it might be useful to set some context around NVP and NSX… The architecture I’m describing here will also be applicable to NSX, which VMware announced in early March. Because NSX will leverage NVP’s architecture, spending some time with NVP now will pay off with NSX later.

Well, the “later” that I referenced is now upon us. I had hoped to be much farther along with this blog series by now, but it has proven more difficult than I had anticipated to get this content written and published. Given that NSX officially GA’d last week at VMworld EMEA in Barcelona, I figured it was time to make the transition from NVP to NSX.

The way I’ll handle the transition from talking NVP to discussing VMware NSX is through an upgrade. I have a completely virtualized environment that is currently running all the NVP components: three controllers, NVP Manager, three nested hypervisors running Ubuntu+KVM+OVS, two gateways, and a service node. (I know, I know—I haven’t written about service nodes yet. Sorry.) The idea is to take you through the upgrade process, upgrading my environment from NVP 3.1.1 to NVP 3.2.1 and then to NSX 4.0.0. From that point forward, the series will change from “Learning NVP” to “Learning NSX”, and I’ll continue with discussing all the topics that I have planned. These include (among others):

  • Deploying service nodes
  • Using an L2 gateway service
  • Using an L3 gateway service
  • Enabling distributed east-west routing
  • Many, many more topics…

Unfortunately, my travel schedule over the next few weeks is pretty hectic, which will probably limit my ability to move quickly on performing and documenting the upgrade process. Nevertheless, I will press forward as quickly as possible, so stay tuned to the site for more updates as soon as I’m able to get them published.

Questions? Comments? Feel free to add them below. All I ask for is common courtesy and disclosure of vendor affiliations, where applicable. Thanks!

Tags: , , , ,

Welcome to part 6 of the Learning NVP blog series. In this part, I’m going to show you how to add an NVP gateway appliance to your NVP environment. In future posts, you’ll use this NVP gateway to host either L2 or L3 gateway services (more on those in a moment). First, though, let’s take a quick recap of what’s transpired so far:

In this part, I’m going to walk you through setting up an NVP gateway appliance. If you’ll recall from our introductory high-level architecture overview, the role of the gateway is to provide L2 (switched/bridged) and L3 (routed) connectivity between logical networks and physical networks. So, adding a gateway would then enable you to extend the logical network you created in part 4 to include either L2 or L3 connectivity to the outside world.

<aside>Many of you have probably seen some of the announcements from VMworld about NSX integrations from various networking suppliers (Arista, Brocade, Dell, and Juniper, for example). These announcements will allow NSX—which I’ve said before will leverage a great deal of NVP’s architecture—to use these hardware devices as L2 gateways, providing bridged/switched connectivity between logical networks and physical networks.</aside>

This post will focus only on getting the gateway appliance set up; in future posts, I’ll show you how to actually add the L2 or L3 connectivity to your logical network.

Building the NVP Gateway

The NVP gateway software is distributed as an ISO, like the NVP controller software. You’d typically install this software on a bare metal server, though with recent releases of NVP it is supported to install the gateway into a VM (refer to the latest NVP release notes for more details). As with the NVP controllers and NVP Manager, the gateway is built on Ubuntu 12.04, and the installer process is completely automated. Once you boot from the ISO, the installation will proceed automatically; when completed, you’ll be left at the login prompt.

Configuring the NVP Gateway

Once the NVP gateway software is installed, configuring the gateway is really straightforward. In fact, it feels a lot like configuring NVP controllers (I suspect this is by design). Here are the steps:

  1. Set the password for the admin user (optional, but highly recommended).

  2. Set the hostname for the gateway appliance (also optional, but strongly recommended).

  3. Configure the network interfaces; you’ll need management, transport, and external connectivity. (I’ll explain those in more detail later.)

  4. Configure DNS and NTP settings.

Let’s take a closer look at these steps. The first step is to set the password for the admin user, which you can accomplish with this command:

set user admin password

From here, you can proceed with setting the hostname for the gateway:

set hostname <hostname>

(So far, these commands should be pretty familiar. They are the same commands used when you set up the NVP controllers and NVP Manager.)

The next step is configure network connectivity; you’ll start by listing the available network interfaces with this command:

show network interfaces

As you’ve seen with the other NVP appliances, the NVP gateway software builds an Open vSwitch (OVS) bridge for each physical interface. In the case of a gateway, you’ll need at least three interfaces—a management interface, a transport network interface, and an external network interface. The diagram below provides a bit more context around how these interfaces are used:

NVP gateway appliance interfaces

Since these interfaces have very different responsibilities, it’s important that you properly configure them. Otherwise, things won’t work as expected. Take the time to identify which interface listed in the show network interfaces output corrsponds to each function. You’ll first want to establish management connectivity, so that should be the first interface to configure. Assuming that breth1 (the bridge matching the physical eth2 interface) is your management interface, you’ll configure it using this command:

set network interface breth1 ip config static 192.168.1.12 255.255.255.0

You’ll want to repeat this command for the other interfaces in the gateway, assigning appropriate IP addresses to each of them.

You may also need to configure the routing for the gateway. Check the routing table(s) with this command:

show network routes

If there is no default route, you can set one using this command:

add network route 0.0.0.0 0.0.0.0 <Default gateway IP address>

Once the appropriate network connectivity has been established, then you can proceed with the next step: adding DNS and NTP servers. Here are the commands for this step:

add network dns-server <DNS server IP address>
add network ntp-server <NTP server IP address>

If you accidentally fat-finger an IP address or hostname along the way, use the remove network dns-server or remove network ntp-server command to remove the incorrect entry, then re-add it correctly with the commands above.

Congrats! The NVP gateway appliance is now up and running. You’re ready to add it to NVP. Once it’s added to NVP, you’ll be able to use the gateway appliance to add gateway services to your logical networks.

Adding the Gateway to NVP

To add the new gateway appliance to NVP, you’ll use NVP Manager (I showed you how to set up NVP Manager in part 3 of the series). Once you’ve opened a web browser, navigated to the NVP Manager web UI, and logged in, then you can start the process of adding the gateway to NVP.

  1. Once you’re logged into NVP Manager, click on the Dashboard link across the top. (If you’re already at the Dashboard, you can skip this step.)

  2. In the Summary of Transport Components box, click the Connect & Add Transport Node button. This will open the Connect to Transport Node dialog box.

  3. Supply the management IP address of the gateway appliance, along with the appropriate username and password, then click Connect.

  4. After a moment, the Connect to Transport Node dialog box will show details of the gateway appliance, such as the interfaces, the bridges, the NIC bonds (if any), and the gateway’s SSL certificate. Click Configure at the bottom of the dialog box to continue.

  5. Supply a display name (something like nvp-gw–01) and, optionally, one or more tags. Click Next.

  6. Unless you know you need to select any of the options on the next screen (I’ll try to cover them in a later blog post), just click Next.

  7. On the final screen, you’ll need to establish connectivity to a transport zone. You’ll want to select the appropriate interface (in my example environment, it was breth2) and the appropriate encapsulation protocol (STT is generally recommended for connectivity back to hypervisors). Then select the appropriate transport zone from the drop-down list. In the end, you’ll have a screen that looks something like this (note that your interfaces, IP addresses, and transport zone information will likely be different):

  8. Adding a gateway to NVP

  9. Click Save to finish the process. The number of gateways listed in the Summary of Transport Components box should increment by 1 in the Registered column. However, the Active column will remain unchanged—that’s because there’s one more step needed.

  10. Back on the gateway appliance itself, run this command (you can use the IP address of any controller in the NVP controller cluster):

  11. set switch manager-cluster <NVP controller IP address>
  12. Back in NVP Manager, refresh the Summary of Transport Components box (there’s a small refresh icon in the corner), and you’ll see the Active column update to show the gateway appliance is now registered and active in NVP.

That’s it—you’re all done adding a gateway appliance to NVP. In future posts, you’ll leverage the gateway appliance to add L2 (bridged) and L3 (routed) connectivity in and out of logical networks. First, though, I’ll need to address the transition from NVP to NSX, so look for that coming soon. In the meantime, feel free to post any questions, thoughts, or suggestions in the comments below. I welcome all courteous comments (even if you disagree with something I’ve said!).

Tags: , , , ,

Welcome to Technology Short Take #36. In this episode, I’ll share a variety of links from around the web, along with some random thoughts and ideas along the way. I try to keep things related to the key technology areas you’ll see in today’s data centers, though I do stray from time to time. In any case, enough with the introduction—bring on the content! I hope you find something useful.

Networking

  • This post is a bit older, but still useful in the event if you’re interested in learning more about OpenFlow and OpenFlow controllers. Nick Buraglio has put together a basic reference OpenFlow controller VM—this is a KVM guest with CentOS 6.3 with the Floodlight open source controller.
  • Paul Fries takes on defining SDN, breaking it down into two “flavors”: host dominant and network dominant. This is a reasonable way of grouping the various approaches to SDN (using SDN in the very loose industry sense, not the original control plane-data plane separation sense). I’d like to add to Paul’s analysis that it’s important to understand that, in reality, host dominant and network dominant systems can coexist. It’s not at all unreasonable to think that you might have a fabric controller that is responsible for managing/optimizing traffic flows across the physical transport network/fabric, and an overlay controller—like VMware NSX—that integrates tightly with the hypervisor(s) and workloads running on those hypervisors to create and manage logical connectivity and logical network services.
  • This is an older post from April 2013, but still useful, I think. In his article titled “OpenFlow Test Deployment Options“, Brent Salisbury—a rock star new breed network engineer emerging in the new world of SDN—discusses some practical deployment strategies for deploying OpenFlow into an existing network topology. One key statement that I really liked from this article was this one: “SDN does not represent the end of networking as we know it. More than ever, talented operators, engineers and architects will be required to shape the future of networking.” New technologies don’t make talented folks who embrace change obsolete; if anything, these new technologies make them more valuable.
  • Great post by Ivan (is there a post by Ivan that isn’t great?) on flow table explosion with OpenFlow. He does a great job of explaining how OpenFlow works and why OpenFlow 1.3 is needed in order to see broader adoption of OpenFlow.

Servers/Hardware

  • Intel announced the E5 2600 v2 series of CPUs back at Intel Developer Forum (IDF) 2013 (you can follow my IDF 2013 coverage by looking at posts with the IDF2013 tag). Kevin Houston followed up on that announcement with a useful post on vSphere compatibility with the E5 2600 v2. You can also get more details on the E5 2600 v2 itself in this related post by Kevin as well. (Although I’m just now catching Kevin’s posts, they were published almost immediately after the Intel announcements—thanks for the promptness, Kevin!)
  • blah

Security

Nothing this time around, but I’ll keep my eyes posted for content to share with you in future posts.

Cloud Computing/Cloud Management

Operating Systems/Applications

  • I found this refresher on some of the most useful apt-get/apt-cache commands to be helpful. I don’t use some of them on a regular basis, and so it’s hard to remember the specific command and/or syntax when you do need one of these commands.
  • I wouldn’t have initially considered comparing Docker and Chef, but considering that I’m not an expert in either technology it could just be my limited understanding. However, this post on why Docker and why not Chef does a good job of looking at ways that Docker could potentially replace certain uses for Chef. Personally, I tend to lean toward the author’s final conclusions that it is entirely possible that we’ll see Docker and Chef being used together. However, as I stated, I’m not an expert in either technology, so my view may be incorrect. (I reserve the right to revise my view in the future.)

Storage

  • Using Dell EqualLogic with VMFS? Better read this heads-up from Cormac Hogan and take the recommended action right away.
  • Erwin van Londen proposes some ideas for enhancing FC error detection and notification with the idea of making hosts more aware of path errors and able to “route” around them. It’s interesting stuff; as Erwin points out, though, even if the T11 accepted the proposal it would be a while before this capability showed up in actual products.

Virtualization

That’s it for this time around, but feel free to continue to conversation in the comments below. If you have any additional information to share regarding any of the topics I’ve mentioned, please take the time to add that information in the comments. Courteous comments are always welcome!

Tags: , , , , , , , , , , , ,

« Older entries § Newer entries »