OpenStack

I had the pleasure of attending the OpenStack Summit in Portland, OR last week. It was my first time at the OpenStack Summit, and it was great to meet lots of folks in the OpenStack community as well as be exposed to some more in-depth and detailed OpenStack information. While I was there I tried to liveblog as many sessions as I was able; here are links to the various session liveblogs that I managed to publish. Enjoy!

Getting From Grizzly to Havana, a DevOps Upgrade Pattern
Nicira NVP Deep Dive
Considerations for Building a Private Cloud, Folsom Update
Building HA OpenStack with Puppet in 20 Minutes
OpenStack Capacity Planning
Networking in the Cloud, an SDN Primer
OpenStack Back to the Enterprise, Keep Calm and Boldly Go On
OpenStack High Availability in Grizzly and Beyond

If anyone has any other liveblog sessions that should be added to this list, drop me a comment and let me know.

This is a session titled “More Reliable, More Resilient, More Redundant: OpenStack High Availability in Grizzly and Beyond.” The presenter is Florian Haas from Hastexo (@hastexo). Florian is one of the founders of Hastexo, which is a services firm that provides OpenStack services, among other things.

There are four things that need to be addressed when discussing high availability:

• Infrastructure
• Storage
• Compute
• Networking

The presentation starts with a discussion of the infrastructure layer. Changes from Folsom to Grizzly are relatively few. Examples of infrastructure services like the databases (typically MySQL, sometimes PostgreSQL), AMQP (message queue; could be using RabbitMQ, ZeroMQ, etc.), and API services. With regard to infrastructure HA, there are 5 types of infrastructure nodes:

1. Cloud controller
2. API node
3. Network node
4. Compute node
5. Storage controller

The cloud controller runs services that underpin OpenStack. It runs a relational database, an AMQP server, registry services, etc. The ability for an implementation to use active/passive or active/active depends largely on the specific back-end applications in use. Largely, cloud controllers will be deployed active/passive (this is due, in part at least, to the persistent data storage found in the relational databases).

API nodes are fundamentally stateless (locally); it interacts with the AMQP message bus. For most API services, we can use active/active scale-out approaches for API services.

The network node is, according to the presenter, “interesting.” This node takes care of routing between tenant networks, provider networks, and upstream networks. The network node typically also runs the DHCP agent to provide IP addresses to tenant systems. We’ll come back to the network node shortly.

The compute node is the hypervisor itself, where guest VMs/instances actually execute.

The storage controller depends greatly on what kind of back-end block storage. The block storage server (Cinder server) might have no local storage (which might be the case if you were running with Ceph, for example) or it might have lots of local storage.

All five of these infrastructure node types can use the same high availability stack, but with a few minor differences. The “recommended” high availability stack is Pacemaker. Hastexo has reference configurations for using Pacemaker with all OpenStack infrastructure services. According to the presenter, “it’s not rocket science—you can do it.”

The presenter next shows a diagram of an architecture using Pacemaker and Corosync for high availability of the cloud controller. From there he shows an example of using Pacemaker/Corosync for high availability of the stateless API services.

From here, he moves on to discuss HA for compute in a bit more detail. Grizzly addresses guest HA. He shows a hack where you “override” a couple of parameters to “trick” Nova into restarting VMs on another host after the first host fails. Unfortunately, this hack breaks live migration and is unsafe with Cinder volumes (although the Cinder issue is fixable).

In Grizzly, Nova has a nova evacuate command, but this command actually only works after a host has failed. This evacuation functionality isn’t present in Horizon (the Dashboard).

Another interesting feature is VM Ensembles. This feature allows you to group guests in a resilient fashion. The presenter provides an example of 6 VMs (2 each of three tiers), and the desire to make sure that the redundant nodes aren’t running on the same compute node. (For VMware users, this would be analogous to anti-affinity rules.) Unfortunately, this feature did not make it into the Grizzly release (it should be in Havana). A workaround would be to use a scheduler filter.

Looking in more detail at Networking, the same HA architecture (Pacemaker/Corosync) worked reasonably well in Folsom for Quantum-server and L3 agent. It didn’t work so well for the DHCP agent. However, this active/passive approach didn’t scale well. Grizzly helps address this by running multiple DHCP agents and multiple L3 agents to provide better scale-out support (via the Quantum scheduler).

Finally, the presenter moves on to discuss storage. Grizzly has dramatically expanded the amount of support within Cinder for block storage options; the presenter highly recommends upgrading to Grizzly if you need expanded Cinder support. He briefly mentions new Cinder drivers for NFS, GlusterFS, 3Par, LeftHand, EMC, and others. Pacemaker/Corosync is needed for the Cinder volume server. There is a hack required (need to override a host value in the database) in order to provide high availability for the Cinder volume server. It’s possible that this hack will be fixed in Havana (a bug has already been filed to fix it).

A few other tidbits:

• Libvirt watchdog support in Nova and Glance is coming
• Heat will bring some additional high availability features (Heat is now an integrated project and will be fully supported with Havana, as will Ceilometer)
• The RabbitMQ library (Kombu) has gained the ability to have a list of queue hosts to which to connect (instead of just a single host)
• ZeroMQ (peer-to-peer messaging) is another interesting option
• MySQL/Galera is firming up to provide write-set replication (which will enable synchronous replication)
• Some changes are occurring within RBD when you use it for both Glance and Cinder (the presenter did not elaborate)

At this point, the presenter wrapped up the session by opening up for questions and answers.

This is a liveblog of a session titled “OpenStack Capacity Planning.” The presenter starts out with a shout-out to the OpenStack Operations Guide that was recently written.

Here’s how to make capacity planning easy and simple:

• A blank check backed by limitless funds
• Unlimited time
• A well-organized team of geniuses
• Perfectly clear expectations that never change (up front & in writing)

Don’t have all that? Well, then you have to worry about capacity planning.

To start with capacity planning, the presenter suggests that the absolute best place to start is DevStack. Using DevStack allows you to test various capacity planning scenarios easily and quickly. However, the presenter warns against trying to use DevStack in production.

Next, you need to answer the question: Public cloud or private cloud? The answer to this question will drive a lot of the follow-up questions that you must answer. If the answer to this first question is private cloud, then it’s typically easier to do capacity planning because you’ll generally have a better idea of what sort of applications and workloads will be deployed. The presenter also feels that limited hardware/networking/storage choices in private cloud deployments makes capacity planning easier (although I’m not sure I agree). Deployment is also easier and can be a bit more leisurely due to lower growth rates.

For a public cloud, capacity planning is much trickier. You’ll have to design against a generic use case/workload because you won’t know the types of applications and workloads that your customers will actually put on this public cloud. In public cloud environments, you’ll likely be standing up new compute nodes constantly, so the speed and frequency of deployment is a key issue. Tools for fast, reliable provisioning and configuration management are very important.

The presenter mentions a few tools in this space: Crowbar, Puppet, Razor, Cobbler, Chef, CFEngine, and Fuel.

Monitoring is often an afterthought, but it really shouldn’t be—it should be done up front as much as possible. Here you can look at tools like Nagios, but there is a lot of debate in this space within the open source community. One of the reasons monitoring is important is to help establish some trending information, which helps in forecasting capacity needs.

What about the hypervisor you’re going to use? The presenter feels like this is an important decision. Pick the best hypervisor for your workload. The presenter prefers KVM, but there are others (he notably doesn’t mention vSphere, but does mention Hyper-V—interesting). He recommends against mixed hypervisor/heterogeneous environments. Hypervisor decision will also drive some storage decisions down the road (among other things).

At this point, the presenter circles back to DevStack, and recommends the use of DevStack to “test the heck out of” your selected hypervisor and anticipated workload. This isn’t necessarily for performance benchmarks, but rather to validate that everything works as expected.

Networking is the next big topic. The presenter recommends being very intentional about network selections, as he says that it is extremely difficult to switch between networking architectures (like from FlatDHCP to Quantum). Also be sure to watch out for bandwidth requirements and design accordingly. Naturally, IP addressing will be another concern you’ll need to consider.

Software-defined networking (SDN) is what the presenter recommends for any sizable deployments. He’s partial to NEC’s OpenFlow solution.

Compute density is a key factor in capacity planning. You’ll need to incorporate physical CPU cores, RAM, oversubscription ratio, and instance storage (ephemeral storage is local or shared). Using shared storage, like NFS or Ceph, means recovering VMs is much easier. Naturally, this means you’ll need to balance IOPS against storage capacity (GB/TB). Based on the speaker’s comments, I’d guess that he heavily favors Ceph.

When it comes to storage, your options for object storage are Swift, Ceph, and Gluster. Bandwidth is a concern (moving data to/from the object storage platform). The presenter stresses the importance of testing to understand the impact of the workloads on the object storage platform. The same goes for persistent block storage.

Finally, the presenter touches on concerns over the scalability of the cloud controller (which hosts the API endpoints and database). At some point you’ll have to consider separating the API endpoints onto dedicated boxes and using a load balancer. The presenter also suggests considering the use of Nova cells to help with growth and partitioning the load on the cloud controllers. (Cells are something that I really need to understand a bit better.)

And that concludes the session.

Overall, I didn’t find this session as useful as I would have liked—I expected this to be about ongoing capacity planning, not implementation design considerations. However, others might have come into the session with different expectations and might have found this session helpful. The key takeaway for me is that, as I saw from a similar session yesterday, it’s just as important when designing an OpenStack implementation to consider the resource demands, workload needs, and similar requirements. Just because it’s “cloud” doesn’t mean that it doesn’t still require some knowledge of how these components work under the covers.

This is a live blog for the session titled “Considerations for Building a Private Cloud, Folsom Update,” led by Ryan Richard of Rackspace (@rackninja on Twitter). As with other sessions here at the 2013 OpenStack Summit, this session is totally full, with people standing in the back, sitting on the floor along the sides, and seated on the floor across the front.

This session is about design considerations for building a private cloud with OpenStack. The focus will be the Folsom release. This session is based on experience after running Folsom for 6 months. Ryan intends to be able to provide a Grizzly-based version of this talk at the next time, after running Grizzly for 6 months.

First he tackles the question, “what is a private cloud”?

• Are you looking for elastic or traditional virtualization? It most likely won’t be both.
• Multi-tenant (or, more likely, multi-application)
• Size (this talk will be limited to discussions of up to 100 nodes)
• Private endpoints (the management endpoints aren’t accessible from the Internet)
• Limited inbound connectivity
• Customized for specific workloads

Ryan’s first recommendation is “Build with the end in mind.” He looks at how deploying the “m1.tiny” flavor would create a mismatch between CPU and RAM utilization, in that 48 vCPUs will be utilized but only a fraction of the host’s RAM would be allocated. The “m1.medium” flavor (4GB RAM, 2 vCPUs) creates a more balanced workload, whereas the larger flavors imbalance utilization the other way.

What this tells me is that capacity planning is just as importan with a private cloud deployment as it is with a “traditional virtualization” solution. Ryan’s recommendations around capacity are:

• Don’t use a disk size of 0.
• For public cloud offerings, you can limit the number of flavors. For private cloud offerings, you can create “customized” flavors for specific workloads. Find a balance for your organization.
• Don’t forget about network utilization.

It’s important to remember that it’s easy to add compute nodes, but you can’t changed the fixed network (without Quantum) once instances are running. Quantum helps address this. (Note that Ryan indicates it’s possible to create multiple networks using the CLI or API in Folsom without Quantum, but the dashboard doesn’t respect or recognize it.)

This limit on the fixed network means that it is critically important to size the number of addresses available by calculating the number of instances that could be spun up within the cloud.

It’s easy to add nodes or networks on the host network side, but you can’t change the fixed network once you go into production (not without destroying your instances). It’s also easy to add more floating networks.

Ryan now switches gears to talk about images and storage. (There’s a session tomorrow in C123 at 1:50 PM.) Rackspace is going with virtio, qcow2 disk format, bare container, cloud-init with dynamic partitioning. I’m not 100% familiar with all of these terms, so you might expect to see some posts soon on some of these. There is a small performance hit with qcow2, so be sure to quantify that when building images (or re-using images that someone else has built).

Snapshotting is another sizing consideration that can’t really be adequately predicted (it’s hard to know if your users will be using snapshots or not). Ryan recommends qcow2 for snapshots. To help maximize the use of host caching of images, try to streamline the number of images.

A few more Glance tips:

• Watch for network utilization. Glance could consume an entire 1Gb NIC.
• Consider RAID–5 for large sequential reads/writes.
• Disk bandwidth is more important than disk IOPS.
• Reduce the number of images to improve host cache functionality.

Storage on the compute nodes, on the other hand, require a different view. Build for random I/O (RAID 10 or SSD or both).

A few architectural examples and thoughts:

• For 1–20 servers, a single controller, a single API, and a single network (1 to 2 Gbps) are probably sufficient. If you need HA, you’ll need to increase those numbers appropriately.
• For 20–100 servers, take the same architecture but add load balancing for APIs (for HA and scalability), use Swift (or CloudFiles or S3) for Glance, consider the use of availability zones, and consider dedicated networks for management, Cinder, VM-to-VM traffic, etc. You might also want to consider a dedicated system for gathering compute node metrics.

Some other general performance considerations:

• Watch random IO and try to get as much random IO off local storage onto Cinder where possible.
• Review hypervisor best practices.

A few other “lessons learned”:

• Floating IPs must be associated with the “public_interface”
• Each piece of OpenStack has its own architecture.
• Folsom is stable.
• Migration (live, block) works but scenarios exist where it doesn’t. Try not to rely on these mechanisms where possible, especially if you’re building an “elastic cloud” as opposed to “traditional virtualization.”
• OpenStack is changing often, so keep up to date with the current state of the projects.
• Don’t do heterogeneous nodes.

A few other operational updates:

• There are some new nova hypervisor calls
• New image types in Glance (including VMDK)
• The policy.json file
• Other things coming in Grizzly: cells, Quantum, and better AD/LDAP support

At this point Ryan opens the session up for Q&A.