Articles by adelp

You are currently browsing adelp’s articles.

VMware Lab Manager Design Considerations

Sunday, October 4, 2009 in Virtualization by adelp | 3 comments

By Aaron Delp
Twitter: aarondelp
FriendFeed (Delicious, Twitter, & all my blogs in one spot): aarondelp

I have done a number of VMware Lab Manager white boarding sessions and I want to share a few of my design notes and the reason for each.  Most of items come from my installation experience and the version 4 release notes.  Here they are in no particular order.

  • You need an LDAP server to import groups - Yes, you can set up user IDs in Lab Manager but you CAN NOT create groups.  Groups must be imported into the LM Server from an LDAP server.  This is critical if you intend to do any kind of restrictions around lease durations of configurations or storage pools.
  • You need fully qualified name resolution (with functioning reverse look-up) between all clients, ESX/vSphere servers, the LM Server, and the vCenter Server - The clients need the ESX/vSphere servers because if this isn’t in place, remote control of the virtual machines will not function (you get a black screen).  You also need DNS entries for the LM server because if you implement LiveLink functionality, LiveLink is hardcoded to the LM server name.  Lastly, you need the vCenter Server for behind the scenes communication of the LM environment.
  • Workflow and Disk Chains will be the key to success or failure of your project - The VMware documentation does a great job of describing how to do things. But, the documentation falls on its face when it comes to describing WHY you should do things.  The behind the scenes architecture must be planned out very specifically for Lab Manager to perform as you would expect. I will be covering Work Flow and Disk Chains in a future article.
  • Lab Manager version 4 Host Spanning requires an Enterprise Plus subscription because the VMware Distributed Switch technology is required - In the previous version of Lab Manager, VMware HA, DRS, and VMotion were not supported if you set up a fenced (isolated by a NAT router) configuration. The configurations (a configuration is groups of VMs in Lab Manager) were pinned to an ESX server at time of creation and stayed with that server until destruction.  LM version 4 gets around this by using the Distributed Switch to span hosts.  Some people will want this feature but in my experience some will not want to pay the extra dollars just to get this one feature. Also, be aware that the Cisco 1000V isn’t supported with Lab Manager.
  • You will need to monitor the number of configurations per server if you do not use Host Spanning - Lab Manager deploys new configurations to the ESX/vSphere servers using round robin.  A configuration is removed from a host when it is undeployed.  It is possible that the workload in your cluster will become out of balance because certain machines “live” longer than others.  Take this example; you have two vSphere servers and ten configurations with only one VM to keep it easy.  The configurations will be deployed split in between the servers for a total of five per server.  A user removes the configurations on the first server but leaves the configurations on the second server.  Now another ten configurations are deployed.  The new ten configurations will again be deployed five each.  You know have one server with five and one server with ten.  Over time your load will become unbalanced.
  • Lab Manager 4 can’t use vSphere’s thin provisioning for disks - Lab Manager uses the concept of Linked Clones for copies of virtual machines.  The first one in the chain is “thick”, the rest of the machines are delta disks of the first one.  This is a technology that is independent and different from both vSphere and VMware View.
  • I haven’t had a chance to test this one yet but it appears VMware is now supporting and suggesting that you run the Lab Manager 4 server in a virtual machine.  Be careful with this one!  Where are you going to install it?  Are you going to install it on the same cluster and ESX servers you are managing?  This will create a circular dependency that I just don’t like.  Same goes for vSphere.  Have you ever tried to patch ESX servers using Update Manager when vCenter Server is on that host?  I have by accident, it doesn’t work!  For my lab I have my vCenter and Lab Manager servers in a different cluster (Scott’s vSphere cluster) than my LM vSphere servers.  In this configuration the servers are virtual, but they are out of the way.  I know with Lab Manager 3 you couldn’t install LM server on a virtual machine if it detected it was hosted on the ESX server you are trying to manage.  I’m not sure if version 4 still has this checking feature included at installation.
  • Make sure you back up your Crypto Key that Lab Manager generates during the installation -  See the Manual for more information.  You will need it if you run into big issues and need to reinstall Lab Manager.
  • VMware Snapshots are not supported because Lab Manager has a Snapshot feature built-in - Lab Manager allows the user to take one (and only one!) snapshot of a configuration.
  • Increase the Service Console on ESX/vSphere servers from 272MB to 800MB - This will help with the overhead of Lab Manager on the servers.
  • VMware Fault Tolerance (FT) and Distributed Power Management (DPM) are not supported with Lab Manager.
  • Storage VMotion and VMware VCB are not supported with Lab Manager.

If you have any other suggestions or design considerations, please let me know!

Tags: , ,

Planning for FCoE in Your Data Center Today

Saturday, August 22, 2009 in Storage, Virtualization by adelp | 6 comments

By Aaron Delp
Twitter: aarondelp
FriendFeed (Delicious, Twitter, & all my blogs in one spot): aarondelp

This week I’ve had the privilege of attending a Cisco Nexus 5000/7000 class. I have learned a tremendous amount about FCoE this week and after some conversations with Scott about the topic, I wanted to tackle it one more time from a different point of view. I have included a list of some of Scott’s FCoE articles at the bottom for those interested in a more in-depth analysis.

Disclaimer: I am by no means an FCoE expert! My working knowledge of FCoE is about four days old at this point. If I am incorrect in some of my situations below, please let me know (keep it nice and professional, people!) and I will be happy to make adjustments.

If you are an existing VMWare customer today with FC as your storage transport layer, should you be thinking about FCoE? How would you get started? What investments can you make in the near future to prepare you for the next generation?

These are questions I am starting to get from my customers in planning/white board sessions around VMware vSphere and the next generation of virtualization. The upgrade to vSphere is starting to prompt planning and discussions around the storage infrastructure.

Before I tackle the design aspect, let me start out with some hardware and definitions.

Cisco Nexus 5000 series switch: The Nexus 5K is a Layer 2 switch that is capable of FCoE and can provide both Ethernet and FC ports (with an expansion module). In addition to Ethernet switching, the switch also operates as an FC fabric switch providing full fabric services, or it can be set for N_Port Virtualization (NPV) mode. The Nexus 5K can’t be put in FC switch mode and NPV mode at the same time. You must pick one or the other.

N_Port Virtualization (NPV) mode: NPV allows the Nexus 5K to act as a FC “pass thru” or proxy. NPV is great for environments where the existing fabric is not Cisco and merging the fabrics could be ugly. There is a downside to this. In NPV mode, no targets (storage arrays) can be hung off the Nexus 5K. This is true for both FC and FCoE targets.

Converged Network Adapter (CNA): A CNA is single PCI card that contains both FC and Ethernet logic, negating the need for separate cards, separate switches, etc.

Now that the definitions and terminology is out of the way, I see four possible paths if you have FC in your environment today.

1. FCoE with a Nexus 5000 in a non-Cisco MDS environment (merging)

In this scenario, the easiest way to get the Nexus on the existing non-Cisco FC fabric is to put the switch in NPV mode. You could put the switch in interop mode (and all the existing FC switches), but it is a nightmare to get them all talking and you often lose vendor specific features in interop mode. Plus, to configure interop mode, the entire fabric has to be brought down. (You do have redundant fabrics, right?)

With the Nexus in NPV mode, what will it do? Not much. You can’t hang storage off of it. You aren’t taking advantage of Cisco VSANs or any other features that Cisco can provide. You are merely a pass thru. The zoning is handled by your existing switches; your storage is off the existing switches, etc.

Why would you do this? By doing this, you could put CNAs in new servers (leaving the existing servers alone) to talk to the Nexus. This will simplify the server side infrastructure because you will have fewer cables, cards, switch ports, etc. Does the cost of the CNA and new infrastructure offset the cost of just continuing the old environment? That is for you to decide.

2. FCoE with a Nexus 5000 in a non-Cisco MDS environment (non-merging)

Who says you have to put the Nexus into the existing FC fabric? We have many customers that purchase “data centers in a box”. By that I mean a few servers, FC and/or Ethernet switches, storage, and VMware all in one solution. This “box” sits in the data center and the network is merged with the legacy network, but we stand up a Cisco SAN next to the existing non-Cisco SAN and just not let them talk to each other. In this instance, we would use CNAs in the servers, Nexus as the switch, and you pick a storage vendor. This will work just like option 3.

3. FCoE with a Nexus 5000 in a Cisco MDS environment

Now we’re talking. Install the Nexus in FC switch mode, merge it with the MDS fabric, put CNAs in all the servers and install the storage off the Nexus as either FC or FCoE. You’re off to the races!

You potentially gain the same server side savings by replacing FC and Ethernet in new servers with CNAs. You are able to use all of the Cisco sexy features of FCoE. Nice solution if the cost is justified in your environment.

4. Keep the existing environment and use NFS to new servers

What did I just say? Why would I even consider that option?

OK, this last one is a little tongue-in-cheek for customers that are already using FC. The NFS vs. traditional storage for VMWare is a bit of a religious debate. I know you aren’t going to sway me and I know I’m not going to sway you.

I admit I’m thinking NetApp here in a VMWare environment; I’m a big fan so this is a biased opinion. NetApp is my background but other vendors play in this space as well. I bet Chad will be happy to leave a comment to help tell us why (and I hope he does!).

Think of it this way. You’re already moving from FC cards to CNAs. Why not buy regular 10Gb Ethernet cards instead? Why not just use the Nexus 5K as a line-rate, non-blocking 10Gb Ethernet switch? This configuration is very simple compared to FCoE at the Nexus level and management of the NetApp is very easy! Besides, you could always turn up FCoE on the Nexus (and the NetApp) at a future date.

In closing, I really like FCoE but as you can see it isn’t a perfect fit today for all environments. I really see this taking off in 1-2 years and I can’t wait. Until then, use caution and ask all the right questions!

If you are interested in some more in-depth discussion, here are links to some of Scott’s articles on FCoE:

Continuing the FCoE Discussion
Why No Multi-Hop FCoE?
There Might Be an FCoE End to End Solution

Tags: , , , , , , , ,

Introduction to Nehalem Memory

Monday, May 11, 2009 in General by adelp | 19 comments

By Aaron Delp
Twitter: aarondelp
FriendFeed (Delicious, Twitter, & all my blogs in one spot): aarondelp

I wanted to relay some information regarding choosing memory speeds and types for the new Intel Xeon 5500 (Nehalem family) processors. As stated in my previous article on the Nehalem CPUs, there are some decisions that need to be made when choosing the memory and processor combinations. Let’s start off with what the memory architecture looks like.

  • The current Xeon 5500 family is a two-socket configuration.
  • Memory will run at 1333 MHz, 1066 MHz, and 800 MHz.
  • Memory is currently produced in single, dual, and quad rank configurations. Dual rank is faster than single rank, quad rank is currently limited to 1066 MHz speed.
  • Each CPU socket has 3 memory channels for a total of 6 channels per server.
  • Each channel can accept up to 3 DIMMS. This is why the servers currently are made with either 12 sockets (2 DIMMS per channel x 3 channels per processor x 2 processor sockets) or 18 sockets (3 DIMMS per channel x 3 Channels per processor x 2 processor sockets).
  • Some servers come in a 16 DIMM arrangement. Please see this IBM Paper for more information.
  • The maximum memory speed is limited by processor. For example, the X5570 has a max memory speed of 1333 MHz, the E5540 has a max memory speed of 1066 MHz, etc.
  • As more memory is added to a channel, the memory will slow down.
  • Better performance is achieved when the memory is “balanced” (the total amount of memory across channels is the same).

Take a look at the Hp Quick Specs for the BL460 G6 server in the Memory section. I found this to be a great source.

So, what does all of that mean? It means that for best performance you should install the memory using the following guidelines:

  • Ideally, install DIMMs in sets of 6, 1 per channel (populate both sockets with CPUs!). Use DIMMs that are dual rank and have the fastest speed you can purchase that the processor supports.
  • Populate the first slot in all channels first, then populate the 2nd slots in all channels, etc. Don’t put all three DIMMs in one channel and leave other channels empty.
  • Balance the amount of memory in each channel whenever possible (3 x 4GB on two channels and 1 x 4GB 1 X 8GB on the last channel).
  • If at all possible, try to keep the system away from the 800MHz memory speed.

Here is link to an awesome IBM white paper explaining everything.

Here’s an example 12 DIMM slot Nehalem configuration:

Speed

Max Mem Speed

Bank 1 in Channel Populated

Bank 2 in Channel Populated

X5570 (2.93 GHZ)

1333 MHz

1333 MHz

1066 MHz *

X5560 (2.80 GHZ)

1333 MHz

1333 MHz

1066 MHz *

X5550 (2.66 GHZ)

1333 MHz

1333 MHz

1066 MHz *

E5540 (2.53 GHZ)

1066 MHz

1066 MHz

1066 MHz

E5530 (2.40 GHZ)

1066 MHz

1066 MHz

1066 MHz

E5520 (2.26 GHZ)

1066 MHz

1066 MHz

1066 MHz

E5506 (2.13 GHZ)

800 MHz

800 MHz

800 MHz

E5504 (2.00 GHZ)

800 MHz

800 MHz

800 MHz

E5502 (1.66 GHZ)

800 MHz

800 MHz

800 MHz

 

Here’s an example 18 DIMM slot Nehalem configuration:

Speed

Max Mem Speed

Bank 1 in Channel Populated

Bank 2 in Channel Populated

Bank 3 in Channel Populated

X5570 (2.93 GHZ)

1333 MHz

1333 MHz

1066 MHz *

800 MHz

X5560 (2.80 GHZ)

1333 MHz

1333 MHz

1066 MHz *

800 MHz

X5550 (2.66 GHZ)

1333 MHz

1333 MHz

1066 MHz *

800 MHz

E5540 (2.53 GHZ)

1066 MHz

1066 MHz

1066 MHz

800 MHz

E5530 (2.40 GHZ)

1066 MHz

1066 MHz

1066 MHz

800 MHz

E5520 (2.26 GHZ)

1066 MHz

1066 MHz

1066 MHz

800 MHz

E5506 (2.13 GHZ)

800 MHz

800 MHz

800 MHz

800 MHz

E5504 (2.00 GHZ)

800 MHz

800 MHz

800 MHz

800 MHz

E5502 (1.66 GHZ)

800 MHz

800 MHz

800 MHz

800 MHz

* According to the HP Quick Spec for the BL460 G6, they are able to keep the speed at 1333 MHz with 2 DIMMS. A BIOS update is required to achieve this. This is HP specific.

Common Questions:

Q: What kind of performance decrease will I see by lowering the clock speed of my memory? For example using 6×2GB DIMMs (running at 1333 MHz) vs 12 x 1 GB DIMMs (running at 1066 MHz) to save a little money.

A: According to the IBM white paper listed above, we have two main areas of performance to worry about, latency and throughput. The latency difference between 1333 MHz and 800 MHz is about 10%. Memory throughput is another story though. The different between 1333 MHz and 1066 MHz is about 9%. The difference from 1066 MHz to 800 MHz is 28%!

Q: What kind of performance increase will I see in a “balanced” (same amount of memory per channel) system?

A: Again, according to the IBM paper, you will see a performance increase if the system is balanced. An exact number isn’t given.

Q: Which is fastest? Single, dual, or quad rank DIMMS?

A: According to the IBM White Paper, dual rank outperforms single rank by 7% in Specjbb2005. Quad rank DIMMs decrease the clock speed to 1066 MHz so they are not faster at this time.

Q: What if I only populate one processor?

A: You want to populate both sockets if performance is a concern. Adding the second processor not only makes the second set of DIMM sockets available, it also doubles the memory bandwidth.

Tags: ,

Implementing IBM BladeCenter Open Fabric Manager

Thursday, May 7, 2009 in General by adelp | 4 comments

By Aaron Delp
Twitter: aarondelp
FriendFeed (Delicious, Twitter, & all my blogs in one spot): aarondelp

A few months ago I implemented the Basic version IBM’s BladeCenter Open Fabric Manager (BOFM) and I wanted to share some of my notes with everyone. It has been awhile so this is very high level; most of the small details have since fallen out of my head.

If you don’t know what Open Fabric Manger is, BOFM allows you to virtualize the FC and Ethernet connections coming from a blade server. BOFM is IBM’s response to HP’s Virtual Connect.

In very simple terms, BOFM overrides the default adapter Ethernet MAC and FC WWN’s and replaces them with MAC’s and WWN’s that are generated by BOFM at the BLADE SLOT level. If you have a blade in the first slot in a chassis and you replace the blade, the MAC and WWN stays the same because OFM will again over ride the defaults on the replacement.

BOFM is offered in two versions, the Basic version and the Advanced version. The Basic version is implemented in the BladeCenter Advanced Management Module and provides BOFM functionality for that chassis and provides slot level redundancy only. Each blade slot in the chassis has a profile and this profile is static.

Advanced BOFM takes this concept a step farther. Advanced BOFM is a plug-in to IBM Director and allows management of multiple chassis from a single location. Furthermore, it also allows for a “hot-spare” blade. A profile on a slot can be copied to another slot in the case of a failure. I did not implement this feature but I will include a link to a GREAT document at the end of this article that explains the hot spare process.

Why is BOFM a good thing? The main reason for BOFM is one time set up of the network and storage. If you work for a large organization where a network team is required to configure the Ethernet ports and a SAN team is required to configure the FC ports, replacement of parts requires interaction with other team members that may or may not be available for changes on short notice. Also, if your organization utilizes change control then tickets may have to be generated, meetings, approvals… all that fun stuff that comes with Enterprise administration with multiple teams.

What does BOFM bring to virtualization? The answer is it depends. If you are using VMWare and you are running HA and DRS in your cluster, does it make sense to have a blade sitting idle in the chassis doing nothing because it is a hot spare? No, it doesn’t. That blade should be in the cluster participating and providing resources. If you are using VMWare in a non-HA and non-DRS environment or some other “stand alone” virtualization product, it could make sense because you would like another blade to fail over and be back up and running as quickly as possible.

Useful BOFM Links:

BOFM Manual

BOFM Technical Doc (explains Advanced BOFM and hot spare set up)

BOFM File Reference (Go under Reference -> Components -> Configuration File)

Tags: , ,

Introduction to “Nehalem” CPUs

Tuesday, April 7, 2009 in General by adelp | 9 comments

By Aaron Delp
Twitter: aarondelp

Hello everyone! It’s Aaron again. I’m sorry for falling off the radar for a bit. A new generation of Intel processors is upon us and I felt the need to come out of seclusion to share some recent findings regarding the new architecture. Today’s article will explore the new processor offerings. I will be following this up with one (or more depending on the length) about the memory architecture and interconnects.

There is one simple reason why I wrote this article. You can no longer pick a processor based on clock speed. The Nehalem processors have “levels” now and each level provides additional features and functionality lacking in the lower levels. You will need to be careful when choosing a processor if you are looking for certain features. Here is a quick table listing the models and the features:

Speed

Watts

Max Mem Speed

Turbo Mode and Hyper-Threading

X5570 (2.93GHz)

95W

1333 MHz

Yes

X5560 (2.80GHz)

95W

1333 MHz

Yes

X5550 (2.66GHz)

95W

1333 MHz

Yes

E5540 (2.53GHz)

80W

1066 MHz

Yes

E5530 (2.40GHz)

80W

1066 MHz

Yes

E5520 (2.26GHz)

80W

1066 MHz

Yes

E5506 (2.13GHz)

80W

800 MHz

No

E5504 (2.00GHz)

80W

800 MHz

No

E5502 (1.66GHz)

80W

800 MHz

No

I find the Max Memory Speed particularly interesting. As you will see in the next article, memory speed can get pretty complex very quickly. The more memory that is installed in the system, the lower the clock speed on the memory. The days of installing in matched pairs and forgetting about it are gone.

What is Turbo Mode and Hyper-Threading you ask? Hyper-Threading as far as I can tell (please leave feedback if this incorrect!) is the same old Hyper-Threading we knew and loved from past chipsets. Turbo mode is interesting though. Think of it as “Burst Mode” for processors. If your OS supports it, the CPU will increase the clock speed as long as you are within the thermal/power thresholds for the chip. The ability to go into Turbo mode depends on the number of active cores. If you are using most of the cores, the chip will be less likely to go into Turbo mode.

UPDATE: Keith from Intel has provided a great explanation of Turbo mode from a hardware perspective in the comments section. I wanted to include it here as a direct quote. Thanks Keith!

Turbo mode is mostly independent of OS support. On CPUs that support Turbo, it is implemented as the P0 p-state in the CPU. It looks & smells like a CPU that is simply running in the highest-frequency P-state. The PCU (power control unit) in Nehalem handles the rest.

Tags: , ,

Blades and Virtualization Aren’t Mutually Exclusive: Part Four, HP Traditional Expansion Options

Monday, February 16, 2009 in Virtualization by adelp | 18 comments

By Aaron Delp

Welcome to Part 4 in our series on blades and virtualization. Please note that I will be covering the HP virtualized I/O offerings in a near future post. I plan to cover Virtual Connect, Flex-10, and the BL495 blade at that time.

As always, let’s start by reviewing the hardware specification we will be using to compare the rack servers against the blade servers. Here are the portions of the spec that concern us:

  • A minimum of 32 GB memory – As many have commented and my own studies have shown, memory is usually the first limiting factor. The more memory you can install in the system, the better.
  • 2 on-board NICs + some combination of expansion cards (FC, iSCSI, NIC, 10Gb)

As we did for the IBM side, let’s start with memory. The maximum amount of memory available on the DL360 is currently 64GB when using 8GB DIMMS in each of the eight slots. The BL460c has the same memory configuration (8 slots), but has a few caveats depending on the processor power consumption. According to the HP Product Bulletin Tool, if the processor is 80W or below, then the max amount of memory is 64GB (8×8GB). If you would like to use processors above 80W, then the maximum amount of memory is 48GB in a 6×8GB configuration. Which processors take above 80W you ask? As of this writing, there are only 2 processors in this category, the 3.0GHz Quad Core (Intel X5450 chip, not the E5450 chip!) and the 3.16GHz Quad Core (Intel X5460).

As with the IBM solution, I will explore four possible transport technologies for virtualization: 10Gb, iSCSI, FC, and NFS. I will also fill any remaining expansion slots with 1Gb Ethernet ports. Just like the IBM Chassis, we have a maximum of eight expansion ports.

HP expansion is a little more straightforward than IBM. Bays 1 & 2 connect to the blade on-board NICs and must be populated with an Ethernet compatible switch. An HP BL460c blade has two expansion slots, labeled Mezzanine 1 & 2. Bays 3 & 4 on the chassis connect to the adapter in Mezzanine Slot 1 on the BL460c Blade. Bays 5-8 connect to Mezzanine bay 2 on the blade. If a dual port card is placed in Mezzanine 2, only Bays 5 & 6 will be active. A four port card is required (Ethernet is the only 4 port card currently) to access all four switch bays.

The only exception to the above port mappings is 10Gb. Like the IBM 10Gb switch, the HP 10Gb switch is a “double wide” switch. For dual 10Gb in the HP chassis, one switch must be placed in Bays 5/6, the other in Bays 7/8.

Here are the possible configurations given the port mappings above:

BL460c blade with 10Gb & 4 NICs:

Dual On board NICs Bays 1 & 2 (Ethernet switches)
Mezz 1 - Dual Port NIC Bays 3 & 4 (Ethernet Switches)
Mezz 2 - Dual Port 10Gb Card 5/6 – 10Gb switch & 7/8 10Gb switch

 

BL460c blade with 10Gb, 2 NICs, and 2 FC:

Dual On board NICs Bays 1 & 2 (Ethernet switches)
Mezz 1 - Dual Port FC Card Bays 3 & 4 (FC Switches)
Mezz 2 - Dual Port 10Gb Card 5/6 – 10Gb switch & 7/8 10Gb switch

 

BL460c blade with FC & 6 NICs:

Dual On board NICs Bays 1 & 2 (Ethernet switches)
Mezz 1 - Dual Port FC Card Bays 3 & 4 (FC Switches)
Mezz 2 - Quad Port 1Gb Card Bays 5-8 Ethernet switches

 

BL460c blade with iSCSI & 6 NICs:

Dual On board NICs Bays 1 & 2 (Ethernet switches)
Mezz 1 - Dual Port iSCSI Card Bays 3 & 4 (Ethernet Switches)
Mezz 2 - Quad Port 1Gb Card Bays 5-8 Ethernet switches

 

BL460c blade 8 NICs:

Dual On board NICs Bays 1 & 2 (Ethernet switches)
Mezz 1 - Dual Port 1Gb Card Bays 3 & 4 (Ethernet Switches)
Mezz 2 - Quad Port 1Gb Card Bays 5-8 Ethernet switches

 

As you can see from the tables above, HP has a nice round portfolio without any holes. In the next section we will be covering the IBM virtualized I/O solutions.

Tags: , , , ,

Blades and Virtualization Aren’t Mutually Exclusive: Part Three, IBM Traditional Expansion Options

Friday, February 13, 2009 in Virtualization by adelp | 15 comments

By Aaron Delp

UPDATE:  I have made some corrections to the 10GB and iSCSI sections based on feedback.  Thank you!

Welcome to part 3 of my series on blades and virtualization. I will be breaking the expansion articles into four parts: traditional IBM options, traditional HP options, virtualized IBM options, and virtualized HP options. By “virtualized options” I mean the IBM and HP blade I/O virtualization solutions (like IBM Blade Open Fabric Manager and HP VirtualConnect), not a server configuration that will run virtualization. Today, we will be covering IBM traditional options.

Let’s take a second to revisit a portion of our hardware standard we have been using for the power series and make a few additional notes. Here is the spec again:

  • A minimum of 32 GB memory – As many have commented and my own studies have shown; memory is usually the first limiting factor. The more memory you can install in the system, the better.
  • 2 on-board NICs + some combination of expansion cards (FC, iSCSI, NIC, 10Gb)
  • 2 x 146GB, 10k SAS drives (except the IBM HS21XM blades which only support one HD)

I will get into the expansion options in a second, but let me start with the memory since this is the area most people are concerned about. With the recent addition of 8GB DIMMS to the HS21XM product line, the HS21XM now supports 64GB of memory. The IBM 3550 supports a max of 32GB and even the IBM 3650 only supports 48GB. If memory is your concern, the IBM blade is actually the BEST fit for virtualization.

Another area of concern is the fact that the HS21XM only supports one hard disk. If you would like to boot the OS from a local disk, this can be an issue. IBM also offers the dual solid state drive in a RAID-1 configuration but both platters are mounted in the same enclosure so if you lose one, you need to replace them both. We know because it happened to one of our customers. In the end that doesn’t make sense either. I don’t see the actual ESX build as critical in a VirtualCenter environment so I usually don’t worry about it excessively.

The expansion cards can get a little tricky depending on the configuration. For virtualization I see four possible options to fill out the blades to meet our needs. They are FC, 10Gb, iSCSI, and NFS. We will fill any remaining expansion with Ethernet 1GB ports to provide as much connectivity as possible.

Above is a picture of an IBM BladeCenter H Chassis from the rear, highlighting the expansion bays. Bays 1 and 2 connect to the onboard Ethernet so the switches must be Ethernet compatible. Bays 3 and 4 connect to the first expansion card on the IBM Blade, often called the CFFv form factor. The v in CFFv stands for “vertical” meaning this card will talk to the vertical expansion switches.

Bays 7-10 often cause much confusion.  They are positioned horizontally in the chassis and connect to any CFFh (h for horizontal, get it?) card on a blade. A number of different switches can be inserted into these bays. The first one is the easiest one, the 10Gb switch.  The first switch would go in bay 7.  The second would go in to bay 9.  Each switch would connect to a port on the dual port 10Gb card.

UPDATE: You can also do a 4×10GB configuration by adding 2 more 10GB switches in Bays 8 and 10.  You will then need the 4 port 10GB card instead of the 2 oprt 10GB card.  I have updated the charts below to reflect the MAXIMUM number of connections.  You can of course go with less.

The second option for Bays 7-10 involves something called the MSIM (Multi-Switch Interconnect Module). This option will allow the FC, iSCSI, and NFS configurations. The MSIM is an adapter that divides the High Speed Bays into Bays 7 to 10 shown above. To take advantage of all four bays, two MSIM’s divide both High Speed Bays and you will need a four port CFFh card to map to all four bays.

The above information yields the following switch and blade expansion card combinations:

Blade with 10Gb & 4 NICs (no MSIM’s – 4 NICs and 4 10Gb per blade):

Dual On board NICs

Bays 1 & 2 (Ethernet switches)

Dual Port Cffv NIC

Bays 3 & 4 (Ethernet Switches)

Dual Port 10Gb Card

Bays 7,8,9,10 (10Gb switches)

 

Blade with 10Gb, FC & Eth (no MSIM’s – 2 NICs, 2 FC and 4 10Gb per blade):

Dual On board NICs

Bays 1 & 2 (Ethernet switches)

Dual Port Cffv FC Card

Bays 3 & 4 (FC Switches)

Dual Port 10Gb Card

Bays 7,8,9,10 (10Gb switches)

 

Blade with FC Option #1 (requires 2 MSIM’s – 2 FC and 6 NICs per blade):

Dual On board NICs

Bays 1 & 2 (Ethernet switches)

Dual Port Cffv FC Card

Bays 3 & 4 (FC Switches)

Quad Port Cffh NIC

Bays 7,8,9,10 (Ethernet switches)

 

Blade with FC Option #2 (requires 2 MSIM’s – 2 FC and 6 NICs per blade):

Dual On board NICs

Bays 1 & 2 (Ethernet switches)

Dual Port Cffv NIC

Bays 3 & 4 (Ethernet Switches)
Dual NIC and Dual FC Cffh Card

Bays 7,9 (Ethernet) & 8,10 (FC)

 

Blade with NFS or software iSCSI (2 MSIM’s – 8 NICs per blade):

Dual On board NICs

Bays 1 & 2 (Ethernet switches)

Dual Port Cffv NIC

Bays 3 & 4 (Ethernet Switches)

Quad Port Cffh NIC

Bays 7,8,9,10 (Ethernet switches)

 

You will notice that only 3 of the 4 options are given. Hardware-based iSCSI is missing. That is because IBM has a hole in their portfolio. The IBM iSCSI card is the older form factor that if used on an HS21XM blade it doesn’t allow any other card. So, you end up with 2 NICs and 2 iSCSI. This isn’t acceptable and isn’t a valid option.

So what do you think? I believe that any of the above configurations will suit virtualization nicely given a proper design. I look forward to your thoughts as we continue on to the next article, HP traditional expansion.

Tags: , , , ,

Blades and Virtualization Aren’t Mutually Exclusive: Part Two, IBM Power Sizing

Saturday, February 7, 2009 in Virtualization by adelp | 6 comments

By Aaron Delp

This is the second part of a series of articles on Blades and Virtualization. The first article is found here. Today, I will present IBM rack servers versus blade servers and explore the power consumption for each.

In case you need it again, here is the hardware platform I will be using as the standard:

  • 2 x E5450 3.0 Ghz quad-core processors
  • 32 GB memory in an 8 x 4GB configuration
  • 2 on-board NICs
  • Additional quad NIC card for a total of 6 NICs
  • Dual FC card (assuming FC SAN storage for VMs but it could easily be NFS or iSCSI as well)
  • 2 x 146GB, 10k SAS drives (except the IBM HS21XM blades which only support one HD)

Here is the link to the IBM System x and Blade Power Configuration Tool I used for the numbers below. Using the hardware specifications above, I will be comparing the IBM System x 3550 server against the IBM HS21XM blade server.

Let me start by giving you a little history of the IBM Power Tool and note some limitations that may skew these results slightly. In case you haven’t used it before, the IBM tool is a little rough around the edges. Not all of the products and configurations are included in the tool. The Power Tool was started as a side project by some members of the IBM Brand Team a few years ago and was picked up by IBM as a product. Although I knew the original authors before they moved on to other jobs, I am not sure who keeps it up to date today.

The System x 3550 calculations are pretty accurate but the IBM Blade Center calculations are missing a few pieces. The tool will only allow for up four switches in the chassis. Even though you can configure two MSIM’s to enable the additional switch bays, you can’t choose the additional switches in the tool. Also, the IBM Blades section of the tool won’t allow for the CFFh expansion card that would attach to the switches. So, these calculations are missing four switches per chassis and one expansion card per blade. Not a big deal but I wanted to point it out. Also, please remember that all calculations assume 100% utilization of the server(s).

With all of that out of the way, what does a fully loaded IBM chassis look like compared to rack servers? An IBM BladeCenter H Chassis holds fourteen servers so we will begin there:

14 Servers (Full IBM Chassis)

3550’s

HS21XM’s

% Savings

Total System Power Input (W)

7028

4869

30.72

Total System Input Current (A)

33.6

23.41

30.33

Total System BTU/Hr

23954

16614

30.64

Total System VA Rating

7168

5098

28.87

 
As you can see, from a power perspective, the IBM blades are pretty impressive! We are looking at approximately 30% savings in all areas versus IBM rack servers. Let me be very clear on one point though: I am not stating that IBM blades are more power efficient than HP blades (I know that comparison is right around the corner for many of you). For many reasons that I won’t go into here, I don’t believe you can make a jump to that conclusion based on the numbers. I am stating that IBM blades are more efficient than IBM rack servers, nothing more.

How about a chassis that is half full with seven servers?

7 Servers (Full IBM Chassis)

3550’s

HS21XM’s

% Savings

Total System Power Input (W)

3514

2732

22.25

Total System Input Current (A)

16.8

13.14

21.79

Total System BTU/Hr

11977

9324

22.15

Total System VA Rating

3584

2873

19.83

 
We are seeing 20% or greater with a half full chassis. Again, the numbers are very nice.

Lastly, what is the breakeven point? For the IBM blades the breakeven point is four servers.

4 Servers (Full IBM Chassis)

3550’s

HS21XM’s

% Savings

Total System Power Input (W)

2008

1805

10.10

Total System Input Current (A)

9.6

8.68

9.58

Total System BTU/Hr

6844

6159

10.00

Total System VA Rating

2048

1907

6.88

 
The most interesting bit of information I find with these numbers is the difference the fourth blade makes. With three blades, you have slightly negative percentages (the rack servers are better). But, add the fourth blade and you are suddenly saving somewhere near ten percent. I find that odd.

My conclusion this time is a cut and paste of last time (almost literally!). As with any consolidation project, there is a breakeven point with blade servers where the initial investment begins to pay off. Also, like virtualization, the more blades (or virtual machines) you add to the environment, the more savings you will see.

The next article will focus on the expansion abilities of both the IBM and HP blade servers. I know that expandability is a big concern for many readers, so stay tuned for Part 3!

Tags: , , , ,

Blades and Virtualization Aren’t Mutually Exclusive: Part One, HP Power Sizing

Wednesday, February 4, 2009 in Virtualization by adelp | 21 comments

By Aaron Delp

I have seen a lot of misconceptions recently when it comes to blade servers and virtualization. Many seem to think that if you are using blades you can’t use virtualization. Most recently, someone shared this article with me. I couldn’t disagree with this viewpoint more. I will explore the reasons why in the next few articles. For today’s article, I will present a foundation for comparison and I will use this basis going forward.

Bar none, the most common misconception surrounding blades is power consumption. Many think that blade servers consume more power than their rack mount counterparts. If you have ever seen a fully loaded blade chassis, I can see why many come to this conclusion. They are loud, move a ton of air, and require larger power circuits that many people don’t normally see in a data center (”Look at the size of that plug! This thing must be a power monster!”).

But, if you are installing more than a few servers, this isn’t true. First, let me present my basis for comparison and the tools I will be using. I have chosen a middle of the road VMware ESX configuration that is supported by both IBM and HP. Here is the configuration:

  • 2 x E5450 3.0 Ghz quad-core processors
  • 32 GB memory in an 8 x 4GB configuration
  • 2 on-board NICs
  • Additional quad NIC card for a total of 6 NICs
  • Dual FC card (assuming FC SAN storage for VMs but it could easily be NFS or iSCSI as well)
  • 2 x 146GB, 10k SAS drives (except the IBM HS21XM blades which only support one HD)

Here are the links to both the IBM and HP Power Tools:

Using the above configuration and tools yielded an IBM System x 3550 and HP DL360 in the rack mount category and the IBM HS21XM and HP BL460c in the blade category. I will cover HP today and IBM in a subsequent article.

If you plug the above specs into the DL360 tool and multiply by sixteen to equal the number of servers in a chassis you will get the results below. I then ran the HP Blade System Sizer tool and filled the chassis with an equivalent spec. In order to properly populate the blades with the expansion cards, I was required to also add all of the switches. I added six VirtualConnect Ethernet and two VirtualConnect FC switches. I also assumed the max number of power supplies and fans in all calculations. Both tools assume 100% utilization of all resources.

16 Servers (Full HP Chassis)

HP DL360’s

HP BL460c’s

% Savings

Total System Power Input (W)

7904

6021

23.82

Total System Input Current (A)

40

29.54

26.15

Total System BTU/Hr

26976

20530

23.89

Total System VA Rating

8432

6143

27.14

As you can see, even with the addition of eight switches in the blade chassis, you still save in excess of 20% in all areas. Pretty impressive no matter what you are running on the blades!

What if the chassis isn’t entirely full? How about 8 blades?

8 Servers (Half HP Chassis)

HP DL360’s

HP BL460c’s

% Savings

Total System Power Input (W)

3952

3396

14.06

Total System Input Current (A)

20

16.66

16.70

Total System BTU/Hr

13488

11579

14.15

Total System VA Rating

4216

3465

17.81

Still very nice results! Again, even with the switches in the blade chassis, you are saving in excess of 14% in all areas. The main story to tell here is this: the more populated the chassis, the more savings you will see.

What is the breakeven point from a power perspective with HP blades? The answer currently is five. Any less and the overhead of the chassis becomes a factor. Here are the numbers with five blades.

5 Servers (Half HP Chassis)

HP DL360’s

HP BL460c’s

% Savings

Total System Power Input (W)

2470

2361

4.41

Total System Input Current (A)

12.5

11.58

7.36

Total System BTU/Hr

8430

8052

4.48

Total System VA Rating

2635

2409

8.57

In conclusion, as with any consolidation project, there is a breakeven point with blade servers where the initial investment begins to pay off. Also, like virtualization, the more blades (or virtual machines) you add to the environment, the more savings you will see.

Thank you for your time! I will be covering the IBM comparison in the near future, so check back regularly.

Tags: , , , , ,