By Aaron Delp
I have seen a lot of misconceptions recently when it comes to blade servers and virtualization. Many seem to think that if you are using blades you can’t use virtualization. Most recently, someone shared this article with me. I couldn’t disagree with this viewpoint more. I will explore the reasons why in the next few articles. For today’s article, I will present a foundation for comparison and I will use this basis going forward.
Bar none, the most common misconception surrounding blades is power consumption. Many think that blade servers consume more power than their rack mount counterparts. If you have ever seen a fully loaded blade chassis, I can see why many come to this conclusion. They are loud, move a ton of air, and require larger power circuits that many people don’t normally see in a data center (”Look at the size of that plug! This thing must be a power monster!”).
But, if you are installing more than a few servers, this isn’t true. First, let me present my basis for comparison and the tools I will be using. I have chosen a middle of the road VMware ESX configuration that is supported by both IBM and HP. Here is the configuration:
- 2 x E5450 3.0 Ghz quad-core processors
- 32 GB memory in an 8 x 4GB configuration
- 2 on-board NICs
- Additional quad NIC card for a total of 6 NICs
- Dual FC card (assuming FC SAN storage for VMs but it could easily be NFS or iSCSI as well)
- 2 x 146GB, 10k SAS drives (except the IBM HS21XM blades which only support one HD)
Here are the links to both the IBM and HP Power Tools:
Using the above configuration and tools yielded an IBM System x 3550 and HP DL360 in the rack mount category and the IBM HS21XM and HP BL460c in the blade category. I will cover HP today and IBM in a subsequent article.
If you plug the above specs into the DL360 tool and multiply by sixteen to equal the number of servers in a chassis you will get the results below. I then ran the HP Blade System Sizer tool and filled the chassis with an equivalent spec. In order to properly populate the blades with the expansion cards, I was required to also add all of the switches. I added six VirtualConnect Ethernet and two VirtualConnect FC switches. I also assumed the max number of power supplies and fans in all calculations. Both tools assume 100% utilization of all resources.
|
16 Servers (Full HP Chassis) |
HP DL360’s |
HP BL460c’s |
% Savings |
|
Total System Power Input (W) |
7904 |
6021 |
23.82 |
|
Total System Input Current (A) |
40 |
29.54 |
26.15 |
|
Total System BTU/Hr |
26976 |
20530 |
23.89 |
|
Total System VA Rating |
8432 |
6143 |
27.14 |
As you can see, even with the addition of eight switches in the blade chassis, you still save in excess of 20% in all areas. Pretty impressive no matter what you are running on the blades!
What if the chassis isn’t entirely full? How about 8 blades?
|
8 Servers (Half HP Chassis) |
HP DL360’s |
HP BL460c’s |
% Savings |
|
Total System Power Input (W) |
3952 |
3396 |
14.06 |
|
Total System Input Current (A) |
20 |
16.66 |
16.70 |
|
Total System BTU/Hr |
13488 |
11579 |
14.15 |
|
Total System VA Rating |
4216 |
3465 |
17.81 |
Still very nice results! Again, even with the switches in the blade chassis, you are saving in excess of 14% in all areas. The main story to tell here is this: the more populated the chassis, the more savings you will see.
What is the breakeven point from a power perspective with HP blades? The answer currently is five. Any less and the overhead of the chassis becomes a factor. Here are the numbers with five blades.
|
5 Servers (Half HP Chassis) |
HP DL360’s |
HP BL460c’s |
% Savings |
|
Total System Power Input (W) |
2470 |
2361 |
4.41 |
|
Total System Input Current (A) |
12.5 |
11.58 |
7.36 |
|
Total System BTU/Hr |
8430 |
8052 |
4.48 |
|
Total System VA Rating |
2635 |
2409 |
8.57 |
In conclusion, as with any consolidation project, there is a breakeven point with blade servers where the initial investment begins to pay off. Also, like virtualization, the more blades (or virtual machines) you add to the environment, the more savings you will see.
Thank you for your time! I will be covering the IBM comparison in the near future, so check back regularly.
Tags: bladevrack, Hardware, HP, IBM, Virtualization, VMware
-
PSC aside, My main beef with blades and virtulization is the utter lack of an upgrade path! It’s been a couple of years since I’ve worked with HP blades so they may have improved, but I have a datacenter full of IBM H series chassis, and I loathe them. Six miles down this road… there is far, far too much I cannot do with a blade (without paying an arm and a leg)!
I have to license double the ESX host’s due to the ram limitations, patch twice as many said host’s, double the 10GBE infrastructure, on and on… ahhh I spoke my piece already, I value your opinion, so I ask you, WHY??? Why is all of IT gaga over blades? It can’t be the flaky must have JRE 1.4.2 web console.
~Glenn
-
The important thing that is missing from this comparison is power density and total potential power per rack. The DL360s are 1U servers. The blade chassis pack all the power consumption into 10U. A standard 42U rack filled with DL360s would use 20,748W. A full rack of blades uses about 16% more power (24,084W). It also leaves 2U free. While the blade servers do indeed give you more servers (64 vs 42), they do have a higher power density than traditional rack mount servers.
Having pointed that out, I don’t disagree with your overall comments about blades and virtualization. In fact I’m about to do a VMware deployment on Dell blades.
-
I’d love to see similar material, but for pricing.
-
@Michael - I would love to do the pricing but I’m not sure it would make sense. The problem is no one pays the same price. I could compare them at list and see where the difference is but most people get some kind of discount and the blades tend to have more profit built into them than the racks so it is very easy to be off by a long shot. Let me know what you think and I could add it in to the end of the series.
@Jefferson The problem with both racks and blades is they are getting too power hungry to fill a rack with either any more. With most of my customers you will have a hard time getting anything above 14-15kW to the rack. I was considering dong a quick calculation on power per server and/or power per U of rack space. This way you have a calculation that you can scale up to your amount of rack power in case you can’t use the full amount. Also, you have to remember the tools assumes 100% utilization for ALL servers ALL the time. This won’t happen real world unless it is some kind of high performance cluster. Thoughts?
@Glenn Yes, blades have come along way, especially on the HP side. I will cover the IBM power shortly but I will then move onto the expansion of the blades (both IBM and HP) for the next article. Please stay tuned for that I would love to hear your opinion on it! I will say this, as an existing IBM customer, you do have valid beefs around expansion (and especially the console!!) and I will point them out in that article. IBM just released a bunch of products to fill the gaps and to catch up with HP. Also, the next generation of both HP and IBM blades are on the horizon from what I hear. As I posted in Twitter, I’m sorry if we called you out, that was never my intention. I’ll present my side and please stick around. At the end we can agree to disagree. Thanks!
-
After almost 3 years running vmware on blades, I love them. I don’t see the difficulty, I run heavy servers and tons of desktops on blades….I just don’t see the downside you do Glenn.
I run servers that before required 4 way xeons with 32 gig of ram now on a 2 way quad core blade with 8 gig of ram and it flies….I run 2400+ desktops on blades, and the “spread” that bothers you, ensures me adequate failover. N+1 every time I have to add another.
I’d rather throw another cheap blade in that pay for a 4 way box, licensing doesn’t cost me anywhere near what a larger server costs alone. I not only get a cheap server, I now split my risk by one more box. Not to mention what I save in power and cooling.
-
While the P-class Blades were less than stellar (to put it charitably) the C-class blades from HP are incredible.
Glenn: HP first came out with the c7000 chassis in Q3 2007, and they are still releasing new blades for it and just recently extended support for the chassis to 2013! Upgrade paths are not an issue, at least on the HP side.
Today you’ve got 10GbE available in the chassis, and soon we’ll see 8Gb FC interconnects soon enough. There’s even Infiniband and Shared SAS connections available, and HP actually has more memory on some of their blades than standard DL servers (the BL495 is a dual socket that takes 16 DIMMs, there’s no HP rackmount server with that RAM density).
I can understand your reservations, but blades have come a very long way since the IBM H-series. I speak of HP blades here, but even the IBM blades have improved… -
@Aaron — yea that’s part of the pricing problem. I guess list might work, but if a general idea of discount on the blades were available, it might be easier. To be sure, I only care about small deployments, say, 5-10 blades, which might make the pricing comparison easier.
-
@ Michael - What I can do is make a quick comparison of rack servers vs. blades (and maybe vs. high end servers) using list price. It won’t be hard at all. I’ll add it to the list. Thank you!
-
If you do math on small amounts, you remove a big portion of savings for blades….Blades pay for themselves and their licensing, somewhere around blade 9 or 10 and every blade added after that brings lower TCO. By doing a pricing on small amounts only, you have taken apples and made them oranges. Might as well do a comparision to the number of cores you can put in a rack at this point, at which time a blade center will demolish scale up environments….But it doesn’t make it a fair comparison. Personally I find them(blades) overkill for small environments. It’s for density, savings, and quick operation. If you only need 5-10 blades, you might as well buy standard servers(although 10 is getting close).
Don’t forget to add in to that cost, cabling and port costs. FC and ETH ports for the total setup, and the cost of the ports themselves. We saved 10’s of thousands of dollars in our environment just in FC port, not even counting how much core ports on our 6509e’s cost….
-
Lots of thoughts around blades (am working with some customers on this right now actually — one is going with 1 RU servers to replace a 3 year old blade chassis as we speak)…some initial thoughts are….
-IP SAN deployments (no FC) - Blades seem to make dense IP SAN IO deployments harder (i.e. can’t shove as many GigE ports per blade as per 1U server even - (10) GigE NICs with the 3550 - lotsa throughput if you put the datastores at different IP #’s over a multi-GigE etherchannel). While I realize there are 10Gig switch modules for chassises, those still only provide GigE connections to the blades (10Gig is for the uplink). Especially when we get Fault Tolerance in vSphere, I’ll actually consider 8 to 10 GigE ports ideal (presuming 10Gig/CNE isn’t mainstream yet) — all the IBM blades I looked at can’t handle that.
-Rack Flexibility - If you have rack constraints, blades require a chunk of consecutive rack space (7U, 9U, etc.) whereas 1U servers can be spread around however you like (and moved around as well if you need to reshuffle stuff).
-Purchase Flexibility - while I can see that you may save some money if you buy a chassis and fill it up at the same time, most of our small to midsize customers don’t have server requirements in sets of 14 (or however many blades you can put in a chassis - I’m thinking IBM). All the scenarios I ran with a chassis half to 2/3 full of blades came out relatively more expensive (time value of money, etc.) especially given uncertain budgets for expansion.
-Rack Density - mostly echoing the comments above - given quad-core processors (with 8 core coming soon), maxing out power is happening before maxing out rack space (even in space-constrained environments….VMware + a good storage array that’s space-efficient (snapshots, dedup, etc.) just go a long way towards pushing density way up).
Somewhat funny note - in one environment with a lot of datacenter space, I actually went with tower servers for VMware — lots of PCIe/PCI-X slots, more memory slots (12), and made the cabling easier actually (all IP SAN so 10+ cables total from the back of each server).
There’s more thoughts in the mush I call brains (it’s been a long week) but that’s what I’ve got so far…would be very curious for feedback from one and all.
-
Andrew,
I don’t know about the IBM blades–I haven’t worked with them as much as the HP blades and there seem to be some quite legitimate complaints regarding expandability–but with the HP blades I’m pretty sure I can get 10 GigE NICs in a half-height blade (2 onboard plus 2 quad-port mezzanine cards) and 16 GigE NICs in a full-height blade (4 onboard plus 3 quad-port mezzanine cards). That’s typically enough connectivity for an IP-based storage implementation. In addition, the new Cisco interconnects (the Catalyst 3120, if I recall correctly) for the HP c-Class chassis support cross-stack EtherChannel for better connectivity when multiple storage targets are involved (keeping in mind that a single storage target–a single NFS export or a single iSCSI target–won’t be helped AT ALL by EtherChannel).
As for the comments regarding rack flexibility, you are certainly correct there. Life is not without its trade-offs.
Regarding purchase flexibility, my own rough calculations typically show that the infrastructure around blades–the chassis, the interconnects in the back, the power requirements–pay for themselves between 4 and 6 servers, after which point buying another blade is cheaper than buying an equivalent rack mount server. I haven’t revisited these calculations in a couple of years, to be fair, so the numbers may have changed since then.
Anyway, good discussion and thanks for sharing your thoughts.
-
Andrew;
I don’t think you get much denser than our enterprise, and SAN io has never been an issue. Think about it this way, from a storage admins position, have you ever maxed your hba’s?, how about the switches? 2400+desktops, hundreds of servers, and I have yet to raise above 20% capacity on my links (though I do balance across fabrics)…but I’ve maxed out a symmetrix! Why? Because you can have 8G fc on all the blades, 8G on all the switches, but the issue is you’ll still roll all of those devices into a bottleneck that’s only 2G or 4G once you get to the array itself. You can get serious expansion with either the IBM or the HP blades, adding in several nic ports to each blade, so that point is not really valid. I can stick 4 or more nics in each blade, and I don’t think I’d exceed 4 nics in your standalone boxes either. Even without expansion, I can plug in 2 gigs to each blade with IBM…Which is more than enough in 99.99% of cases. Add in expansion ports, and I can bump that quite a bit higher.As to rack power, yes, I’ll max power if I fill the rack (as I have done), but that’s fine, I can still get far more virtual machines in that 36U’s of space (or less with HP), than you can get in a rack of 1 or 2U boxes. Why? Because in 9U’s you have 9 servers, and I have 14. And instead of a mess of cables, I have just a few (Trust me I have 2 legacy datacenters of 1u boxes, and getting the bad one out of the middle of a rack gets to be not fun).
To TCO, once you hit 9 blades, it pulls ahead of the competition (standard racks and towers)…but no one is saying it’s the way you have to go, and if you are using 9 or less, by all means go the other way. But if you’re going with 9 blades, think how easy it is to build out that 10th server…you just slide it in, install esx on it, zone it, and you’re done. No racking, no cabling, no futzing with it. Period. And I didn’t have to add a single port on my very expensive san switches, or my nearly expensive 6500’s. Sure there are tradeoffs, maybe I just don’t have the need. But neither do most people if you think about it. If you’re willing to put the time and energy into planning carefully so that you get it right the first time, you’ll be exceedingly happy with the bladed approach and virtualization.
-
Rereading I was not clear in that first paragraph. In my IBM’s I get 8G FC and 2G Eth to each blade by default, and that can be increased.
-
Hi Scott,
I’m a frequent reader of your blog, and some of your information on VMware and Netapp has been gold to me over the last year. I started my own blog to detail some experiences of mine that I think would be helpful to others, that I haven’t found elsewhere online.Since mine is around the same technology you often discuss, I thought I’d share my site: philiparnason.blogspot.com
-
Thank you to everyone for the comments. A couple of things:
@Andrew Miller - The most nics you can get on either IBM or HP blades is 8 nics on a “normal size” (HP half height, IBM single size) blade. 2 nics onboard, 2 on the first expansion card, 4 port in the second bay.
You can do 10GB down to the blade from both IBM and HP, you need the 10GB card. For both IBM and HP the 10GB switches take up two switch bays so you reduce the overall number of bays from 8 down to 6 on the back of the chassis.
@Scott Lowe - The max NICS you can put on the HP half height is 8 (mentioned above) but you will be able to get at least 10 on the full height. It has 4 on-board and I think you could put a bunch of quads. I would have to look it up.
I’ll make sure I point all of this out in the next article…. Expansion!
-
Since we’re talking about HP blades, it is possible now to get more than 8 “nics’ in a half height blade, depending on what your definition of a NIC is:
http://h18004.www1.hp.com/products/blades/virtualconnect/index.html
WARNING: annoying video autoplays with sound, my apologies…HP has a new 10GbE interconnects called Flex10 which gives two 10GbE pipes into the blades that are split into 8 virtual NICs total. That means if all you wanted was ethernet, you can get 60Gbps into a single half-height blade, spread across potentially 24 NICs. You could also present that only over 6 NICs each running 10Gb each, it’s pretty flexible. And they’re supported in ESX 3.5 Update 3.
-
Sorry for the extra post…
@adelp: You can get 16 NICs in a full-height HP blade, but then you can’t have any fibre, obviously. -
Disclaimer: I did not enjoy my first experience with IBM blades. However, I do understand both sides of this argument and blades have significantly improved in the last few years.
All of these comparisons are almost meaningless without real facts, figures, and dollar amounts (List price is just fine). We need to see RAM, CPU, power, etc. totaled, per server, per rack.
In my environment the ESX host will always run out of memory before CPU. If I have to run more blades to get to the same total amount of ram then I have to factor in the additional ESX licenses. Also, be careful when maxing out the RAM in some servers because it can double the price.
If your datacenter is short on space then you need to run blades.
-
@ Justin - That is good to know about the number of max nics on the full-height blade. I have intentionally stayed away from the HP “full height” and the IBM “double wide” because it will muddy the waters too quickly. Thank you for the update!
Also, about the Flex-10. I am now thinking I’m going to split up the expansion into two sections, traditional and the virtualized offerings. Flex-10 is worth talking about!
Thanks again!
Site Sponsors
21 comments
Comments feed for this article
Trackback link: http://blog.scottlowe.org/2009/02/04/blades-and-virtualization-arent-mutually-exclusive-part-one-hp-power-sizing/trackback/