Technology Short Take #9

Welcome to Technology Short Take #9, the last Technology Short Take for 2010. In this Short Take, I have a collection of links and articles about networking, servers, storage, and virtualization. Of note this time around: some great DCI links, multi-hop FCoE finally arrives (sort of), a few XenServer/XenDesktop/XenApp links, and NTFS defragmentation in the virtualized data center. Here you go—enjoy!


  • Brad Hedlund has a great post discussing Nexus 7000 connectivity options for Cisco UCS. I’ll include it in this section since it focuses more on the networking aspect rather than UCS. I haven’t had the time to read the full PDF linked in Brad’s article, but the other topics he discusses in the post—FabricPath networks, F1 vs. M1 linecards, and FCoE connectivity—are great discussions. I’m confident the PDF is equally informative and useful.
  • This UCS-specific post describes how northbound Ethernet frame flows work. Very useful information, especially if you are new to Cisco UCS.
  • Data Center Interconnect (DCI) is a hot topic these days considering that it is a key component of long-distance vMotion (aka vMotion at distance). Ron Fuller (who I had the pleasure of meeting in person a few weeks ago, great guy), aka @ccie5851 on Twitter and one of the authors of NX-OS and Cisco Nexus Switching: Next-Generation Data Center Architectures (available from Amazon), wrote a series on the various available DCI options such as EoMPLS, VPLS, A-VPLS, and OTV. If you’re considering DCI—especially if you’re a non-networking guy and need to understand the impact of DCI on the networking team—this series of articles is worth reading. Part 1 is here and part 2 is here.
  • And while we are discussing DCI, here’s a brief post by Ivan Pepelnjak about DCI encryption.
  • This post was a bit deep for me (I’m still getting up to speed on the more advanced networking topics), but it seemed interesting nevertheless. It’s a how-to on redistributing routes between VRFs.
  • Optical or twinax? That’s the question discussed by Erik Smith in this post.
  • Greg Ferro also discusses cabling in this post on cabling for 40 Gigabit and 100 Gigabit Ethernet.


  • As you probably already know, Cisco released version 1.4 of the UCS firmware. This version incorporates a number of significant new features: support for direct-connected storage, support for incorporating C-Series rack-mount servers into UCS Manager (via a Nexus 2000 series fabric extender connected to the UCS 61×0 fabric interconnects), and more. Jeremy Waldrop has a brief write-up that lists a few of his favorite new features.
  • This next post might only be of interest to partners and resellers, but having been in that space before joining EMC I fully understand the usefulness of having a list of references and case studies. In this case, it’s a list of case studies and references for Cisco UCS, courtesy of M. Sean McGee (who I hope to meet in person in St. Louis in just a couple of weeks).



  • Using XenServer and need to support multicast? Look to this article for the information on how to enable multicast with XenServer.
  • A couple of colleagues over at Intel (I worked with Brian on one of his earlier white papers) forwarded me the link to their latest Ethernet virtualization white paper, which discusses the use of 10 Gigabit Ethernet with VMware vSphere. You can find the link to the latest paper in this blog entry.
  • Bhumik Patel has a good write-up on the “behind-the-scenes” technical details that went into the Cisco-Citrix design guides around XenDesktop/XenApp on Cisco UCS. Bhumik provides the details on things like how many blades were using in the testing, what the configuration of the blades was, and what sort of testing was performed.
  • Thinking of carving your storage up into guest OS datastores for VMware? You might want to read this first for some additional considerations.
  • I know that this has seen some traffic already, but I did want to point out Eric Sloof’s post on the Xenoss XenPack for ESXTOP. I haven’t had the opportunity to use it yet, but would certainly love to hear from anyone who has. Feel free to share your experiences in the comments.
  • As is usually the case, Duncan Epping has had some great posts over the last few weeks. His post on shares set on resource pools highlights the need to adjust the shares value (and other resource constraints) based on the contents of the pool, something that many people forget to do. He also provides a breakdown of the various vCenter memory statistics, and discusses an issue with binding a Provider vDC directly to an ESX/ESXi host.
  • PowerCLI 4.1.1 has some improvements for VMware HA clusters which are detailed in this VMware vSphere PowerCLI Blog entry.
  • Frank Denneman has three articles which have caught my attention over the last few weeks. (All his stuff is good, by the way.) First is his two-part series on the impact of oversized virtual machines (part 1 and part 2). Some of the impacts Frank discusses include memory overhead, NUMA architectures, shares values, HA slot size, and DRS initial placement. Apparently a part 3 is planned but hasn’t been published yet (see some of the comments in part 2). Also worth a read is Frank’s recent post on node interleaving.
  • Here’s yet another tool in your toolkit to help with the transition to ESXi: a post by Gabe on setting logfile location, swap file, SNMP, and vmkcore partition in ESXi.
  • Here’s another guide to creating a bootable ESXi USB stick (on Windows). Here’s my guide to doing it on Mac OS X.
  • Jon Owings had an idea about dynamic cluster pooling. This is a pretty cool idea—perhaps we can get VMware to include it in the next major release of vSphere?
  • Irritated that VMware disabled copy-and-paste between the VM and the vSphere Client in vSphere 4.1? Fix it with these instructions.
  • This white paper on configuration examples and troubleshooting for VMDirectPath was recently released by VMware. I haven’t had the chance to read it yet, but it’s on my “to read” list. I’ll just have a look at that in my copious free time…
  • David Marshall has posted on a two-part series on how NTFS causes I/O bottlenecks on virtual machines (part 1 and part 2). It’s a great review of NTFS and how Microsoft’s file system works. Ultimately, the author of the posts (Robert Nolan) sets the readers up for the need for NTFS defragmentation in order to reduce the I/O load on virtualized infrastructures. While I do agree with Mr. Nolan’s findings in that regard, there are other considerations that you’ll also want to include. What impact will defragmentation have on your storage array? For example, I think that NetApp doesn’t recommend using defragmentation in conjunction with their storage arrays (I could be wrong; can anyone confirm?). So, I guess my advice would be to do your homework, see how defragmentation is going to affect the rest of your environment, and then proceed from there.
  • Microsoft thinks that App-V should be the most important tool in your virtualization tool belt. Do you agree or disagree?
  • William Lam has instructions for how to identify the origin of a vSphere login. This might not be something you need to do on a regular basis, but when you do need to do it you’ll be thankful you have the instructions how.

I guess it’s time to wrap up now, since I have likely overwhelmed you with a panoply of data center-related tidbits. As always, I encourage your feedback, so please feel free to speak up in the comments. Thanks for reading!

Tags: , , , , , , , , , , ,


  1. Richard Anderson’s avatar

    Merry Christmas, and thanks for the mention/link.

  2. Melonhead’s avatar

    Thank you very much for your hard work!

  3. Nigel Poulton’s avatar

    Hi Scott,

    Hope you had a great Christmas.

    Re your comment that VSP and VMAX. While VSP and VMAX clearly share many similarities, however, “in this regard” as you put it I do not personally believe they share a similarity. (Assuming Ive understood what you are inferring)

    In VMAX all processors are tied to either a front end or back end port. Unless I’m getting rusty on VMAX, these processors manage front end and back end I/O as well as things like T/F and RDF…. The processors/slices are named something like A through H with some being tied to backend and other to front end ports. E.g. H0 and H1 ports on the front a VMAX are mapped to processor/slice H. The resources of slice H cannot be utilised by other front or backend ports (at least not in the same way…).

    VSP has ASICs that are tied to front and back end ports in a similar way (these are purpose built I/O processors), but VSP also has a pool of intel CPUs that are pooled and that execute the equivalents of T/F and RDF etc. These general purpose intel chips dont handle I/O, they execute most of the additional functions such as thin provisioning, dynamic provisioning….

    To be honest, I like what Hitachi have done from an architectural perspective. Although Im not sure if its worth the investment (custom ASICs etc) and if it makes mush difference from a performance, availability and maintenance perspective. I imagine there are some benefits but I don’t know how well they actually work. Will be interesting to see how the differences pan out. I also like how VSP and VMAX are going down different architectural roads – keeps things interesting.

    I think I’ll have a crack at a VMAX vs VSP post over the next week or so, but one that wont see me lose my job :-P


  4. slowe’s avatar

    Nigel, thanks for your comment, and I appreciate the discussion! I see your point regarding the difference between the way CPUs are used in the VMAX and the way CPUs are used in the VSP. There still seems to be some confusion (perhaps only on my part) as to whether the director slices in a VMAX are tied to a particular CPU core or whether they are “software constructs” that are scheduled onto available CPU cores. I’ve seen information that supports both conclusions. If indeed the slices are software constructs, then VMAX uses a “quasi-pool” approach whereby slices are scheduled on a pool of cores within each engine. Not quite as radical as VSP’s approach, perhaps, but similar in nature. Again, the question really boils down to the relationship between the director slices and the available CPU cores.

    I do agree with you regarding the ROI of custom ASIC development. It would initially seem to make more sense to leverage the R&D of Intel and take advantage of the x86 price/performance curve, but as you stated it really depends on how much the custom ASICs impact performance and RAS (reliability, availability, and serviceability).

    Thanks for the discussion! I’m looking forward to your VSP vs. VMAX post.

  5. Hu Yoshida’s avatar

    Hello Scott – thanks for linking to my blog. I would like to correct your impression that the Hitachi VSP and the EMC VMAX share architectural similarities. In fact they are contrasting architectures. The Hitachi VSP is a tightly coupled pool of global processors, global cache, Front End processors, and Back End processors, where the tight coupling is done through an internal switch matrix. With this architecture, we can scale out individual components like Front End processor blades or scale up performance with global processor blades without having to add other components. The VMAX architecture consists of storage nodes that are loosely coupled over an external switch. Each storage node is basically a two controller modular storage system with a limited number of storage ports, cache, and disks. The two controllers consist of two processors, which support all the functions like replication, tiering, snap shots, etc. in addition to the front end and back end processing. When an application talks to a VMAX system it can only talk to one of the storage nodes and is limited to the resources in that one storage node. Additions to a VMAX has to be done by adding an entire storage node, even only additional disk capacity is required.

    When an application talks to a VSP it can access a common pool of resources that can scale across the entire VSP system, including up to 247 PB of external storage, and the scaling can be done by individual components.

    Amrith Kumar provided a good analogy in my post on “Redefining scale-up and scale-out” by comparing the difference in storage architecture to that of server MPP and SMP architectures.

    Each subsystem in an MPP is self contained with its own processor, memory, operating system, and application which communicates with the other subsystems over a high speed interconnect. This type of system does not scale unless there is additional software that can distribute the application workload across the subsystems. This is not required with an SMP since it shares the same memory, operating system, and application. The VMAX architecture is similar to the MPP architecture while the VSP is similar to the SMP in that it can share all the resources in the support of a storage application.

  6. slowe’s avatar

    Hu, thanks for your comment. I have a better idea now of how you see the two architectures as different. I think that a few of your conceptions about VMAX are slightly skewed—for example, additional disk capacity can be added in some cases by daisy chaining DAEs off an engine—but these are minor nits in the grander scheme of things.

    Thank you again for taking the time to respond!

  7. Hu Yoshida’s avatar

    Happy New Year Nigel,

    Thanks for helping to clarify the difference between the VMAX and VSP architectures. The concept of having a pool of Intel processors to handle the general purpose functions of Dynamic Provisioning, tiering, replication, etc is one that many people over look. As you note, Hitachi continues to use a customized processor for our front and backend I/O processing. Hitachi learned many years ago that I/O processing can be optimized by using an instruction set that is optimized for I/O. However, over the years as the storage systems took on more workload like, replication and dynamic tiering, etc, these processors were required to do more general processing. In the VSP, Hitachi separated the I/O processing from general processing to distribute the processing workload and enable the I/O processors to be refocused on processing I/O. Using a reduced instruction, the dual core I/O processors in the VSP can process more I/O than a general purpose quad core processor, with less cost and power consumption. This separation also enables more scaling flexibility in adding separate pairs of I/O and general purpose processors as required.

    I am looking forward to your post on VMAX vs. VSP. Let me know if you need any information from Hitachi.

  8. Hu Yoshida’s avatar

    Scott, Thanks for giving me the chance to clarify the difference. I suppose you can add disks by daisy chaining DAEs off another engine, but that would require an additional VMAX node with dual quad core processors and cache to get the additional DAEs. Isn’t there a limit as to how many disks are attached to each engine?

    Best regards for the new year

  9. slowe’s avatar

    Hu, thanks for your response. Certainly there’s a limit on how many disks can be attached to each engine, but the key point is that there is some room for storage expansion without adding engines. Great discussion!

  10. Nigel Poulton’s avatar

    Scott, Hu, While I love talking about stuff like this, AND
    truly like some of the architecture behind VSP, the question is “is
    it worth it”. It looks on the outside to be a more expensive
    approach than taking the 99.9% commodity approach. Im interested in
    what you think about such differences Scott? Taking off your EMC
    employee hat, nd coming from a VMware Guy perspective, do you care
    about such architectural differences in storage arrays?

  11. the storage anarchist’s avatar

    For the record, depending upon performance and capacity
    requirements, each VMAX engine can support either 16, 24 or 40
    DAEs, or 240/360/600 drives; DAEs can be added to existing engines
    non-disruptively in groups of 8 (additional engines can also be
    added non-disruptively).

  12. slowe’s avatar

    Nigel, from a VMware perspective all that really matters is efficiency, performance and integration. If the array vendor chooses to use custom ASICs, it makes no significant difference to the hypervisor.

    Personally, I question the use of custom designed ASICs and I/O-optimized RISC processors. It seems to me to make more sense to leverage commodity x86 CPUs and take advantage of Intel’s massive R&D budget. It would seem to be easier to do that and also gain the ability to respond more quickly to market demands.

    That being said, I do see value in providing separate processing power for I/O functions (front-end or back-end) and other “higher level” functionality. Whether this means separate CPUs or simply separate cores on a set of multi-core CPUs is yet to be seen.

  13. Hu Yoshida’s avatar

    Nigel and Scott, great discussion.

    What Scott says is the bottom line. It doesn’t matter what is inside as long as it provides efficiency, performance, integration, and functionality at a competitive price.

    As far as VMware goes, we believe the VSP provides better scalability and better support for the VAAI offload functions like write Zeros, copy, and SCSI Reserve offload, because of the way we separate the general purpose computing from the I/O processing and tightly couple them through a global cache. In other storage systems, the same commodity processor must service all these functions out of an active/passive cache, which is bound to affect performance and scalability.

    In terms of time to market, Hitachi feels that using custom designed ASICs gives us an advantage, since we do not need to wait for the development of off the shelf components to deliver new functionality. Hitachi is a vertically integrated company with a heavy investment in R&D and we can develop what we need when we need it. Hitachi was the first to introduce the internal switch architecture in storage systems, as well as virtual ports, dynamic cache partitioning, dynamic tiering, Dynamic provisioning for internal and external storage, etc.

    The question of separate cores or separate processors, is one of modularity. Adding I/O FED and BED processors can be done at a more granular level than adding VSD general purpose multi-core processors. Hitachi is using both approaches in the VSP.

    Barry — so for the record, if you need 601 drives, do you need to add another engine? With 8 VMAX engines do you support 4800 drives?

  14. slowe’s avatar

    Thanks for your reply, Hu. Clearly both approaches have advantages and disadvantages, and therefore customers need to weigh the advantages and disadvantages of each approach when selecting the solution. I think that you and I agree that in the end, what matters most is that the customer gets the right solution for their needs.

    As for your question to Barry—which I’ll let Barry answer if he so chooses—I think that *every* vendor has certain inflection points where additional hardware has to be added in order to cross over into a new threshold. Are you saying that the VSP does not have any such points, that drives could be added all the way up to the maximum without having to add FED, BED, or global processors? Or that users can add functionality like thin provisioning, replication, or tiering without having to add more global processors? Probably not–and that’s OK. Hitachi establishes limits on what can be supported with certain levels of hardware in order to ensure that customers receive the expected levels of performance and availability. EMC does the same. So does NetApp, HP, and all the other major storage vendors. To call out these inflection points as a perceived limitation is, in my opinion, disingenuous.

  15. Hu Yoshida’s avatar

    Thanks Scott for opening up this discussion. We are in
    agreement. Certainly all systems have inflection points where they
    have to add more hardware, but it is the granularity of the
    hardware we add that makes a difference. If you need more
    processing power in a VSP you add VSD blades or cache adapters, for
    connectivity or external storage – FEDs, for internal drives –BEDs.
    There is a point where we need to add another control frame to
    house more than 4 VSD blades, or 4 BED blades, or 8 Cache adapters,
    or 12 FED blades and I also need to add drive frames when I exceed
    3 drive modules. The point is that the VSP is more granular in
    scaling since it can add components as needed without having to add
    whole storage nodes with dual quad core processors, dual cache
    modules, FEDs, BEDS, and DAE’s.

  16. the storage anarchist’s avatar

    Hu- For the record, 601 drives can be supported with as few
    as 2 engines, or as many as 8, dependent upon performance
    requirements. Standard configurations support a maximum of 2400
    drives per array.

  17. slowe’s avatar

    Barry, thanks for taking the time to clarify the specifics on VMAX expansion.

Comments are now closed.