September 2009

You are currently browsing the monthly archive for September 2009.

I recently had the opportunity to look through a copy of Cisco Routers for the Desperate, 2nd Edition, written by Michael W. Lucas and published by No Starch Press.

Having worked fairly extensively with both Cisco routers and Cisco switches (see this tag archive for my Cisco-related posts), the information provided in this book was well below my knowledge level. It’s highly likely, then, that this is not a book that I would have purchased for my own personal use—simply stated, I’m not in the target market for this title. Instead, it’s administrators who have very little prior exposure to or experience with Cisco routers and Cisco switches who are the target market for this book. And for those people, this is a book that will get them started down the path of configuring and managing Cisco network equipment.

The book is comprehensive and thorough in introducing concepts that would be new to future network administrators, making sure that the basics are covered before moving on to other topics. The author does a good job of explaining each concept in an order that I felt was logical and made sense (again, trying to think like someone who would be new to this environment). On the down side, I personally felt the book could have used a bit more information in some areas, and I think the inclusion of BGP and HSRP was a stretch for the target audience. It’s my opinion the space taken on BGP and HSRP would have been better used to provide a few detailed examples of some of the concepts or configurations.

These “drawbacks”—and I use that term very loosely—don’t outweigh the overall usefulness of the book for the intended market. The author and the book have a strong focus on getting new users comfortable in the environment, so if you are brand new to managing a Cisco router and need somewhere—anywhere—to get started, Cisco Routers for the Desperate might be just the right starting place. For the more advanced users out there, this book wasn’t written for you, and you’re better off looking elsewhere for the in-depth content you are seeking.

Disclosure: No Starch Press provided a copy of this book for review at no cost. The fact that the book was provided at no cost did not influence my review.

Tags: , ,

Books or Blogs: My Perspective

A short while ago, I came across this article (I believe it was in a tweet posted by co-worker and fellow blogger Colin McNamara). In the article, the author asks this question: For technical folks interested in writing, what’s the best approach—books or blogs?

To me, this is akin asking the question, “What’s the best kind of fruit: apples or oranges?” Both are types of fruit, like writing a blog and writing a book are types of writing. But there are many more types of fruit, and there are many more types of writing. Just as apples and oranges each have their unique flavor and unique properties, writing a blog and writing a book have their unique qualities and unique requirements. One isn’t really a substitute for the other; each has its own place and purpose.

I think I am qualified to speak on this topic because I have not only recently completed my first book (Mastering VMware vSphere 4, Sybex, 2009; available on Amazon) but I am also close to wrapping up my second book (VMware vSphere 4 Administration Instant Reference, Sybex, 2009; also available on Amazon). And, as many of you probably already know, I’ve been writing here on this site for almost four years now. So I’ve been on both sides. This is not to say that Jeremy Filliben, the author of the original article, or any of the commenters on his article, are not qualified; rather, it is to point out that I have seen and experienced several types of writing, including books, blogs, and periodicals.

Jeremy seems to imply in his post that writing a blog is an acceptable substitute for writing a book, and he cites the “drawbacks” of writing a book as reasons to support the claim. After all, why work under onerous deadlines to produce a work that will not generate significant amounts of income? By all indications, Mastering VMware vSphere 4 has been tremendously successful thus far (and I hope it continues to be successful!), but it’s not going to make me rich. Why not just publish the content on your own site, at your own pace? Wouldn’t that be better? Sure, you give up a little bit of income (you will make some money writing a book if you do it right), but as Jeremy points out some of that lost income can be regained through smart monetization of the site. So why write a book?

In my opinion, the answer is this: difficulty of entry. What do I mean? Look at it this way: just about anyone can purchase a hosting package, set up a site, and start writing. That doesn’t make immediately make their content reputable. That doesn’t immediately make their writing easy to read, or simple to understand. There are tons of technology-centric blogs out there. The bar for entry is ridiculously low.

With a book, on the other hand, the bar for entry is much higher. Authors have to show that their work is going to make money for the publisher, and the publisher isn’t going to sink money into publishing something that isn’t worth the investment. Not every book idea gets published, and not every author who wants to write a book gets the opportunity. I wanted to write a book for years, but only this year got the opportunity. How many other authors are out there who haven’t gotten that opportunity yet? I’d say there are many. Because the bar for entry is higher, more difficult, this generally (yes, there are exceptions!) means that the resulting products are higher quality. Self-publishing eliminates some of these barriers to make publishing a book easier for just about anyone, but self-published books also generally don’t get the broad support and broad distribution like titles from publishers such as Wiley, Addison-Wesley, or others.

These fundamental differences between books and blogs—in addition to some of the differences that Jeremy points out in his article—make me believe that the question “Books or blogs?” isn’t a valid question. These are fundamentally different, with different audiences, different entry points, and different results. Neither is better than the other; they are just different. The same could be said of writing for online magazines or print periodicals—they each have their own audiences, entry points, and results.

It’s like choosing a technology. You would choose a technology based on how well that technology satisfies a need. The same goes for writing. Depending upon what you are trying to achieve, you should use the writing outlet (books, blogs, or periodicals) that best meets your needs and helps you achieve your desired outcome. All of them are good—but they are not all equal.

Tags: ,

I’m a Bad Student

As I mentioned a short while ago, I’m in the process of learning about EMC storage. To that end, I managed to get my hands on a copy of Information Storage and Management: Storing, Managing, and Protecting Digital Information (Wiley, 2009; available on Amazon). While general in nature, the book has a fair number of EMC-specific product examples.

The book, though, isn’t the focus of this post. As I’ve started reading this book and trying to expand or solidify my knowledge of the concepts found in it, I’ve come to a disappointing realization: I’m a bad student.

Yes, that’s right. I’m a bad student. Over the past few years, my studying skills have really deteriorated. In the past, the constant flow of certification tests kept my studying skills reasonably sharp. And while we can all debate the value of IT certifications, I think we can also agree that they do, at least, help sharpen the study skills.

Normally, I wouldn’t be too terribly worried about this, but I have to ask myself: is this a symptom of something more? Is my ability to learn, assimilate, process, and relate new information also deteriorating? I pride myself on being a smart guy that can pick up just any new technology—is that still accurate? What can I do to reverse these trends, if indeed they are occurring?

So I need to ask your help. What studying techniques do you guys (and gals!) find useful? If you were in my situation—specifically, let’s say you need to acquaint yourself with an entirely new set of products from a new vendor—what techniques would you use in the process? Additionally, what techniques are readers using now to help with certification exams? I have at least two very important certification exams looming on the horizon (one much closer than the other, but both very close), so any tips readers can provide are welcome. Help me become a better student!

Tags: , ,

I had a customer contact me about scaling network throughput when using NFS datastores. Specifically, this customer was interested in knowing if it was possible to utilize more than 1 NIC with IP-based storage. The customer is currently using link aggregation (EtherChannel on a Cisco switch). I pointed the customer to my post on NIC utilization, in which I explain the prerequisites for utilizing more than 1 NIC in this sort of configuration. To refresh your memory, those prerequisites are:

  • The vSwitch must be configured for “Route based on IP hash”
  • The physical NICs connected to the vSwitch as uplinks must all be configured as active in the failover order
  • The physical switch must be configured for link aggregation
  • There must be multiple, unique source-destination IP address pairs involved

The customer responded with a question (which I’m paraphrasing here): “That’s all? It will just automatically use more than one link?”

Well…sort of.

There is one little caveat. Cisco IOS uses a hashing algorithm to determine which link a particular traffic flow between a source and destination will use. This algorithm is controlled by the port-channel load-balance command. Assuming that you’re using source-destination IP hashing, that means the Cisco switch will use a hash of the source IP address and the destination IP address to determine which link it will use. This page has more detailed information.

It’s theoretically possible, based on the number of links in the port channel, that some traffic flows between different pairs of source-destination IP addresses might end up on the same link. That means it’s not necessarily just as simple as setting up multiple NFS exports or iSCSI targets on different IP addresses—you also need to know if the IP addresses you are using will actually result in the traffic being distributed across the links.

How does one tell? Good question, and one I’m glad you asked. You can tell using this command (this command assumes you are using IP-based hashing):

switch# test etherchannel load-balance interface <Port channel interface> ip <Src IP Addr> <Dst IP Addr>

So, let’s say that you have an ESX/ESXi host with a VMkernel interface whose address is 172.16.5.10. Let’s say that you have a storage array (NetApp FAS, EMC Celerra, etc.) that supports NFS and you want to mount two different NFS exports on two different IP addresses so that traffic from this ESX/ESXi host to the storage array. You could use the test etherchannel load-balance command to help you determine which address could help ensure traffic distribution across the links:

switch# test etherchannel load-balance interface Po3 ip 172.16.5.10 172.16.5.100

For more examples of what the output would look like, take a look at this image. This was taken off a Cisco Catalyst 3560G running my test lab (and yes, the IP addresses have been changed to protect the innocent).

This would give you one way of testing whether your link aggregation configuration would actually use multiple links, or only a single link due to the IP hash calculation. Also, don’t forget that esxtop can also show you NIC utilization; here’s an example of both uplinks being used in this sort of configuration.

Unfortunately, what I can’t tell you right now is what algorithm the vSwitch itself uses to place traffic onto the uplinks. Does it follow the same sort of mechanism as the Cisco switch? I don’t know. If anyone has any information on that, it would be tremendously helpful.

If anyone has any other pertinent information or resources on this topic, please add them to the comments below.

UPDATE: Duncan Epping pointed out an article by Ken Cline from earlier this year provides the mechanism VMware uses to determine which uplink on a vSwitch will be used. This algorithm performs an XOR operation on the Least Significant Byte (LSB) of the source and destination IP addresses, then finds the modulus of that result and the number of uplinks. Thanks, Duncan and Ken!

Tags: , , , ,

I was reading this article by Eric Gray about CSV and followed his link to this ZDNet post about VMFS-3. I was a little taken aback.

One of the first questions that popped into my head was, “Why not use VMware Converter?” After all, it’s the tool that was expressly designed to do exactly what Jason Perlow was attempting to do in the DR exercise with his customer: import data back into a VMware vSphere environment.

Further—and I could be wrong here—but it seems to me that I’ve accessed USB storage devices from the Service Console before. Assuming that I’m correct (again, a big assumption, and one that I’ll test in the lab today), why not just use vmkfstools to import the VMDKs back into a VMFS datastore? And yes, Jason does provide a passing reference to ESXi, so perhaps he was using ESXi at the DR site and therefore didn’t have access to the Service Console.

I wasn’t there in the DR exercise, so there may have been other mitigating circumstances of which I am not aware. In addition, using VMware Converter wouldn’t have addressed the lengthy time taken to copy the data across the network which, as I understand it, was one of the primary complaints in Jason’s article.

In the end, though, why is Jason complaining about VMFS-3 and the time taken to restore a bunch of VMware vSphere virtual machines when the real root of the problem was the “consumer grade hard disks that you buy at the local Staples or Best Buy hooked up to some random Linux server”? He would have faced the exact same problem if he were trying to restore large Exchange databases from these commodity 1TB hard disks, or if he were trying to restore large Oracle or SQL databases from commodity 1TB hard disks. Does that mean Microsoft and Oracle are at fault as well? In this case, since VMware is the new “whipping boy” in the data center, VMware—specifically, VMFS-3—gets blamed.

I fail to see how better VMFS interoperability would have helped. Even if he’d been able to run VMs off USB-attached consumer grade hard disks, would you have wanted to? Then, VMware vSphere would have been blamed for “awful, horrible performance”. Like I said, VMware is the new whipping boy, and everything is VMware’s fault. Hey, at least the networking guys are happy—it’s no longer the network’s fault!

And while this wouldn’t have helped in Jason’s situation (he was trying to get data into VMFS, not get data out of VMFS), there is this open source VMFS driver available. So VMFS-3 is as much the “black box” that he lets on, in my opinion.

Let me know what you think: am I way off here?

Tags: , , ,

Learning About EMC Storage

Despite repeated incidences of being called a “storage expert,” I remain firmly convinced that I am far from being a storage expert. Case in point: I really don’t know that much about EMC storage, how to configure it, the caveats/gotchas/pitfalls when you’re setting it up, etc. Fortunately, I’ve recently been given the opportunity to spend some hands-on time with an EMC Celerra. Due to the Celerra’s architecture, this also means that I get some hands-on time with the EMC CLARiiON CX4 that sits behind the Celerra.

So far, my hands-on time has been fairly limited, but I expect that will all be changing very soon. I can tell you that I’m no fan of the web-based Celerra management interface. The web-based Navisphere 6 interface for the CX4 is better, but I haven’t tried it on a non-Windows system yet. I suspect I won’t be quite so happy when I try it on some other system. (Will it even run on a non-Windows platform?) I guess I’ll be learning the command-line interface (CLI)…

So, here’s a request to my readers: what resources have you found useful in learning how to work with EMC storage products? I’d welcome any and all feedback on useful white papers, blogs, free training, etc. Thanks in advance!

Tags: ,

VMworld is always a very busy time of year. Press releases, product announcements, new product or technology releases, companies emerging from stealth mode—it all happens around VMworld. Now that I’m back home again from VMworld, I thought I’d clear out my Inbox from all the various VMworld-related news items. Perhaps you’ll find something interesting or useful here!

VMware and HP Unveil Solution to Simplify Datacenter Management
World’s Leading Service Providers Build New Cloud Services on the VMware Platform
Altor VF 3.0 Meets VMware’s VMsafe Certification Requirements (no hyperlink available)
VMware Submits VMware vCloud API Specification to the Distributed Management Task Force (DMTF) — First Ever Submission of Key Cloud Interface
VMware Leads in Virtual Desktops With VMware View(TM) — Simplifying Desktop Management, Lowering Costs and Enriching User Experience
Cisco and VMware Validated Architecture for Long Distance VMotion
VMworld 2009 Hello Freedom videos (this one is funny)
VMworld 2009 Virtual Infrastructure Design – Lab Manager vPODS Enable Conference Cloud
VMworld 2009: VMware, Cisco, and EMC Super-Session (SS5240)
VMworld 2009: Best Practices for Multipathing in VI3.5 and vSphere (TA2467)
VMworld: Is it a scalability issue to run drivers in the Hyper-V parent partition? (Answer: No)

That’s it this time around. I promise that some original content is coming soon…

Tags: , , ,

There is no Internet connectivity in this session, so I’ll have to publish this after the session has concluded.

The Cisco Nexus 1000V is, of course, a Layer 2 distributed virtual switch for VMware vSphere built on Cisco NX-OS (the same operating system that drives the physical Nexus switches). It’s compatible with all switching platforms, meaning that it doesn’t require physical Nexus switches upstream in order to work. The Nexus 1000V brings policy-based VM connectivity, network and security property mobility, and a non-disruptive operational model.

The Nexus 1000V has two components: the Virtual Supervisor Module (VSM). Interestingly enough, the slide shows that the VSM can be a virtual or physical instance of NX-OS; there has been no formal announcement of which I know that has discussed using a physical instance of NX-OS as the VSM for the Nexus 1000V. The second component is the Virtual Ethernet Module (VEM), which is a per-host switching module that resides on each ESX/ESXi host. A VSM can support up to 64 VEMs in a distributed logical switch model, meaning that all VEMs are centrally managed by the VSM. Each VEM appears as a remote line card to the VSM.

The VEM is deployed using vCenter Update Manager (VUM) and supports both ESX and ESXi. The Nexus 1000V supports both 1Gbps and 10Gbps Ethernet uplinks and works with all types of servers (everything on the HCL) and upstream switches.

The Nexus 1000V supports a feature called virtual port channel host mode (vPC-HM). This feature allows the Nexus 1000V to use two uplinks (NICs in the server) connected to two different physical switches and treat them as a single logical uplink. This does not require any upstream switch support. Multiple instances of vPC-HM can be used; for example, you could use four Gigabit Ethernet uplinks, two to each physical switches, could be used to create two different vPC-HM uplinks for redundancy and separation of traffic.

For upstream switches that support VSS or VBS, you can configure the Nexus 1000V to use all uplinks as a single logical uplink. This requires upstream switch support but provides more bandwidth across all upstream switches. Of course, users can also create multiple port channels to upstream switches for traffic separation. There are lots of flexiblity in how the Nexus 1000V can be connected to the existing network infrastructure.

These network designs can be extrapolated to six NICs (uplinks), eight NICs, and more.

One interesting statement from the presenter was that Layer 8 (the Human layer) can create more problems than Layers 1 through 7.

Next, the presenter went through the use and configuration of the Cisco Nexus 1000V in DMZ environments. Key features for this use case include private VLANs (private VLANs can span both physical and virtual systems). Network professionals can also use access-conrol lists (ACLs) and remote port mirroring (ERSPAN) improve visibility and control over the virtual networking environment.

At this point, I left the session because it was clear that this session was more about educating users on the features of the Nexus 1000V and not about best practices on how to deploy the Nexus 1000V.

Tags: , , , , , ,

Earlier I posted some notes on meetings I’d had with Virsto and Xangati. In this post I’d like to discuss some additional meetings I’ve had with Virtual Instruments and Tranxition.

Virtual Instruments

Virtual Instruments makes a solution that is intended to help troubleshoot and optimize storage environments. I had the opportunity to grab some coffee with them this morning and hear about what they’re doing and how they’re doing it. As a company carved out of Finisar and taken private, their goal is to help drive higher levels of virtualization by providing more visibility into the storage fabric.

Clearly, this message will really only resonate with larger customers, and that is their target market: multiple hundreds of terabytes into the single petabyte range. At this scale, providing visibility into the thousands of virtual machines across hundreds of ESX/ESXi hosts attached to hundreds of Fibre Channel ports is almost impossible. Virtual Instruments tackles this with a multi-prong approach:

  • First, they use a SAN tap to plug into the Fibre Channel fabric and mirror traffic information to a collection device for analysis. If you’re a networking person, you can think of this as using a SPAN port to mirror traffic. This is done on the storage side to reduce the scale due to fan in-fan out ratios.
  • Second, they gather SNMP information from the Fibre Channel switches. This enables visibility at the switch level.
  • Third and finally, Virtual Instruments collects information from VMware vCenter Server. This information provides the final piece necessary to correlate per-host and per-VM traffic to the information being gathered by the fabric taps and the switch monitoring.

What this allows Virtual Instruments to do is to feed information back to vCenter Server to enable I/O-based recommendations for VM movement. It also enables visibility into path utilization so that multipathing information can be configured for optimal performance. Finally, more detailed storage information is exposed that enables organizations to more effectively place VM storage on Tier 1, Tier 2, or Tier 3 according to its storage needs. In some cases, in fact, money saved on buying additional Tier 1 storage can more than pay for an implementation of Virtual Instruments.

Overall, this is very interesting soltuion, albeit limited in scope to larger environments. If this describes your organization, though, it may definitely be worth a closer look.

Tranxition

Tranxition makes software to do “personality virtualization.” Apparently they’ve been around since 1998 and are just now becoming more visible, creating a partner program, and starting to expand coverage. Their key product is Adaptive Persona, which some have said can be called “Softricity for user personality data”. The product seems to work a lot like ThinApp in that it creates a virtual file system and virtual Registry that captures all user personality data. This user personality data, which can reside either inside or outside the traditional user profile file system structure, is then continuously streamed back to a central server. When a user logs off, whatever data has not been synchronized to the server is then copied up to the server, and the local system is scrubbed of user personality data. Then, when that same user logs on to a different system, Tranxition streams down only those portions of the user personality that are needed at that moment. All other data is fetched “on demand”. This helps speed up the logon process by decoupling the size of the profile from the time required to log on.

Overall, I was fairly impressed with the product. They seem to have done a reasonably good job of taking the principles behind application virtualization and applied them to user personality management. If anyone has any additional feedback on Tranxition (vendors, please disclose yourselves!), I’d love to hear it in the comments.

Tags: , , , ,

I’ve had the opportunity to speak with a few different vendors over the last couple of days here in San Francisco at VMworld 2009. Here are some notes on my meetings.

Virsto

My first meeting of the week was with Virsto, a early storage startup (they just closed Series A funding in the last few weeks). Virsto is led by some long-time storage professionals from companies such as StorageTek, Veritas, and others.

Virsto is unique, to me, in that they have an interesting view of the storage component. I met with Alex, one of the founders, and he used a term that I found quite illustrative and useful: “the I/O blender”. This is the term he applied to the effect that the hypervisor has on I/O as it moves from the virtual server to the physical server to the storage layer. If you think about it, it makes sense: I/O from each virtual server has to be multiplexed onto the same HBAs as the I/O from every other virtual server. The end result is, of course, that the storage array ends up having to deal with small, random I/O workloads instead of large, sequential workloads. This impairs performance.

The Virsto solution combines a software portion that is currently architected only for Microsoft Hyper-V. Virsto’s software component illustrates both the strength and the weakness of Hyper-V’s indirect I/O model. It’s a strength in that it’s very easy to write a filter driver to run in the management partition to modify VM I/O; the weakness is that it’s really easy to write a filter driver to run in the management partition to modify VM I/O. I’m being partially facetious here, but I hope you get the point. In any case, what Virsto’s software layer does is help undo the I/O blender effect by working in conjunction with a storage staging layer. Typically this would be some sort of high-speed local storage, such as an SSD. As a result of the software working in conjunction with the hardware, Virsto can “re-assemble” I/O into workloads that are better suited for performance at the physical layer and thus undo the I/O blender effect.

Virsto’s solution also allows for some forms of storage virtualization, in that different types of underlying block storage can be combined and managed by Virsto. Virsto’s solution also offers snapshots (checkpoints), the ability to split data streams for replication, and better support for disk-to-disk (D2D) backups via their snapshots.

My biggest concern with Virsto is that they are competing in a space with lots of very large, very well-funded organizations that are laser-focused on making their storage work extremely well with VMware vSphere and virtual environments, including Hyper-V. Think of NetApp integration with Hyper-V, or EMC integration with Hyper-V (remember that Virsto supports only Hyper-V at this time). These companies have lots of development talent, lots of money, and an established presence. I fear it will be difficult for Virsto to really gain a foothold in that space.

Xangati

Xangati (pronounced “zan-gotti”) is an application performance solution. I had a spirited discussion with the Xangati folks about what differentiates them versus other solutions like AppSpeed, BlueStripe, etc., in that Xangati relies upon network traffic information to measure application performance. In Xangati’s case, they rely upon NetFlow (or the various vendor-specific implementations of NetFlow). At first, I found this a bit limiting because I wasn’t aware that NetFlow v5 was supported in ESX 3.5 on vNetwork Standard Switches (I know, this is probably something everyone knows). But it indeed is (see here); the real question is whether it will continue to be supported on vNetwork Standard Switches on vSphere. In any case, Xangati insists that using NetFlow to gather network information is very different (and yields different results) than performing packet analysis. I must admit that I don’t fully see the difference; perhaps a network guru can explain it?

Having stated all that, what Xangati does it pretty interesting. It requires that you enable NetFlow throughout the environment—on both virtual and physical switches and other physical network equipment—and then allows you to see end-to-end network usage on an application-by-application basis. From there, Xangati allows the organization to take “network recordings” of the network behavior and then replay that recording later. Different views can be created for different roles within the organization, allowing IT pros to see only the information they need or want to see.

I’ll go back to my earlier statements and say that while Xangati offers some unique functionality—such as the ability for an end-user to initiate a network recording and submit that as a “Visual Trouble Ticket” to the help desk—I’m still at a loss to explain how, in the end, they are different from AppSpeed, BlueStripe, and others who provide end-to-end application performance and correlation. Yes, they use a different way (perhaps a superior way) of gathering information, but what the customer ultimately wants to know is this: “Which part is slow?” It seems that there are other, more well-established solutions already on the market that are trying to address this. Whether or not Xangati is successful in the space is yet to be seen, but I do wish them the best of luck!

I also met with Virtual Instruments and had a great discussion with them, but as I’m running a bit short on time I’ll have to do their write-up later on. Check back here later for the write-up on Virtual Instruments.

Tags: , , ,

« Older entries