A Comment Policy Reminder

I encourage open discussion and conversation here on my site, and I’m thrilled that readers feel welcome to share their viewpoints (even when those viewpoints differ from my own). To help foster this sense of free discourse, there are two rules upon which I insist for all comments:

  1. First, all comments should be courteous. There’s no reason to personally attack another reader or author—simply state your position, why that is your position, the facts you feel support your position, etc. Leave the personal attacks somewhere else.
  2. Second, all commenters should provide full disclosure. This helps avoid even the appearance of wrongdoing. Where a vendor’s products helps to address readers’ needs, I don’t mind a vendor mentioning their products. That vendor just needs to be sure to provide full disclosure. If you have a business relationship with an organization, disclose that. Be transparent and provide full disclosure.

Recently, I’ve had one commenter leave a series of comments on the site that blatantly and bluntly promote his employer’s products. Unfortunately, this commenter has failed to provide full disclosure. For that reason, I’ve been simply deleting this commenter’s comments. And I’m going to continue to delete this commenter’s blatant, outright comment spam as long as he/she refuses to provide full disclosure. Other readers deserve the right to know why a commenter is pushing a particular product or feature!

Tags: ,

Author’s Note: This content was first published over at Storage Monkeys, but it appears that it has since disappeared and is no longer available. For that reason, I’m republishing it here (with minor edits). Where applicable, I’ll also be republishing other old content from that site in the coming weeks. Thanks!

In this article, I’m going to tackle what will probably be a sensitive topic for some readers: VMware over NFS. All across the Internet, I run into article after article after article that sings the praises of NFS for VMware. Consider some of the following examples:

That first link looks to be mostly a reprint of this blog post by Nick Triantos. Now, Nick is a solid storage engineer; there is no question in my mind that he knows Fibre Channel, iSCSI, and NFS inside out. Nick is certainly someone who is more than qualified to speak to the validity of using NFS for VMware storage. But…

I am going to have to disagree with some of the statements that are being propagated about NFS for VMware storage. Is NFS for VMware environments a valid choice? Yes, absolutely. However, there are some myths about NFS for VMware storage that need to be addressed.

  1. Myth #1: All VMDKs are thin provisioned by default with NFS, and that saves significant amounts of storage space. That’s true—to a certain point. What I pointed out back in March of 2008, though, was that these VMDKs are only thin provisioned at the beginning. What does that mean? Perform a Storage VMotion operation to move those VMDKs from one NFS datastore to a different NFS datastore, and the VMDK will inflate to become a thick provisioned file. Clone another VM from the VM with the thin provisioned disks, and you’ll find that the cloned VM has thick VMDKs. That’s right—the only way to get those thin provisioned VMDKs is to create all your VMs from scratch. Is that what you really want to do? (Note: VMware vSphere now supports thin provisioned VMDKs on all storage platforms, and corrects the issues with thin provisioned VMDKs inflating due to a Storage VMotion or cloning operation, so this point is somewhat dated.)
  2. Myth #2: NFS uses Ethernet as the transport, so I can just add more network connections to scale the bandwidth. Well, not exactly. Yes, it is possible to add Ethernet links and get more bandwidth. However, you’ll have to deal with a whole list of issues: link aggregation/802.3ad, physical switch redundancy (which is further complicated when you want to use link aggregation/802.3ad), multiple IP addresses on the NFS server(s), multiple VMkernel ports on the VMware ESX servers, and multiple IP subnets. Let’s just say that scaling NFS bandwidth with VMware ESX isn’t as straightforward as it may seem. This article I wrote back in July of 2008 may help shed some light on the particulars that are involved when it comes to ESX and NIC utilization.
  3. Myth #3: Performance over NFS is better than Fibre Channel or iSCSI. Based on this technical report by NetApp—no doubt one of the biggest proponents of NFS for VMware storage—NFS performance trails Fibre Channel, although by less than 10%. So, performance is comparable in almost all cases, and the difference is small enough not to be noticeable. The numbers do not, however, indicate that NFS is better than Fibre Channel. You can read my views on this storage protocol comparison at my site. By the way, also check the comments; you’ll see that the results in the technical report were independently verified by VMware as well. Based on this information, someone could certainly say that NFS performance is perfectly reasonable, but one could not say that NFS performance is better than Fibre Channel.

Now, one might look at this article and say, “Scott, you hate NFS!” No, actually, I like using NFS for VMware Infrastructure implementations, and here’s why:

  • Provisioning is a breeze. It’s dead simple to add NFS datastores.
  • You can easily (depending upon the storage platform) increase or decrease the size of NFS datastores. Try decreasing the size of a VMFS datastore and see what happens!
  • You don’t need to deal with the complexity of a Fibre Channel fabric, switches, WWNs, zones, ISLs, and all that. Now, there is some complexity involved (see Myth #2 above), but it’s generally easier than Fibre Channel. Unless you’re a Fibre Channel expert, of course…

So there are some tangible benefits to using NFS for VMware Infrastructure. But let’s be real about this, and not try to gloss over technical details. While NFS has some real advantages, it also has some real disadvantages, and organizations choosing a storage protocol need to understand both sides of the coin.

Tags: , , ,

This is another one of my “thinking out loud” posts. This time, the question I’m mulling is this one: why deploy FCoE?

I haven’t hid the fact that I’m not really a fan of FCoE (see here or here), but I was starting to warm to the technology and thought that I was beginning to see some benefits to deploying FCoE. Namely, the fact that FCoE is inherently very compatible with “traditional” FCP, allowing organizations to leverage their existing FCP installation while transitioning to FCoE. Some hands-on time I’d recently spent with a Cisco Nexus 5000 switch showed me just how closely aligned the two technologies are and how (relatively) easy it was to extend an FC fabric using FCoE. OK, I think I get this.

Then, a few days ago, I read this article on FCoE divergence. Given that The Register can sometimes be quite sensationalist (and that’s putting it mildly), I contacted a colleague of mine whose input and knowledge I trust. He informed me that FCoE was currently limited in that FCoE is not multi-hop enabled—meaning, you can’t connect FCoE initiators on one switch to FCoE targets on another switch. (Apparently, this shortcoming is due to be corrected shortly.)

Whoa! That’s a limitation of which I was not aware. And with that limitation in mind, knowing that FCoE will—for the time being at least—be limited to convergence at the edge, I have to ask: why deploy FCoE at all? What real and specific benefits does an organization seek to gain by deploying FCoE as opposed to just deploying FC? Is the edge convergence really that worthwhile and valuable?

Tags: , , ,

I was going through my list of actions in OmniFocus, looking at my projects and actions and evaluating each of them. In my “Potential Posts” project, where I keep links to articles that I might use in a blog post, I found the URL for this article by Steve Kaplan about virtualization, Cisco Nexus, and blade servers. The basic idea of his article is that virtualization and the Cisco Nexus—specifically, the unified fabric—are going to combine to kill blade servers.

I do agree with Steve that there is no innate relationship that means running VMware on blades is somehow “automagically” better:

It is amazing how frequently we hear IT managers talk about deploying blade servers as an integral component of their new virtual infrastructures - as if there were an obvious synergy between VMware and blade server architectures.

Absolutely! Blades are an option, just like rack mounted servers, and it’s up to the customer to choose (or us as consultants to recommend) the form factor that best meets the business needs. It might be blade servers, or it might be rack mounted servers. It just depends. So, on this one point, I agree with Steve.

Yet, at the same time, I also disagree with this point that Steve makes in his article:

Blade servers have always been an impediment to an optimal virtual infrastructure because they introduce limitations in efficiently utilizing power and cooling resources, budget, flexibility, manageability, bios and firmware updates, performance and troubleshooting.

Here is where Steve and I start to disagree. In fact, this specific article was something of the catalyst for a series of posts, written by colleague and friend Aaron Delp, detailing how blade servers and virtualization work well together:

Blades and Virtualization Aren’t Mutually Exclusive: Part One, HP Power Sizing
Blades and Virtualization Aren’t Mutually Exclusive: Part Two, IBM Power Sizing
Blades and Virtualization Aren’t Mutually Exclusive: Part Three, IBM Traditional Expansion Options
Blades and Virtualization Aren’t Mutually Exclusive: Part Four, HP Traditional Expansion Options

While this series of articles doesn’t squarely address all of the arguments against blades and virtualization, the series does make it clear that blades can produce power savings vs. rack mounted servers, and that blades do offer enough expansion options to accommodate the majority of virtualization deployments.

I also disagree with Steve about the value of the unified fabric, especially considering that right now unified fabric can exist only at the edge of the network and not at the core. That being the case, I find it hard to say that unified fabric is going to kill blade servers. So, again, I have to disagree with Steve’s position.

However, Steve’s not entirely wrong—virtualization, FCoE and 10Gb Ethernet, and yes even unified fabric will change how blade servers are designed and deployed. Cisco’s Unified Computing System (UCS) is one example of how blade servers are going to adapt to these agents of change, and I believe we’ll see more examples from other leading vendors in the coming months and years. But will blades die away entirely? No, I don’t think so.

Think I’m crazy? Think I’m out of my mind? Feel free to speak up in the comments—courteous comments are always welcome.

Tags: , , , ,

Reader Kyle Ross shared with me a potential issue with VMware’s new backup product, VMware Data Recovery. Others within the VMware blogging scene have also covered this, but I wanted to mention it as well so that others didn’t run into the problem. Here’s Kyle’s write-up:

I was made aware of a serious (in my opinion) bug with VDR during a call with VMware support that I haven’t seen discussed anywhere. This is an internally known issue that causes snapshots to build up on VM’s that are members of VDR backup jobs.

During the backup process a new snapshot is created and VDR updates the snapshot descriptor file (vm_name-000001.vmdk) to mark the snapshot as un-removable. The bug is introduced when the backup process completes, it fails to mark the snapshot as removable causing them to remain.

The tricky part of the problem is that the snapshots are not visible through the vSphere Client, nor are they listed in apps like ‘RVTools’ that use the VMware CLI to gather data. They could potentially be listed in the new datastore views but I didn’t think to look there before I resolved it in my environment. I ran across them by logging into the service console and running the following command to list all the delta files on the datastores attached to the server.

find /vmfs/volumes/ -name \*delta\*

In my environment I noticed numerous VM’s with multi-gigabyte delta files that I couldn’t account for via snapshots listed in the GUI. Here is the solution I was given by VMware. Via the service console, browse to the location of the VMDK files for the affected VM. Run this command to identify the descriptors that need to be corrected, replacing ‘virtual_machine_name’ with the actual name of the VM.

grep –I ddb.dele *virtual_machine_name*-000???.vmdk

This command will quickly identify the delta files that are marked as non-deletable. The workaround is to edit the affected VMDK descriptor files and change “ddb.deletable” from “false” to “true”. You will probably also need to edit the root VMDK file and change this field as well, otherwise you may be left with one open snapshot. Note that due to a change in how ESX 4 performs file locking, you will probably need to SSH into the host that is currently running the VM to edit these files. Once you have edited all the files, create a new snapshot for the VM either via the GUI or command line. Then issue the “Delete All” snapshots command to force ESX to combine all the files and close all the visible and hidden snapshots.

As soon as more information is available, I’ll post it here. If any other readers have more information to share, please speak up in the comments.

Tags: , , ,

You know, it’s really irritating when you pour your heart and soul into something, only to find someone else riding your coattails and leeching off your efforts. It would appear that NetworkVirtualization.com is one such leech.

I have no problem with other sites syndicating my content as long as proper attribution of the original author and original site is provided. Do me a favor: visit some of the URLs below (I’m not going to hyperlink them and give the site a traffic boost) and tell me how any of the examples I’ve listed below provide proper attribution of the original author and the original site:

http://networkvirtualization.com/content/unified-fabric-inevitability
http://networkvirtualization.com/content/vmware-io-queues-micro-bursting-and-multipathing
http://networkvirtualization.com/content/tap-vsphere-pvscsi-performance-separate-vm-boot-and-data-drives

Let’s see…content from my site, Chad Sakac’s site, and Rich Brambley’s site, all syndicated on their site without any clear attribution back to the original post—except for a very small link near the bottom of the article. If you hadn’t been looking for that link, or if I hadn’t told you that the articles above were written by me, Chad, and Rich, respectively, would you have known? And those are just the authors I recognized! How many more are there that I don’t recognize?

To whomever is running NetworkVirtualization.com: if you are going to syndicate content, you need to provide proper attribution. Otherwise, taking someone else’s content and allowing people to believe that it’s yours is called plagiarism, and it’s wrong.

Tags: ,

Here’s Virtualization Short Take #27, a collection of news, tidbits, thoughts, articles, and useless trivia I’ve gathered over the last week or so. Perhaps you’ll find a diamond in the rough among these items!

  • Interested in more information on how it is, exactly, that Cisco is going to provide so much memory in their UCS blades and rack mount servers to make them ideal virtualization hosts? This article from CommsDesign and this “Catalina” article by Rodos Haywood both provide some great information on how Cisco is working around the Intel reference architecture limitations introduced with the Xeon 5500 and Quick Path Interconnect (QPI).
  • This article provides a handy reference on how to unregister the Nexus 1000V vCenter Server plug-in. I wish I’d known this information several weeks ago…
  • Need to view some configuration files on an ESX host? Just browse to http://<IP address of ESX server>/host and you’re all set. I learned of this handy little trick via Virtual Foundry.
  • And speaking of handy little tips, here’s one Eric Sloof shared regarding the vCenter Ops Dashboard. Again, just browse over to http://<IP address of vCenter Server>/vod/index.html to view the vCenter Ops Dashboard.
  • Adam Leventhal describes using the latest version of VirtualBox—which now includes OVF support and host-only networking—to run the Sun Storage 7000 Simulator. This is pretty cool stuff. I hope Oracle doesn’t kill it like Virtual Iron…
  • I just mentioned Virtual Foundry a bit ago, but forgot to mention this great post on hardening the VMX file. Good stuff.
  • For others who are, like myself, pursuing the elusive VMware Certified Design Expert (VCDX) certification, Duncan’s recent post describing the VCDX design defense is a great resource. Thanks, Duncan!
  • The VMware networking team addresses some questions around using VMware for virtualized DMZs, and how to protect against Layer 2 attacks.
  • Want to do manual linked clones in VMware Fusion? Here’s how.
  • Via Matt Hensley, I found this VIOPS document on configuring a VMware vCenter Data Recovery dedupe store.
  • This article has more information on installing ESXi 4.0 to a flash drive, a process I have yet to try. (Instructions for burning ESXi 3.5 to a flash drive can be found here.) Anyone else done it yet? I’d be interested in how it went.
  • If you have any questions about SAN multipathing, Brent Ozar’s two part series on the topic may help straighten things out (here’s Part 1 and Part 2). I’m not sure that I agree with Brent’s statement regarding the ability of desktop-class SATA drives to saturate 4Gbps Fibre Channel, but I’m no storage expert so I could very well be wrong.
  • VMware SE and friend Aaron Sweemer provides a handy script that can help fix Service Console networking when performing automated builds of VMware ESX.

That wraps it up for this edition of Virtualization Short Takes. Feel free to share thoughts, questions, or corrections in the comments, and thanks for reading!

Tags: , , , , , ,

VMware has completed Microsoft Server Virtualization Validation Program (SVVP) certifications for both VMware vSphere 4.0 as well as ESX/ESXi 3.5 Update 4. This brings the list of SVVP-certified products to include:

  • VMware vSphere 4.0 (ESX 4.0 and ESXi 4.0)
  • VMware ESX 3.5 Update 4
  • VMware ESXi 3.5 Update 4
  • VMware ESX 3.5 Update 3
  • VMware ESXi 3.5 Update 3
  • VMware ESX 3.5 Update 2

According to my contacts within VMware—and many of you have probably heard the same—the company is seeking to achieve SVVP certification for every ESX and ESXi release from 3.5 Update 2 onward for the maximum supported configuration of both CPUs and RAM on both Intel and AMD platforms. That includes both 32-bit and 64-bit versions of Windows Server.

You can view the full list of SVVP-certified platforms here.

Tags: , , , ,

A lot of the content on this site is oriented toward VMware ESX/ESXi users who have a pretty fair amount of experience. As I was working with some customers today, though, I realized that there really isn’t much content on this site for new users. That’s about to change. As the first in a series of posts, here’s some new user information on creating vSwitches and port groups in VMware ESX using the command-line interface (CLI).

For new users who are seeking a thorough explanation of how VMware ESX networking functions, I’ll recommend a series of articles by Ken Cline titled The Great vSwitch Debate. Ken goes into a great level of detail. Go read that, then you can come back here.

Before I get started it’s important to understand that, for the most part, the information in this article applies only to VMware ESX. VMware ESXi doesn’t have a Linux-based Service Console like VMware ESX, and therefore doesn’t have a readily-accessible CLI from which to run these sorts of commands. There is a remote CLI available, which I’ll discuss in a future post, but for now I’ll focus only on VMware ESX.

The majority of all the networking configuration you will need to perform on VMware ESX boils down to just a couple commands:

  • esxcfg-vswitch: You will use this command to manipulate virtual switches (vSwitches) and port groups.
  • esxcfg-nics: You will use this command to view (and potentially manipulate) the physical network interface cards (NICs) in the VMware ESX host.

Configuring VMware ESX networking boils down to a couple basic tasks:

  1. Creating, configuring, and deleting vSwitches
  2. Creating, configuring, and deleting port groups

I’ll start with creating, configuring, and deleting vSwitches.

Creating, Configuring, and Deleting vSwitches

You’ll primarily use the esxcfg-vswitch command for the majority of these tasks. Unless I specifically indicate otherwise, all the commands, parameters, and arguments are case-sensitive.

To create a vSwitch, use this command:

esxcfg-vswitch -a <vSwitch Name>

To link a physical NIC to a vSwitch—which is necessary in order for the vSwitch to pass traffic onto the physical network or to receive traffic from the physical network—use this command:

esxcfg-vswitch -L <Physical NIC> <vSwitch Name>

In the event you don’t have information on the physical NICs, you can use this command to list the physical NICs:

esxcfg-nics -l (lowercase L)

Conversely, if you need to unlink (remove) a physical NIC from a vSwitch, use this command:

esxcfg-vswitch -U <Physical NIC> <vSwitch Name>

To change the Maximum Transmission Unit (MTU) size on a vSwitch, use this command:

esxcfg-vswitch -m <MTU size> <vSwitch Name>

To delete a vSwitch, use this command:

esxcfg-vswitch -d <vSwitch Name>

Creating, Configuring, and Deleting Port Groups

As with virtual switches, the esxcfg-vswitch is the command you will use to work with port groups. Once again, unless I specifically indicate otherwise, all the commands, parameters, and arguments are case-sensitive.

To create a port group, use this command:

esxcfg-vswitch -A <Port Group Name> <vSwitch Name>

To set the VLAN ID for a port group, use this command:

esxcfg-vswitch -v <VLAN ID> -p <Port Group Name> <vSwitch Name>

To delete a port group, use this command:

esxcfg-vswitch -D <Port Group Name> <vSwitch Name>

To view the current list of vSwitches, port groups, and uplinks, use this command:

esxcfg-vswitch -l (lowercase L)

There are more networking-related tasks that you can perform from the CLI, but for a new user these commands should handle the lion’s share of all the networking configuration. Good luck!

Tags: , , , ,

vSphere Book News

There’s good news on the book front. My book, Mastering VMware vSphere 4, is into the production phase. All the writing is complete, edits are complete or very nearly complete, and things are winding down. It’s been a crazy road, but one that I’m glad I took. So what’s the good news? The book will be available at VMworld!

Yes, that’s right—you’ll be able to buy Mastering VMware vSphere 4 at VMworld 2009 in San Francisco at the conference bookstore. I’m sure that also means there will be a book signing (by yours truly, of course!), although the details for that have not yet been determined.

But that’s not all, I have more good news! If you can’t wait until VMworld to order your copy, you can pre-order your copy via Amazon, Barnes & Noble, and Borders. In addition, if you use the code D3A7V3V you should be able to get 20% of the book at Barnes & Noble. I had an online coupon for Borders, too, but it expired. Sorry.

So, if you haven’t already pre-ordered your copy, feel free to use one of the links above, or you can wait and pick up your copy in San Francisco at the VMworld 2009 conference bookstore.

Tags: , , ,

« Older entries