blog.scottlowe.org

This session is titled “Best Practices for Architecting VMware Consolidated Backup Enabled Solutions,” and it’s being led by a couple of different presenters, one of whom is Dan Anderson, who also led the VCB session on Partner Day and leads the VCB labs (and led the VCB labs last year at VMworld 2006).  So far, it looks like this session won’t be a repeat of Monday’s Partner Day session, which is good.

The session started out with a review of VCB architecture, VCB components, and the interplay between the various portions of the VCB solution—the VCB proxy server, the pre-freeze and post-thaw scripts, file-level backups versus full VM backups, etc.  This stuff I already knew and had already been covered in detail in previous sessions.

The discussion then moved into a mention of the various items that affect the design of a VCB solution.  This would include things like the SAN architecture (Fibre Channel vs. iSCSI), software components (VCB version, VC version, ESX version, third-party backup software version), recovery mechanism, backup types, etc.  Dan mentioned again the command-line interface for VMware Converter that was mentioned on Monday; I really need to dig up that information so that I can explore that possibility in greater detail.

It was stated again today that there are bad candidates for VMware snapshots—high I/O, high transaction VMs are bad candidates for snapshots, and therefore are bad candidates for use with VCB.  (Recall that VCB uses snapshots to unlock the base VMDK for use by the VCB proxy, so a VM that is a bad candidate for snapshots is therefore a bad candidate for VCB-enabled backup solutions.)

I ended up leaving the session early because it turned out that a great deal more of the information that John and Dan were presenting was identical to the information that was presented on Monday at the partner session.  This is not a reflection on the presenters, or the session materials; I’d just already seen most, if not all, of the materials before.

I had the opportunity to attend three Partner Day sessions today, all three from the “Advanced In-Depth Technical Track.”

The first was this morning, and it was focused on an in-depth technical review of VMware Consolidated Backup.  I’ve written a few articles about VMware Consolidated Backup before:

• VMware Consolidated Backup
• Restoring VCB Full Backups with VMware Converter
• Strange VCB Error

The presenter was very knowledgeable; in fact, he was the one running the VCB labs at VMworld (and ran the labs at VMworld last year, which I attended).  I got some great information from the session, and when I have more time I’ll compile that information here.  Some of the key points I took away from the session included information on a command-line interface (CLI) for VMware Converter that allows for automated restores of VCB full VM backups using Converter (I’m really excited about looking into that one); good information on the minimal permissions needed for the user that logs into VirtualCenter and whose login information is hard-coded in config.js (this is a big security concern for many customers); and some RDM compatibility mode issues (RDM in virtual compatibility mode versus physical compatibility mode).  It was a great session; I thoroughly enjoyed it.

Unfortunately, the next session (the first session after lunch) was not nearly as useful.  Although included in the same in-depth track, there was a remarkable lack of technical information.  Fortunately, it was only 45 minutes long.

The final session of the day was on VMware’s VDI and ACE solutions, including a first look at VDM (Virtual Desktop Manager) 2.0, which VMware just publicly announced earlier today.  The presenter, Tommy Walker of VMware, was a great presenter and I enjoyed the presentation.  I’m hoping to be able to catch up with Tommy later this week to conduct some in-depth comparisons of VDM with other brokers, such as Leostream (a broker with which I’ve worked fairly extensively—and which, by the way, just released version 5.0 of their broker).

As soon as I have some additional time, I’ll try to post some additional information about these sessions and some of the in-depth technical details presented.

While in the process of verifying the operation of VMware Consolidated Backup (version 1.0.3) today using the command-line vcbMounter.exe utility, I kept receiving an error from vcbMounter and the full VM backups would fail.  Nothing seemed obvious at first, so I added the “-L 6” parameter to the command line, which was something like this:

vcbmounter -h vcserver.example.com -u username -p password
-r e:\mnt -a ipaddr:10.1.1.107 -t fullvm -L 6

Nothing terribly complicated there, just a simple full VM backup of the VM whose IP address is 10.1.1.107.  (For those of you that aren’t familiar with the vcbMounter.exe command-line syntax, it looks worse than it actually is.  Trust me.)  Upon running this command with the increased logging, I kept getting these errors:

[2007-08-07 12:13:47.418 ‘App’ 2144 warning] Could not
obtain inquiry page 128 for device on path 0, target 4, lun 0
 
[2007-08-07 12:13:47.418 ‘App’ 2144 warning] Sending SCSI inquiry
failed: Unknown error. (No proper error code was returned.)
 
[2007-08-07 12:13:47.418 ‘App’ 2144 warning] Could not
obtain inquiry page 128 for device on path 0, target 5, lun 0
 
[2007-08-07 12:13:47.418 ‘App’ 2144 warning] Sending SCSI inquiry
failed: Unknown error. (No proper error code was returned.)

The odd thing was, target ID 4 and target ID 5 were local SCSI targets, not anything SAN-related.  In fact, they were the system (C:) and data (D:) drives that had been created when the server was built and Windows Server 2003 was installed.

Google turned up nothing obvious, so I decided to try running the command directly against an ESX server.  The modified command now looked like this:

vcbmounter -h esxserver.example.com -u root -p password
-r e:\mnt -a ipaddr:10.1.1.107 -t fullvm -L 6

The operation still failed, but now I had a critical piece of missing information:

[2007-08-07 12:29:43.798 ‘BlockList’ 2052 error] Your VirtualCenter or the ESX server hosting the virtual machine you are dealing with needs to be upgraded to work with this version of VCB. (VCB attempted to invoke the method “acquireLeaseExt” on a remote object of type “vim.host.DiskManager”, but this method is unknown to this object type.)

Aha!  A quick review of the environment showed that the ESX host this particular VM was hosted on was indeed running version 3.0.1.  With a quick VMotion to a nearby host running ESX Server 3.0.2 and a repeat of the command (changed to target the new host, obviously), and the backup operation worked.  I moved the guest back to the original host again, and the operation failed again.  This pattern held true regardless of whether the vcbMounter.exe command targeted the VirtualCenter server (which was running version 2.0.2) or the ESX Server. Anytime the VM was hosted on the ESX server running 3.0.1, the command failed.

<aside>Now why didn’t the error message just say that the first time, instead of complaining about local SCSI disks?</aside>

A quick review of the VCB 1.0.3 release notes turns up this fairly brief blurb:

VMware Consolidated Backup (VCB) 1.0.3 is compatible with VirtualCenter 2.0.2 and ESX Server 3.0.2 (or newer) only. This release is not supported if used with older version of ESX Server and VirtualCenter.

OK, fair enough—I should have more closely read the release notes before getting too deep into the testing.  But what this means is that customers won’t be able to start using VCB 1.0.3 until all their hosts have been upgraded to ESX Server 3.0.2 and VirtualCenter has been upgraded to 2.0.2.  I don’t know at this time if VCB 1.0.2 will work against both the newer and older versions; if not, that will put organizations in a situations where they may end up with two different sets of VCB proxy servers: one set to support the hosts running ESX Server 3.0.1, and another set to support hosts running ESX Server 3.0.2.  And that doesn’t even take VirtualCenter into consideration!

Anyone out there testing VCB 1.0.2 against the newer releases of ESX and VC?  This will help tell us if we can leverage the existing VCB infrastructure until after all the hosts have been upgraded, and then upgrade VCB, or if a parallel VCB infrastructure will have to be established to support the newer of ESX and VC.

I think I’ve mentioned before that I use NetNewsWire for subscribing to various RSS and Atom feeds for various topics.  Of course, given my interest in virtualization, VMware and VMware-related tags are things that are included in my feed list, and quite often I come across a link from a tag search or del.icio.us subscription that I find interesting.  It may not justify a dedicated blog post about that particular item, though, because it may not be anything about which I can contribute useful information.  After all, there’s enough useless junk floating around the Internet as it is; why should I contribute to the problem?

Likewise, I typically won’t post a link to my del.icio.us bookmarks unless it’s something that I know I’ll need to reference again, and these tidbits of information sometimes aren’t useful reference articles.  So, instead of blogging about these or just posting them to del.icio.us, I thought I’d post the links here.  It seems like a reasonable compromise between the two, if you ask me.

• VMware: new patches available for ESX 3.0.1
• Getting VMFS datastore information with PowerShell
• Shutting down a Windows VM when the console is locked
• VMware: VCB & existing snapshots
• Virtual Mac Tool

There you go.  Enjoy!

When performing full VM backups with VMware Consolidated Backup (VCB), the end result of the backup operation is a copy of the VMs VMDK files, in 2GB blocks.  This format is similar to the format used by VMware Server and VMware Workstation, VMware’s hosted virtualization products.  Given that VMware Converter can convert VMs between ESX Server and the hosted virtualization products, I thought, “Why not use VMware Converter to restore VCB backups?”

So, with that question in mind, I set to see if it would work.  After conducting some tests, I have good news and bad news.  The bad news first:  it doesn’t work without some manual massaging of the VCB backup.  When you attempt to laod the VM’s files into VMware Converter, you receive an error message that VMware Converter “can’t determine the guest OS”.

Fortunately, the good news is that it isn’t hard to make it work.  Thanks to this VMTN forums thread, we can see that the process is actually pretty straightforward:

1. Edit the .vmx file to remove the “-NNNNNN” suffix that is automatically appended to the names of any VMDK files referenced in the .vmx file.  Make sure the VMDK filenames referenced in the .vmx file are indeed unique (they should be).
2. Rename the actual VMDK files by removing the “scsiN-N-N-” prefix from the filenames.  You’ll want to ensure that the filenames match the filenames referenced in the .vmx file.
3. Edit the VMDK index file (not the -sNNN files).  This is just a text file that references the rest of the files that comprise the virtual hard disk.  You’ll need to edit the filenames referenced in this file to ensure that they match the names of the actual “-sNNN” files on the disk.

That should be it.  After making those changes, VMware Converter should read the VM files without giving the “can’t determine guest OS” error, and then will let you select your final destination (which would typically be VirtualCenter).

Note that this will create a new VM, rather than restoring over the existing VM.  As a result, this may be most applicable when you need to recover individual files within a VM and aren’t performing file-level backups using VCB.  By using VMware Converter and creating a new VM, you can boot the VM up, get the files that are needed, then shut the restored VM down and blow it away.  This allows you to have the restore functionality of file-level backups but the speed of full VM backups.

VMware Consolidated Backup (VCB) is a new component of Virtual Infrastructure 3, and is designed to facilitate both full-VM and file-level backups of ESX-hosted virtual machines on a SAN.  In its current release, it only supports Fibre Channel SANs, but support for iSCSI is supposedly coming in the next release.  When used in conjunction with a third-party backup application (such as Backup Exec) and the appropriate integration software, VCB can provide the ability to backup VMs across the SAN (instead of across the network) without the need to install backup agents on every VM.  The speed of backups is pretty good, too.

We performed a number of backups with VCB during our tests:

• File-level backup of Windows-based guest with the guest OS running
• File-level backup of Windows-based guest with the VM powered off
• Full VM backup of Windows-based guest with the guest OS running

All of the preliminary VCB tests listed above were performed using vcbMounter, a command-line tool installed on the VCB proxy server.  The command we used looked something like this:

vcbmounter -h vcenter.example.com -u vcbservice
-u <password> -a ipaddr:10.1.1.100 -r E:VMmountVM1
-t file -m san

In the specific environment in which this testing was conducted, the hostname (of the VirtualCenter server, in this example) had to be changed from the default of 902, as the customer was using a non-default port number.  This threw us for a minute, until we could determine exactly what port on which the server was listening.

This mounted the contents of this VM’s virtual hard disks on the path E:\VMmount\VM1\letters\C, E:\VMmount\VM1\letters\D, etc.  We could then, of course, manually launch a backup of the files, but instead we continued with our preliminary testing and chose to wait on the Backup Exec testing until we were comfortable with vcbMounter.

This command worked, but in order to back up the virtual machine while it was shut down, we had to change the command slightly:

vcbmounter -h vcenter.example.com -u vcbservice
-u <password> -a name:VM1 -r E:VMmountVM1
-t file -m san

Here, the “name:VM1” parameter was the name of the VM that we wanted to back up exactly as it appears in VirtualCenter, including case.  (We did try this command using the same name with in different case, but it failed.)  We could also have used the BIOS UUID for the VM, which can be retrieved using this command:

vcbvmname -h vcenter.example.com -u vcbservice
-u <password> -s name:VM1

Again, the name has to match exactly what is listed in VirtualCenter.  One of the parameters returned by this command is the BIOS UUID, which you can then use in a vcbMounter command like this:

vcbmounter -h vcenter.example.com -u vcbservice
-u <password> -a uuid:<BIOS UUID> -r E:VMmountVM1
-t file -m san

By using the VM name or the VM BIOS UUID, we were able to make vcbMounter work both when the VM was running as well as when the VM was shutdown.  Using the VM’s IP address or DNS name, on the other hand, only worked when the VM was up and running.

Once we felt comfortable with vcbMounter and vcbVmName, we begain testing of the Backup Exec Integration Module (BEIM), a set of freely downloadable scripts designed to provide some automation between VCB and Backup Exec.  Although the syntax of Backup Exec’s integration scripts was a bit odd (and I think the documentation was incorrect in spots), the scripts worked well.  Using the supplied scripts, we were able to perform both file-level and full-VM backups of selected virtual machines without any manual intervention required (the scripts handled all the mounting/dismounting/etc.).  Watch out for spaces in the VCB path or in the VM names, though; they’ll cause the supplied scripts to fail.  This article offers some extensions to the Backup Exec scripts that help address some of the shortcomings, including correcting the problem with spaces in the path or the VM name.

While VCB does have its shortcomings, it’s still a very useful tool to have in your backup arsenal.  Between VCB full-VM and file-level backups and agent-assisted backups with an agent in the guest OS, there are plenty of ways to protect your virtualized servers.