February 2010

You are currently browsing the monthly archive for February 2010.

Congrats to HyTrust

Congratulations are in order for virtualization security company HyTrust, who within the last few days has had a flurry of activity. I’ve met HyTrust’s CEO several times (he’s a great guy, by the way) and I’ve followed the company since their early days; personally, I think it’s great that they’re seeing some success.

The first big piece of news was that HyTrust hired Jim Gannon, a former VMware executive, to serve as the VP of Sales. The full press release for that announcement is here.

The second big piece of news comes in the form of this press release announcing that HyTrust has secured $10.5 million in Series B financing, including an investment from Cisco Systems.

The third and final piece of news, and the one that I personally find most exciting, is that HyTrust has been named one of ten finalists for the “Most Innovative Company at RSA Conference 2010″ award. Congratulations on all three counts!

Tags: ,

A Potential Use for the iPad

There’s a lot of hype surrounding the Apple iPad. Some people are proclaiming it’s the end of traditional media like newspapers, magazines, and books. I’m not so sure about that, but I have found one potential use for the iPad that—for me, at least—might be compelling enough to make me go buy one later this year.

One task that I’m finding as a member of EMC’s vSpecialist team is that there is a lot of reading. We’re responsible for reading all sorts of documents. I don’t mind doing this in the evenings, when I’m not writing for one of my upcoming books or studying for a certification exam, but I’d really much rather prefer to do this in a way that makes it possible for me to be with my family. So, having some sort of device that would allow me to review documents while I’m sitting in the den with the kids would be great.

My thought is that I could leverage something like Dropbox to synchronize documents between my MacBook Pro and an iPad. With the documents easily accessible on (or from) the iPad, I could sit on the couch and read or review documents while the kids sit next to me and watch TV or read a book. This would help me stay on top of the document reviewing without pulling me into my office and away from the family.

What do you think? Good idea, or not? Anyone else have any uses for the iPad that you’d like to describe? Speak up in the comments.

Tags: ,

On the VCDX Defense

I’ve been thinking about how to write this post since last Friday afternoon after I completed my VCDX defense panel. Even now, a week later, I’m not sure that I have the right words to use.

After many long hours of preparation on the application and the submitted design, after days and weeks of waiting, after hours spent reviewing the design, and after reading numerous tips and tricks from established VCDXs, it all culminated in the defense panel. There, in the defense panel, I would have to stand before three knowledgeable, established design experts and defend the choices made in the design. I’d have to explain why I chose block storage over NFS, why Fibre Channel over iSCSI, why blade servers vs. rack-mount servers. I’d have to explain why I chose the LUN size I chose, and I’d have to defend the zoning that controlled the presentation of those LUNs. Clusters, cluster sizes, features enabled or not enabled, networking layout, VM density on the LUNs, projected IOPS—nothing was safe from their inquisition. And yes, I’d have to explain why the design used NetApp storage instead of EMC storage. (It was a customer requirement, i.e., a design constraint.)

Surprisingly, when the time for the VCDX defense panel arrived I found myself a lot more nervous than I had expected I would be. After all, this was just a friendly conversation with technical peers, right? In many ways it could be viewed that way, but the underlying purpose behind the conversation was ever-present: there was a reason I was there standing before these three people. It wasn’t just “shooting the breeze” with friends; there was a purpose there. It wasn’t just bouncing ideas off co-workers or industry colleagues; there was a reason for the conversation. It’s not that the panelists did anything to cause this feeling; they were completely fine, very courteous and quite friendly. (In case you’re wondering, I’m not going to disclose who was on my panel. They are welcome to disclose if they so choose, but that will be their decision.)

Looking back on it now, I realize that I should have gotten better control of my nerves. I spoke too quickly. I rushed through questions that probably deserved more explanation. I forgot details about my design. I got tripped up by relatively simple questions. I’ve made no secret of the fact that I wasn’t pleased by how well I performed—or didn’t perform—in the VCDX defense panel. I was upset that I had been thrown off and that I wasn’t able to recall all the details from my design. For a few hours after completing the defense panel, I beat myself up over how things had gone. But it didn’t take me too long to make peace with not having passed. I knew that if I had not passed, the experience was still worthwhile as a learning experience. Even if I hadn’t gained the VCDX certification, I’d still gained knowledge and experience. And hey, there was always another chance to defend at VMworld, right?

After returning home from Las Vegas, I spent the week thinking about what I would write after I’d finally gotten my results. I tried to prepare for the questions like “How in the world could Scott not pass?” I thought about explaining that the defense panel was only doing their job; they were preserving the value of the certification. After all, if the bar is not held high, what is the value of VCDX? All the while I secretly hoped that the result would be something other than what I was confident it would be.

And so it was that as I was driving my kids to a church youth group function tonight—after a long and unproductive day working with some rather stubborn equipment in the lab—that I received an e-mail from VMware. The first line of the message was this:

Congratulations! You have achieved the VCDX3 certification. Your VCDX number is: VCDX39

Unbelievable! I’d passed! I was so excited. I’d hoped for this result, but I honestly did not believe that I had managed to pull it off. I immediately called Crystal to tell her the news. I think she might have been even more excited than me.

Having now been through this entire process, what advice do I have for aspiring VCDX candidates?

  • As many others have stated, know your design. If I had only one thing to change about my entire process, this would be it. You should know it forward and backward: every detail, every choice, and every reason behind the design.
  • Don’t be too nervous. I allowed my nerves to get the best of me, there’s no question. I also don’t doubt that I would have done better had I not been so nervous. (As a side note, it’s interesting to me that I can stand up and speak in front of large crowds and not be nervous, but standing in front of those panelists really threw me. Odd.)
  • Understand the impact of your choices. As Duncan pointed out in this recent blog post, it’s really about the impact. Be prepared to discuss the reasons for the decisions in your design and the impact of the decisions in your design.

That’s it from the latest VCDX to join the ranks. I’ll post another update later with more tips and tricks that I learned from the experience, but those are some that jump to my mind immediately.

Have a great weekend!

Tags: , ,

A Collection of UCS Posts

There’s been quite a few good Cisco UCS posts published recently; I thought it might be handy to collect a list of some of them (I’m sure that I will miss some). Here are a few that I’ve seen over the past few weeks (in no particular order):

Swapping UCS Blades with Local Boot Policies
Get Spidey Powers with UCS; but with Great Power comes Great Responsibility
25 ways that Cisco UCS frees you to do other things
Cisco UCS – How Many FEX Uplinks Do I Need?
UCS local disk policy + some vBlock
Cisco UCS: different workload, different configuration, same blade. Simple.
Cisco UCS Information for “Server People”
Cisco UCS vs. IBM and HP – Where are the Brains?
UCS Gotchas? and how much time does it take day to day?

Anyone else have any UCS posts that have surfaced recently? Add them in the comments below.

Tags: , ,

It’s that time again: time for another Virtualization Short Take! Here’s a collection of links, articles, posts, and other tidbits that I’ve found interesting, informative, or useful over the last few weeks. I hope that you find something useful as well!

  • Tom Howarth has been spending some time with Microsoft’s App-V application virtualization solution; he’s written a three-part series (Part 1, Part 2, and Part 3). Part 1 discusses domain and certificate setup, Part 2 centers around policies and GPO settings, and Part 3 covers the client-side setup for App-V. While Tom’s overview is extremely helpful, I don’t recall seeing any thoughts on App-V as a product. Tom, did you like it? Not like it? What was good or bad about it? It would be great to have a post that brings this sort of information together.
  • Interested in getting a better feel for the communications that occur between an ESX/ESXi host and vCenter Server? This post discusses decoding SSL traffic with Wireshark so that you can see what’s happening.
  • Jeremy Waldrop of Varrow has a good “getting started” post on using vCenter Server’s storage alarms. If you’re looking for an introductory piece, this is a good place to start.
  • If you’re using Hyper-V and have VMs that are generating lots of network traffic, this post from the Windows Server Performance Team discussing increasing the VMBus buffer size is probably worth a look for you.
  • And while I’m mentioning Hyper-V, Ben Armstrong aka Virtual PC Guy discusses an RDP ActiveX control that provides RDP connectivity to a VM (not RDP connectivity to the guest OS, which is distinct and separate). I’ve never been a huge fan of ActiveX controls, but this could be useful in certain environments.
  • Is defragmentation of VMs a good thing? Scott Drummonds asks the same question in this blog post. My only comment: avoid defragmentation with thin provisioned disks (array-level or hypervisor-level thin provisioning).
  • Of course, Scott Drummonds also had a flurry of very useful posts over the last few weeks: missing Perfmon counters, inaccuracy of guest performance counters, and Las Vegas taxi rates. (The Las Vegas taxi post actually helped me save some money when headed to the airport after PEX. Your mileage may vary—pun intended.)
  • Eric Sloof’s home-grown tests of running linked clones on an SSD aren’t definitive, but they definitely back up the value that has been seen with the deployment of EFDs (Enterprise Flash Drives) in virtualized environments.
  • This PowerShell script will show you the logged-in user for a given VMware View desktop. Handy!
  • Readers seeking more information on guest OS alignment should read this article by Jeff Muir. While the focus of the article is on VHD and NTFS alignment, the underlying principles are also applicable to VMDK files in VMware environments.
  • Frank Denneman, VCDX 29, has had a few good posts recently. He had a post that discusses the use of local storage for VM swap; this post was then parlayed into a greater discussion on understanding the impact of design decisions. It’s a pretty fitting discussion given the timing around all the VCDX defense panels at Partner Exchange and Frank’s own elevation to the VCDX priesthood. Frank’s article on VM sizing and NUMA was also a great read. Keep up the good work, Frank! (And I’m still waiting to see all the info about memory reservations you promised me…)
  • Jason Boche recently highlighted his adventures in using Round Robin multipathing with his EMC Celerra. One key takeaway is that he had to reboot the ESX/ESXi host after changing the SATP, so keep that in mind. There is also a very specific CLARiiON configuration that needs to be set: the Failover Mode needs to be set to 4.
  • Jonathan Medd provides some great information on users who might be new to vCenter Update Manager in this article.
  • If you are planning on virtualizing any SQL Server systems, be sure to check out this list of best practices for SQL Server, written by Scott Drummonds. The document is a bit old (December 2008), but the recommendations are still valid.
  • It appears that VMware has updated this KB article recommending the use of the LSI Logic vSCSI controller for low I/O environments. I’m glad to see VMware has added more information and clarification; the previous version of the article was a bit spartan, to say the least.
  • I think that Figure 1 on this page on Cisco solutions for VMware View environments would give even Hany Michael a run for the money! While Figure 1 is pretty complex, the information in the article is useful and helps underscore some of the many different ways Cisco products can be put to use in a VMware View environment.
  • Here’s a useful document on integrating Cisco UCS with VMware DPM.
  • This weekly summary of new KB articles is quite useful. OK, I know this isn’t new and many people probably already knew about it but it’s still useful. So get off my case, OK?

There’s more that I could include, but I should probably wrap this up. Here are a few other links worth mentioning:

The Backup Blog: Avamar and VMware Backup Revisited
VMware KB: ESX 4.0 and ESXi 4.0 shutdown and reboot commands
VMware KB: Masking a LUN from ESX and ESXi 4.0 using the MASK_PATH plug-in
Rethinking vNetwork Security
Announcing NVSPBind

That’s it for this time around. Thanks for reading and feel free to submit any interesting links you’ve found in the comments!

Tags: , , , , ,

In case you’ve haven’t already heard, David Davis and the good folks over at Train Signal recently released an advanced VMware vSphere training course called “VMware vSphere Pro Series Training, Vol 1″. You can get more information about this new course from Train Signal’s web site.

The new video course features not only David Davis, but also well-known virtualization figures Hal Rottenberg and Rick Scherer. David Davis takes viewers of the training course through a section on VMware View, VMware’s product for virtual desktops, and ThinApp, VMware’s application virtualization solution. Hal provides coverage of PowerCLI (is anyone surprised?), and Rick discusses the Cisco Nexus 1000V. All in all, the new video course is almost 11 hours in length.

Train Signal also includes multiple digital formats as well to make it easier for busy administrators to be able to view or listen to the content.

I do have to say that I haven’t yet had the opportunity to actually view any of these videos. However, I do know both David Davis and Rick Scherer personally (sorry Hal, I haven’t met you personally yet). I’m confident that this is a good quality product. If you’re a VMware vSphere administrator looking to expand his or her knowledge of VMware View, ThinApp, PowerCLI, and/or the Nexus 1000V, this new training course is an excellent place to start.

Disclaimer: Train Signal is a paid sponsor of this site.

Tags: , ,

This is a two-hour session on VMware View security architecture and security benefits titled “VMware View Security Benefits, Architecture, and Best Practices”.

So what is VMware’s security strategy? First, start with core platform security. This encompasses all the various features and functions of the hypervisor like memory protection and isolation, kernel module protections, hypervisor attack surface, etc. Next, continue with operational security. This is about integrating VMware’s products into your organization’s existing operational security policies and includes things like the vSphere Security Hardening Guide that was recently released. Using security virtual appliances is another step that enables broad-based security for all VMs in the environment. Finally, VMware is striving for a “better than physical” model where virtual security is better than physical security. Consider VMsafe as an effort in this area.

The presenter next reviewed the VMware View infrastructure and all the various components that are included in this infrastructure. To ensure security, all of these various components need to be reviewed with an eye on security. For example, componentizing the different parts of a View infrastructure—for example, separating access points, user data, applications, data, and operating system helps to secure each of these different pieces.

A further benefit of this separation is that it allows for the creation of a true “gold master” for VMs. Products like ThinApp and VMware View Composer helps to simplify this process and help maintain a true “gold master” image. This means that all the various security guidelines can be more easily incorporated into this master image, the master image can be patched more easily, configuration drift is reduced, and you can recover more easily and more quickly after an attack.

Using virtual desktops also allows organizations to more easily create “desktop security zones” that help isolate higher-risk PCs from lower-risk PCs, thus containing potential security risk to a limited subset of all desktops. This might also help with meeting compliance requirements (the presenter specifically mentioned PCI).

Thin clients are helpful in reducing complexity at the edge, which can (in some cases) help reduce the attack surface and limit the amount of work that IT organizations need to do to help secure the endpoints.

What about backing up data? Using View to centralize desktops allows organizations to more easily implementation full data backups for the various types of data that are being created within the virtual desktop environment.

The presenter next moves on to vSphere security. Because VMware View depends upon vCenter and ESX/ESXi, the security of View is dependent upon the security of vCenter and ESX/ESXi. This led into a discussion of the benefits of virtualization vs. the security impact of virtualization. The topics covered here include all the usual suspects: greater impact of misconfiguration or attack; loss of visibility in the network access layer; loss of separation between network admins and server admins; potential VM sprawl without consistent configurations and properly defined procedures; possible security problems resulting from VM mobility; and unauthorized access to VMs because of VM encapsulation (users copying a VM by copying the VM’s files).

So how does one protect the virtual infrastructure? You use existing techniques such as hardening and lockdown; defense in depth; and authorization, authentication, and accounting.

The same goes for protecting virtual machines. Use anti-virus, IDP/IDS systems, firewalls, etc. VMsafe and the functionality enabled by VMsafe will be very helpful here.

Be sure to isolate the management interfaces using physically separate management networks or by using VLANs. You should also control access to the management network using ACLs, jump boxes, VPNs, or other access controls. Only authorized individuals should have access to the management network and “ordinary end-users” should absolutely not have access.

The separation of duties is also important. Use vCenter Server’s built-in roles to enable the principle of least privilege to help enforce separation of duties. Third-party products like HyTrust might also be helpful.

The presenter argues that moving to a vNetwork Distributed Switch is a security benefit. One big plus is the mitigation of the risk associated with misconfiguration. In addition, there is support for private VLANs (PVLANs), inbound traffic shaping, Network VMotion, and (with the Nexus 1000V) ACLs and a natural separation of duties.

At this point the presenter moves on to a discussion of secure access to virtual desktops.

Authentication is one key area; View supports AD authentication as well as RSA SecurID. View Manager does not store any of the authentication information; this is all offloaded to Active Directory or the RSA Authentication Manager. Smart Card authentication is an alternative to standard username and password authentication. The certificate on the Smart Card contains a Subject Alternative Name (SAN); the SAN is matched against the User Principal Name (UPN) in Active Directory. Smart Card authentication is not supported with PCoIP.

View does support a form of single-sign on so that users log on to the View Client and is authenticated all the way down to the virtual desktop.

Future support with regard to authentication will include Kerberos realm authentication; UPN authentication; RADIUS support in the View Connection Server; and improved SSO to virtual desktops.

Moving on to access options, PCoIP requires direct access to the virtual desktop; it won’t work with SSL tunneling. Fortunately, PCoIP is already encrypted (wirespeed encryption using AES 128-bit encryption). For non-PCoIP connections, HTTPS tunneling of RDP is supported by VMware View. This can greatly simplify firewall configuration (only TCP port 443 is required). Secure tunneling also has the benefit of helping to maintain sessions in the event of a dropped connection.

Some advantages of PCoIP is the built-in encryption and support for blocking USB Plug events (to control USB device usage).

The View Security Server enables you to create a DMZ infrastructure that prevents end points from having direct access to virtual desktops or the Connection Server. The use of load balancers is supported with both Security Servers and Connection Servers.

VMware does recommend replacing the self-signed certificates that are supplied with VMware View with valid SSL certificates. Note that the specific SSLv3/TLSv1 ciphers that are used with secure connections can be configured to enable or disable specific ciphers.

The use of a VPN can also help provide a single point of entry and simply the firewall configuration.

The next topic is VMware View’s entitlements model. View uses Microsoft ADAM on Windows Server 2003 or Microsoft AD LDS on Windows Server 2008. Back-end Active Directory is still leveraged for authentication. View uses the idea of foreign security principals (FSPs), which means that Active Directory doesn’t have to be synchronized with the local LDAP instance. In addition, user authorizations and entitlements don’t have to be stored in Active Directory (which would require schema extensions).

At this point the presenter moves into a discussion of View security best practices:

  • Harden the base OS within the virtual desktops and enforce refresh intervals and OS patching.
  • Choose the proper authentication model and use a Security Server or VPN for secure remote access.
  • Be sure to understand the firewall requirements and configure the firewall accordingly.
  • Be sure to harden the Connection Server and the underlying Windows Server OS upon which it is installed.
  • Replace the default self-signed certificates.
  • Set appropriate entitlements within the Connection Server. Zone users according to use case and risk.
  • Avoid direct remote access to virtual desktops where possible. Don’t allow users to connect without going through the Connection Server.
  • Control USB access, redirection of clipboard, printers, and drives.
  • Leverage Active Directory Group Policy to help with virtual desktop OS lockdown and some View-specific settings. (You might need to use Loopback Policy Processing in this instance.)
  • Know the different ports and the directions that are required when configuring firewalls. Refer to the View Architecture Planning Guide for full details.
  • Install anti-virus, but use a minimal installation to reduce bloat.
  • Use a staggered or randomized scanning policy to avoid overwhelming the infrastructure. Use policies or corporate configuration tools to enforce staggered scanning and signature updates and to configure exclusion lists (only need to scan the user data disk; the base OS is locked down through the use of linked clones).
  • Consider a VMsafe Ready AV product.
  • Include Network Access Control (NAC) management agent in the parent VM prior to cloning.
  • Use ThinApp to gain some security benefits (prevents the OS from getting infected through the actions of a ThinApped application). Consider using ThinApp for browsers.
  • Specific to ThinApp and anti-virus, don’t install AV on the Capture/Build system if at all possible. If AV is installed, no on-demand scanning of the ThinApp project directory.

The next topic of the session was a discussion of using VMware vShield Zones. vShield Zones provide virtual firewalls that operate as transparent Layer 2 bridges and allow you to create different security zones. This can provide some technological enforcement of zones for different user environments (different pools for web browsing vs. internal CRM access and these pools cannot communicate with each other because of vShield Zones).

The presenter wrapped up the session with an overview of VMsafe and how VMsafe can help contribute to the security of a VMware View environment. VMsafe enables greater protection of VMs through APIs that allow deepened inspection of CPU/memory, networking, and storage. For example, VMsafe allows knowledge of specific CPU state or inspection of specific memory pages. VMsafe allows networking traffic to be inspected, intercepted, modified, or even replicated (consider vShield Zones integrated with the VMsafe APIs). With regard to storage, VMsafe allows the ability to mount VMDKs, inspect storage I/Os, and do so transparently and inline to the storage stack.

The session wrapped up with a list VMsafe-integrated solutions from companies like Altor Networks, TrendMicro, McAfee, and Checkpoint.

Tags: , , , ,

This is a session on vCenter Chargeback deployment, configuration, and best practices. The presenter is Naeem Malik from VMware. Naeem works in VMware’s PS organization and specializes in Chargeback, CapacityIQ, and Capacity Planner assessments.

There are three things to keep in mind when you are thinking about a vCenter Chargeback implementation: hierarchy, cost models, and cost templates. Malik will discuss those in more detail later.

So why chargeback? Chargeback is necessary to handle the new model of shared resources instead of dedicated physical servers. There is no longer a one application-to-one server model; now many applications run on the same server. Why is this new model necessary? Malik quotes Gartner that “the speed and flexibility of virtualization makes some form of chargeback mandatory”. Otherwise, organizations run the risk of VM sprawl. After all, VMs are not free—they require CPU, memory, storage, and network capacity.

vCenter Chargeback is a resource accounting tool that helps users and organizations understand that VMs are not free. It features support for fixed, allocation, and utilization-based costing; provides the ability to charge different amounts for different tiers of infrastructure; and can schedule reports and e-mail results.

In a vCenter Chargeback implementation, Malik believes that a large part of a consultant’s time is taken up helping organizations define the resource costing (if the organization has not already established those costs).

From an architectural perspective, vCenter Chargeback uses a separate database but also pulls information from the vCenter Server database. vCenter Chargeback can run as a VM and integrates with an organization’s existing e-mail systems (via SMTP) and existing Active Directory/LDAP infrastructures. Both SQL Server and Oracle are supported for Chargeback. The Chargeback server will interact with vCenter Server to pull performance and utilization information.

A single Chargeback server supports up to 5 vCenter Server instances and up to 5,000 VMs/entities. An embedded data collector is found on the Chargeback server itself. When an implementation goes beyond 5 vCenter Server instances, an additional data collector is necessary. The data collectors can be easily deployed from the Chargeback server itself. Extremely large implementations (up to 75 vCenter Server instances and up to 20,000 VMs/entities) require multiple Chargeback servers behind a load balancer with multiple data collectors.

Chargeback uses HTTP over TCP port 8080, HTTPS over TCP port 443, load balancing configuration over TCP port 8009, LDAP over TCP port 389, and SMTP over TCP port 25 (the slide had a typo and listed port 24).

Earlier Malik had mentioned three things to keep in mind. The first of these is hierarchy. The hierarchy controls how reports are created. The second thing to keep in mind is the cost model. There is fixed costing (fixed cost for a VM instance), allocation-based costing (variable costs per VM based on allocated resources), and utilization costing (variable cost per VM based on actual resources utilized). Many customers are using a hybrid model that is somewhere between fixed costing and allocation-based costing. The third thing is cost templates. Cost templates combine cost accounting information with fixed costs.

Cost accounting works with the cost model to determine how resources are actually priced. If a customer hasn’t already determined costs for their resources (CPU, RAM, storage, networking), VMware has a tool that can help determine this number. This can be difficult and requires “buy in” from all applicable stakeholders within the environment. Items that require costs assigned to them include CPU, memory, disk, disk I/O, and network I/O.

Fixed costs (not the same as the fixed cost model) include stuff like power/cooling, software licenses, real estate in the data center, labor, etc.

Cost templates combine the cost accounting for the various resources with the fixed costs and allow you to apply a cost multiple to the metering element (GHz of CPU cycles used, GB of RAM used, etc.). Multiple cost templates can be created to help allow for flexible costing of VMs.

vCenter Chargeback also has extensive reporting functionality. Scheduled reports are available, and reports can be generated at any point within the hierarchy (datacenter object, cluster object, host object). Reports can be customized for a company-specific look and feel. Reports are available via e-mail or via the Chargeback web UI.

At this point, Malik now moves into a product “demo”, which is essentially a collection of screenshots from vCenter Chargeback that help illustrate the various components and features of Chargeback.

Tags: ,

This is a liveblog for VMware Partner Exchange session TECHBC0320, “How VMware Leverages Microsoft Volume Shadow Services for Virtual Machine Snapshots”. The presenter is Paul Vasquez with VMware; he works within the Technical Alliances Organization at VMware with a focus on backups.

The session starts out with an overview of VMware snapshots followed by a quick overview of Microsoft Volume Shadow Copy Services.

Vasquez is careful to distinguish VMware snapshots from array-based snapshots, which is good since that seems to confuse a number of people. VMware snapshots can include the state of memory (optional), settings, and disk. Snapshots are taken at the VM level, and up to 32 snapshots can be taken. Over 20 snapshots can cause performance concerns and, in Vasquez’s words, “can cause undesirable results”.

In general, a snapshot will include all disks although there are ways to exclude disks from a snapshot.

Operations involving VMware snapshots include taking a snapshot (self-explanatory), reverting to a snapshot (reverts the VM to the snapshot state, the delta file remains until the snapshot is deleted), and deleting a snapshot (delta file is removed, VM continues running in the current state).

Some use cases for snapshots include: rollback capability for testing patches or updates; rollback for failed software installation; protection against unwanted results of OS reconfigurations or testing; backups (for creating consistent copies of a VM); and replication.

The delta file grows as-needed; over time, the delta file will grow larger and larger. Vasquez cautions attendees to be sure to plan datastore sizes to account for snapshots for VMs and the delta file growth caused by the changes to those VMs.

A good question was raised about read I/Os and the impact of snapshots (does

The presentation now moves on to a discussion of VSS. One component of VSS is the requestor; the requestor makes a request from a provider, and the writer provides information on how to provide information to a requestor. Providers are included with Windows and are responsible for intercepting I/O requests to create and represent volume shadow copies on the file system. There are also 3rd party providers. In this context of this discussion (VSS integration with VMware snapshots), VMware Tools is the requestor.

There is a wide range of applications that provide VSS support, including Exchange, SQL, SharePoint, Active Directory, BITS, DHCP, and WINS. The vssadmin list providers command will show all the providers. (Note that you won’t see the VMware Tools when you run this command; it is dynamically loaded only at snapshot time and then unloaded.)

The vssadmin list writers command will show a list of writers.

The general flow of operation with VSS runs like this:

  1. Requestor makes a shadow copy.
  2. The writer is told to freeze all I/O.
  3. The provider creates a shadow copy.
  4. The writer is told to “thaw,” or resume, I/O to the application.
  5. The requestor now has access to the shadow copy.

The writer can support multiple enumerations, or different ways of coordinating the creation of the shadow copy. Exchange, for example, supports Full (backs up databases, logs, and checkpoints; truncates logs), Copy (backs up databases, logs, and checkpoints; does not truncate logs), Incremental (backs up and truncates logs), Differential (backs up logs but does not truncate). Of these, VMware uses the Copy enumeration when requesting shadow copies. Supposedly, the reason this is the case is to prevent interfering with backup applications that aren’t aware that logs were truncated. In addition, when VMware calls VSS, all writers are engaged, so it’s not possible to selectively choose which VSS writers should be engaged (can’t engage VSS for Exchange but not SQL within the same VM, for example).

In the future, VMware Tools will offer granular control over which VSS enumeration is used. Granular control over which VSS writers can be engaged is also planned.

Vasquez now moves into a discussion of how VMware snapshots and VSS integrate together. When a VMware snapshot is taken, this is when VSS integration comes into play. Obviously, for VSS integration the VM must be powered on (the guest OS must be running in order for VSS to be operational).

Some form of quiescing is always used when a snapshot is taken (unless the VM is powered off). The VMware Sync driver provides a crash-consistent copy of the VM but doesn’t interact with applications. This option is available in vSphere 4.0 and can be used when no VSS support from the application is available. Obviously, there is VSS support (hence this session), and there are pre- and post-quiesce scripts that can be used to create homebrew solutions as well. Both VSS and the Sync driver can be enabled using VMware Tools.

VSS support is enabled in VMware ESX 3.5 Update 2 or higher.

Going back to the VSS flow earlier, an additional step is present before the writer resumes I/O to take the VMware snapshot. After the VMware snapshot is taken, the shadow copy created by the provider is discarded because it is no longer needed. Once again, Vasquez reminds attendees that the VMware Tools Requestor only supports the copy enumeration.

An attendee asked if any plans were in place to do quiescing at the VMFS layer (supposedly to assist with hardware-based snapshots); Vasquez responds that some form of VMFS quiescing would be helpful, but there are challenges with that arrangement that make it currently very difficult to actually achieve.

(Vasquez also commented on the end-of-life policy for the ESX Service Console, but I’ll hold on mentioning what was said until I verify the confidentiality of the statement.)

Some additional things to remember:

  • VMware Tools build must be 110268 or higher.
  • VMware Tools must be running and VSS must be functioning properly.
  • VSS Service must be set to Manual or Automatic.
  • ESX 3.5 Update 2 is required for VSS support.
  • Be sure VSS support is installed with VMware Tools.
  • Try not to keep VMware snapshots around for a long time. Manage snapshots carefully.
  • Sync driver can be used as a failback in the event VSS support fails.
  • VSS snapshot has a 10 second timeout. Rare cases could cause a failure of getting the VSS shadow copy.

Most of the information contained in this presentation are found in the current vSphere documents and in Microsoft’s VSS documentation. (I’ll update this post with URLs when possible.)

And that’s it for the session.

Tags: , , , ,

You might have noted a slight incompatibility between VMware vCenter Lab Manager and one of VMware vSphere’s core features, Storage VMotion, in this earlier post on VMware Lab Manager design considerations by former co-worker Aaron Delp:

Storage VMotion and VMware VCB are not supported with Lab Manager.

Obviously, this could present a problem for users who might need to migrate Lab Manager datastores from one LUN or array to another LUN or array. So what’s a user to do?

Fortunately, an internal discussion on this earlier today turned up some great information on a utility called SSMove. What is SSMove?

SSMove is a utility installed on the Lab Manager server that allows you to move data from one datastore to another. You can move a specific tree of related virtual machines. See “Viewing Virtual Machine Datastore Directories” in the Lab Manager User’s Guide for more information on trees. To move an entire datastore, you must move all its trees individually.

Credit goes to rockstar team member Denis Guyadeen for pointing out this utility. More information is available at this links:

VMware KB: Moving a datastore using SSMove
VMware KB: SSMove does not work if a datastore is disabled (3.0.2 only)
VMware Lab Manager 3 Online Library – Managing Datastores

So, if you are needing to migrate data in Lab Manager from one datastore to another, this is your tool.

I haven’t yet found any information on whether SSMove is also included in Lab Manager 4. (To be fair, I haven’t really searched too hard.) If anyone knows, please speak up in the comments.

Tags: , ,

« Older entries