2008

For those that follow me on Twitter, you may have noticed that I purchased and installed OmniFocus for iPhone over the weekend. If you are a GTD lover and a Mac user, you are no doubt familiar with the “regular” OmniFocus application from The Omni Group. I’ve been using OmniFocus for quite some time now (I wrote about that back in February), and I was really excited about the possibility of taking it mobile with me on my iPhone.

A number of things stand out after using this combination for a few days:

• I’m too cheap to pay for a MobileMe subscription, so I just setup a WebDAV site on my hosting package. It’s not the greatest in the world (I’d prefer to run WebDAV over SSL on the standard HTTPS port, for example), but it works. However, less technically inclined folks would have had a problem without a MobileMe subscription. I hear that Omni is planning on adding Bonjour syncing support, but I can’t imagine that will be anything other than Wi-Fi only.
• It takes quite a while to sync my iPhone after a day of working in OmniFocus on my MacBook Pro. This is even with a good 3G data connection. I don’t know that there’s anything that can be done about this, nor is this even anyone’s “fault”; it’s just an observation I’ve seen thus far.
• In an effort to stretch the battery out as long as possible, I generally keep Location Services turned off. This limits some of the location-aware functionality that OmniFocus for iPhone features. Since the release of the 2.1 firmware, my battery life has improved; perhaps I can turn on Location Services and not suffer too much of a battery hit. If anyone has any feedback on how much of a hit it is to keep Location Services turned on, I’d certainly appreciate it.
• I’m finding that my current set of contexts don’t necessarily translate well to the iPhone version. Currently my contexts are more for grouping similar tasks than by location or resource; I’m thinking I may need to move to more location-based contexts. I’m not yet sure how that will work or how I’ll integrate that with my workflow. Again, suggestions are more than welcome.

Overall, I’m pretty pleased with OmniFocus for iPhone. (In fact, I’m pretty pleased with my iPhone in general.) There’s always room for improvement, but given my experience with The Omni Group with applications like OmniGraffle and OmniOutliner I’m quite confident that the application will improve over time.

This edition of Virtualization Short Takes is mainly a collection of items from the VMworld 2008 conference in Las Vegas. Some of these are session transcripts from various bloggers; some are just VMworld-related blog posts.

• Blogger Rich Brambley has some great coverage of various VMworld sessions: BC3819, BC2370, PO2575, and VD3261, among others. Excellent work, Rich.
• The whole vStorage thing has prompted quite a flurry of attention. I’ll probably tackle this myself soon, but for now I’ll just comment on what others are saying. Stephen Foskett comments that “VDC-OS has legs!”, meaning that this vision has some substance behind it; I agree. He also has a nice collection of related vStorage links. There’s also Chad’s discussion of vStorage, which is quite helpful coming as it does from the perspective of a storage vendor seeking to take advantage of these new capabilities. Mark Twomey also had a little bit to say about vStorage as well.
• Michael Keen’s VMworld 2008 analysis provides a good review of VMware’s announcements and strategy in relation to the competition and the partner and the partner ecosystem.
• UK magazine Computing was disappointed by Paul Maritz’s vision and the VDC-OS announcement, expecting “more granular details on how the initiative would actually take off.” Personally, I felt that Steve Herrod’s keynote on Day 2 did a fairly reasonable job of showing the mechanics behind Paul’s vision.
• Redmond Developer News provided another view of VMware’s VDC-OS announcement, although to be honest the article looked pretty much like every other discussion of VDC-OS, with the same interviews of the same people.
• Kevin Fogarty’s take on the VDC-OS pitch, published on CIO.com originally, then republished by Computerworld and touched upon briefly by Tarry Singh, is that it’s “too much of a leap of faith for me and, I suspect, most of the VMware faithful as well.” I get that VMware’s new strategy is radically different from anything VMware has done before; up until now, their releases have been product-focused. Now Maritz is driving the company with a long-term vision that will be fulfilled through a series of product releases. It’s a shift in thinking, and one that will require an adjustment. Personally, I don’t like the new VDC-OS branding and I told VMware straight up that it would be confusing to customers. Nevertheless, the vision behind the brand is, in my opinion, solid and so I must disagree with Kevin’s analysis.
• Massimo thinks Paul plagiarized the VDC-OS concept. If he were serious (which he’s not), he actually has a pretty good case.
• Eric Sloof has a bit of additional Cisco Nexus 1000V coverage.

I think that about wraps up my collection of VMworld 2008-related links. If I’ve missed anything significant—and I’m sure I have—please feel free to add it in the comments.

My wife, a couple of the kids, and I just got back home after watching Fireproof at the local movie theater. You’ll recall that I first mentioned Fireproof back in August, and today was the opening day. I thoroughly enjoyed the movie. If you haven’t seen it, make plans to go see it right away! The movie is great, but the Message in the movie is even greater.

Having previously discussed Marathon Technologies’ everRun VM product in conjunction with XenServer HA as part of XenServer 5, I think that I have some useful information to bring to the recent discussion that has come to light about everRun VM vs. VMware HA vs. VMware FT.

Apparently, this discussion started with a blog entry by Marathon titled VMware FT - The Top Four Reasons it’s Kinda Sorta Fault Tolerance. Mike DiPetrillo of VMware responded from his personal blog, first tackling Marathon’s blog post and then again tackling comparisons posted on Marathon’s web site. Various others also weighed in, such as Duncan at Yellow Bricks and TechTarget’s Server Virtualization Blog.

I’ve spoken with the folks from Marathon a couple of different times about everRun and its functionality, so let me attempt to compare these three products—everRun VM, VMware HA, and VMware FT—with an eye toward understanding the differences between them.

• Marathon everRun VM provides two levels of protection: Level 1 and Level 2. Level 1 is basic failover, and is included in XenServer 5 as XenServer HA. In this regard, it is essentially the same as VMware HA in that it will restart VMs in the event of host failure. Both products calculate available capacity for failover but do not reserve those resources in advance; hence, neither of them can provide guaranteed failover. VMware HA seems to have an upper hand here because admission control can actually prevent users from powering on VMs if there are not enough resources to provide failover for that VM. From all information I have been able to obtain, everRun VM Level 1/XenServer HA lacks that ability, and it’s possible therefore that users could power on more VMs than the resource pool could sustain in the event of hardware failure. Both products should be considered “best effort” as a result. Users wanting to make comparisons between Marathon everRun VM and VMware HA should constrain their comparison to everRun Level 1. Otherwise, the comparison is not a like-to-like comparison.
• Marathon everRun VM goes on to add Level 2 protection for component-level failure. It’s true that this level of protection exceeds anything that can be provided via VMware HA today. With component-level protection, I/O to or from a failed storage device or a failed network device is transparently redirected to another host, where an identical VM environment has been established. Please note that the two VM environments are not both executing at the same time, but that resources on the secondary host are reserved and cannot be used by any other VMs. These resources include not only RAM, but also storage and networking. If there is a host failure, the VM is restarted on the secondary host. Because resources were pre-allocated, everRun VM is able to provide guaranteed restart on the secondary host. The functionality provided by everRun VM when configured for Level 2 protection exceeds any functionality that VMware HA has today.
• On the flip side, however, it’s also fair to note that VMware has not needed to provide component-level fault tolerance because they’ve supported storage multipathing and NIC teaming for quite some time. It’s my understanding that those features have only recently made it into the XenServer product line.
• VMware Fault Tolerance (FT) and everRun VM Level 3 are comparable. Both establish an identical VM on another host and keep that VM “mirrored” with the original VM. If there is a host failure, the “mirrored” VM will automatically take over right where the primary was when it failed. It appears that everRun VM might have an edge here because it again supports component-level failover, but given that neither product is available yet it’s still a bit too early to be making calls on which product is “better”.
• As for the “complexity” of one product versus the other, both have their own complexities. Marathon everRun VM requires a dedicated network link, called the “Availability Link”, in order to provide the component-level protection. I would assume the Availability Link will be needed for everRun VM Level 3 as well. That corresponds directly to VMware FT’s logging NIC. VMware HA does not require any special NICs or unique configurations; it’s unclear if the same is true for everRun VM Level 1/XenServer HA protection. I’ll have to call Marathon out on their knocks against setting up NIC teaming and storage multipathing; those tasks may be complicated in XenServer environments but are drop-dead simple in VMware ESX environments. The same goes for enabling VMware HA and VMware DRS.

As you can see, each product has its own set of strengths and weaknesses.

As a final note, as SearchServerVirtualization.com stated, comparisons between these two product sets are a bit irrelevent anyway: VMware’s functionality works only with VMware ESX environments, and Marathon’s functionality works only with XenServer. It’s not like users have to choose between them in the same virtualization environment.

I welcome everyone’s input and thoughts on this matter. Please contribute in the comments to this article.

While trying to clear out the backlog of articles that have accumulated in my Reading/Review context in OmniFocus, I’ve come across a number of Hyper-V articles on various topics:

• Robert Larson has a good article at VirtualizationAdmin.com that covers how to work with VLANs with Hyper-V. That includes information on configuring the parent partition to use VLANs as well as configuring child partitions to use VLANs. The title of the article is completely wrong for the content, but it’s still a useful article nevertheless.
• Via Ben Armstrong, I saw that there is a VMC to Hyper-V import tool. More information on the tool is available here. In addition, Ben also recently mentioned a Hyper-V VSS hotfix that fixes a problem with VSS failing to backup any VMs if even a single VM has a corrupt or invalid configuration file.
• And while we’re on the subject of migrating to Hyper-V from Virtual Server or Virtual PC, this blog post provides a checklist of things to do when migrating virtual machines from those older platforms to Hyper-V.
• Hyper-V’s Linux Integration Components—the paravirtualized drivers (or synthetic devices, or virtualization-optimized components, or whatever else you want to call them) that provide better performance under Hyper-V—have been officially released.
• By the way, this page provides a comparison of Hyper-V Server 2008 and more “traditional” implementations of Hyper-V with a full Windows Server 2008 parent partition.
• Looking for a comparison of performance with dynamic VHDs and fixed VHDs? Inquiring minds want to know! Get the scoop here.

That’s it for now. If any readers have other useful or helpful Hyper-V links, please share them in the comments.

I was alerted to this by a friend of mine within the Borg mothership…er, who works at Microsoft. An update has been released for the Hyper-V role of Windows Server 2008 x64 (available for download here) that increases the number of processors and virtual machines supported. With the update, Hyper-V can support up to 24 logical processors and up to 192 virtual machines. My contact also tells me that the update doubles the address space cache limit (up to 384 from 192), although I didn’t see that documented anywhere.

The related Microsoft KB article for the Hyper-V update also makes mention of a couple of interesting tidbits:

• Although the update does allow Hyper-V to run up to 192 virtual machines, a Registry edit is required if users want to run more than 150 virtual machines.
• If you want to run Windows Server 2008 on a 6-core CPU, you may also have to install hotfix 950182. Apparently, Windows expected CPU core counts to always be powers of 2 (i.e., 2, 4, 8). This update fixes that assumption.

Thanks to my contact for the tip.

UPDATE: My contact (need to think up a cool code name for her/him) supplied a link to the documentation on the increase of the address space cache limit. That information is here.

Via Duncan on Twitter, I saw that VMware Workstation 6.5 and VMware Server 2.0 have been released:

VMware Workstation 6.5 Download Page
VMware Server 2.0 Download Page

Some of the key features in Workstation 6.5 include “Unity,” the seamless windowing technology that first debuted in VMware Fusion; support for Windows Server 2008; enhanced VMware ACE authoring; and virtual machine streaming. (This last feature was originally demonstrated at VMworld 2007, so it’s nice to see VMware finally getting this technology into production.)

New features in Server 2.0 include support for Windows Server 2008; support for 64-bit Linux hosts and expanded 64-bit guest support; new VMware Remote Console to allow access to virtual machine consoles independent of the VI Web Access interface; VSS support; and VMI support for transparent paravirtualization.

Refer to the VMware web site for more information on either of these products.

A few weeks ago I blogged about SVVP validation of VMware ESX 3.5 Update 2. Just last week, though, I learned that Microsoft’s validation may be quite limited. According to this article, the validation is handled per CPU architecture, per memory configuration. So, apparently VMware ESX is validated on an AMD system with 4GB of RAM, but that’s about it.

Does anyone have any information to back this up? Or any information on additional CPU architectures and memory configurations that are currently being tested for validation?

This is PO2061, VMware VirtualCenter 2.5 Database Best Practices. The presenter is Bruce McCready with VMware. My battery is starting to run low, so I may lose some or all of the rest of this session partly through the session. I have another battery that I can swap in if necessary.

The session starts out with the standard disclaimer. Most of this stuff is already out there, but the disclaimer is required anyway.

The VirtualCenter database is clearly an important part of the overall virtualization implementation. The goals of this presentation will be to help users get a better understanding of the VC DB inner workings, to help users take advantage of changes in VC 2.5 and VC 2.5 Update 1, talk about performance statistics and how they affect DB performance, and answer some FAQs regarding VC DB size and performance.

As an overview of the VC DB, it stores host and VM details. The DB is slightly larger in VC 2.5 as it keeps about 40% more data about hosts. The VC DB also stores information about alarms and events. Old data about alarms and events is not purged over time. Finally, the VC DB also stores performance statistics. Up to 90% of the DB is taken up by performance statistics. Performance statistics are the primary factor in scalability and performance of the VC DB, which is what led to a major refactoring of the database in VC 2.5 as opposed to VC 2.0.

Bruce briefly describes the components that add load to the VirtualCenter database: performance statistics inserts, performance statistics rollup stored procedures, and ad hoc queries.

There are four levels of performance statistics. Level 1 has only about 10 statistics, Level 2 goes to 25 items, Level 3 ups that to about 65, and Level 4 increases to around 100 items. You can multiply the number of statistics being captured by the number of objects in inventory to get a rough idea of how much data is being inserted into the VC DB.

Stored procedures are run as scheduled jobs in SQL Server every 30 minutes, 2 hours, and 24 hours. Every half hour the 5 minute statistics are rolled up into the half hour roll up. Every 2 hours the half hour statistics are rolled up, and every day the two hour statistics are rolled up. After a statistic is a year old, it is automatically purged. Some statistics (like the 5 minute statistics) are kept for shorter periods of time.

Noting that performance statistics are purged periodically, it’s important to understand the most of the unchecked table growth comes from alarms and events.

The performance improvements in VC 2.5 allow the VC DB and stored procedures to scale much higher and greatly reduce the amount of time required for the operations to complete. In addition, the refactoring also allowed the VC 2.5 database to reduce the amount of space required to store the same amount of performance data.

Bruce next shows some performance results about response times for various operations (rollup, purge, and insert).

When it comes to sizing for optimal performance, memory is the most important thing. Disk I/O and CPU are important, but memory will most likely have the greatest impact. SQL Server’s TEMPDB isn’t as important in VC 2.5 as it was used in VC 2.0. Keep in mind that SQL Server will take advantage of multi-core situations, and the VC DB is optimized to help take advantage of SQL Server’s parallelism.

Bruce again shows how more memory will greatly reduce the time required to perform two-hour rollups on a very large inventory with Level 4 performance statistics. This just reinforces the need to give VC plenty of memory.

To help calculate the sample size of performance statistics, use this formula:

Sample size = (number of entities) * (statistics per level)

Based on the sample size, VMware has come up with some recommendations for VC DB hardware:

Up to 40K: 1 core, 1GB of RAM, less than 120 IOPS
40K to 80K: 2 cores, 2GB of RAM, 120-200 IOPS
80K to 120K: 4 cores, 4GB of RAM, 200-300 IOPS
120K to 160K: 4 cores, 6GB of RAM, 400-500 IOPS
160K to 200K: 6 cores, 8GB of RAM, 500-600 IOPS
More than 200K: 8 cores, 12GB of RAM, 650 IOPS

Next he moves on to some performance tips for VC 2.5. One trick to configure different collection levels for different time periods. For example, longer-term data can be kept at Level 1, but short-term data can be gathered at Level 3.

From the SQL Server perspective, it may be beneficial to separate the data and log onto different physical drives. In addition, monitoring page fullness in vpx_hist_stat1 and vpx_hist_stat2 may also help improve performance. Use the NOLOCK option for ad hoc queries to prevent blocking of other operations that need to occur, and correctly size the memory that is allocated to SQL Server. In addition, configuring parallel processing in SQL Server may be helpful. This is typically on by default, but if it is off then you may want to turn it on.

It is important to decide on a suitable backup strategy as well.

Enable vardecimal format for vpx_hist_stat can reduce database size about 60%. However, vardecimal format is only supported on SQL Server 2005 SP2, and only in certain editions of that version (such as the Enterprise Edition).

Purging old data can also help with DB performance. Remember that performance statistics are purged automatically, but alarm and event data is not. See VMware KB article 1000125.

The presentation is still going and looks like it has about 20 minutes left at the most, but the battery is running low now so I’m going to go ahead and publish this entry. Readers who attended this session are invited to flesh out my notes in the comments for additional information that I wasn’t able to cover.

This is PO1644, VMware Update Manager Performance and Best Practices. The presenter is John Liang.

Covering some terminology before moving forward, the presenter covered the idea of a patch store (a location where patches are stored), baseline, compliance (the state of the host or VM when compared to the baseline; falls into compliant/not compliant/unknown/not applicable), scan (either VM scan or host scan; VM scans can be either online or offline); and remediation (the idea of applying patches to a host or VM).

VUM has two deployment models. In the Internet-connected model, the VUM server knows and has connectivity to the VMware patch repository, and VUM will work closely with VirtualCenter. VUM can also be connected to multiple VC servers on multiple subnets.

In an Internet-disconnected model, VUM has no direct connectivity to the Internet and is not able to download patches for deployment. In this model, a separate Update Manager Deployment Server (UMDS) instance can download the patches. The patches can be exported to physical media and transferred to the VUM server for use in scanning and remediation.

Next, the presenter moved into a discussion of VUM sizing. VUM uses a separate database. A small deployment (20 hosts, 200 VMs, 4 scans/month for VMs, 2 scans/month for hosts) will generate 17MB/month in database storage. A medium deployment ups to 109MB/month, and a large deployment would generate 552MB/month in database storage.

The presenter provided some guidelines for patch store disk space, but I couldn’t capture that information before he proceeded to the next slide.

There are a number of VUM deployment models. VUM can be deployed on the same server as VC and use the same database server as VC. However, for medium deployments, consider separating the VUM database to a different database server. Consider a medium deployment to be 500 VMs or 50 hosts. For even larger deployments, both VC and VUM should use separate servers and separate databases. The recommendation is to separate VUM from VC is there are more than 1000 VMs or 100 hosts. In addition, the VUM database should be on different disks than the VC database, use at least 2GB of RAM for caching (more is better), and finally to separate VC and VUM onto separate servers for maximal performance.

Next, the presenter discussed some performance results for VUM. The host running VC and VUM and the database was a dual socket/dual core host with 16GB of RAM, managing VMware ESX hosts with 32GB of RAM. The results that were presented:

• 8 seconds to download the VM guest agent
• 27 seconds to scan a powered-off Windows VM
• 36 seconds to scan a powered-on Windows VM
• 8 seconds to scan a Linux VM

Next, the presenter showed some results of resource consumption during these various tasks. Compared to other operations, scanning a powered-off Windows VM took the most CPU usage on the VUM server itself. For the same task, the VC server CPU was not tremendously impacted, VMware ESX CPU was not impacted, and disk and database performance was essentially equivalent across all operations. Again, these results are all for single-operation scenarios. Out of all the tasks, the most (relative) expensive operation was an offline Windows VM scan.

VUM is limited to 5 VM remediation tasks per VMware ESX host (48 per VUM server), 6 powered-on Windows scans per VMware ESX host (42 per VUM server) , 10 powered-off Windows scans per VMware ESX host (10 per VUM server), 6 powered-on Linux scans per VMware ESX host (72 per VUM server), 1 VMware ESX scan per VMware ESX host (72 per VUM server), and 1 VMware remediation task per VMware ESX host (48 per VUM server). Some of these limits make sense; you can’t scan more than one VMware ESX host per VMware ESX host, for example.

The presenter gave a quick of wanting to scan 5000 Windows VM across 100 hosts, each host with 50 VMs and each scan taking 60 seconds? The answer: just shy of 105 minutes. I won’t go into the math details.

Entering maintenance mode can be blocked for the following reasons:

• VMware HA is configured with only two hosts
• VMware DRS fails to VMotion a VM to another host
• VMware DRS is configured for manual mode instead of automatic mode
• Without VMware DRS, there is a VM powered on

To correct these problems, use VMware DRS and configure it to use automatic mode, and use more than 2 hosts in a cluster.

It’s important to remember that the guest agent is single-threaded, and the Shavlik scan and remediation are also single-threaded. Using multiple vCPUs won’t necessary help with guest OS performance with regards to patching.

What is the impact of patching on guest memory? VUM will create virtual CD-ROM images, attach them to the guest, and then issue the remediation command to the VM. This will trigger a fairly significant amount of network traffic between the VUM server and the VMs being remediated. This can have a significant impact on network performance. The remediation process itself is also memory intensive, which can be further exacerbated by larger patches (Windows XP SP3 is 331MB, for example). To help with performance, VMware recommends at least 1GB of RAM for Windows VMs.

Next, the presenter tackles the subject of the impact of high-latency networks on VUM operation. The time taken by various operations (online scans, offline scans, remediation, etc.) is directly related to network latency; the higher the latency, the longer the operation takes. Online VM scans are the only exception; they remain constant and very low.

To help address this potential problem, VMware recommends deploying the VUM server as close to the VMware ESX hosts as possible. This will help reduce network latency and packet drops. In addition, use online scans on high-latency networks to minimize the impact of network latency.

An offline scan works by having the VUM server mount the VMDKs for the offline Windows VMs and then scan them directly from the VUM server. This explains why the CPU utilization on the VUM server is so directly impacted as a result of performing offline Windows VM scans; it has to mount the VMDK and scan the Registry and disks locally on the VUM server.

To help optimize offline scan, exclude “\Device\vstor*” in any on-access anti-virus software on the VUM server itself. This will prevent the VUM server from performing more I/O operations than necessary. Making this optimization helps improve performance by reducing latency by almost 50% on a high-latency network. The impact is almost negligible on a low-latency network. The presenter walks through excluding the appropriate device/location in anti-virus, something with which most users in here are probably already familiar.

• Use VMware DRS in automatic mode for host patching.
• Separate physical disks for patch store and VUM database.
• Use at least 2GB RAM for VUM server host to cache patch files in memory.
• Separate the VUM server database from the VC database if the inventory is large enough.
• Using multiple vCPUs in guests won’t necessarily help performance.
• Deploy VUM close to ESX hosts where possible.
• Prefer online scan on high-latency networks.
• Configure on-access anti-virus appropriately on the VUM server.

At this point, the presenter closes the session with a summary of VUM and its role in a VMware Infrastructure deployment.