blog.scottlowe.org

Apple has just given me one more reason to possibly switch to an iPhone…

iPhone opens to Exchange e-mail

I’m currently using a Samsung BlackJack II, a Windows Mobile 6-based device with 3G connectivity. The e-mail functionality is great, but not having an easy way to keep my Address Book contacts in sync with my phone is a major hassle. It looks like using an iPhone would help address that.

Now all we need is 3G (UMTS/HSDPA) functionality and a price drop…OK, at least the first is an actual possibility.

It’s that time again, folks…time for another Virtualization Short Take!

• VMware’s performance team published some interesting information on running Exchange Server 2007 on VMware ESX Server 3.5. 16,000 mailboxes on 1 physical server, eh? Now all we need is for Microsoft to actually support running Exchange Server on top of ESX Server and we’ll be good to go.
• Speaking of vendor support, Simon Gallagher at vinf.net has started an open discussion of vendor support for virtualization. This is truly a difficult sticking point for many organizations. Visionary vendors, like SAP, will fully support their solutions on virtualized platforms, but others—which I won’t mention by name but are based in the Pacific Northwest area of the United States—won’t. At least, they won’t until they need to build market share for their own virtualization solution, and then they’ll only support their own platform. There is no easy answer to the dilemma that Simon brings to light, but I am interested in the same question as Simon: what is everyone out there doing about vendor support?
• vinternals.net has discovered that the 2.5 version of the VI Client finally supports passthrough authentication, including—if you are willing to edit the vpxd.cfg file—Kerberos support. Good information!
• The team over at xtravirt.com has updated their information on vimsh, a woefully underdocumented but extremely useful command-line utility. This is must-have information for ESX Server engineers and architects.
• Chris Hoff has responded with some thoughts on VMsafe, the new set of security APIs that VMware announced at VMworld Europe last week. Chris, along with others, has been trumpeting the need for a sea change in security to accommodate the changes wrought by virtualization. In his words, it looks like the security vendors have been given another chance at life. Let’s hope they don’t blow this one.
• Many of you are probably aware that not all virtualization solutions support memory page sharing. The implications of supporting or not supporting memory page sharing may be greater than you think, though; have a look at this analysis.
• OK, take a deep breath here and don’t faint, but I’m getting ready to do something I don’t normally do: I’m going to defend Microsoft. That’s right. Via VMblog, I was turned on to some apparent controversy about VM performance under Hyper-V during guest OS installation. As Ben Armstrong aka “Virtual PC Guy” explains at his site, this is due to the use of emulated drivers during the installation process. People, this is no different than VMware. Before you install VMware Tools, VMware uses emulated drivers as well. Perhaps VMware’s emulated drivers are a bit more efficient than Hyper-V’s at this point, but Hyper-V is still in beta. And which would you rather have—highly optimized “synthetic drivers” (equivalent to VMware’s paravirtualized drivers) or efficient emulated drivers? Personally, I’ll take the second. So give Microsoft and Hyper-V a break for once. Save your energy for after the product is released.
• Information on booting VMs from an iSCSI LUN under Hyper-V is posted here. I don’t know that I would actually call that booting from an iSCSI LUN, since to the VM it’s a local drive, but I guess it all depends on your perspective.

That about wraps it up for now. As always, your thoughts, corrections, rants, and raves are welcome below.

I’m in the midst of a non-virtualization-related project right now, which is a bit odd; a great majority of my work these days is centered around virtualization.  Nevertheless, I try to view every project as one from which I can learn.  I have definitely learned some things with this project, that’s for sure.

Here are a few tidbits that I’ve learned so far this week, most of them centered around the installation of Microsoft Exchange Server 2007:

• First, if you have even one Active Directory domain controller that isn’t running Windows Server 2003 SP1, you can’t use the GUI setup routine for installing Exchange Server 2007.  That’s right, no GUI setup for you.  Instead, you’ll have to install from the command line like this:

  setup.com /mode:install /roles:mb, ca, ht, mt /EnableLegacyOutlook /LegacyRoutingServer:oldserver.domain.com /dc:win2k3sp1.domain.com

  Nice, eh?  Supposedly this will be fixed in Exchange 2007 SP1.

• Apparently, about 20% of the installations run from the command-line fail with an error about being unable to access the source files.  This is even when installing from local CD-ROM, as I was.  The Microsoft tech I spoke with recognized that this was a problem; the suggested solution was to copy the files from the CD to a local hard drive and run setup from there.
• The use of the “/LegacyRoutingServer:” command-line switch, which is required for interoperability with “legacy” Exchange 2000/2003 servers, can only be used when installing the very first server with the Hub Transport role (the “ht” in the command line above).  If the installation of that first server dies for some reason—say, like due to some strange error about not being able to access the source files—then you won’t be able to use this command-line switch again.  This means you’ll need to create the appropriate connector yourself manually after installation.
• If Exchange Server 2007 setup fails while installing the Client Access server role (the “ca” in the setup command line above) citing an error about not being able to find an object (see this URL), then you’ve got some damaged attributes in Active Directory.  In my case, while sitting on hold with Microsoft Support for an hour, I resolved it by doing a full dump of the domain and configuration naming contexts of Active Directory using LDIFDE:

  ldifde -f example.domain.ldf -d “dc=example,dc=com”
ldifde -f example.config.ldf -d “cn=configuration,dc=example,dc=com”

  I was then able to find the specific object with which Exchange Server 2007 Setup was having a problem and fix it.  In my case, the Offline Address Book server had somehow gotten damaged and was causing setup to fail.  I was able to manually correct the problem using Exchange 2000 System Manager and then Exchange Server 2007 setup proceeded to completion.

• Specifying a smart host on the SMTP virtual server properties on your “legacy” Exchange servers will cause a routing loop, and mail won’t flow between the new and old servers.  Apparently this is documented somewhere, although the Microsoft tech I spoke to could only point me to some articles about how to configure a smart host.  I haven’t seen any documentation yet that recommends checking and fixing this potential problem.  Furthermore, the troubleshooting tools in Exchange Server 2007 pick this up, but fail to tell you that it could be a problem.
• Oh, yes, I almost forgot about one: ASP.NET is required for Exchange Server 2007, but what happens when you can’t install it via Add/Remove Programs > Add/Remove Windows Components?  That’s right, back to the command line again:

  %systemroot%\Microsoft.NET\Framework64\
v2.0.50727\aspnet_regiis.exe -ir -enable

  This is assuming, of course, that you’ve already installed the .NET Framework 2.0 on your server in preparation for Exchange.

You are welcome to tell me that I’m an idiot for not knowing this stuff, on one condition: you provide a URL where information about the problem is posted and a workaround provided.  That way, when someone else runs into the issue, we’ll at least know where to point them for help.

Migrating from one Microsoft Exchange organization to another Exchange organization is always a troublesome task.  There’s free/busy time, public folders, the Global Address List (GAL), permissions, mailbox data, etc., to be worried out during the migration.  With recent versions of the Outlook client, there’s also the nickname cache to worry about, too.

The nickname cache is that functionality in Outlook that lets you start typing a recipient’s name (one that you’ve used before) and Outlook will “auto-complete” the name for you.  It’s a pretty handy feature, to be honest, and I’m sure that quite a few users out there rely on this functionality.  In fact, I’ve had people tell me that their nickname cache is more important than their contacts folder!

In a straight STMP-only situation, the nickname cache can be quite harmless during the migration.  In an Exchange migration, however (such as when migrating from one organization to a new organization, perhaps due to acquisition, rebranding, whatever), the nickname cache can be quite a pain in the rear.  Why?  Because the nickname cache doesn’t store the SMTP address of the Exchange recipients to which you’ve been sending e-mail; instead, it stores the X.500 address.

For example, when you mailbox-enable (i.e., create a mailbox for) a user object, Active Directory and Exchange will stamp the legacyExchangeDN attribute with an address that looks something like this:

/o=VMware Test Lab/ou=Raleigh/cn=Recipients/cn=scott.lowe

This is an X.500 address, and if an Outlook user sends an e-mail to this mailbox-enabled user object, this X.500 address will get added to the nickname cache.  For an Outlook user creates a contact for an Exchange recipient from the GAL, this X.500 address will be the address that is saved with the contact.  If this mailbox is moved to a new organization, this X.500 address—by default—won’t go with it, and that is the root cause of the most of nickname cache problems in a migration.

So how do we fix it?  Easy, actually.  Let’s say that you want to represent John Doe, who has a mailbox in Organization B, in the GAL for Organization A.  You create distinct SMTP namespaces for the two organizations, create SMTP connectors with the appropriate namespaces, and you test mail flow between the two.  Everything works fine, and so you create contacts in the GAL for Organization A to be able to send e-mail messages to those recipients in Organization B.

Thinking you’re pretty clever (which you likely are, since you’re visiting this site), you create contacts in Organization A to represent users in Organization B and vice versa.  Since these contacts are all SMTP based, routing messages based on the SMTP namespace, all should be well, right?  Nope.

Unfortunately, because these contacts were created on the server (we’re talking Active Directory Contact objects here, not Outlook contacts), users sending e-mail messages to them will be adding the X.500 address of the contact to their nickname cache, not the SMTP address.  When these users migrate from Org A to Org B and send mail to these recipients again, the system will generate an NDR (non-delivery report).  Why? Because the nickname cache has the X.500 address for the Contact object in the source AD tree.

But enough of the background.  How do we fix the problem?  Check out these steps:

1. Review the legacyExchangeDN attribute on the target mailbox.
2. Add the value of legacyExchangeDN (on the target mailbox) to an X.500 address on the Contact object in the source domain.  To create an X.500 address, select the “Custom Address” and specify the address type as “X500” (no quotes).

That’s it!  When users send e-mail to the Contact objects, the X.500 address will be stored in the nickname cache.  The Contact object’s targetAddress attribute will, of course, point to an SMTP address assigned to the target mailbox; that’s what allows Exchange Server to route the e-mail messages appropriately.  After the users are migrated to the new Exchange organization, the X.500 address for the users to which they used to send mail will still be the same as the Contact objects they used to use.  Perfect!

You’ve probably all run into the “Unable to relay” error message before.  The usual fixes to this problem are pretty straightforward:

• Add the address space to a recipient policy in Exchange System Manager; this tell Exchange to accept the messages as inbound; or
• Add IP addresses, subnets, or host names to the SMTP virtual server in Exchange System Manager as allowed to relay through this host;

Generally speaking, either one of these two options will generally fix the problem.  If the address space (say, example.com) should truly be accepted inbound, then adding it to an Exchange recipient policy will tell Exchange to accept it as inbound—thus eliminating the relay issue.  Likewise, if the messages should truly be relayed to their final destination, then adding the host(s) to the list of machines allowed to relay via the Exchange SMTP virtual server will indeed do that.  Pretty simple, right?

In this case, the e-mail address space should have been accepted inbound, so I first added the address space to an Exchange recipient policy (leaving the check box to apply that address space to recipients unchecked, since I didn’t really want addresses from this domain applied to users).  That didn’t work.  OK, maybe the check box to apply this domain needs to be checked, so I add it to a recipient policy that has no filter criteria (and therefore won’t be applied to any users).  It still doesn’t work.

Next, I add the appropriate servers to the list of hosts allowed to relay (by modifying the properties of the Exchange SMTP virtual server), restart the SMTP service, and try again.  It still doesn’t work, reporting a “5.7.1 Unable to relay for <username>” message.

My colleague searched the Microsoft Knowledge Base and turns up KB article 323669, titled “XFOR: 550 5.7.1 Unable to relay Error Message When E-Mail is Sent to Local Exchange Recipient”.  In the article, it mentions using MetaEdit to edit the IIS metabase and delete the LM/DS2MB key.

Good idea, but MetaEdit doesn’t support IIS 6.0, and this problem is occurring on Windows Server 2003 with Exchange Server 2003.  Fortunately, the IIS 6.0 Resource Kit comes with the Metabase Explorer, which allows me to delete the key as per the KB article and restart the services.  After restarting the services, the problem is fixed!  (Kudos go to Chauncey, who found both the article and the Metabase Explorer.)

Apparently, the DS2MB portion is involved in one-way replication of configuration data from Active Directory (the directory service, hence the “DS” in the name) to the metabase.  If the metabase (or even just the DS2MB portion of the metabase) gets damaged, then changes made in Exchange System Manager—which are written to Active Directory via the configuration domain controller—will never get synchronized to the IIS metabase.  Since the IIS metabase controls the SMTP service, then SMTP will continue to deny what it believes to be unallowed requests to relay.  Removing the DS2MB key and restarting the services rebuilds that portion of the metabase from scratch, thus pulling in the correct configuration from Active Directory and fixing the problem.

The Exchange Best Practices Analyzer (ExBPA) has been around for a while now, and is a very useful tool in making sure that your Exchange implementation is following recommended best practices from Microsoft for optimal performance, reliability, and scalability.  The ExBPA was recently updated to include the ability to check the overall Exchange topology for readiness to upgrade to Exchange Server 2007.  The Exchange team blog has full details on the new “Readiness Check” in this posting.

In addition, Microsoft has released the Internet Security and Acceleration (ISA) Server Best Practices Analyzer (IsaBPA), which is designed to do the same thing for ISA implementations.  Supporting both ISA 2004 and ISA 2006, IsaBPA helps make sure that the ISA deployment adheres to a set of best practices outlined by Microsoft, and identifies critical configuration errors or potential problems.  The IsaBPA can be downloaded from the Microsoft Download Center.

Also available for download are the SQL Server Best Practices Analyzer and the BizTalk Server Best Practices Analyzer.

A while ago, I discussed the use of Apache to protect OWA from web-based attacks.  This configuration placed an Apache HTTP server in front of a server running Microsoft Exchange Server 2003 to protect it against web-based attacks, offload SSL encryption, and enable name-based virtual hosts (for the conservation of public DNS hostnames, especially important for smaller organizations).  While this is a useful configuration, it is not without its drawbacks.

First, let’s review some of the advantages of this type of configuration:

• You can use the open source mod_security module to protect OWA against virtually all forms of URL-based attacks.  Mod_security is an extremely powerful and useful module that can greatly increase the protection against web-based attacks.  See the mod_security web site for more information.
• Even without mod_security, deploying Apache in front of OWA can protect the OWA server against many IIS-specific attacks.
• This configuration can be used in addition to IIS-specific protection such as URLScan.
• You can terminate the SSL connection at the Apache server instead of on the OWA server, freeing up CPU resources on the OWA server for other tasks (this is especially useful in smaller Exchange deployments where the OWA server may also be a mailbox server).

Now, for some of the disadvantages of this type of configuration:

• Apache lacks the intelligence of an Exchange server configured as a true front-end, and therefore cannot direct requests to multiple back-end mailbox servers.  In this type of configuration, the Apache reverse proxy always directs requests to the same OWA server and cannot determine which mailbox server the user is homed on.
• Organizations with expertise in Microsoft products won’t necessarily see any real benefit from this due to the added overhead and learning curve of supporting Linux and Apache.  (Don’t snicker, this is a real concern for organizations.)

I’m sure there are other disadvantages as well.  Anyone care to comment and share their experiences?

While performing some testing and research at the office today, I found myself in need of a way to mass-create some Exchange mailboxes.  A very quick Google search revealed just the tool I needed to perform the task:  ExchMbx, a freeware utility by the same author of AdFind and AdMod.

The real power of ExchMbx is demonstrated when combined with Dsquery (a Microsoft-supplied command-line tool for pulling lists of objects from Active Directory) or AdFind.  In these examples I’m using AdFind because tools such as Dsquery don’t work on Windows 2000 (at least, not in my experience).

So, let’s say you wanted to mailbox-enable all the users in a particular OU.  With AdFind, you could enumerate all the users in an OU like this:

adfind -dsq -b “OU=Users,OU=Department,DC=example,DC=net”
-f “(objectclass=user)”

(Be sure to type commands like this on a single line, not broken across lines for appearance’s sake as shown here.)

This will produce a quoted DN listed similar to the output of Dsquery (hence the “-dsq” switch).  Then, this output can be fed to ExchMbx:

adfind -dsq -b “OU=Users,OU=Location,DC=example,DC=net”
-f “(objectclass=user)” | exchmbx -cr “SERVER1:First Storage
Group:Mailbox Store (SERVER1)”

Here, the DN output from AdFind is piped to ExchMbx to create a mailbox on the database named “Mailbox Store (SERVER1)” in the First Storage Group on SERVER1.

Or, you could move all the user objects for the HR personnel to a new Exchange server or database:

adfind -dsq -b “OU=Users,OU=Location,DC=example,DC=net”
-f “(&(objectclass=user)(department=HR))” |
exchmbx -move “SERVER1:First Storage Group:Second Database”

To find which accounts don’t have an Exchange mailbox (perhaps you only created Exchange mailboxes for a subset of your users), this command will help you out:

adfind -dsq -b “ou=RTP,dc=legacyad,dc=net”
-f “(&(objectclass=user)(!(homeMDB=*)))”

You could then pipe this to ExchMbx again to create mailboxes, repeating the process until the AdFind command did not find any more accounts out there without mailboxes.

Of course, there’s a lot more to ExchMbx than just creating and moving mailboxes; you can also mail-enable objects, hide or unhide objects from address lists, and set mailbox quotas.  All in all, a very handy tool!

Earlier this week, Microsoft released a couple of patches on its standard monthly schedule.  These patches are designed to plug a couple of critical security flaws, including what appears to be a very serious problem with Microsoft Exchange Server.

The two Windows flaws are not terribly serious, in my opinion.  One, MS06-020, is rated “Critical” and plugs a problem with the Flash player.  So, technically, this isn’t a problem with Windows but with Flash, and Adobe has also released a security bulletin as well.  The second, MS06-018, fixes a flaw with the Distributed Transaction Coordinator (DTC).  This flaw can only cause a Denial of Service (DoS) condition and can be blocked at perimeter firewalls (but this, of course, won’t protect against internal threats).

Other related security advisories:
Secunia:  Microsoft Distributed Transaction Coordinator Two Vulnerabilities
Secunia:  Microsoft Windows Flash Player Code Execution Vulnerabilities

However, it is the Microsoft Exchange Server vulnerability, MS06-019, that is more troubling.  Remotely exploitable via anonymous connections (such as SMTP), this exploit is ripe for an automated worm.  What’s worse, typical perimeter firewall protections won’t help and no user intervention is required.  Simply getting spammed may be sufficient to affect your server!  This is one patch to get installed as quickly as possible (after appropriate testing has occurred, of course).

Read the Secunia advisory on the Exchange flaw here.

Also, a third party has uncovered an additional flaw in Windows that has not yet been patched.  This vulnerability affects compiled Help files (see more detailed information).  This one requires user intervention, so isn’t quite as likely to spread via a worm.

I started work at ePlus Technology yesterday.  I had originally thought that my first day would be on March 13, but things got delayed until the 20th.  So far, things are going pretty well.

One of my first assignments is to get up to speed with VMware’s high-end products, ESX Server and VirtualCenter.  I’m signed up for a VMware class in May and hope to be getting some hands-on experience with the products in the next week or so.  Having already worked with GSX Server (soon to be replaced by VMware Server) fairly extensively, I don’t expect the learning curve for ESX Server to be too bad.  I have a feeling that most people getting into ESX Server for the first time probably aren’t command-line people; I, on the other hand, sometimes feel like I live at the command line.  I should feel pretty comfortable.  I’ve managed to dig up some good resources with the organization as well; that’s one key advantage of working for a larger company.

I knew that integration of my PowerBook onto their corporate network would be a challenge, and I was right.  I had spent a fair amount of time on my network making sure that everything was cross-platform, but not here.  I’m going to be working first on universal e-mail access, so that I can use IMAP both inside and outside (without having to establish a VPN first).  I haven’t quite worked out all the details yet, but I imagine it will involve the use of the Perdition IMAP proxy and some DNS cunning.  I also need to get LDAP queries from the Mac OS X Address Book working against the corporate Global Address Book, so that I don’t have to add a bunch of people manually to my address book.  Snerdware, whose Groupcal product I do use (even now) to push iCal data up into my Exchange calendar, also offers a product for helping with that (called AddressX), but I’d rather avoid having to purchase yet another license if at all possible.  (And yes, I know that the Address Book has an option to synchronize with Exchange.)  Unfortunately, all of these changes will have to be done on some sort of local scope; that’s one key disadvantage of working for a larger comapny.

All in all, I’m excited about the opportunities that are ahead of me, particularly with regards to learning some new products.  Stay tuned!