Exchange

I started work at ePlus Technology yesterday.  I had originally thought that my first day would be on March 13, but things got delayed until the 20th.  So far, things are going pretty well.

One of my first assignments is to get up to speed with VMware’s high-end products, ESX Server and VirtualCenter.  I’m signed up for a VMware class in May and hope to be getting some hands-on experience with the products in the next week or so.  Having already worked with GSX Server (soon to be replaced by VMware Server) fairly extensively, I don’t expect the learning curve for ESX Server to be too bad.  I have a feeling that most people getting into ESX Server for the first time probably aren’t command-line people; I, on the other hand, sometimes feel like I live at the command line.  I should feel pretty comfortable.  I’ve managed to dig up some good resources with the organization as well; that’s one key advantage of working for a larger company.

I knew that integration of my PowerBook onto their corporate network would be a challenge, and I was right.  I had spent a fair amount of time on my network making sure that everything was cross-platform, but not here.  I’m going to be working first on universal e-mail access, so that I can use IMAP both inside and outside (without having to establish a VPN first).  I haven’t quite worked out all the details yet, but I imagine it will involve the use of the Perdition IMAP proxy and some DNS cunning.  I also need to get LDAP queries from the Mac OS X Address Book working against the corporate Global Address Book, so that I don’t have to add a bunch of people manually to my address book.  Snerdware, whose Groupcal product I do use (even now) to push iCal data up into my Exchange calendar, also offers a product for helping with that (called AddressX), but I’d rather avoid having to purchase yet another license if at all possible.  (And yes, I know that the Address Book has an option to synchronize with Exchange.)  Unfortunately, all of these changes will have to be done on some sort of local scope; that’s one key disadvantage of working for a larger comapny.

All in all, I’m excited about the opportunities that are ahead of me, particularly with regards to learning some new products.  Stay tuned!

Outlook Web Access (OWA) is the web-based interface for accessing e-mail and other resources handled by Microsoft Exchange.  Unfortunately, OWA’s popularity also makes it the target of numerous worms and security exploits.  As a result, many organizations seek to deploy OWA behind a reverse proxy that can help shield OWA from web-based attacks and exploits.  In this posting, I’m going to share information to help build a reverse proxy using Apache 2.0.

Here’s a skeleton of an httpd.conf file to support Apache as a reverse proxy in front of OWA:

NameVirtualHost 1.2.3.4:80
NameVirtualHost 1.2.3.4:443
ProxyRequests Off

<VirtualHost 1.2.3.4:443>
ServerAdmin webmaster@domain.com
ServerName webmail.domain.com
DocumentRoot /var/www/webmail
RequestHeader set Front-End-Https “On”
ProxyRequests Off
ProxyPreserveHost On

SSLEngine On
SSLCertificateFile conf/webmail-ssl-cert.pem

<Location /exchange>
ProxyPass http://mail.domain.com/exchange
ProxyPassReverse http://mail.domain.com/exchange
SSLRequireSSL
</Location>

<Location /exchweb>
ProxyPass http://mail.domain.com/exchweb
ProxyPassReverse http://mail.domain.com/exchweb
SSLRequireSSL
</Location>

<Location /public>
ProxyPass http://mail.domain.com/public
ProxyPassReverse http://mail.domain.com/public
SSLRequireSSL
</Location>

</VirtualHost>

The key portions of this configuration are described below, along with some supporting information.

• NameVirtualHost:  The NameVirtualHost directive enables Apache to use name-based virtual hosts on the specified IP addresses and ports.  The parameter to the NameVirtualHost directive must match one of the VirtualHost definitions, as shown in the sample configuration, or else the content will be served from the default virtual host (the first virtual host listed in the configuration).  Note that if the Apache reverse proxy will not be using name-based virtual hosts (instead using IP address-based virtual hosts or running only a single server instance), then this directive is not required.
• RequestHeader:  This directive instructs Apache to add a header “Front-End-Https: On” to requests sent to the internal OWA server.  This header is proprietary to OWA and forces OWA to build URLs using “https://” references instead of ordinary “http://” references.  This directive is required in order to terminate the SSL tunnel at the reverse proxy and use clear-text HTTP between the reverse proxy and the internal OWA server.  This directive requires the mod_headers module.
• ProxyPreserveHost:  This directive configures Apache to pass the original host header, supplied by the client, to the server to which the request is being proxied.  (This is instead of the host name supplied in the ProxyPass directive.)  Again, this facilitates the construction of URLs with the correct hostname when accessing resources inside OWA.
• SSLCertificateFile:  Apache expects the web server’s SSL certificate to be in PEM format.  If the certificate’s key is encrypted, Apache will prompt upon startup for the passphrase to the key (this prevents any form of automated startup).  It is considered a security best practice to keep the key in a separate file (using the SSLCertificateKeyFile directive) in encrypted form and supply the password upon the startup of Apache.

With this configuration in place, the following benefits are realized:

1. Name-based virtual hosts are supported. This allows other URLs to also be proxied through this same reverse proxy server.
2. SSL encryption is offloaded from the Exchange server to the reverse proxy server. Traffic from the reverse proxy server itself to the Exchange server is standard, unencrypted HTTP.
3. When used in conjunction with mod_security (another Apache module), OWA is protected against a very significant majority of all web-based attacks.

Using Apache to serve as a reverse proxy for OWA is a cost-effective way to add another layer of security to an Exchange-based messaging infrastructure.

Every now and then, I like to post out here a list of my current “tech projects.”  These are the things that I’m working on for my own network, things that I may or may not start recommending to or supporting for customers.

Here’s my current list:

• InterNetNews (INN):  I had an installation of INN up and running a short while back, but had to resort to an ugly hack with stunnel in order to make SSL work from a newsreader.  To get a clean build, I’ve decided I’ll just start from scratch with a clean installation.  I’ll be using CentOS 4.1 again as I work on transitioning all my Linux-based servers to a newer Linux distribution, and I’ll be compiling INN from source instead of using a package.
• OpenBSD-based antispam gateway:  I’ve got an antispam gateway running right now (uses Red Hat Linux, Postfix 2.1, SpamAssassin, Postgrey, Razor, DCC, and ClamAV), but I want to try building one using OpenBSD 3.8 (just recently released) and newer builds of Postfix, SpamAssassin, and Amavisd-New.  In particular, I’m interested in the advanced integration of newer versions of Postfix and Amavisd-New.
• XC Connect:  I’ve also mentioned XC Connect before as well, but a previous installation proved to be unstable, and the Apache integration was less than stellar.  In fact, the integration was nonexistent.  I’m going to try a clean build of CentOS 4.1 and XC Connect to see if that will correct the stability and integration problems.

I also need to wrap up the documentation for a few completed items, such as the Cisco VPN integration with Active Directory.  Mac OS X integration with Active Directory is also on the “to do” list, but it will have to wait a little while—I’ll need to find another Mac to “experiment” with instead of using my own PowerBook.

As you may already know, I’ve been struggling with a bug in an environment running Exchange Server 2003 and Windows Server 2003 with SP1.  The bug is manifested as an error that appears when users with the properly delegated permissions attempt to add or modify an e-mail address on an already mail-enabled or mailbox-enabled object.  The error, listed as error ID c10308a2, contains text along the lines of being unable to determine if the Microsoft Exchange System Attendant service is running.

The underlying issue is a change that Windows Server 2003 SP1 makes to the security descriptors applied to the Service Control Manager.  This change in security descriptors now makes it impossible for non-administrators to query service status; hence, the error message.

In trying to apply the fix suggested by Microsoft in KB905809 (the use of the SC.EXE command), the error was never resolved.  It turns out that the workstation I was using the test environment was configured not to use the primary DNS suffix, but instead use a predetermined DNS suffix search list.  This configuration resulted in the system’s AD domain name not being in the suffix search list.  As a result, even though the fix from Microsoft was applied, errors still occurred.

This morning I double-checked everything on the test servers as well as the test workstations, corrected the problem described above, and reset the environment to match the production environment.  Then, walking through the tests again, I confirmed that running the SC command to add permissions to the Authenticated Users group (see the KB article linked above for more details, then see this explanation of SDDL syntax) does indeed resolve the issue.

So, finally, we can put this issue to rest.  If you are running Exchange Server 2003 with Windows Server 2003 SP1 and finding that your non-administrative users can’t add or modify e-mail addresses using Active Directory Users & Computers, see KB905809 and run the SC command listed in there.

As the next-to-last article in a series of postings about the problem described in this KB article, it appears that Microsoft will not be producing a hotfix for this bug.  In speaking with Microsoft earlier today, it appears that no hotfix will be produced for this, even though it is a confirmed bug.  As of this afternoon, the only known fix for the problem is using both methods outlined in the KB article.  I hope to confirm that for sure on Monday.

I’m also going to confirm on Monday whether membership in the local Administrators group (on the actual Exchange servers) will also fix the issue.

The next posting about this problem will be the last one regarding this issue.  Really.

If you are planning an in-place upgrade of your server running Exchange 2000 to Exchange Server 2003, beware of the Badmail folder.  Apparently, during the Exchange Server 2003 setup process, the setup application tries to go back and stamp ACLs (access control lists) on all the objects in the installation directory.  This, by default, includes the Badmail directory.  If your Badmail directory contains lots of items (which, in an Exchange 2000 installation, it probably does), then this can cause the Setup process to appear to be hung.  Microsoft has published this KB article discussing the issue and the resolution.

Fortunately, in Exchange Server 2003 SP1, Microsoft has changed the behavior of Exchange to use the Badmail folder only if explicitly configured to do so (see this KB article).  No more monitoring the Badmail folder!

In addition, for those networks that have not yet deployed Exchange Server 2003 SP1, Microsoft has released the BadMailAdmin tool.  I’ve tested this, and it works as advertised.

I finally managed to get Perdition working.  Still unable to confirm if Mac OS X’s Mail.app supports STARTTLS (my experience thus far says No), I had to resort to using Stunnel to wrap IMAP inside an SSL tunnel, then forward the IMAP traffic to Perdition on the same host.  The Perdition proxy then passes the traffic to the back-end mail server.  It’s not the solution that I really wanted, but it will do for now.  At least the Exchange Server 2003 IMAP server isn’t exposed directly to external networks.

On a slightly related note, the Slipstick Systems web site has a link to an IMAP proxy server that implements STARTTLS as a workaround for Exchange’s lack of native support for STARTTLS.  The IMAP proxy can be found at http://www.slipstick.com/files/imapproxysvc.zip.  So, if you have an IMAP4 client that supports STARTTLS and want to connect it to Exchange, you can use this IMAP proxy.  At least, until Microsoft puts STARTTLS support into Exchange directly.

In my experiments with Perdition, I learned a couple of very interesting facts.  First, the IMAP4 implementation on Exchange Server 2003 does not support the STARTTLS command, as described in RFC 2595 and re-affirmed in RFC 3501.  Instead, Exchange expects an SSL session to be established immediately, and then IMAP is spoken.  This is similar to the “smtpd_tls_wrappermode” directive that Postfix supports.

Second, it appears that the Mac OS X Mail application (commonly referred to as Mail.app) also uses this IMAP-over-SSL approach, since I’ve been using Mail.app to connect to Exchange using IMAP with SSL for quite some time.  I’m trying to confirm that now, but having precious little luck finding any definitive information one way or the other.  If anyone knows for certain, please let me know.  I’m going to keep searching.

This is one of those things that just makes me crazy.