January 2006

A while back, I experimented with a multicast DNS (mDNS) responder for Linux.  (For those not already “in the know,” so to speak, multicast DNS is one of the key components of Bonjour, Apple’s automatic service discovery functionality—formerly known as Rendezvous).  For some strange reason, I had an urge to try it again today.  Here’s what I found.

First, I started looking for an “official” RPM package for a CentOS 4.2-based server that I manage.  Despite numerous Google hits that implied an official RPM existed, I could not find one.  (Pointers and/or URLs are welcome.)  I finally found a few RPMs on one of the CentOS mirrors, and installed it without any major issues.  The problem was, there was no documentation.  It installed an executable file called mdnsd, along with a directory in /usr/share/doc and a matching init script.  But how to configure it?  How to tell it what services to advertise via mDNS?

Having no luck whatsoever finding any additional documentation, I turned to a POSIX-compliant mDNS responder I had downloaded from Apple’s developer site and compiled on Red Hat Linux 9.0 some time ago.  I also had a simple init script for it, which (if I recall correctly) had been created by Rui Carmo of Tao of Mac (great site, by the way—I recommend it).  Fortunately for me, all I had to do was just copy the files over to the CentOS-based server and place the files in the right place, and it worked flawlessly.

Sure enough, I could now see this Linux-based server in Terminal.app’s “Connect to Server” dialog box.  I could not, however, see the server as an SFTP server in Cyberduck.  I briefly searched to see what kind of advertisements Cyberduck was expecting to see, but couldn’t find any information.  (Note, strangely enough, that Terminal.app could see the server as an SFTP server, but Cyberduck couldn’t.)

Now don’t ask me why exactly I was driven to tinker with this today, because I couldn’t tell you.

More information on multicast DNS, DNS Service Discovery, and related technologies can be found at the sites linked below:

DNS Service Discovery (DNS-SD) - http://www.dns-sd.org/
Multicast DNS - http://www.multicastdns.org/

For the past few weeks, I’ve been struggling with a couple of very difficult decisions to make, and searching to find the Lord’s will in each of these decisions.  I pray about these things, and every time it looks like one door is opening something happens that makes me wonder.  Is it the Lord telling me I’m leaning the wrong direction, or the Enemy seeking to distract me?

There are two key areas in which I’m really wrestling with issues right now.  One of these is my ministry to the church.  Some months ago, I felt like I was being called into a “higher” form of ministry within the church, specifically directed at youth/teens.  (I can’t really describe it better than that—I don’t know exactly what it is I’m supposed to do, just that I’m supposed to be doing more than I’m doing now.)  Then some things happen, and it looks like that’s not supposed to happen.  I also start to doubt myself, questioning my skills, my dedication, and my testimony.  “Surely the Lord doesn’t want someone like me ministering to the youth?  There are others that are more qualified and more educated than me.”  That’s what I tell myself.

And then something happens like what happened this past Sunday.  I had been down all morning, all weekend, and then I walked into my Sunday School classroom and someone (a “silent messenger”) had written this on my whiteboard:

The Lord doesn’t call people who are qualified to do His will.  He qualifies them in the act of calling them to do His will.
 
This is for you, Scott.

Now that may not really mean anything to you, but a few months earlier my wife had called me in the middle of the day one day to tell me she’d just heard something on the radio.  She’d been listening to the radio, and the radio show host said, “God doesn’t call the qualified, He qualifies the called.”  The same message, from two different people, at two different times?  I don’t believe in coincidence.

Well, that’s one decision settled…it seems pretty clear that the Lord wants me, that He is calling me.  I don’t know about you, but when the Lord starts calling, I’m not going to ignore Him.

Now if the second decision were so clear.  I have a big career-related decision to make—a really big decision.  There are a great number of advantages and disadvantages either way, and no clear or overwhelming reasons to go one way versus the other.  From a personal perspective, I have reasons to go either way.  This is a really tough position to be in, but I’m trying very hard to let the Lord have His will and His way in this situation, so that He will be glorified.

For any of you Christian readers out there that may have stumbled my way, I certainly desire your prayers of guidance and direction.

One of my projects involved the configuration of GRE (Generic Routing Encapsulation) tunnels, encrypted by IPSec, between two locations.  I was having some problems getting the tunnels to work properly, but now I’ve managed to resolve that problem, and the configuration is working well.  Here’s some additional information on the problem and how it was finally corrected.

This was my first project using GRE tunnels.  I’d used IPSec tunnels many times, and on many different platforms, but this time around we needed an interface that could be tracked for HSRP (Hot Standby Router Protocol) purposes, and until recently Cisco didn’t offer IPSec tunnel interfaces.  (I just came across some documentation last night that indicated very recent releases of IOS offer this functionality.)  So, the idea was to use GRE tunnels, track the GRE tunnels using HSRP for failover with another router, and encrypt the traffic using IPSec in transport mode.

The GRE tunnel configuration (scrubbed for sensitive data) looked something like this originally:

interface Tunnel0
 description GRE tunnel to other location
 ip address 192.168.254.1 255.255.255.252
 tunnel source FastEthernet0/0
 tunnel destination 172.31.254.1
 crypto map tunnel-ipsec-map

Of course, there was an appropriately configured interface at the other end of the tunnel as well.  The tunnels came up, and appeared to work just fine, until we added the keepalive statement.  (The keepalive statement is required for the tunnel to report an actual up/down status, necessary for HSRP interface tracking.)  Then they went down and stayed down.

A “debug tunnel” statement showed that the keepalives were being sent, but none were being received.  Thinking perhaps the IPSec configuration was incorrect, I removed the “crypto map” statement from the tunnel interface.  It still didn’t work.

After reviewing the configuration again, I began to suspect an MTU issue—the “show int tun0” output listed an MTU of 1514.  I consulted with a Cisco expert (recently obtained his CCIE), and he confirmed that it was most likely an MTU issue.  So I modified the configuration to look like this:

interface Tunnel0
 description GRE tunnel to other location
 ip address 192.168.254.1 255.255.255.252
 ip mtu 1400
 keepalive
 tunnel source FastEthernet0/0
 tunnel destination 172.31.254.1

At that point, the tunnel finally came up and I was able to pass traffic through the tunnel.  I re-added the “crypto map” statement to enforce encryption, and the tunnels promptly went back down again.

Once again, debug output saved the day.  The output from a “debug crypto” statement was constantly reporting “packet too small”.  A search of the Cisco web site turned up a result (I can’t find it now) that indicated a bug within IOS and suggested the addition of a “tunnel key” statement.  So, I modified the configuration again:

interface Tunnel0
 description GRE tunnel to other location
 ip address 192.168.254.1 255.255.255.252
 ip mtu 1400
 keepalive
 tunnel source FastEthernet0/0
 tunnel destination 172.31.254.1
 tunnel key 12345
 crypto map tunnel-ipsec-map

With this configuration, the IPSec/ISAKMP SAs were established and the tunnels came up, passing traffic as expected.  The debug output showed no crypto errors, and keepalives were being sent and received.  Success!

Earlier today I completed another GSX Server upgrade (from version 2.5 to version 3.2.1) for a customer, and fortunately this upgrade was much smoother than the last GSX Server upgrade.

No BSODs (Blue Screens of Death) this time during the uninstallation, and the previous version uninstalled itself cleanly.  The installation of the new version went quickly and smoothly, and in practically no time we were booting up “legacy” VMs under the new version of GSX Server.

As with the previous upgrade, those VMs running Red Hat Linux 9.0 detected “new” hardware (specifically, a “new” network card and a “new” SCSI card; ironically, they are exactly the same virtual hardware as before) and seamlessly migrated over the configuration without any issues whatsoever.

Unfortunately, I was dismayed to find that a VM running CentOS 4.2 did not maintain the network configuration while setting up this “new” hardware, so I had to go back in and re-enter the network settings.  This was only a minor inconvenience, as the network reconfiguration was quick and easy.  It’s also nice to note that the time synchronization problems with CentOS 4.2 appear to have resolved themselves now under the new version of GSX Server.  (Note that even after the changes that mostly resolved the NTPd problems, time was still slightly off and lots of NTPd messages were being logged; even those issues have disappeared as well.)

The network reconfiguration on a VM running Windows Server 2003, on the other hand, was not quick and easy.  As before, the network configuration simply disappeared during the discovery of “new” hardware.  In the last upgrade for this customer, the only Windows-based VM we had to work with was an older Windows 2000-based server.  I had hoped that the problems we’d seen with that server would be resolved in Windows Server 2003.

Not so.  The VMware Tools installation removed the driver and installed a new driver, so even if the network configuration had made it through the “new hardware” discovery process, it would have been hosed at that point.  Eventually, after a couple of different reboots, I finally had the Windows server up and running with its original network configuration again.  If there is one area I’ve found so far that VMware really needs to work on, it’s this one.

Last summer, I wrote about my concerns with regards to fourth-generation rootkits and their supposed beneficial intentions.  Now that the same approach is being applied to Oracle databases, I ask again:  isn’t there a better way?

A security researcher recently announced that he has created a better “rootkit” for Oracle that improves upon the earlier version unveiled last year at the Black Hat Conference in Amsterdam.&nbps; This new version makes it more difficult for database administrators and security professionals to locate the rootkit.  Supposedly, this is all being done to underscore the vulnerabilities and flaws in the Oracle database (and, to a lesser extent, Microsoft SQL Server, IBM DB/2, and others).

Isn’t there a better way?  As IT professionals—whether we be security experts, database experts, or networking experts—we ought to be able to find a way to openly discuss security flaws and vulnerabilities without actually creating tools for exploiting them.  Now what’s going to happen when this “rootkit” (my definition of rootkit is a bit more stringent than the one used in the referenced eWeek article) falls into the wrong hands and is used to steal hundreds of thousands of credit card numbers from a leading financial institution?  What if it was YOUR financial institution that was compromised using this tool?  Would you still be in favor of this approach then?

I suppose that’s the real value behind open source software; the flaws and vulnerabilities are out there for anyone to see in the source code itself.

I had a situation today where a customer forgot the root password to a Debian GNU/Linux 3.1 system in their office.  That left it up to me to try to find a way to get into the system.  Here’s how I managed to gain access.

(Note: As far as I am aware, NONE of the information I’m going to list in this article will work across the network; you MUST have physical access to the server.  Therefore, I’m not too terribly worried about “making it easier for the hackers”.  If you don’t have physical security, then no amount of electronic security is going to help you!)

Here’s how it works:

1. With physical console access, reboot the server.
2. When the Grub menu comes up, press “e” to edit the menu selections.
3. Use the arrow keys to select the Kernel line, then press “e” again.
4. Add “single init=/bin/bash” to the end of the existing line.
5. Press “b” to boot the modified line.
6. The system will boot up into single-user mode.  Unfortunately, the root filesystem will be mounted read-only, so you’ll need to remount it using “mount -o remount,rw /”.
7. Use the “passwd” command to change the password for root to whatever you like.
8. Reboot the computer again and log in as root with the new password.

There are ways to protect against even this (a BIOS-based power-on password, or passwords in Grub to prevent casual editing of the boot configuration), and those steps may be necessary depending upon the other aspects of physical security.  If this system is out where people can get to it, then I’d highly recommend taking these additional steps to secure the server.

Please note that I’ve only done this on Debian GNU/Linux 3.1, but I would be reasonably confident that the steps will work elsewhere as well.

A short while back I mentioned that I was having a bit of a problem with network address translation (NAT) on a Cisco router.  I’ve managed to get the issue resolved, so here’s the solution in case someone runs across this problem in the future.

In this instance, the original configuration of the router provided a means for both dynamic NAT and static NAT.  Specifically, all the workstations on the LAN would be dynamically translated using port address translation (PAT) behind the external interface of the router, while the web server itself would be statically translated.  Normally, this would not be a problem.

This feat was accomplished using an access list, like this:

access-list 1 permit 172.16.1.0 0.0.0.255

This access list was then applied to the interface using a route map.  The problem here, though, is that this dynamic NAT setup includes the IP address of the web server (let’s just say for the purposes of this example that the web server is 172.16.1.10).

So, to fix the problem, we modified the access list to specifically exclude the web server’s IP address:

access-list 1 deny 172.16.1.10
access-list 1 permit 172.16.1.0 0.0.0.255

This took care of the apparent conflict between the dynamic NAT setup and the static NAT setup (which was accomplished using an ordinary “ip nat inside source” command), and the setup has worked without any problems since then.

Now, if I could just figure out why my GRE-over-IPSec tunnels aren’t working, I’d be in really good shape…

Over this past week, a wide collection of vendors have released patches for various vulnerabilities and flaws in their applications.  Here’s a quick recap of some of the security patches released this past week.

• Oracle:  Oracle released patches for more than 100 flaws in their database and application server products.
• Symantec/Veritas:  Symantec released patches for its NetBackup application (acquired with Veritas) after exploit code was released on the Internet.  It does not appear that this flaw also affects the more-popular Backup Exec application.
• “Do It Yourself” Trojan Builder:  Sunbelt Software claims to have uncovered a do-it-yourself Trojan builder that allows unsophisticated crackers to build applications for identity theft and fraud.  Does this signal a new era for the “script kiddie”?

Of course, this list doesn’t include the patches from Microsoft that were released last week, which included fixes for two ‘critical’ e-mail server flaws.

As usual, the activity in software patching is another reminder that we must continue to be vigilant in the construction and maintenance of our networks.

Is it just me, or is anyone else bothered by the recent stories of MSN and Yahoo turning over customers’ search terms to the Federal government?  Apparently, the same request was made to Google as well, and Google is battling the request.

Supposedly, the data being requested by the government is intended to be used as evidence in a trial that could revive a child privacy law intended to protect children online.  As a father myself, I know how important it is to safeguard our children while they are online, but I’m loath to let the government do it via legislation—the track record isn’t really all that great.  Remember CAN-SPAM, anyone?

In any case, perhaps I’m just paranoid, and some would claim that my resistance to this and other similar moves indicates a desire to hide something from the government.  No, it’s just that as a free society that desires to remain free we’ve got to draw the line somewhere.  Why not here?

I wouldn’t dare to describe myself an expert on the use of such sites or services as del.icio.us, Technorati, or other sites that promote the use of “tags” as a means of organizing links or other content.  I do use these sites (you’ll note the “Technorati Tags” added to each post), and I do find them very useful.  However, I’m beginning to shift tactics in my use of tags, and hoping that this change will make these services even more useful.

Up until now, I’ve used tags primarily as categories, as a way of grouping links (or other content) together.  For example, my Linux tag on del.icio.us contains links that all primarily center around Linux.  That makes perfect sense, but what about links that discuss Linux integration with Windows?  With my old system, that creates a quandary, a conflict that is not necessarily easy to resolve.

Moving forward, I’m going to start using tags more as keywords than categories.  This will mean that more items will have multiple tags associated with them (and that makes sense), but it will also mean more tags than I have used in the past.  This will allow me to more easily tag links and content in such a way that I hope will be more useful to myself and others.

I will continue, however, to attempt to use the same sets of tags across multiple sites and services, just as I do now with my del.icio.us links and my weblog content.

If anyone has any feedback on other ways I can make tagging work more effectively, I’d love to hear about it—perhaps someone out there would care to share their tips, tricks, and techniques?