Exchange 5.5, open relays, and why I hate Microsoft

[ShowMe]

Tale of a Systems Administrator
Or: Like I need another reason to hate Microsoft

As a System Administrator I’m used to people telling me that their mail couldn’t get through. It’s almost always a problem on the other end. After all, if mail wasn’t working why would someone send me an email asking if it was working?

I run Exchange 5.5 and have been for several years. I've never had a problem. I constantly check it against ordb.org and abuse.net. I've been pretty confident that I was secure.

So, earlier this week, I am completely taken aback when a user forwarded me an email which is a non-delivery report (NDR), sent from a customer site who was blocking us via spamcop.

*MY* server? Blocked for sending spam? This cannot be!

I checked the log (in this business you learn to NEVER take anything for granted) and, lo and behold, I am shocked to find over 2000 messages. And none of them have an originator that resides in my domain.

Various hilarity ensues; hours of checking, researching and testing my server. Could my port forwarding system be compromised? Or is it my Exchange server? I know I'm not an open relay, all the tests prove that. And I've got all the patches and hot fixes applied; I checked and everything was up to date...yet I was still sending spam.

I hit the newsgroups and begin searching. And searching...and testing, and searching. I set up logging on Exchange so I can trace the problem.

Exchange logging, I discover, is next to useless. It can keep copies of all sent mail, so I had a lot of sample spam. Just what I need, more letters telling me how to enlarge certain parts of my body. And the logs show who I’m connecting to, which really isn’t my concern at the moment. I KNOW I’m sending out spam & all that is going to do is allow me to tell who I’m pooching.

But what is the FROM IP address? Not that it matters, it’s probably spoofed anyway.

I dropped Ethereal (a packet sniffer) onto the Exchange box and started logging. Do I need to mention that Ethereal caused my machine to BSOD? No, of course not. Since it's a Microsoft OS that is simply assumed. So I set Ethereal to write to a file instead of a screen and I was able to capture packets when the spammer started spamming.

I turn off the Internet Mail Connector to stop the sending of spam and open up my capture file. I am told that 1 MB = approx. 640 typed pages and after the last day and a half that sounds about right. I had 4 MB of worth of capture and 2500 pages seems correct, snce I spent the night going through every...single...one....

After I poured through the capture, trying to decipher what I was looking at (and calling several friends to add to the brain trust) I narrow it down to the sequence where the spammer first logs in. It has an AUTH LOGIN with a hashed Username and Password.

To someone who is familiar with mailbox protocol this is probably the equivalent of a Dr. Seuss primer to a college graduate. To me it was nonsense & I spent several hours trying to figure out what it all means. The most interesting part, to me, was the 334 sequences. Those looked like hashed passwords; in fact, they looked like UUE encoding. (Hey, downloading all that porn years ago finally paid off!)

So I take a bit and find a uudecode program that will allow me to decode these things to a file. Unhashed here's what it reads:

9/2/2003 6:18:45 PM : <<< IO: |AUTH LOGIN
|
9/2/2003 6:18:45 PM : <<< AUTH LOGIN
9/2/2003 6:18:45 PM : >>> 334 Username
9/2/2003 6:18:46 PM : <<< IO: |Administrator
|
9/2/2003 6:18:46 PM : <<< Administrator
9/2/2003 6:18:46 PM : >>> 334 Password
9/2/2003 6:18:46 PM : <<< IO: |
|
9/2/2003 6:18:46 PM : <<< ########
9/2/2003 6:18:46 PM : >>> 235 LOGIN authentication successful


OK...WTF? WTF? WTF? Someone is logging in as Administrator???? With no password?? And Exchange is ALLOWING it??

Oh, I’d love to have a drink right about now. Except that I don’t drink, and if I start I know I’ll never get this thing fixed.

It's now roughly 11:00 pm and I'm going absolutely insane. My server, which (in theory) is completely locked down & has been for several years, is now allowing administrative access? The system is virus free, several scans have shown that. Every test on I could find shows that I am NOT an open relay. All Service Packs and Hot fixes have been applied. Routing is set up to only allow authenticated users to send mail, and the Administrator account most certainly does NOT have a null password.

Back to the news groups and more Internet research, looking for any mention of this bug. Oddly enough I find it in a throw-away line on one of the news groups: Someone is trying to set up his sendmail program to communicate with an Exchange server and someone else mentioned to be sure to plug the "smtp auth hole" in Exchange.

SMTP Auth hole? says I. That's a new one on me.

Finally someone mentions how to plug this hole and refers me to a KB article. There's an explanation of EHLO and AUTH LOGIN, and something about Exchange not supporting it. It's still unclear to me, but it does lead me to this article:

http://support.microsoft.com/default.aspx?scid=kb;en-us;235627

You will note that this article doesn't mention that it also fixes a bug where a user can simply log into your network using the AUTH LOGIN command.

Ominously I also saw this at the bottom of the post that fixed this problem for me:

"All we need now is a similar fix for Exchange 2000 - I haven't found one yet."

So...have I stopped the spammers? I hope so. No spam sent since I implemented the patch at 11:30 pm Tuesday night. I get a lot of "AUTH LOGIN - Security Not Available" in my logs now. I hope I've fixed the spam and haven't shut down our system to half the Internet, though it seems that the spam lists did that anyway.

That's 30 hours of my life, distilled to several paragraphs on a web page. I will end with this remarkably apt quote, sent to me by a friend of mine when I was in the midst of this:

From "Secrets and Lies:"
"The first firewalls were on trains. Coal-powered trains had a large furnace in the engine room, along with a pile of coal. The engineer would shovel coal into the engine. This process created coal dust, which was highly flammable. Occasionally the coal dust would catch fire, causing an engine fire that sometimes spread into the passenger cars. Since dead passengers reduced revenue, train engines were built with iron walls right behind the engine compartment. This stopped fires from spreading into the passenger cars, but didn't protect the engineer between the coal pile and the furnace. (There's a lesson for sysadmins in this somewhere.)" -Bruce Schneier.

[bignickel]

There's probably quite a few Unix admins chuckling at your post.

The rest of us Microsoft users who have admin-ed one time or another are chuckling while grimacing. Your post was like watching a guy in a movie take a hit to the family jewels.

Ouch.

Godd@mn Microsoft!

[a_unique_person]

From what I have read of Unix security, it was as leaky as Microsoft software too. It just has the benefit of maturity. Ditto for MVS. When I was an MVS systems programmer, there were constant digs at the lack of security in Unix. Eg, when you started up the mail program, at ran with the 'auth' bit on. When you escaped to the shel, you were still running as root.

Note that while all this was going on, (several years ago), there were still patches for MVS security issues coming along. Normal patches, or PTFs, had a commentry on what the bug was and what was being done to fix it. Security patches were always blank in the commentary section.

I think the main issue for virus writers is, why bother with the other operating systems, when the majority of users out there are Windows users?

MVS was always more secure right from the start because it had a proprietary networking system and OS, and there weren't too many installations around. There was the odd security hole in it, though, one of which I found. It was exposed by a third party product.

[Wudang]

After a few years working on AIX and NT I'm back home on MVS as a sysprog. In my previous incarnation I ran penetration tests on live systems for my company. My record from getting a standard userid and password (for TSO) to being superuser (RACF system Special) was 20 minutes. 3rd party SVC not verifying registers correctly. My record lasted 2 months when 2 other guys managed 10 minutes via a brain-dead security administrator. The main problem I kept seeing, and still do, is lazy programmers who don't know as much they think they do.

[a_unique_person]

Originally posted by Wudang
After a few years working on AIX and NT I'm back home on MVS as a sysprog. In my previous incarnation I ran penetration tests on live systems for my company. My record from getting a standard userid and password (for TSO) to being superuser (RACF system Special) was 20 minutes. 3rd party SVC not verifying registers correctly. My record lasted 2 months when 2 other guys managed 10 minutes via a brain-dead security administrator. The main problem I kept seeing, and still do, is lazy programmers who don't know as much they think they do.

With MVS, that doesn't surprise me. No one knows much about it. I was just your run of the mill systems programmer in a small shop. 20 minutes? Hmm.

I didn't know they were handing out new jobs for MVS, or that anyone would want to take one even if it was there. Every year, the number of installations is getting smaller, not bigger.

[Wudang]

Big bank, lots of CICS and DB2. I was lucky enough to be job hunting in 1999 when the Y2K panic was at its height and after all this "the mainframe is dead" rubbish had died down amongst grown-ups. Acres of intel and unix systems but the core data is on the big iron where it belongs. Interestingly one job offer at the time was for a credit card company opening a shiny new mainframe site. It'll keep me employed for a few years and at home I have a mixed linux/windows network where I mess about with new software (http://www-106.ibm.com/developerworks/offers/cd/index.html)