It all started about a week ago. I got an alert email from a server, complaining about a RAID array. That’s nothing terribly out of the ordinary, as I have all of my systems set to alert me if something seems out of place. However, the problem in this case is that the alert didn’t actually come from one of my servers… even though the message claimed it did.
The message read (with domain names changed for security):
Since I’m using Gmail via Google Apps, I clicked the small drop-down arrow next to the recipient’s name (which was “root”) to display more details about the message. The details showed mailed-by: MyPersonalDomain.org as one of the items, meaning the message had actually passed an SPF check! This concerned me, since my personal domain’s SPF record explicity disavows all mail sent from servers not specifically mentioned in the record. Here’s my current SPF record for MyPersonalDomain.org:
The above SPF record translates to mean that the server with IP (v4) address 123.45.678.90 is allowed to send mail on behalf of MyPersonalDomain.org, as can any server that has a valid A or MX record in the domain’s zone file, and to also “include” the outbound Google Mail servers (since I’m using Google Apps). The -all directive at the end means that no other servers are allowed to claim they’re mine. And yet, one was… and it was successfully delivering mail to me as such.
Meaning something wasn’t right.
To investigate further, I selected the “Show Original” option in the message (which is one of my favorite features of Gmail). Here’s a copy of the full message shown, with all the headers:
For fellow mail geeks, you should be able to read the above and realize right away that a misconfigured mail server in Russia was causing all the problems by pretending to be MyPersonalDomain.org.
But for the slightly less geeky, here’s a walk-through of what was happening at every step of the message’s journey from the naughty server in Russia server all the way to my Google Apps inbox:
A server claiming to be MyPersonalDomain.org (but that actually lives at 91.242.161.26 — which is an IP in Russia and not my server) attempted to deliver a message to the authoritative inbound MX server for MyPersonalDomain.org, which is Google.
The Google MX server (10.112.72.70) receives the mail, Google checks the sender’s SPF record, and finds that the message “fails” the check, since 91.242.161.26 is not a “permitted sender.” However, instead of rejecting it here (like I believe it should have), Google decides to keep processing the message and allow it to continue on its journey. That’s strike 1 against Google, IMHO.
Because I have MyPersonalDomain.org configured as a domain alias in Google Apps with split domain routing, and because the root@MyPersonalDomain.org address doesn’t match any local Google Apps mail accounts, a Google outbound mail server (mail-lb0-f171.google.com [209.85.217.171]) sends the message on to the Backup.MyPersonalDomain.org server, which is the Postfix server I have set up to receive mail for all recipients on the MyPersonalDomain.org domain that don’t have Google Apps accounts (and bounce messages for any invalid addresses).
The Backup.MyPersonalDomain.org server scans for viruses and spam, and finds the message is clean, and then delivers the message to the “steve” account on Backup.MyPersonalDomain.org server.
The Postfix server on MyPersonalDomain.org is configured to forward any messages for steve@MyPersonalDomain.org on to stevej@MyGoogleAppsDomain.com, so that’s what it does. Note, however, that because the message was forwarded from the IP address that’s allowed in the MyPersonalDomain.org SPF record, the accepting Google mail servers declare that it passes the SPF check for the original sender! So even though it failed an SPF check earlier in the journey, because it eventually gets forwarded to Google by a permitted server, it’s allowed through!
The message gets delivered to my Google Apps/Gmail inbox at stevej@MyGoogleAppsDomain.com.
I was disappointed that the first failed SPF check hadn’t stopped the message dead in its tracks. This message was clearly a hacked or misconfigured mail server that was spoofing my domain and sending out mail on my domain’s behalf… and doing so every two hours! This is precisely what SPF records are supposed to protect me against… and yet Google let it slip through.
So I dug deep into the Advanced Settings sections of Google Apps, and found what I thought to be a promising featured called Blocked Senders. My optimism faded, however, when I realized it was impossible to block senders by IP address. Google only allows users to block individual sender email addresses or domains. But since I receive valid server messages from root@MyPersonalDomain.org (that really are sent from my servers) every day, I couldn’t block the spoofed sender or the domain! That’s strike 2 against Google Apps. You can whitelist IPs, but you can only blacklist domains or email addresses (and an hour on the phone with Google Apps support confirmed this). That makes no sense!
My next idea was to forego the split domain routing for the root@MyPersonalDomain.org address, so I created an alias in Google Apps for root@MyPersonalDomain.org so that any messages received for that account would be delivered directly to my Google Apps inbox. I hoped that by eliminating the server from the delivery route with IP address allowed by the SPF record, Google Apps would block or send the next message received from the misconfigured server to my Spam folder.
However, two hours after the previous message, a new one arrived, with these headers:
Google still “hardfailed” the message when checking the SPF record, but still delivered it directly to my newly aliased Google Apps inbox. Sigh. That’s really bad. Why bother setting strict SPF records if large mail handlers like Google won’t honor them?!
I tried contacting the admins of the IP address in question, but with no luck. Probably some Russian hackers, anyway. I ran a port scan on the remote IP, and found:
So that tells me they’ve got the system locked down pretty tight, including some BlackICE anti-intrusion software. I tried poking around the open ports for anything that could help me identify and get in contact with the owners, but found nothing. I’m guessing it’s using those 8083, 9080, and 9081 ports for its outbound mail — which could be attempts at spam, or could just be misconfiguration.
Finally, I took the least elegant approach of all. I set up a content filter in Gmail to send any message with “Report from /etc/init/d/mpt-statusd on MyPersonalDomain.org” in the body directly to the Trash. I call that the least elegant, because that doesn’t bounce anything back to the misconfigured server, which tells the remote admin he/she has a problem.
So I’ve stopped the bogus messages every two hours, but I’ve also exposed what I believe to be a couple of weaknesses in the way Google handles hosted Google Apps mail domains. First, if they won’t want to strictly enforce SPF records, they should at least give the mail customer (me) the ability to do so. And second, they should allow blacklisting by IP address, so that customers can quickly silence misconfigured or spammy mail servers that are sending useless crap.
Related posts:
Fixing Postfix “certificate verification failed for gmail untrusted issuer” Error Message
Cuong Nhu IATC 2013 Black Belt Meeting Google Hangout Instructions
Rant: Navigating Gerber’s Bear Grylls Parang recall process is harder than drinking your own pee