Send SPF verified gmail from a non-gmail address without “on behalf of” using a lightweight python SMTP relay

e-mail, iphone

The gmail blog   describes a wonderful feature of gmail which allows you to use gmail to  send e-mail from youraddress@yourdomain.com  (rather than yourgmailaddress@gmail.com or yourothergmailaddress@googlemail.com).

As somebody with their own domain, who has loads of carefully seperated different e-mail identities – all of which funnel incoming e-mails to a single gmail account – I sometimes find it necessary to reply from one of these addresses.

In this article we’ll discuss how it’s possible to arrange for this to happen.

Challenges I faced (which are likely to be common to many people with more complex e-mail setups!) include:

  1. I want to be able to continue using the excellent gmail front end
  2. I don’t want to pay google to host my domain (just yet – although this may one day change)
  3.  I use many forwarding aliases which do not have their own  actual unique mailbox (but which do funnel mail into my gmail account)
  4. I don’t want my e-mail password being sent in plain text  (and, as I eventually discovered, gmail’s servers  do not support CRAM-MD5)
  5. My ISP, in common with other Internet Service Providers,  restricts access to specific IP ranges.  Specifically, it does not permit  use of their SMTP server from any address that does not belong to one of their customers (e.g. gmail mail servers).  This means that the “without on-behalf-of”  solution described in the gmail blog just doesn’t work for me.
  6. I want to be able to send e-mails from multiple from addresses in my iPhone
  7. I do not want to run an open mail relay
  8. I want to be able use SPF to avoid e-mail from my domain being marked as SPAM
  9. I do not want to have to store and resend e-mails if remote domain’s mail servers failed – I want to use my ISP’s high availability servers for this task  (not my own)

However, resources I have available to me (that not everyone will have available to them) include:

  1. A home server  (in fact it’s an ultra-low power server that I always leave switched on)
  2. A static IP address
  3. The ability to set up a port forwarding rule from my firewall to the server
  4. My own domain name (and control of the DNS records for that domain)
  5. An ISP-hosted SMTP server that does allow access from my home server

The custom mail forwarder solution I chose played to these strengths.

Despite hunting around, I wasn’t happy that existing mail servers really met my all my needs in a straightforward manner – and most seemed overkill for this solution.

So,  starting from  the python standard library’s smtpd.py I added the following features:

  1. TLS support  (to secure the connection from gmail to my mail server)
  2. Password-based authentication
  3. e-mail address whitelist  (an extra level of obscurity in case)
  4. A smattering of crude syslog logging

(oh, and the basics of CRAM-MD5 support – although that turned out to be a total waste of time as google’s servers don’t support that. )

To use the server you’ll need some basic python coding skills. You’ll have to:

  • deploy python2.6  and openssl
  • Get the source
  • generate certificates as per the instructions at the end of the source code (or use certificates you already have)
  • Modify lines the e-mail addresses, usernames and passwords in lines 21-44 of the script to suit your setup
  • Write some kind of auto-startup script to run this as a low-privileged user when the system starts up
  • Configure DNS smtp2.yourdomain.com to point to your public IP address
  • Set up port forwarding rules to forward ports 487 and 465 to port 60025 of the server running this code
  • If you want SPF, set up a DNS TXT record appropriately for your domain (authorizing your ISP’s email server to send e-mail for your domain e.g. “v=spf1 include:smtp.myisp.com ~all”)
The source:
#! /usr/bin/env python

import smtplib
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
import time
import sys
import smtpd
from smtpd import DEBUGSTREAM, EMPTYSTRING, NEWLINE
import asyncore
import asynchat
import base64
import ssl
import hashlib
import random
import hmac
import syslog

__version__ = 'My First Authenticating SMTP proxy version 0.8'

# The credentials and connection information to use when sending to our upstream SMTP server
outgoing_username="myISPmailserverusername"
outgoing_password="myISPmailserverpassword"
outgoing_mailserverhost="smtp.myISP.com"
outgoing_mailserverport=25

# The credentials Mail User Agents (e.g. gmail etc) must use to be permitted to use this relay
INCOMING_SMTP_CREDENTIALS = {"asecretusername":"somereallystrongpassword"}

# A whitelist of permitted "from" accounts that may be permitted to use this relay
# Each of the following addresses is in ***lower*** case
acceptable_from_email_addresses = [
"myaddress@mydomain.com",
"mysecondaddress@mydomain.com",
"myotheraddress@myotherdomain.com",
"yetanotheraddress@mydomain.com"
]

# You'll need to set up a port forward from port 465 and port 587 on your firewall's public IP
# address to port 60025 on your server.
incoming_port = 60025
incoming_listen_address = "0.0.0.0"
RELAYFQDN= "smtp2.mydomain.com"

def logmessage(message):
    syslog.openlog("emailproxyd", syslog.LOG_PID, syslog.LOG_MAIL)
    print message
    syslog.syslog(syslog.LOG_ERR, message)

class CredentialRepository(object):
    def __init__(self):
        self.username_to_password_lookup_dict = INCOMING_SMTP_CREDENTIALS

    # will throw an exception if username is unknown
    def getPasswordForUser(self, username):
        return self.username_to_password_lookup_dict[username]

class NotAuthenticatedException(Exception):
    def __init__(self, value):
        self.parameter = value
    def __str__(self):
        return repr(self.parameter)

def forwardmess(fromaddr, toaddrs, message):
    server = smtplib.SMTP(outgoing_mailserverhost, outgoing_mailserverport)
    server.set_debuglevel(1)
    server.login(outgoing_username, outgoing_password)
    server.sendmail(fromaddr, toaddrs, message)
    server.quit()
    logmessage("Message forwarded")

#
# After laboriously implementing and debugging this python CRAM-MD5 password authenticator,
# I was *gutted* to discover that gmail did not (and had no plans to ever) support CRAM-MD5. Darn.
# See http://www.google.com/support/forum/p/gmail/thread?tid=40351de0112c5962&hl=en
# It's left here purely for posterity.
class CRAMMD5PasswordVerifier:
    """Responsible for verifying credentials"""

    #See: http://www.fehcom.de/qmail/smtpauth.html
    # The challenge 'PDI0NjA5LjEwNDc5MTQwNDZAcG9wbWFpbC5TcGFjZS5OZXQ+'
    # translates to '<24609.1047914046@popmail.Space.Net>'.
    # The leading and trailing brackets ('<', '>') are mandatory, as
    # well the portion of the challenge which provides the hostname
    # after the '@'. '24609.1047914046' is a random string,
    # typically build from the 'pid' and the current time stamp to
    # make that challenge unique.

    def __init__ (self, server_fqdn = RELAYFQDN):
        self.fqdn = RELAYFQDN

    def get_b64_challenge(self):
        randomstr = hashlib.md5(str(time.time())  + str (random._urandom(32))).hexdigest()
        challenge = "<%s@%s>" % (randomstr, RELAYFQDN)
        return base64.b64encode(challenge)

    def is_authenticated (self, username, password, b64_challenge, response):
        try:
            expected_response = hmac.new(password, base64.b64decode(b64_challenge)).hexdigest()
            print "CRAMMD5PasswordVerifier.is_authenticated response='%s'; expected_response='%s'" % (response, expected_response)
            return response == expected_response
        except Exception, e:
            print "Unexpected exception (%s); authentication treated as failing", (str(e),)
        return False

class PlainPasswordVerifier:
    """Responsible for verifying credentials"""
    def __init__ (self, credentialrepository):
        self.credentialrepository = credentialrepository

    def is_authenticated (self, response):
        try:

            plainauthresp = response[len("PLAIN "):]
            junk, username, supplied_password = base64.b64decode(plainauthresp).split("\x00")
            actual_password = self.credentialrepository.getPasswordForUser(username)
            return actual_password == supplied_password
        except Exception, e:
            print "Unexpected exception (%s); PLAIN authentication treated as failing", (str(e),)
        return False

# Class capable of doing PLAIN or CRAM-MD5 password authentication
class CustomSMTPCRAMMD5Channel(asynchat.async_chat):
    COMMAND = 0
    CRAM_MD5_RESPONSE = 1
    DATA = 2

    def __init__(self, server, conn, addr,  fqdn = "sesame"):
        asynchat.async_chat.__init__(self, conn)
        self.__server = server
        self.__conn = conn
        self.__addr = addr
        self.__line = []
        self.__state = self.COMMAND
        self.__greeting = 0
        self.__mailfrom = None
        self.__rcpttos = []
        self.__data = ''
        self.__fqdn = fqdn
        self.__peer = conn.getpeername()
        print >> DEBUGSTREAM, 'Peer:', repr(self.__peer)
        self.set_terminator('\r\n')
        self.is_authenticated = False
        self.credentialrepository = CredentialRepository()
        self.crammd5verifier = CRAMMD5PasswordVerifier()
        self.plainverifier = PlainPasswordVerifier(self.credentialrepository)
        self.b64challenge = self.crammd5verifier.get_b64_challenge()
        print >> DEBUGSTREAM, 'About to send 220'
        self.push('220 %s %s' % (self.__fqdn, __version__))

    def smtp_EHLO(self, arg):
        print >> DEBUGSTREAM, 'CustomSMTPCRAMMD5Channel::smtp_EHLO ', arg
        dir(self)
        if not arg:
            self.push('501 Syntax: EHLO hostname')
            return
        if self.__greeting:
            self.push('503 Duplicate HELO/EHLO')
        else:
            self.__greeting = arg
            self.push('250-%s' % self.__fqdn)
            #self.push('250-AUTH CRAM-MD5') # PUT THIS BACK??
            self.push('250-AUTH PLAIN')
            self.push('250-DELIVERBY 300')
            self.push('250 SIZE 15728640')

    def smtp_AUTH(self, arg):
        print >> DEBUGSTREAM, 'CustomSMTPCRAMMD5Channel::smtp_AUTH(%s)' %(arg,)
        if self.is_authenticated:
            print 'Already authenticated!'
            #  a server MUST reject any further AUTH commands with a 503 reply.
            self.push('503 Already authenticated')
        else:
            if arg.upper() == "CRAM-MD5":
                print "Client requesting authentication using CRAM-MD5"
                # send a challenge
                self.push('334 ' + self.b64challenge)
                # and "special case" the fact that the response will be a garbage-looking b64 string
                self.__state = self.CRAM_MD5_RESPONSE
            elif arg.upper().startswith("PLAIN "):
                print "Client requesting PLAIN authentication"
                if self.plainverifier.is_authenticated(arg):
                    logmessage("Authenticated OK using PLAIN authentication")
                    self.is_authenticated = True
                    self.push('235 PLAIN authentication successful')
                else:
                    logmessage("client connecting to our proxy FAILED authentication; we will not relay")
                    time.sleep(10)
                    self.push('535 invalid user ID or password')
            else:
                logmessage("client connecting to our proxy does not support a common authentication scheme; we will not relay")
                self.push('535 invalid user ID or password')

    def smtp_MAIL(self, arg):
        print >> DEBUGSTREAM, 'CustomSMTPChannel::smtp_MAIL ', arg
        if self.is_authenticated:
            print >> DEBUGSTREAM, '===> MAIL', arg
            address = self.__getaddr('FROM:', arg) if arg else None
            if not address:
                self.push('501 Syntax: MAIL FROM:<address>')
                return
            if self.__mailfrom:
                self.push('503 Error: nested MAIL command')
                return
            self.__mailfrom = address
            print >> DEBUGSTREAM, 'sender:', self.__mailfrom
            self.push('250 Ok')
        else:
            self.push('503 Error: need AUTH command')

    # Overrides base class for convenience
    def push(self, msg):
        asynchat.async_chat.push(self, msg + '\r\n')

    # Implementation of base class abstract method
    def collect_incoming_data(self, data):
        print >> DEBUGSTREAM, "collect_incoming_data: ", data
        self.__line.append(data)

    def process_cram_md5_response(self, response):
        try:
            print 'CustomSMTPChannel::process_cram_md5_response:raw="%s"  ' % (response)
            print 'CustomSMTPChannel::process_cram_md5_response: decoded = "%s"  ' % (base64.b64decode(response))
            # validate the username/password e.g. response should be
            #       base64.b64encode("username " + hmac.new("password", "<24609.1047914046@popmail.Space.Net>").hexdigest())

            # i.e.  base64( username, then space, then digestedpassword)
            # where digestedpassword = hmac.new("plainpassword", "challenge").hexdigest()
            username, response = base64.b64decode(response).split(" ")
            password = self.credentialrepository.getPasswordForUser(username)
            self.is_authenticated = self.crammd5verifier.is_authenticated(username, password, self.b64challenge, response)
            if not self.is_authenticated:
                raise NotAuthenticatedException("Username ok but response bad")
            print "Authenticated OK"
            self.push('235 CRAM-MD5 authentication successful')
        except Exception, e:
            logmessage("Failed to authenticate user; exception=(%s)" % (str(e),))
            self.is_authenticated = False
            self.push('535 invalid user ID or password')

    # Implementation of base class abstract method
    def found_terminator(self):
        line = EMPTYSTRING.join(self.__line)
        print >> DEBUGSTREAM, 'Data:', repr(line)
        print "found_terminator; Data=", repr(line)
        self.__line = []
        if self.__state == self.COMMAND:
            if not line:
                self.push('500 Error: bad syntax')
                return
            method = None
            i = line.find(' ')
            if i < 0:
                command = line.upper()
                arg = None
            else:
                command = line[:i].upper()
                arg = line[i+1:].strip()
            method = getattr(self, 'smtp_' + command, None)
            if not method:
                self.push('502 Error: command "%s" not implemented' % command)
                return
            method(arg)
            return
        else:
            if self.__state == self.CRAM_MD5_RESPONSE:
                self.process_cram_md5_response(line)
                self.__state = self.COMMAND
                return
            if self.__state != self.DATA:
                self.push('451 Internal confusion')
                return
            # Remove extraneous carriage returns and de-transparency according
            # to RFC 821, Section 4.5.2.
            data = []
            for text in line.split('\r\n'):
                if text and text[0] == '.':
                    data.append(text[1:])
                else:
                    data.append(text)
            self.__data = NEWLINE.join(data)
            status = self.__server.process_message(self.__peer,
                                                   self.__mailfrom,
                                                   self.__rcpttos,
                                                   self.__data)
            self.__rcpttos = []
            self.__mailfrom = None
            self.__state = self.COMMAND
            self.set_terminator('\r\n')
            if not status:
                self.push('250 Ok')
            else:
                self.push(status)

    # SMTP and ESMTP commands
    def smtp_HELO(self, arg):
        if not arg:
            self.push('501 Syntax: HELO hostname')
            return
        if self.__greeting:
            self.push('503 Duplicate HELO/EHLO')
        else:
            self.__greeting = arg
            self.push('250 %s' % self.__fqdn)

    def smtp_NOOP(self, arg):
        if arg:
            self.push('501 Syntax: NOOP')
        else:
            self.push('250 Ok')

    def smtp_QUIT(self, arg):
        # args is ignored
        self.push('221 Bye')
        self.close_when_done()

    # factored
    def __getaddr(self, keyword, arg):
        address = None
        keylen = len(keyword)
        if arg[:keylen].upper() == keyword:
            address = arg[keylen:].strip()
            if not address:
                pass
            elif address[0] == '<' and address[-1] == '>' and address != '<>':
                # Addresses can be in the form <person@dom.com> but watch out
                # for null address, e.g. <>
                address = address[1:-1]
        return address

    def smtp_RCPT(self, arg):
        print >> DEBUGSTREAM, '===> RCPT', arg
        if not self.__mailfrom:
            self.push('503 Error: need MAIL command')
            return
        address = self.__getaddr('TO:', arg) if arg else None
        if not address:
            self.push('501 Syntax: RCPT TO: <address>')
            return
        self.__rcpttos.append(address)
        print >> DEBUGSTREAM, 'recips:', self.__rcpttos
        self.push('250 Ok')

    def smtp_RSET(self, arg):
        if arg:
            self.push('501 Syntax: RSET')
            return
        # Resets the sender, recipients, and data, but not the greeting
        self.__mailfrom = None
        self.__rcpttos = []
        self.__data = ''
        self.__state = self.COMMAND
        self.push('250 Ok')

    def smtp_DATA(self, arg):
        if not self.__rcpttos:
            self.push('503 Error: need RCPT command')
            return
        if arg:
            self.push('501 Syntax: DATA')
            return
        self.__state = self.DATA
        self.set_terminator('\r\n.\r\n')
        self.push('354 End data with <CR><LF>.<CR><LF>')

class CustomSMTPTLSServer(smtpd.DebuggingServer):
    def __init__(self, localaddr, remoteaddr, sslcertfile, sslkeyfile):
        smtpd.SMTPServer.__init__(self, localaddr, remoteaddr)
        print "My Socket = ", self.socket
        securesock = ssl.wrap_socket(self.socket,
                                     do_handshake_on_connect=True,
                                     server_side=True,
                                     ssl_version=ssl.PROTOCOL_SSLv23,
                                     suppress_ragged_eofs=True,
                                     certfile=sslcertfile,
                                     keyfile=sslkeyfile)
        self.socket = securesock
        print "My Socket Now = ", self.socket

    def may_forward_mail_from_address(self, address):
        try:
            print "** Checking whether %s is  an OK from address" % (address, )
            if address.lower() in acceptable_from_email_addresses:
                print "** %s is clearly an OK from address" % (address, )
                return True

            # Address could be in chevronned "<wacky@Domain.com>" format
            # '<wacky@Domain.com> size=257' => 'wacky@domain.com'
            normalised_address =  address[address.find("<")+1:address.find(">")].lower()
            if normalised_address in acceptable_from_email_addresses:
                logmessage("** %s is an OK from address" % (normalised_address, ))
                return True
        except:
            logmessage("Error determining whether '%s' was an acceptable e-mail address; assuming it was *not*" % (address,))
        return False

    def handle_accept(self):
        try:
            print "Entering handle_accept..."
            conn, addr = self.accept()
            print >> DEBUGSTREAM, 'Incoming connection (handled in a custom way, with negotiated TLS) from %s' % repr(addr)
            logmessage('Accepting incoming TLS connection from from %s' % repr(addr))
            print >> DEBUGSTREAM, "Instantiating Channel object to handle this connection"
            channel = CustomSMTPCRAMMD5Channel(self, conn, addr)
        except ssl.SSLError, e:
            logmessage("Unexpected SSL Exception while setting up connection: %s\n" % (str(e)))

    def process_message(self, peer, mailfrom, rcpttos, data):
        logmessage('Receiving message from: "%s"' % (peer,))
        logmessage('Message addressed from: "%s"' % (mailfrom,))
        logmessage('Message addressed to  : "%s"' % (rcpttos,))
        logmessage('Message length        : "%s"' % (len(data),))
        if self.may_forward_mail_from_address(mailfrom):
            logmessage("We will attempt to forward message")
            forwardmess(mailfrom, rcpttos, data)
            time.sleep(1)
        else:
            logmessage("Message not forwarded")
            time.sleep(10)
            return '553 Requested action not taken mailbox name not allowed '

if __name__ == '__main__':
    DEBUGSTREAM = sys.stdout
    smtpd.DEBUGSTREAM = sys.stdout
    certfile="servercert.pem"
    keyfile="serverkey.pem"

    if len (sys.argv) == 3:
        certfile = sys.argv[1]
        keyfile = sys.argv[2]
        server = CustomSMTPTLSServer((incoming_listen_address, incoming_port), None, certfile, keyfile)
        asyncore.loop()
    else:
        print  >> DEBUGSTREAM "No key files specified. Run 'smtpproxy.py servercert.pem serverkey.pem'"

############################################################################################
# To run this you need python2.6 or later (On Ubuntu - "sudo apt-get install python2.6")
# To run the server you'll need a  certificate.
# To create an (unsigned) private key and certificate I used the following commands:
#    openssl genrsa -out serverkey.pem 2048
#    openssl req -new -x509 -key serverkey.pem -out servercert.pem -days 1095

# To run the server do:
#    python smtpproxy.py servercert.pem serverkey.pem

############################################################################################

Was this article useful for you?  Please comment!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s