At Shanje, we run dozens of Windows Server 2003 machines with XMail for POP3 and SMTP duties for tens of thousands of domains
and hundreds of thousands of email accounts. We currently need to filter millions of spam emails each day on behalf of our customers.
The purpose of this document is to explain in detail how we deployed a mixed operating system spam filtering solution
that scales well and maintains excellent reliability.
Setting the stage
Our previous SpamAssassin solution included running both the SpamC and SpamD components of SpamAssassin on each individual Windows
server to scan all of the email destined for that machine. While that solution was obviously better than launching SpamAssassin
once per email, it still did not provide enough performance for the amount of email we were dealing with. After doing some additional
research and testing, we found that SpamD currently runs much better on Unix platforms than it does on Windows 2003. The performance
differences between SpamAssassin for Unix and SpamAssassin for Windows are not really the fault of the underlying OS. With some clever
programming work, we could have found a way to get SpamAssassin to scale effectively on Windows, but we felt it would be a more
effective use of our time to simply find a way to deploy it on Unix instead of rewriting SpamAssassin. Since we are a 100%
Windows based web hosting company, we could not simply dump Windows in favor of Unix just to get SpamAssassin to work better. So we decided
to deploy a mixed environment with some Unix based machines to handle SpamAssassin duties.
Old hardware & new solutions
We have been hosting web sites since 1997, so we have deployed a considerable number of rack mounted servers over the years.
In 2003, we yanked all of our old Pentium III 800MHz servers out of the racks and replaced them with newer and much faster
Pentium IV 3GHz servers. As a result, we had dozens of old and seemingly useless Pentium III 800MHz servers sitting around.
We had often heard tales of sysadmins deploying barebones installations of FreeBSD on really old PCs to get some additional life
from outdated hardware. We decided to try this approach with ten of our old P3-800 servers. We blew the dust off, replaced the CPU fans,
and checked the hardware to make sure they would still boot. Then we tossed a barebones install of FreeBSD onto each box and proceeded
to installed SpamAssassin 3.0. We soon had ten working SpamAssassin boxes to use in our clustered spam filtering project at Shanje.NET.
CoyotePoint Equalizer
Once we had our ten FreeBSD boxes working, we needed to find a way to have SpamC link to these boxes. There are switches in the SpamC
tool that let you specify the IP of the server that is running SpamD. There is even a round robin functionality built into the SpamC tool
in case you want to load balance multiple SpamD boxes. We wanted to take things one step further. We had previously used CoyotePoint
Equalizers to balance HTTP traffic to ensure uptime for websites that we host. We wanted to do something similar with SpamAssassin.
So we placed all ten SpamAssassin servers behind the CoyotePoint Equalizer and set up a cluster for the SpamAssassin port in the Equalizer.
Then our XMail servers were able to launch SpamC and SpamC would connect to the Equalizer, which would in turn route the connection to the
FreeBSD box with the least amount of SpamAssassin traffic at that moment. The CoyotePoint Equalizer monitors which systems are online,
which systems aren’t, and the load of each machine. This solution works great, and keeps all of the SpamAssassin FreeBSD boxes
equally loaded. We are also able to pull real time stats from the CoyotePoint Equalizer box to see in real time what the level of SpamC traffic is.
Testing
The real test of any spam filtering solution is not just whether or not it works, but whether or not it scales. So
in addition to testing to make sure SpamC was able to talk to the FreeBSD SpamAssassin boxes, we also did a battery of tests
to see what the peak throughput of the solution would be. We found that the clustered SpamAssassin solution was able to easily
handle hundreds of simultaneous SpamC connections without any errors or timeouts. We also found that the clustered SpamAssassin
solution could easily parse millions of emails every day without using very much CPU time on the FreeBSD boxes. Initially we were
worried the old P3-800 servers would not stack up to the demand, but the old boxes handled the task with ease.
Tweaking Settings per mailbox
Since we host hundreds of thousands of mailboxes for customers around the world, it is safe to say that a considerable amount of
flexibility is needed on a per customer basis. What works for one customer in terms of filtering may not work for others. So we
wrote a script that XMail calls to decide what to do with the email after SpamAssassin has scored the email. Each mailbox owner
can set their own custom filtering level. This lets users drop email that scores above the threshold they have set instead of forcing
everybody to adopt the same threshold. Customers can even configure their mailbox to not drop the email no matter how high it scores,
so they can receive everything that was sent to them and then do client side filtering rules within applications like MS Outlook. So
even though we have deployed a very high traffic solution, we have also deployed tools that let the customer feel like SpamAssassin can be
set up specifically to meet their unique needs. In the future, we also plan to deploy tools to let customers build whitelists within a
SpamAssassin database, so customers can avoid accidentally filtering an email from an important friend or business contact.
Conclusion
Some of you may be asking, “Why did they choose to use Unix and an open-source spam filter if they are a Windows 2003 host?”
The answer is simple. We want to provide the best solution to our customers, and at this point in time, SpamAssassin is the most
effective spam filter that we have found. We also found that SpamAssassin runs best under Unix, so we took the initiative to break
from our Windows world, and deploy the product that we knew worked the best with SpamAssassin. All in all, we did this so that we
could better serve our customers, and provide them with another valuable service at Shanje.NET.
