Email Spam Filtering at Shanje
Shanje is serious about blocking spam. We use every tool imaginable to filtering spam, while still preserving the flow of legitimate emails.
Shanje uses several real-time blocking lists (RBL) to detect if a remote SMTP server is probably a spam source. We place a score on each RBL that we use instead of simply blocking the email if the remote SMTP server is listed in one RBL. Some legitimate email servers are listed in at least one RBL. When the email servers are listed in several RBLs, then we can safely assume they are probably spamming. This is the advantage of using a scoring system instead of simply blocking email if the email server is in one RBL.
We also use scoring to interpret a number of other factors beyond the real-time blocking lists. We score based on SURBL tests, SPF, HELO, existence of DNS-MX records, and the DKIM-Signature header. The results of these tests are taken into account when scoring along with the RBL test results.
We use greylisting to delay and reject incoming spam. Greylisting is a spam filtering technique that relies on the remote SMTP server to retry the email after an initial delay period. The vast majority of spammers won't try to resend delayed messages, so greylisting can be very effective for blocking spam. There is an excellent Wiki article about greylisting.
Greylisting can delay some legitimate email, so we give customers the option to enable or disable greylisting on a per domain basis. We also have greylist bypassing enabled for any hosts that have valid SPF and/or valid MX or A records. This means that most legitimate emails will automatically bypass the greylisting delay, while most spam is blocked by the greylisting system. As an additional measure for ensuring speed of the inbound email queue, we have also manually whitelisted the SMTP servers of every major host within the greylisting system. For example, all of the Gmail servers are allowed to automatically bypass the greylisting system, so email from Gmail is not delayed by the greylisting system.
We designed our own custom cluster of SpamAssassin servers for doing detailed scoring of the messages and message headers. SpamAssassin is a powerful spam scoring tool. Since it uses a lot of CPU time, it is not always practical to run on email servers. We built a cluster of SpamAssassin servers running on Linux with a load balancer running on FreeBSD located in front of the SpamAssassin cluster. All of our email servers connect to the load balancer to access SpamAssassin. This completely offloads the CPU usage of SpamAssassin to the SpamAssassin cluster, so our email servers do not suffer any performance penalty for using SpamAssassin. This is yet another way Shanje has been able to scale resources to meet the demands of a changing world.
The Shanje SpamAssassin spam filtering can be set to drop emails based on the spam scoring from SpamAssassin. The aggressiveness of this spam filtering option can be configured on a per mailbox basis.
In addition to spam filtering, it is important to filter out virus emails as well. We run ClamAV as a spam rule within our SpamAssassin cluster. This provides us with a scalable virus filtering solution, which is important since virus scanners will use a lot of CPU time. When other companies try to run virus scanners on the email servers, their email servers get slowed down. Our virus scanning is offloaded to the SpamAssassin cluster, so our virus scanning does not slow down any of our email servers.
So far, we have discussed what we do to clean up the incoming email. We realize we also have a duty to make sure our customers don't send spam. Shanje does not allow bulk email sending. To prevent bulk email sending, Shanje actively limits the number of email messages per day that each mailbox is allowed to send. If you need to send more than a few hundred messages per day, you should find a company that specializes in bulk email sending. We don't allow bulk email sending, because we don't want our email servers getting listed in the real-time blocking lists.
Disclaimer:
We do not guarantee that we can block all spam. While we try to block as much spam as possible, some spam will still get past even our best filtering attempts.