Canned Spam Filters

The ACCC has now turned on anti-spam filtering using SpamAssassin for everyone at UIC and has the same system working for everyone, regardless of where your account is. (Or isn't -- we now do antispam filtering for people who just use our uic.edu re-direction service also.)

The default settings are described in this Web page, along with information on how to change them if you wish. To change anti-spam filters settings, go to our Email Filters utility page, login with your UIC netid and ACCC common password, select the machine that your account is on if necessary, and click on ANTI-SPAM Settings/Filter.

Are you interested in an "Incredible Satellite TV offer?" Would you like to "Consolidate your debts in an offshore Visa card?" No? Well then, certainly you'd like to "Dig up Dirt on your Coworkers!" Still not interested? Good, because that's what the ACCC's automatic spam filtering system is turning away.

You've probably seen phrases like those above in email messages you received but didn't request, that is, spam. We discussed the spam problem in the October/November/December 2000 issue of the A3C Connection in the article called "Slamming Spamming." That article covered the basics, what spam is, how it works, and some possible options to minimize the amount of spam you receive.

One such option is to filter spam out before it reaches your Inbox. But anti-spam filters can be very complicated to set up, and they can dispose of valid email if not crafted carefully. (For example: The word "Specialist" has "Cialis" in the middle of it. And yes, that does happen.)

We are now remedying this for you in an automatic and simple way: we've set up effective anti-spam email filters, i.e., Canned Spam Filters, and we're running them for everyone, automatically. The most egregious spam we're throwing away before it even gets to you.

The Mailtools anti-spam filters won't catch all of of your spam -- a perfect anti-spam filter is impossible. And there is some stuff that really looks like it's spam but we just can't be sure. This spam we deliver to you, marking it as spam.

Why? Although we have taken care to make this unlikely, it is possible that a piece of valid email will somehow be sorted in with the spam. You should check the spam messages we deliver to you just in case there is any real email that gets caught among the spam. We will help you with this too, by sending you an email message with a list of your probably spam email -- its From address and its Subject. This way you can make sure you didn't miss anything that you wanted to see -- that notification of your lottery winnings, for example.

What We Do Now

This is what happens by default now:

• Our anti-spam filter applies to all ACCC email accounts and any @uic.edu address (even if it doesn't go to an ACCC account) is also now automatically scanned for spam.
  
  
• Messages that are identified as having a very high spam score, that is, messages that most certainly are spam, are thrown away before ever reaching you. Comparing the number of messages I used to have in my spam folder to the number that I have now, I'd say that about 60% of the spam that I used to get is now getting thrown out.
  
  
• But there are always some messages that we can't be 100% sure are spam. In fact, now and then, a legitimate message will get filtered out as spam. So we deliver the lower scoring spam to you, but we tag it as spam for you.
  
  
• By default, we will tag the spam messages as being spam by putting ***SPAM*** in their Subject:.
  
  
• By default, if your account is on an ACCC server, we also move them the spam messages into to a spam mailbox on the server.
  
  
• Also by default, we will change each spam message into an attachment, and send it to you as an attachment, with the SpamAssassin report. This is good because it saves you from viruses that might automatically execute or other malicious or intrusive items in the message.
  
  
• Every day, if your account is on an ACCC server, we send you a Spam Report Email message listing today's new spam messages, giving you a chance to redeliver them or whitelist their senders.
  
  
• Every day, if your account is on an ACCC server, we delete your spam messages that are three days old.
  

You can change all these settings and decide what exactly you want to be done with your spam once it gets to your account. You might not want it moved to a spam mailbox on the server. You can also make exceptions to the spam filter rules, or set the filter to be more or less aggressive, or various other options. Login to the Email Filters Utility to view and adjust your current settings. These settings are discussed below in more detail.

If your account is on an ACCC server, your main interaction with the ACCC SpamAssassin anti-spam service will be with the Spam Report email messages that it sends, for example, Figure 1.

Figure 1: Sample Spam Report Email Message The bold number before the From: address is the message's spam score. Do not reply to the message; no one will see your reply. If you have questions about ACCC spam filtering, send them to spamfiltering@uic.edu.

The Email Message Lists

There are two groups of spam email messages listed;

• the top messages with spam scores less then 10,
• the bottom messages with scores other greater than 10.

(Any messages with a spam score of less than 5 are not considered to be spam. Though lots of email with lower scores are spam; but lots with lower scores aren't also.)

For each message,

• the From: address
• and the Subject:

will be listed.

If you see a message that isn't spam, you can retrieve it with the links in your Spam Report email message.

Beside the Subject: is a (redeliver) link; click it to have the message re-delivered to your Inbox. You might also want to forward it, with full headers, to spamfiltering@uic.edu ; we are interested in seeing false-positive messages. And sometimes we can fix false-positive problems.

And to prevent a message from this sender being mis-identified as spam again,

Beside the From: address is a (whitelist) link; click it to add the From: address to your whitelist.

By default, these spam messages will be deleted automatically after three days is you don't do anything else about them.

The Text at the Bottom of the Message

The text at the bottom of the Spam Report message has links to various Web pages that will be useful for you in working with your spam.

• First there is a link to WebMail. If your regular email program is set up to use POP -- if you login, download all you new incoming email, and then logout -- then you can't work with your spam email folder with your regular email program. Click on this Webmail link if you want to directly access your spam email folder.
  
  
• Next there is a link to the ACCC Email Filters Utility, http://mailfilter.accc.uic.edu
  Then click ANTI-SPAM Settings/Filter. Use this Web page to check out your ACCC anti-spam filter and change details if you wish to.
  
  
• Next are links to this page and to the ACCC Web home page.
  
  
• Finally, there is a link that allows you to toggle between text and HTML format for your Spam Report email messages.

The ANTI-SPAM Settings/Filter are introduced below.

To configure your anti-spam filters, visit our Email Filters utility page and login with your UIC netid and ACCC common password. To work on your anti-spam filters, click ANTI-SPAM Settings/Filter.

Changing Your "Account"

If your uic.edu email is sent to an ACCC email server, the Email Filters utility will automatically open for that server. However you can do filters for all ACCC servers you have accounts on and for your uic.edu email alone, if you wish. At the top of the Email Filters Utility Page, just below the heading, it says what netid and "account" -- the server -- that you're working on the filters for. If you want to work on the filters for another machine, click on (change account). That displays a screen which allows you to select another of your ACCC accounts to do filters for or for No Account -- your uic.edu email.

These choices will only be offered when you are setting options for an ACCC account.

There are two choices as to what should be done with spam once detected. With the new Spam Report email messages, the default moving to spam mail folders on the server, should work just fine for everyone. Having the spam messages on the server used to be a problem for people using POP, because they didn't have contact with email on the server with their email program, but the Spam Report email message (Figure 1) solves that problem now.You can check to see whether there is any legitimate email in your spam folder by checking the listing of your spam email messages in the Spam Report email message (Figure 1) that we'll send you listing of your spam email, and use the links in the email message to redeliver any messages you want to keep to your Inbox. So it's no longer as important to use the Tag only option with POP.

However, there still are two choices. They are: (1) Move to a spam mailbox on the server or (2) tag the message as spam. Filter spam into your spam mailbox on the server is the default.

• The Tag only option was designed for people who use POP. Here's how it works:
  
• When spam comes into your email Inbox, it will stay there unless YOU make a filter to move it to another mailbox.
• A hidden tag is added to each spam email message: X-Uicclass: UICClass Spam
  (This is an X-header ; see the online version of Figure 2: Headers of a Legit Email Message, from the October/ November/ December 2000 A3C Connection.) You can use this hidden tag as a criterion in a Eudora local filter so that Eudora will recognize the email message as spam and move it into a local spam mailbox on your personal computer. See Creating a Eudora Filter with the SPAM Tag. To see an example of the X-Uicclass header, see Figure 3 , which is the full headers of a spam message identified by the ACCC SpamAssassin filters.
• It also tags the Subject: as ***SPAM***
  You can also use this tag as a criterion for a local email filter so that your email program will recognize a spam email as spam and move it into your local spam mailbox. For example, see Figure 4 , which is a spam message identified by the ACCC SpamAssassin filters.
• Make sure that you check your spam messages to see if any are legitimate messages. (It's a lot easier to do this with the other option.)
  
  
  
• The default option Filter spam into your spam folder was designed for people who use IMAP, but with the new Spam Report message email (Figure 1) can be used by everyone. Once an email message is identified as probably being spam, it will automatically be sorted into another folder on the server called spam. When you use this option the system will send you a summary of the titles of the probable spam messages you receive and will delete older spam on a regular basis. The choices are for the scheduling are two:
• Daily summary of filtered spam, spam deleted when it's three days old. This is the default.
• Weekly summary of filtered spam, spam deleted when it's 10 days old. You might want to choose this if you don't check your email every day.
This option also tags the Subject: as ***SPAM***
Make sure that you check the spam messages listed in the Spam Report message email message (Figure 1) for legitimate messages before they are will be deleted, so that you can redeliver any legitimate messages that were falsely identified as spam.

For example, see Figure 4 , which is a spam message identified by the ACCC SpamAssassin filters.

This and all the following choices are "Anti-Spam Settings" and will be offered for all types of accounts.

The anti-spam filter works on a number scale. Each email message is given a numeric score, the sum of the points for various infringements are given specific numbers of points depending on how spam-like they are. All email messages with a spam score worse than a specific number is marked as spam.

• Clicking Aggressive will lower the bar, causing more spam to be filtered out, but it also raises the risk of accidentally filtering out your legitimate email.
• Clicking Low Sensitivity will raise the bar, causing less spam to be filtered out, but also lowering the risk of accidentally filtering out your legitimate email.

I recommend that you leave it at Normal. It works well at normal, with a minimum of false positives (real messages that you want to see that are marked as spam).

Your whitelist is a list of email address that you know won't send your spam. The ACCC's SpamAssassin will give an automatic pass to any email that is sent to you from any address that's in your whitelist.

Email messages from email lists are often identified as spam when they aren't; to prevent messages from your email discussion lists from being counted as spam, type the email address of all lists the you are subscribed to and other email addresses that you trust in the box. For instance, do you subscribe to bubblegumweekly@stickylists.com? Then you put that address into the box in this section.

If you subscribe to a number of lists from the same domain, you can cover all of them by just entering the domain name, @yahoogroups.com, for example.

By default we already exempt any mail that was sent from within UIC or was sent through a UIC emailing list (Listserv).

We automatically whitelist all addresses in your Webmail address book on a daily basis, so you shouldn't have to add any addresses you have there.

We'd be glad to whitelist your personal address books too (Eudora, Outlook, etc), but we don't have access to them. If you would like all your address book entries to be whitelisted, just upload/transfer them to your Webmail address book. By the next day they will all be whitelisted.

You can do this easily with our address book conversion utility.

If your account is on an ACCC server, every day, the ACCC anti-spam filters will send you an email message listing the messages that have been identified as spam.

For each message,

• the From: address
• and the Subject:

will be listed.

• Beside the From: address is a (whitelist) link; click it to add the From: address to your whitelist.
• Beside the Subject: is a (redeliver) link; click it to have the message re-delivered to your Inbox.

This is opt-in not opt-out.

These days, a lot of the spam email comes not only in foreign languages, but also in different character sets. This sections has a list of foreign language character sets (Japanese, Korean, Cyrillic, etc.).

If you ever get legitimate email in any of these character sets, select them. If you don't, don't. They will be used to identify probable spam.

There are two additional message modification options:

Rewrite the Subject: line of spam messages by putting ***SPAM*** before the message's original subject. You can choose to have this done or not; the default is add the ***SPAM***. It's best to select this; it's much easier to write a filter in your email program to sort on ***SPAM*** in your Subject: than the X-header.

Change the spam message into an attachment and send it to you that way. This will disable any auto-execute viruses or Web bugs. Or you can leave the messages as is. The default is to change the spam messages into attachments. The message in A Look at SpamAssassin Spam Messages Headers and Creating a Eudora Filter with the SPAM Tag is an example of why you might want to turn this option on.

When normal email messages are routed through the internet, it often happens the machine on the receiving end isn't able to accept the message right away. In such cases, the receiving machine replies to the sending machine with a "temporary error", which tells the sending machine to try again in a little while. After a few minutes, or perhaps even as much as an hour, the sending machine will try again. Normally it will continue to try for a few days before giving up completely. When the receiving machine is finally ready, it will accept the message and it will be delivered to the target user's mailbox.

Greylisting takes advantage of the fact that spammers don't usually keep trying, they tend to give up after one try. If the incoming message is from a spam server, and we give the incoming messages such a temporary error as described above, the result is that they never try to send the message again. So giving a temporary error to any message that we're seeing for the first time has the effect of keeping a whole lot of spam from ever being delivered, while also not stopping real email, as legitimate email is re-delivered shortly after the first rejection.

Normally the whole process takes under 10 minutes, but the time it takes to retry a message ultimately depends on the particular remote server.

That's the basic principal of greylisting, although there are many details. The main disadvantage to such method is that it has the potential to slow down some incoming messages, whose immediate reception might be important to some people. However, we take many steps to make this very minimal:

• All mail from within UIC is whitelisted (meaning it bypasses greylisting, see above for more on whitelisting),
• Messages from senders in your whitelist or your Webmail address book are whitelisted,
• After a message is successfully delivered once, the sender is whitelisted, so that the delay only happens once per sender. Messages from the same sender after that are not delayed again. (It's actually a little more complex than that, but that's the general idea.)
• You can also add any addresses to your permanent whitelist if you want to make sure that delivery from those senders is always immediate.

Additionally, we have provided an interface so that you can view exactly what messages are in which states of greylisting. Normally you shouldn't need to use this, but if you suspect there is a problem you can check this list. Unfortunately, however, the only information we have is the sender's address -- no subject, no message, etc.

The interface is available from the anti-spam settings utility where you set the greylisting option.

A second problem with greylisting is that it's possible, although very rare, that the remote server, if not properly configured, might NOT redeliver a legitimate message.

Because of this possibility, when we find servers like this, we globally whitelist them to prevent any future problems. Likewise, the interface mentioned above can be of use in finding those types of situations. If you ever come across such a situation, please inform us so that we may globally whitelist that server for all other UIC users.

Would you like to know more about the details of how greylisting works? You can read a greylisting "whitepaper" on the topic here.

Click the Save Settings button.

The neat thing is you're not turning on the anti-spam filter, you're just adjusting the settings. It's already been turned on for you!

Of course, if you forget to identify any email discussion lists or other email senders that you trust, the filter might identify messages from then as spam, so you'll want to be sure to check your spam folder frequently at first to see if any messages got through from addresses you forgot to include. You can imagine that, if you subscribe to many lists, it may take a few tries before you get it quite right.

Did you forget to add an email discussion list or another email address that you always trust to send you email that you want to see? No problem. Just return to Email Filters page, click ANTI-SPAM Settings/Filter, then move down to the Whitelist section. You'll see that the whitelist addresses that you entered before in the box where you typed them. Simply edit the list and click the Save Settings button again and your changes will take effect immediately.

If you use POP, you won't generally want to use the move to a different folder options because these folders will be online and you don't have direct access to online folders when you use POP. The Tag only: action option was designed to be used in this case, and should be used if you use the Email Filters attachment filters. If you use this option, you have to then filter the tagged messages with a local (Eudora) filter.

The Tag only option action adds a hidden tag: X-Uicclass: UICClass Spam to each spam email message. (What they actually add is an X-header ; see the online version of Figure 2: Headers of a Legit Email Message, from the October/ November/ December 2000 A3C Connection.) Notice the last header line in this long header view of a spam message.

Using X-Uicclass to Sort Spam

An Easier Way To Sort Spam: ***SPAM*** in the Subject:

Figure 4: An easier way: use the ***SPAM*** in the Subject: This is the whole message that goes with the headers above. SpamAssassin added the Spam Report Card. Notice that is says that the original note wasn't plain text and therefore could have been malicious. This message shows why it is a good idea to allow SpamAssassin to deliver spam messages as attachments.

The anti-spam email filter is only one of the types of filters the ACCC email filters tools allows you to make. There are also specialized filters for attachments and customizable filters for just about anything else. There is a Web page help file for the ACCC email filter tools; there are links to it on all of the email filters setup pages.

A legitimate message was filtered out as spam, how do I get it back?

You can also use Webmail: If you simply want to transfer the message from your spam folder to your Inbox without too much extra reading, you can use our Webmail email system. Login at https://webmail.uic.edu and then click on the spam folder from the left frame. Once you've found your message in the right frame, select it and then click the MOVE button at the top to move it your INBOX.

For more information on using Webmail, see the ACCC Webmail Users Guide.

What if I don't want to receive ACCC's spam summaries at all?

Use the "CUSTOMIZABLE email filter setup" option to create a new filter. Choose Subject: field of the incoming mail starts with this text, and here you type in ACCC Spam Summary. Then below for what action to take, select Delete the message.

How can I automatically delete all my spam?

Nonetheless, you can do this simply by creating another filter in the same manner described above. Use any header line, contains, and then type in X-UICClass: UICClass Spam for the text. Then for the action you can select delete the message. See Creating a Eudora Filter with the SPAM Tag.

I am suddenly receiving more/less spam in my inbox now than before. Why?

Spammers continually come up with new tricks, so we write filters for them, they come up more tricks, we write more filters, and so on. Sometimes they get ahead, and sometimes we make progress. Sometimes, especially when they deluge us with huge amounts of spam at once, the time it takes to respond is too slow, so more spam gets through.

Another factor is how much your email address has been spread through the internet and in spammer databases. A new account will not receive much spam, but the longer you have it the more spam you can expect.

The best way to avoid spam is preventative -- think carefully before entering your email address into any web forms.

Also look out for tricks they use to gather addresses, such as "mail this news story to a friend", "send a greeting card to your sibling", etc. Many addresses are on spam lists because some well-intentioned friend fell for one of these tricks.

We are continually seeking out new software solutions to make this better, but there fluctuating amounts of missed spam are inevitable. Please bear with us.

Can I report spam that wasn't properly filtered? Should I report legitimate messages that were improperly filtered out as spam?

Note: If you use an email client like Eudora or Outlook, be sure to create the folder on the server, not on your local disk.

Legitimate messages in your spam folder:

You can report legitimate messages that are filtered out as spam to us. You can forward those messages (with FULL headers please) to spamfiltering@uic.edu so that we can take a look at what happened. Sometimes we are able to find and fix problems the problem by looking at this. But do not expect a reply to every forwarded message.

Nonetheless, the best thing you can do in that case is not depend on a better filter, but rather add the sender to your list of exemptions (your whitelist).

Can I make my spam filter even more aggressive?

Use the CUSTOMIZABLE email filter setup option to create a new filter. Create the filter to say:

If any header line of the incoming email starts with this text: X-Spam-Level: ***

Then below for what action to take, select that you want it filed into your "spam" folder (if that's where you want it).

Note the number of stars (*) in the filter text. The above example has three -- that means that it scored 3 on the SpamAssassin tests -- but you could make it more aggressive yet by using only two, and even more so by using only one. Be sure to have at least one, however, or else it will catch ALL of your messages.

How can I personalize my anti-spam filter to make it more accurate?

First set your filter to be "aggressive". Or, for even better results, you can set it to be even more aggressive than the 'aggressive' setting -- see the question above, "Can I make my spam filter even more aggressive?"

Next, you've got to put all the email addresses from which you normally receive email into your whitelist:

You shouldn't have to worry about adding any UIC or UIUC addresses, they should come through without being on your whitelist.

Any addresses in your Webmail addressbook will automatically be counted in your whitelist as well, even though they won't appear in your whitelist listing on the anti-spam filter page. If you don't use Webmail, you can still take advantage of this by uploading your addressbook to Webmail. Click the EMAIL button at the top of this page and then find the link "Email - Address Book Tool".

If there are still address from which you get mail besides those, you need to add them to your whitelist manually to make sure mail from them still gets through. It may help to go through the legitimate mail that you've sent or received in the last month or two and make a note of all the addresses there.

Doing this will probably keep almost all spam out of your inbox. But, especially at first, it will probably cause some of your legitimate messages to end up in your spam folder. You'll probably want to carefully check the "Spam Summary" for such messages. If you continue to add addresses to your whitelist as you find them, your filter will become more and more accurate over time.