Milter-greylist

Hello

I thought about the next anti spam tool and I'd like some feedback about
my ideas. Please try to find a weak point. If there is none, we have our
countermeasure.

1) The context
greylisting have been a good protection against spam, but spamware able 
to defeat it by resending spam are now available. We need a way of adressing
the new problem

2) The idea
The sender machines does not stay idle between the two resends. It tries to
send spam to other locations. If we have spam honeypots (aka spam traps) 
everywhere on the Internet, we have good chances that the sender machine 
will send a spam to a honeypot before the resend. If we have a Distributed 
Spam Trap (DST) netwowrk, one site can catch a sender machine and inform 
all the sites participating to the DSTnet that it found a spammer.

The spam traps would be e-mail addresses released in web pages. The DSTnet
would work by exchanging messages in real time from site to site. I already
wrote the software that does that.

3) Counter measure for spammers
In order to work around DST, spammers need to send mail to honey pots
from IP addresses we don't want to blacklist: ISP SMTP servers. That will
kill the ability of DST to report spamming IP, since it will also report
IP we really don't want to refuse mail from.

4) Preventing the counter measure
I found a very simple way of dealing with the problem: each site in the 
DSTnet could advertise its whitelisted IP netblocks. This would build a 
global whitelist containing as much real SMTP servers as possible. If 
a honeypot address starts getting mail from such a whitelisted address, 
no spam report would be generated.

Whitelist advertisement would have a lifetime and would be sent periodically.
That way if a site gets out of the DSTnet, its stale whitelist  entries 
will not remain.

Of course we need to avoid bad information to be entered in the global
whitelist (so do we need to avoid fake spam trap reports). This can be done
by signing any message sent on the DSTnet, and having a web of trust to
decide what trust to give to a newcomer. 

Opinions, comments?

-- 
Emmanuel Dreyfus
manu@...

On Tue, 4 Apr 2006, Emmanuel Dreyfus wrote:

ED> 
ED> The spam traps would be e-mail addresses released in web pages. The 
ED> DSTnet would work by exchanging messages in real time from site to 
ED> site. I already wrote the software that does that.
ED> 

I had a page on my website for a few weeks during 2004 with my musings on 
co-operative spam fighting (The page is still there 
http://www.clifford.ac/cooperativespamfighting.html and is visited by 
crawlers and I get email to the example honeypot address, even though 
there are no links to it from my website).

My premise then was, basically, that if everyone in the world had a second 
email address, indistinguishable from a real address, and the databases of 
the spamming fraternity could be polluted with those addresses, then spam 
to real addresses would be halved.  What if we all had a thousand false 
addresses?

Expanding on that, suppose I gave you a subdomain (like as done at 
http://www.projecthoneypot.org)) such that any mail sent to, say, 
@...-seine.net, was sent to, and accepted by, your server.  
You (you being some co-operative entity) would pollute the spamworld with 
addresses at donated subdomains and collect the ips of any mail sent to 
those addresses.  

Your scoring scheme, with the score rising and falling over time, would 
fit in here.  If you received a large quantity of spam to the honeypot 
domains, even from legitmate isp servers, then they would deserve to be 
blacklisted.  You could publish a level such as "192.168.0.1 has emitted 
spam at a rate of 1500 per hour over the past 24 hours" and I, as a user, 
could decide what trigger level to put in the milter that you are going to 
write.

Maybe this has all been tried before, I don't know.  However, I think the 
key is publicity.  If a large ISP gets a "Drefus score" above 1000, they 
need to be outed in the national and international press and that would 
need to be worked on as well as the technical side.

-- 
Alan

( Please do not email me AS WELL as replying to the list.  Please 
  address personal email to alan+1@ as lists@ is not read. A
  password autoresponder may be invoked if this email is very old. )

Emmanuel Dreyfus wrote on Tue, 4 Apr 2006 14:50:27 +0000:

> 1) The context 
> greylisting have been a good protection against spam, but spamware able 
> to defeat it by resending spam are now available. We need a way of adressing 
> the new problem 

yes, that was to be expected.

>  
> 2) The idea 
> The sender machines does not stay idle between the two resends. It tries to 
> send spam to other locations. If we have spam honeypots (aka spam traps) 
> everywhere on the Internet, we have good chances that the sender machine 
> will send a spam to a honeypot before the resend. If we have a Distributed 
> Spam Trap (DST) netwowrk, one site can catch a sender machine and inform 
> all the sites participating to the DSTnet that it found a spammer.

Isn't that very similar to existing blacklists, either RBLs for sending hosts, 
URIBLs for advertised URLs in emails or message hash comparing systems like 
Pyzor and Razor (which also take automated submissions from other systems after 
these identified messages as spam)? So, there's already quite a few distributed 
systems out there. I'm not sure if we need another although a bit different 
one.
Also, spamtraps have a high potential of false positives. As you said spammers 
might send over a legitimate ISP and that mailer then gets blacklisted simply 
because spam by a customer was sent thru it. It's almost impossible to stop 
this unless you are actively scanning all outgoing messages against spam. It's 
by far the no. 1 reason for false positives in the existing RBLs. F.i. I use 
SORBS-SAFE instead of SORBS because SORBS contains SORBS-SPAM which is a 
spamtrap filled list. The FP-rate isn't high, but even 0,1% is too much if you 
block at MTA level based on this. Spam-trap lists are only usable if you 
maintain huge whitelists at the same time.

I'd rather build on top of the existing greylist method and try to expand on 
it. There are two possible ways I can think of: distribute it globally or 
combine it with other local feedback methods.

-- distribute it
- use milter-greylist as the base of your Distributed Network. I only have a 
vague idea how to do that. Maybe one could distribute all hosts that do not 
retry after one hour to the network as "probably spammer". This could be 
weighted with scores. So, after one hour the weight could start with 1 and 
increase hour by hour or so and make the host more "spammy" over time. The 
other machines in the network could then start greylisting this host with a far 
higher retry period than usual and also exchange results again.

-- local integration
- check if the user exists and if it doesn't exist set the greylisting time for 
this host much higher
- allow feedback from spamassassin and other spam-content-checkers and if a 
message was spam wipe the host from the whitelist and set the greylisting time 
for this host much higher

-- maybe distribute these findings to the network as well?

In general I think it is a good idea to try combining methods and integrating 
feedback. Fighting all spam with one method is bound to fail. Greylisting works 
marvelously at the moment, but it can't catch all spam and other methods won't 
do this single-handedly as well.
F.i. I use (in this order):

-milter-greylist
-sendmail with access.db and three good RBLs
-MailScanner which tests for phishing, bad content and spam checks with the 
help of Spamassassin. Spamassassin is filled with good rulesets from SARE in 
addition to the packaged ones. Only that makes it really effective.

In the end almost nothing spammy or nasty gets thru. I used to kill 50-70% at 
the sendmail level with RBLs and access.db before I used greylisting. Now 
greylisting accounts for somethign like 70-80%, sendmail for another 15-20% and 
only 5% or less is left for MailScanner/Spamassassin. On my main machine my 
Spamassassin wouldn't barely get a spam if I did't get some mail from a 
forwarder which is of course whitelisted. I already fear that SA gets so few to 
do that the bayes database cannot be trained.
I could just go with Spamassassin and don't use any other method and it would 
catch all spam that is now caught before it as well, so the end result of what 
I get is the same. But putting sendmail and greylisting before it saves me 
ample ressources.


Kai

-- 
Kai Sch\ufffdtzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

Kai Schaetzl wrote on Wed, 05 Apr 2006 01:05:02 +0200:

> > 1) The context 
> > greylisting have been a good protection against spam, but spamware able 
> > to defeat it by resending spam are now available. We need a way of adressing 
> > the new problem 
>  
> yes, that was to be expected.

Ahm, one more aspect on this. There are spammers which use zombies and what-not 
to flush out their crap. But there are also spammers that use normal SMTP 
software and send from their own static mailservers and if they don't play to 
foul they may not get kicked by the ISP or they send from a safe haven, anyway. 
So, you would need to determine if the resends really came from special bulkware 
or where not just normal MTAs.

Kai

-- 
Kai Sch\ufffdtzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com