Milter-greylist

Hi

Recently I saw a new kind of spam. The messages contain a text part with 
nonsense that are obviosuly here to ruin bayesian filtering. The words 
are non-spam words, and if the user classify the message as spam, the 
bayesian filter efficiency will go down.

The message also contains an image, which carry the spam message. Because
the spam message is in an image, it is unreachable for bayesian filters.

That's not a real problem for me because I don't use bayesian filtering. 
I am more worried by the fact that a lot of such message get through 
milter-greylist.

Headers show that the message come from DSL and cable pools, so IMO it's
from a botnet. X-Greylist header reports that the sender retried only one
time and after 5 minutes and a few seconds. My greylist delay is 5 mn, 
so I wonder if this is a coincidence, or if the spam engine reads the
text message in the SMTP reply that says "please come back in 00:05:00".

Do we have to face spam engine that implement resends? What is your 
experience with that problem? 

I will try raising the greylist parameter (delay before the mail is accepted)
from 5 mn to 30 mn. If that does not cure the problem, it probably means
we have to hunt for new ideas again and code a new tool. Any suggestion
is welcome.

-- 
Emmanuel Dreyfus
manu@...

> Headers show that the message come from DSL and cable pools, so IMO
it's
> from a botnet. X-Greylist header reports that the sender retried only
one
> time and after 5 minutes and a few seconds. My greylist delay is 5 mn,

> so I wonder if this is a coincidence, or if the spam engine reads the
> text message in the SMTP reply that says "please come back in
00:05:00".
>

You may also try removing the polite and helpful "...in 00:05:00" and
just say "Greylisting in progress, please come back later...". Or
perhaps even remove the word "Greylisting" as well.

The more fun & crazy (experimental) way would be to fill the the time in
the SMTP message with letters/non-numeric characters, to see how they
react to that :-) Hopefully they will crash...

(I haven't heard of MTAs that notice the SMTP message and greylisting
time printed there...)

/P

Emmanuel Dreyfus wrote:
 > [...]
 > I am more worried by the fact that a lot of such message get through 
 > milter-greylist.
 > 
 > Headers show that the message come from DSL and cable pools, so IMO it's
 > from a botnet. X-Greylist header reports that the sender retried only one
 > time and after 5 minutes and a few seconds. My greylist delay is 5 mn, 
 > so I wonder if this is a coincidence, or if the spam engine reads the
 > text message in the SMTP reply that says "please come back in 00:05:00".

A quick and simple fix would be not to include the information
in the SMTP reply, don't you think?  For example, just say
"please come back later".  Standard MTAs never look at the
reply sentence anyway, so it shouldn't hurt legal mail.  But
spammers might lose an important piece of information.

 > Do we have to face spam engine that implement resends? What is your 
 > experience with that problem? 

I got a few of those spams that you described, too.  But not
many, which is probably due to the fact that I use NJABL and
DSBL for blacklisting dynamic DSL IPs and similar things, so
many of those botnet-generated spams might not reach me.  :-)

 > I will try raising the greylist parameter (delay before the mail is accepted)
 > from 5 mn to 30 mn. If that does not cure the problem, it probably means
 > we have to hunt for new ideas again and code a new tool. Any suggestion
 > is welcome.

You should increase it from 5 to 6.  If your spam mails then
start coming in at 6 minutes plus a few seconds, you have
proof that their mailers indeed read the SMTP reply sentence
and interpret the time specification.  You should then modify
that sentence, so spammers don't get a hint when to retry
your address.

Best regards
   Oliver

-- 
Oliver Fromme,  secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing
Dienstleistungen mit Schwerpunkt FreeBSD: http://www.secnetix.de/bsd
Any opinions expressed in this message may be personal to the author
and may not necessarily reflect the opinions of secnetix in any way.

"C++ is the only current language making COBOL look good."
        -- Bertrand Meyer

On Tue, Apr 04, 2006 at 11:04:11AM +0200, fredrik.pettai@... wrote:
> You may also try removing the polite and helpful "...in 00:05:00" and
> just say "Greylisting in progress, please come back later...". Or
> perhaps even remove the word "Greylisting" as well.

Yes, I will try this if I see just one retry after my new greylisting 
delay (30 mn).

If the spam engine reads that message, this make new counter measures
possible. I could use a real greylisting delay of 5 mn, display a 
message telling it's 30 seconds, and blacklist any machine that perform
retries within less than one minute. Publishing a DNSRBL of such a 
blacklist could be useful too. 

> The more fun & crazy (experimental) way would be to fill the the time in
> the SMTP message with letters/non-numeric characters, to see how they
> react to that :-) Hopefully they will crash...

The really fun and crazy idea would be to obtain the spam engine binary, 
find an overflow in it, and send data that would cause an exploit to
take control of the sender. Any taker? :-)

-- 
Emmanuel Dreyfus
manu@...

Emmanuel Dreyfus wrote:
> On Tue, Apr 04, 2006 at 11:04:11AM +0200, fredrik.pettai@... wrote:
>>You may also try removing the polite and helpful "...in 00:05:00" and
>>just say "Greylisting in progress, please come back later...". Or
>>perhaps even remove the word "Greylisting" as well.
> 
> Yes, I will try this if I see just one retry after my new greylisting 
> delay (30 mn).
> 
> If the spam engine reads that message, this make new counter measures
> possible. I could use a real greylisting delay of 5 mn, display a 
> message telling it's 30 seconds, and blacklist any machine that perform
> retries within less than one minute. Publishing a DNSRBL of such a 
> blacklist could be useful too. 
> 
Many normal MTA dont read the string with the delay time and retry after 
less then one minute. I think ther is a high risk that you blacklist 
normal MTA.

Greylisting works because Spamers often are not RFC conform. But it was 
clear that Spamers would become RFC conform if many Mailadmin use 
Greylisting.


>>The more fun & crazy (experimental) way would be to fill the the time in
>>the SMTP message with letters/non-numeric characters, to see how they
>>react to that :-) Hopefully they will crash...
> 
A nice idea but i hope normal MTA won't crash.


> The really fun and crazy idea would be to obtain the spam engine binary, 
> find an overflow in it, and send data that would cause an exploit to
> take control of the sender. Any taker? :-)
> 


-- 
--------------------------------------------------------------------------------
M.Menge                                 Tel.: (49) 7071/29-70316
Universitaet Tuebingen                  Fax.: (49) 7071/29-5912
Zentrum fuer Datenverarbeitung          mail: menge@...-tuebingen.de
Waechterstrasse 76
72074 Tuebingen

> Headers show that the message come from DSL and cable pools, so IMO it's
> from a botnet. X-Greylist header reports that the sender retried only one
> time and after 5 minutes and a few seconds. My greylist delay is 5 mn, 
> so I wonder if this is a coincidence, or if the spam engine reads the
> text message in the SMTP reply that says "please come back in 00:05:00".
> 
> Do we have to face spam engine that implement resends? What is your 
> experience with that problem? 

Yes, it is 5 minutes and some seconds for me too.
I had on one minute greylist time,
increased now it to 6 minutes.
(by the way, would be nice to specify in second granularity)
Since I am in quite mode (not communicating the delay),
pretty sure that the spamware does not read the SMTP reply text.
Anyhow, the spamware retries now in double time (~10m20s).
I think nothing really to do. We expected this to happen.
It just costs a bit more to send the spam.

Bests,
Attila

<attila.bruncsak@...> wrote:

> Anyhow, the spamware retries now in double time (~10m20s).
> I think nothing really to do. We expected this to happen.
> It just costs a bit more to send the spam.

We have a old counter measure that we now have to deploy: distributed
honeypots. That will catch botnet members between resends. I wrote some
software to do that some time ago, and I guess it's time to work on it
again.

But there is a problem: spammers will probably use ISP regular SMTP
servers as relays so that they get blacklisted by the distributed
honeypot network, thus making the counter measure useless. So we have to
invent a way of finding that a host is a regular SMTP server that should
not be blacklisted.

I thought about using some scoring scheme. Each time you accept a mail
from an IP, raise the karma of the IP. If the IP is caught by a
honeypot, lower its karma. But the idea is not fully mature. 

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

>
>But there is a problem: spammers will probably use ISP regular SMTP
>servers as relays so that they get blacklisted by the distributed
>honeypot network, thus making the counter measure useless. So we have
to
>invent a way of finding that a host is a regular SMTP server that
should
>not be blacklisted.
>
>I thought about using some scoring scheme. Each time you accept a mail
>from an IP, raise the karma of the IP. If the IP is caught by a
>honeypot, lower its karma. But the idea is not fully mature. 
>

Spamcop, (a DNSBL) uses the same idea with honeypots (at least, they did
use em...). They added IPs of MTAs to there blacklist (bl.spamcop.net)
for 24 hours if a honeypot caught the MTA sending spam. I don't know
exactly how Spamcop classed the MTAs that was caught by one or more
honeypots, since that was a "trade secret" of Spamcop design as I
remember it (I could be wrong, though).

We tried there services about a year ago, and on some occasion one of
the ISPs in Sweden got some of there MTAs blacklisted for this reason
:-)

/P

wrote on Tue, 4 Apr 2006 11:04:11 +0200:

> You may also try removing the polite and helpful "...in 00:05:00" and 
> just say "Greylisting in progress, please come back later...". Or 
> perhaps even remove the word "Greylisting" as well.

And on top of this it might be interesting to randomize this a bit. F.i. 
if your "base" timeout is 10min you may actually choose a time from 5 - 
15min for each host or even by recipient. If done by recipient it's even 
possible that one message can get delivered after 10 minutes, but the next 
is still greylisted, although both were waiting the same time.

And as a special for those engines that *really* read the notice and act 
accordingly to it. You could advertise a wrong greylisting time. I doubt 
that a legitimate SMTP server will ever read and act after this unless it 
is written down in some RFC in 50 years.

Kai

-- 
Kai Sch\ufffdtzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

Emmanuel Dreyfus wrote:
> Hi
> 
> Recently I saw a new kind of spam. The messages contain a text part with 
> nonsense that are obviosuly here to ruin bayesian filtering. The words 
> are non-spam words, and if the user classify the message as spam, the 
> bayesian filter efficiency will go down.
> 
> The message also contains an image, which carry the spam message. Because
> the spam message is in an image, it is unreachable for bayesian filters.
> 
> That's not a real problem for me because I don't use bayesian filtering. 
> I am more worried by the fact that a lot of such message get through 
> milter-greylist.
> 
> Headers show that the message come from DSL and cable pools, so IMO it's
> from a botnet. X-Greylist header reports that the sender retried only one
> time and after 5 minutes and a few seconds. My greylist delay is 5 mn, 
> so I wonder if this is a coincidence, or if the spam engine reads the
> text message in the SMTP reply that says "please come back in 00:05:00".
> 
> Do we have to face spam engine that implement resends? What is your 
> experience with that problem? 
> 
> I will try raising the greylist parameter (delay before the mail is accepted)
> from 5 mn to 30 mn. If that does not cure the problem, it probably means
> we have to hunt for new ideas again and code a new tool. Any suggestion
> is welcome.
> 

Hi,

I think SPF (see www.openspf.org)  may be a googd idea to use.
Whitelist if SPF returns "Pass", reject if SPF returns "Fail"
and in the other cases greylist. The time should depend on the return 
value of SPF 5 min if SPF returns "Neutral" or "none" and
1h if SPF returns "Softfail" or Errors.

cu

	Michael


-- 
--------------------------------------------------------------------------------
M.Menge                                 Tel.: (49) 7071/29-70316
Universitaet Tuebingen                  Fax.: (49) 7071/29-5912
Zentrum fuer Datenverarbeitung          mail: menge@...-tuebingen.de
Waechterstrasse 76
72074 Tuebingen

On Fri, Apr 07, 2006 at 04:57:02PM +0200, Michael Menge wrote:
> I think SPF (see www.openspf.org)  may be a googd idea to use.

SPF is quickly becoming worthless because spammers register new domains,
create fine looking SPF records for their spam bot networks and start
delivering spam from them.

I'm currently using SPF for automatic white listing in Milter Greylist
and am seriously considering to drop it.

	Kind regards

-- 
Matthias Scheler                                  http://scheler.de/~matthias/

Matthias Scheler wrote:
> On Fri, Apr 07, 2006 at 04:57:02PM +0200, Michael Menge wrote:
>> I think SPF (see www.openspf.org)  may be a googd idea to use.
> 
> SPF is quickly becoming worthless because spammers register new domains,
> create fine looking SPF records for their spam bot networks and start
> delivering spam from them.

Duh.. this is exactly what SPF is intended to do.. Force spammers to create
their own domains instead of abusing existing ones.

Anyone who thinks SPF is intended to stop spam is fooling themselves. SPF is an
anti-forgery technology. This has some utility in fighting spam, but it isn't
intended to stop spam.

Also on the upside for the spam front, forcing spammers to create their own
domains costs them money.
> 
> I'm currently using SPF for automatic white listing in Milter Greylist
> and am seriously considering to drop it.

You really should. IMHO, this is one of the most misconceived features of
milter-greylist. Passing SPF is a horribly poor indication the message is in any
way "good" from a spam perspective.

In general the only thing you can treat with much credibility on the spam front
is treating SPF failures as an indication the message is likely spam. Period.

really, milter-greylist should generalize SPF results into ACL rules, so we can
all choose what to do at different SPF levels.

This way we can do things like this:

acl spf_softfail dark_greylist
acl spf_fail blacklist
acl default greylist

which makes a lot more sense than the current use of SPF in milter-greylist.

(note- read the dark-grey thread for the idea behind dark_greylist)

Matt Kettler <mkettler@...> wrote:

> really, milter-greylist should generalize SPF results into ACL rules, so
> we can all choose what to do at different SPF levels.
> 
> This way we can do things like this:
> 
> acl spf_softfail dark_greylist
> acl spf_fail blacklist
> acl default greylist

Yes, this has been a desirable feature for a while. Feel free to
contribute it...

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

On Fri, 7 Apr 2006, Matthias Scheler wrote:
> On Fri, Apr 07, 2006 at 04:57:02PM +0200, Michael Menge wrote:
>> I think SPF (see www.openspf.org)  may be a googd idea to use.
> SPF is quickly becoming worthless because spammers register new domains,
> create fine looking SPF records for their spam bot networks and start
> delivering spam from them.

SPF IS NOT DESIGNED TO BLOCK SPAM!!!

-Dan

I agree and the SPF group agrees too. SPF is to prevent domain spoofing 
not stop Spam. There's a big difference. Unfortunetly there's an easy 
way around SPF so the user sees the spoofed domain/email address even if 
the spoofed domain has SPF records. Fortunetly only a couple of the 
Spammers have figured this out and are using it.

goemon@... wrote:

>On Fri, 7 Apr 2006, Matthias Scheler wrote:
>  
>
>>On Fri, Apr 07, 2006 at 04:57:02PM +0200, Michael Menge wrote:
>>    
>>
>>>I think SPF (see www.openspf.org)  may be a googd idea to use.
>>>      
>>>
>>SPF is quickly becoming worthless because spammers register new domains,
>>create fine looking SPF records for their spam bot networks and start
>>delivering spam from them.
>>    
>>
>
>SPF IS NOT DESIGNED TO BLOCK SPAM!!!
>
>-Dan
>
>
> 
>Yahoo! Groups Links
>
>
>
> 
>
>
>
>  
>

-- 

----------------------------------
Dennis Willson
mailto:taz@...
http://www.taz-mania.com

You just need to be careful here. SPF is expensive (lots of overhead) so 
you should do as few SPF lookups as possible.
A single SPF record could cause as many as 10 DNS lookups, so if you 
verify SPF more than once you're multiplying the DNS overhead (Yes I 
know that the second, third, etc.. lookups would be cached in your local 
caching DNS server, but it still has to make the mail server wait for 
the additional lookups even if their are a bit faster after the first 
one). So as few modules as possible should be doing SPF lookups. The 
best way would be to have one module do it as early as possible and put 
a marker as an email header so other modules later on could see the 
results and act accordingly (I also know that the first module that was 
doing the actual SPF lookups would have to be sure there wasn't a 
spoofed header saying it passed SPF when it didn't so it would have to 
remove any SPF headers and then do its lookups and then add its own header).

Just some thoughts on the subject...

Matt Kettler wrote:

>Matthias Scheler wrote:
>  
>
>>On Fri, Apr 07, 2006 at 04:57:02PM +0200, Michael Menge wrote:
>>    
>>
>>>I think SPF (see www.openspf.org)  may be a googd idea to use.
>>>      
>>>
>>SPF is quickly becoming worthless because spammers register new domains,
>>create fine looking SPF records for their spam bot networks and start
>>delivering spam from them.
>>    
>>
>
>Duh.. this is exactly what SPF is intended to do.. Force spammers to create
>their own domains instead of abusing existing ones.
>
>Anyone who thinks SPF is intended to stop spam is fooling themselves. SPF is an
>anti-forgery technology. This has some utility in fighting spam, but it isn't
>intended to stop spam.
>
>Also on the upside for the spam front, forcing spammers to create their own
>domains costs them money.
>  
>
>>I'm currently using SPF for automatic white listing in Milter Greylist
>>and am seriously considering to drop it.
>>    
>>
>
>You really should. IMHO, this is one of the most misconceived features of
>milter-greylist. Passing SPF is a horribly poor indication the message is in any
>way "good" from a spam perspective.
>
>In general the only thing you can treat with much credibility on the spam front
>is treating SPF failures as an indication the message is likely spam. Period.
>
>really, milter-greylist should generalize SPF results into ACL rules, so we can
>all choose what to do at different SPF levels.
>
>This way we can do things like this:
>
>acl spf_softfail dark_greylist
>acl spf_fail blacklist
>acl default greylist
>
>which makes a lot more sense than the current use of SPF in milter-greylist.
>
>(note- read the dark-grey thread for the idea behind dark_greylist)
>
>
>
> 
>Yahoo! Groups Links
>
>
>
> 
>
>
>
>  
>

-- 

----------------------------------
Dennis Willson
mailto:taz@...
http://www.taz-mania.com

Dennis Willson wrote:
> You just need to be careful here. SPF is expensive (lots of overhead) so 
> you should do as few SPF lookups as possible.
> A single SPF record could cause as many as 10 DNS lookups, so if you 
> verify SPF more than once you're multiplying the DNS overhead (Yes I 
> know that the second, third, etc.. lookups would be cached in your local 
> caching DNS server, but it still has to make the mail server wait for 
> the additional lookups even if their are a bit faster after the first 
> one). So as few modules as possible should be doing SPF lookups.

I agree whole heartedly.. but, milter-greylist already has SPF support, so we
already have the option of paying this penalty. To top it off, milter-greylist
is using SPF unwisely. My general goal would be "if you're going to do SPF, do
it well, and do it for the right reason"

As for multiple queries:  In theory, if the whole acl generalization was
implemented correctly, the SPF result should be cached in milter-greylist
itself, so that multiple ACL objects can check against it without causing more
than one SPF check..

The only thing you'd have to worry about then is if you had another tool further
down the mail chain that did SPF as well.

However, none of this is any worse than the existing situation if you use m-g's
SPF support. In many ways it's better, because you can use the SPF data to do
something useful (use SPF-fail perform more severe greylisting, or
blacklisting), instead of doing something pointless (using SPF pass to whitelist).