Milter-greylist

Hi

Anyone want to give this a try?
http://ftp.espci.fr/shadow/manu/blacklist.diff

That adds the acl blacklist feature. 

-- 
Emmanuel Dreyfus
manu@...

> -----Original Message-----
> From: milter-greylist@yahoogroups.com 
> [mailto:milter-greylist@yahoogroups.com] On Behalf Of Emmanuel Dreyfus
> Sent: 30 June, 2005 14:40
> To: milter-greylist@yahoogroups.com
> Subject: [milter-greylist] patch for blacklist support
> 
> Hi
> 
> Anyone want to give this a try?
> http://ftp.espci.fr/shadow/manu/blacklist.diff
> 
> That adds the acl blacklist feature. 

Do you also have plans to expand this with:

acl blacklist helo <unwanted helo>

Best regards,

Adri.

> 
> --
> Emmanuel Dreyfus
> manu@...
> 
> 
>  
> Yahoo! Groups Links
> 
> 
> 
>  
> 
> 
>

On Thu, 30 Jun 2005, Emmanuel Dreyfus wrote:
> Anyone want to give this a try?
> http://ftp.espci.fr/shadow/manu/blacklist.diff
> 
> That adds the acl blacklist feature. 

It would be nice to specify a custom reject message. This is the main
thing special about /etc/mail/access.

acl blacklist blabla1 text "554 5.7.1 fuck spammers"
acl blacklist blabla2 text "554 5.7.1 twit filter"
acl blacklist blabla3 text "554 5.7.1 mail refused. your domain does not have a working abuse or postmaster mailbox."
acl blacklist blabla4 text "554 5.7.1 yumago dong-mogo"

-Dan

Dan Hollis <goemon@...> wrote:

> It would be nice to specify a custom reject message. This is the main
> thing special about /etc/mail/access.
> 
> acl blacklist blabla1 text "554 5.7.1 fuck spammers"
> acl blacklist blabla2 text "554 5.7.1 twit filter"
> acl blacklist blabla3 text "554 5.7.1 mail refused. your domain does not
> have a working abuse or postmaster mailbox."
> acl blacklist blabla4 text "554 5.7.1 yumago dong-mogo"

I added this (well, it's the message keyword instead of text... maybe it
should be named status?), and I added helo as well.

http://ftp.espci.fr/shadow/manu/blacklist2.diff 

Now, this seems buggy: try this
acl blacklist from manu@... rcpt toto@...
acl blacklist from manu@... rcpt titi@... message "551
5.0.0 go away"

Send a message from manu@... to two recipients: toto@...
and titi@.... The second message is accepted.

Someone wants to debug it? I'm getting bored with it.

-- 
Emmanuel Dreyfus
Le cahier de l'admin BSD 2eme ed. est dans toutes les bonnes librairies
http://www.eyrolles.com/Informatique/Livre/9782212114638/livre-bsd.php
manu@...

On Thu, 30 Jun 2005 manu@... wrote:
> Now, this seems buggy: try this
> acl blacklist from manu@... rcpt toto@...
> acl blacklist from manu@... rcpt titi@... message "551
> 5.0.0 go away"
> Send a message from manu@... to two recipients: toto@...
> and titi@.... The second message is accepted.
> Someone wants to debug it? I'm getting bored with it.

Sounds like this might happen with greylisting too. Anyone tested this 
case?

-Dan

--- In milter-greylist@yahoogroups.com, manu@n... wrote:
> Someone wants to debug it? I'm getting bored with it.

I wouldn't be surprised, if the last char of the message was lost, but
I have no idea, what this is about.


    Elrond

Emmanuel,

I have just installed the blacklist2 patches against milter-greylist 2.0
and it seems there's a serious problem with the acl blacklist helo!
When I add to my greylist.conf:

acl blacklist helo mydomain.com

Each message with a to: for a user of mydomain.com, which is NOT
whitelisted, will get a '551 5.0.0 Go away, you are permanently
blacklisted' error!
It seems that the 'acl blacklist helo' is compared against the to, from
or hostname of the connecting MTA, instead of comparing it with the
supplied helo string.
Also it is not possible to blacklist a helo string like '192.168.0.1',
since only FQDN's seem to be accepted, while some spammers will do a
'helo 'my.ip.add.res' (like in helo 192.168.0.1).

Best regards,

Adri Koppes

> -----Original Message-----
> From: milter-greylist@yahoogroups.com 
> [mailto:milter-greylist@yahoogroups.com] On Behalf Of manu@...
> Sent: 30 June, 2005 23:31
> To: milter-greylist@yahoogroups.com
> Subject: Re: [milter-greylist] patch for blacklist support
> 
> Dan Hollis <goemon@...t> wrote:
> 
> > It would be nice to specify a custom reject message. This 
> is the main 
> > thing special about /etc/mail/access.
> > 
> > acl blacklist blabla1 text "554 5.7.1 fuck spammers"
> > acl blacklist blabla2 text "554 5.7.1 twit filter"
> > acl blacklist blabla3 text "554 5.7.1 mail refused. your 
> domain does 
> > not have a working abuse or postmaster mailbox."
> > acl blacklist blabla4 text "554 5.7.1 yumago dong-mogo"
> 
> I added this (well, it's the message keyword instead of 
> text... maybe it should be named status?), and I added helo as well.
> 
> http://ftp.espci.fr/shadow/manu/blacklist2.diff 
> 
> Now, this seems buggy: try this
> acl blacklist from manu@... rcpt toto@... acl 
> blacklist from manu@netbsd.org rcpt titi@... message 
> "551 5.0.0 go away"
> 
> Send a message from manu@... to two recipients: 
> toto@... and titi@.... The second message is accepted.
> 
> Someone wants to debug it? I'm getting bored with it.
> 
> --
> Emmanuel Dreyfus
> Le cahier de l'admin BSD 2eme ed. est dans toutes les bonnes 
> librairies 
> http://www.eyrolles.com/Informatique/Livre/9782212114638/livre-bsd.php
> manu@netbsd.org
> 
> 
>  
> Yahoo! Groups Links
> 
> 
> 
>  
> 
> 
>

Adri Koppes <adrik@...> wrote:

> I have just installed the blacklist2 patches against milter-greylist 2.0
> and it seems there's a serious problem with the acl blacklist helo!

Of course it's buggy, else I would have released a new version instead
of a patch. :-)

I'm stuck with bugs in ipsec-tools. If someone wants to work on the
patch...

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

On Fri, 1 Jul 2005, Adri Koppes wrote:
> Emmanuel,
> 
> I have just installed the blacklist2 patches against milter-greylist 2.0
> and it seems there's a serious problem with the acl blacklist helo!
> When I add to my greylist.conf:
> 
> acl blacklist helo mydomain.com
> 
> Each message with a to: for a user of mydomain.com, which is NOT
> whitelisted, will get a '551 5.0.0 Go away, you are permanently
> blacklisted' error!
> It seems that the 'acl blacklist helo' is compared against the to, from
> or hostname of the connecting MTA, instead of comparing it with the
> supplied helo string.
> Also it is not possible to blacklist a helo string like '192.168.0.1',
> since only FQDN's seem to be accepted, while some spammers will do a
> 'helo 'my.ip.add.res' (like in helo 192.168.0.1).

have you made any process on this patch?

-Dan

On Tue, Jul 26, 2005 at 02:32:19PM -0700, Dan Hollis wrote:
> have you made any process on this patch?

No, I have no time for hacking thoses days, unfortunately :-(

-- 
Emmanuel Dreyfus
manu@...

Yo all. How does this blacklist support work? Just by acl blacklist entries, or can it also interact with results of other mitlers and sendmail? What I would

Benoit Panizzon <panizzon@...> wrote:

> How does this blacklist support work? 

It does not work. :-)

I just drafted a patch and made it public. It's not finished, there are
issues to fix.

> Just by acl blacklist entries, or can it 
> also interact with results of other mitlers and sendmail?

Except if the other milter and milter-greylist can agree on some
protocol, there is no way to use another milter's results.
 
> I see more and more spamtools that are greylist aware.  They retry sending
> that email after a few minutes. So greylisting does not avoid them.

For how long do you greylist? A long enough delay should do the trick...

-- 
Emmanuel Dreyfus
Publicité subliminale: achetez ce livre!
http://www.eyrolles.com/Informatique/Livre/9782212114638/livre-bsd.php
manu@...

Hi Manu ... (10minutes) Not realy. As example, our favourite swiss spamer is ordering a new bullet proof server in china for allmost every spamrun. He does not

Benoit Panizzon wrote:
> Hi Manu
> 
> 
>>>I see more and more spamtools that are greylist aware.  They retry
>>>sending that email after a few minutes. So greylisting does not avoid
>>>them.
>>
>>For how long do you greylist? A long enough delay should do the trick...
> 
> 
> (10minutes)
> 
> Not realy. As example, our favourite swiss spamer is ordering a new bullet 
> proof server in china for allmost every spamrun. He does not just run a 
> 'spambot' on them, but a real mailserver. So just greylisting has become 
> useless.
> Better would be what I suggested: blacklist tuples (or their IP) which had a 
> positive hit in spamassassin. But I understand this is not trivial :-)
> 
> Btw, I have whitelisted all known Mailservers from the list in:
> 
> http://antispam.imp.ch/swinog-dnsrbl-whitelist
> 
> others might find this list usefull too as it eliminates delays from servers 
> which would resend that email anyway.
> 
> -Benoit-


You could always write up a little script that pulls results from your 
spam marked emails, and uses the relay address from that to add to the 
blacklist.

Benoit Panizzon <panizzon@...> wrote:

> Not realy. As example, our favourite swiss spamer is ordering a new bullet
> proof server in china for allmost every spamrun. He does not just run a
> 'spambot' on them, but a real mailserver. So just greylisting has become
> useless.
> Better would be what I suggested: blacklist tuples (or their IP) which had a
> positive hit in spamassassin. But I understand this is not trivial :-)

Well, we can add such a communication capability (well, in fact you
could write the code, and I could add it to milter-greylist :-). Here
are some ideas of how it could be done:

1) Finish the blacklisting support. That means trying the patch I
released, testing it in various situation, sit down for a while thinking
of how things should really be handled, fix the patch and contribute it
back.

2a) Then there is the communication between other tools and
milter-greylist. A complex approach is to enhance the simple protocol
used for MX sync. We could add dynamic blacklist tuples to that. And it
would be trivial to write a command line tool for using that protocol to
remote-control milter-greylist. 

Then you just have to fork that command line tool from your spamassassin
machinery when you find someone that needs to be blacklisted.

2b) An alternative: you just rebuild a milter-greylist config file from
your spamassassin machinery and you let milter-greylist automatically
reload the config with the new blacklist. 

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

--- In milter-greylist@yahoogroups.com, Benoit Panizzon
<panizzon@w...> wrote:
> Hi Manu
> 
> > > I see more and more spamtools that are greylist aware.  They retry
> > > sending that email after a few minutes. So greylisting does not
avoid
> > > them.
> >
> > For how long do you greylist? A long enough delay should do the
trick...
> 
> (10minutes)
> 
> Not realy. As example, our favourite swiss spamer is ordering a new
bullet 
> proof server in china for allmost every spamrun. He does not just run a 
> 'spambot' on them, but a real mailserver. So just greylisting has
become 
> useless.
> Better would be what I suggested: blacklist tuples (or their IP)
which had a 
> positive hit in spamassassin. But I understand this is not trivial :-)
> 
> Btw, I have whitelisted all known Mailservers from the list in:
> 
> http://antispam.imp.ch/swinog-dnsrbl-whitelist
> 
> others might find this list usefull too as it eliminates delays from
servers 
> which would resend that email anyway.
> 
> -Benoit-

I agree on the points that have come about in this thread.

What I am seeing, is, two phenomenon, usually working in conjunction:

* SPAM hosts are overcoming greylisting and sendmail's greet_pause by
reconnecting every 30 seconds on up to a few minutes and waiting
increasingly longer after connecting to port 25 before blasting SPAM.

Now, I have not thoroughly delved into the RFCs to see if what I
propose would break standards, but it is my opinion that legitimate
(non-spam) hosts would attempt to redeliver on the order of minutes
(say 10 at the least, but I'd argue more like 15.)

So, what I would like to see is a configurable blacklist window in
milter-greylist that, if a tuple shows up as attempting to redeliver
mail within a window (say 3 times in less than 5 minutes), that the
tuple be blacklisted.
 
I was curious to get some comment on this idea from the author of
milter-greylist and/or other mail system administrators as to the
viability (at least in terms of not breaking the mail RFC).

Thanks much.

bytemastr wrote:
> 
> I agree on the points that have come about in this thread.
> 
> What I am seeing, is, two phenomenon, usually working in conjunction:
> 
> * SPAM hosts are overcoming greylisting and sendmail's greet_pause by
> reconnecting every 30 seconds on up to a few minutes and waiting
> increasingly longer after connecting to port 25 before blasting SPAM.
> 
> Now, I have not thoroughly delved into the RFCs to see if what I
> propose would break standards, but it is my opinion that legitimate
> (non-spam) hosts would attempt to redeliver on the order of minutes
> (say 10 at the least, but I'd argue more like 15.)
> 
> So, what I would like to see is a configurable blacklist window in
> milter-greylist that, if a tuple shows up as attempting to redeliver
> mail within a window (say 3 times in less than 5 minutes), that the
> tuple be blacklisted.

Sounds very dangerous, for multiple reasons.

First, I've seen several legitimate hosts that retry every minute.

Usually this is a byproduct of a site that relays mail to an internal server and
the internal server is unreliable (ie: any kind of groupware). In order to
reduce the time to receive mail that got backed up while the groupware server
was down, the admin has retry interval set short. Yes, a smart admin would set
this up so only local mail gets retried quickly, but there's not nearly as many
smart admins out there as there should be.

Second, milter-greylist can only track the tuple. It doesn't know if the message
is the same message, or multiple different messages, say from a busy mailing
list you forgot to whitelist. Usually all the messages on a mailing list will
have the same tuple: return-path (the list manager), recipient (you) and source
IP (the list server). Usually the return-path doesn't match the From: header
unless the listserv is completely broken.

I know some mailing lists that easily break 3 messages every 5 mins, and if you
signed up for those lists you'd auto-blacklist your subscription.

On Mon, Aug 15, 2005 at 05:04:37PM -0000, bytemastr wrote:
> Now, I have not thoroughly delved into the RFCs to see if what I
> propose would break standards, but it is my opinion that legitimate
> (non-spam) hosts would attempt to redeliver on the order of minutes
> (say 10 at the least, but I'd argue more like 15.)

Queue handling in "sendmail" works differently. It doesn't remember when
to retry delivering a particular e-mail. It will instead try to redeliver
every queued e-mail in regular intervals. An e-mail that got rejected
by greylisting 1 minute before the next global queue run will therefore
be redelivered 1 minute later.

	Kind regards

-- 
Matthias Scheler                                  http://scheler.de/~matthias/

On Mon, Aug 15, 2005 at 05:04:37PM -0000, bytemastr wrote:
> Now, I have not thoroughly delved into the RFCs to see if what I
> propose would break standards, but it is my opinion that legitimate
> (non-spam) hosts would attempt to redeliver on the order of minutes
> (say 10 at the least, but I'd argue more like 15.)

Matthias Scheler replied:
> Queue handling in "sendmail" works differently. It doesn't remember when
> to retry delivering a particular e-mail. It will instead try to redeliver
> every queued e-mail in regular intervals. An e-mail that got rejected
> by greylisting 1 minute before the next global queue run will therefore
> be redelivered 1 minute later.

Sendmail does remember the last time delivery of a particular email
message was attempted.  It has a tunable parameter MinQueueAge which
can be set to avoid the situation where a message delivery attempt
failed and then a queue run makes another delivery attempt shortly
thereafter.  The default value of MinQueueAge is 0 (always try to
deliver all messages in a queue run regardless of how recently the
last delivery attempt was made).

The default behaviour is as you stated:  email rejected by greylisting
just before a scheduled queue run will have delivery attempted twice
within a short period of time.  Because of this default, requiring three
delivery attempts within a short period of time (is there any vendor
shipping an MTA with a default configuration that will cause a message
delivery to be retried three times in less than 15 minutes?) would be a
reasonable minimum requirement to blacklist a server or tuple.

Section 4.5.4.1 of RFC 2821 states in part:

   The sender MUST delay retrying a particular destination after one
   attempt has failed.  In general, the retry interval SHOULD be at
   least 30 minutes; however, more sophisticated and variable strategies
   will be beneficial when the SMTP client can determine the reason for
   non-delivery.

   Retries continue until the message is transmitted or the sender gives
   up; the give-up time generally needs to be at least 4-5 days.  The
   parameters to the retry algorithm MUST be configurable.

--d

bytemastr <sinkr@...> wrote:

> I was curious to get some comment on this idea from the author of
> milter-greylist and/or other mail system administrators as to the
> viability (at least in terms of not breaking the mail RFC).

I'm not sure my opinion is worth such a consideration, but you asked for
it, here it is:

Your idea will kill legitimate mail from RFC compliant systems. This is
extremely bad. You should not do it.

On the other hand, a MTA that retries every 30 seconds is RFC compliant
but such a beahvior a shame. It wastes peer's bandwidth and CPU, it
fiils logs with garbage, and I can understand some system administrator
will want to setup a filter that does this.  

Such a setup is not annoying for me, as the MTA I administer retry every
15 or 30 minutes. The admins that will do it will loose mail, but not
mail comming from my systems. Such a setup will be annoying for spammers
and badly configured legitimate MTA that retry every 30 seconds. If
enough servers filter that way, it will gradually broken MTA and
spammers them to retry sending at a reasonable speed, leading to a
better Internet.

So in the end if some admins do it, it's not bad for me, and it will
even have a good side effect. If I can help providing tools to filter
that way, I'll do it (but with appropriate documentation and feature
turned off by default, so that someone doing it does it while knowing
the consequences). 

-- 
Emmanuel Dreyfus
Publicité subliminale: achetez ce livre!
http://www.eyrolles.com/Informatique/Livre/9782212114638/livre-bsd.php
manu@netbsd.org