Milter-greylist

At Thu, 2 Nov 2006 15:59:36 +0000,
Jonathan Perkin wrote:
> 
> Hi,
> 
> I'm trialling milter-greylist on the BBC mail infrastructure, which
> receives around 1 million emails per day.  Recently I added
> 
>   acl greylist domain /[0-9][0-9]*\-[0-9][0-9]*\-[0-9][0-9]*/
>   acl greylist domain /[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*/
>   acl greylist domain /[0-9]{12}/
> 
> to the config to greylist anything which looks like a dynamic address,
> and since making that change my monitoring has shown milter-greylist
> to fail an awful lot more.
> 
> The milter-greylist processes are sitting at around 600M resident
> memory, and are causing the system to swap.
> 
>   1. Can I limit the amount of memory milter-greylist will use to
>      cache lookups?  Obviously with a large number of connections this
>      is going to grow, but I cannot add more memory to the MX easily.
> 
>   2. Why is the increased load causing more failures?  I test the
>      filter with something similar to
> 
>        acl greylist from /greylist-test.*@host/
> 
>      and generate a random string after "greylist-test" for MAIL FROM
>      so that it won't get cached.  Today the number of failured for
>      this test has been extremely high (previously I saw a number of
>      cases where it wasn't being greylisted, but it appears to get
>      worse with load).
> 
> This is sendmail 8.13.7 with security fixes, milter-greylist 2.0.2,
> Solaris 9 and everything compiled with Sun Studio 11.

Frankly speaking, don't do that just for now because:

* The milter-greylist has not been designed for such highly loaded
servers. For example, it holds everything in core. Scalability and
performance improvements are important TODOs.

* There is a known bug in libmilter which leads information loss in
greylist.db when stopping or restarting the milter-greylist. I hope it
is fixed in sendmail 8.13.9.

* There is also a known bug in all versions of milter-greylist when
handling mail addresses such as <foo@[ip.add.re.ss]>. It will be
partially fixed in the next release but the full fix will be available
in 3.1.x.

* There are known race conditions in all versions of milter-greylist.
Nobody has reported problems due to these bugs but I can't tell what
happens on heavily loaded multiple CPU servers. We already have a new
threading implementation, which will be available in 3.1.x.