Milter-greylist

Hi all,

On the web page it says don't use 3.1.x for production servers, yet 
from postings to this mailing list, it seems that 3.1.x is 
recommended, and someone just suggested starting 4.0.

Which version should be used in a production server?

Collin

Collin Baillie <collin@...> wrote:

> On the web page it says don't use 3.1.x for production servers, yet 
> from postings to this mailing list, it seems that 3.1.x is 
> recommended, and someone just suggested starting 4.0.
> 
> Which version should be used in a production server?

3.0.x is battle-tested
3.1.x has been under active developement, but feature addition seems to
settle, so I'm going to start a new stable release cycle. I do run
latest 3.1.x in production environement.
4.0a1, which I'm about to release in a few minutes, is just 3.1.x
renamed as 4.0a1.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

On Tue, 3 Apr 2007, manu@... wrote:

> Collin Baillie <collin@...> wrote:
>
>> On the web page it says don't use 3.1.x for production servers, yet
>> from postings to this mailing list, it seems that 3.1.x is
>> recommended, and someone just suggested starting 4.0.
>>
>> Which version should be used in a production server?
>
> 3.0.x is battle-tested
> 3.1.x has been under active developement, but feature addition seems to
> settle, so I'm going to start a new stable release cycle. I do run
> latest 3.1.x in production environement.
> 4.0a1, which I'm about to release in a few minutes, is just 3.1.x
> renamed as 4.0a1.

Do any of the above actually use any sort of db/file based storage for the 
active greylist, or is it all still in main memory?

-Dan

--

"When I'm lost, and confused, and trying to make a U-turn, nothing annoys
me more than someone telling me to watch out for the tombstone!"

"How often does that happen, Fab?"

-David Feld & Tom Fabry, sometime in High School.

--------Dan Mahoney--------
Techie,  Sysadmin,  WebGeek
Gushi on efnet/undernet IRC
ICQ: 13735144   AIM: LarpGM
Site:  http://www.gushi.org
---------------------------

On Tue, Apr 03, 2007 at 12:55:26PM -0400, Dan Mahoney, System Admin wrote:
> Do any of the above actually use any sort of db/file based storage for the 
> active greylist, or is it all still in main memory?

Nobody have been annoyed enough to change that code. 

Well, it's almost true: I did try BDB at once, but I gave it up since I had
less garantees about integrity in case of crash. The code is available from
CVS if someone is interested (cvs co -rBDB milter-greylist). It was a fork 
from version 1.3.2 and it has not been updated since. I seriously doubt it
is usable at all.

-- 
Emmanuel Dreyfus
manu@...

On Tue, 3 Apr 2007, Emmanuel Dreyfus wrote:

> On Tue, Apr 03, 2007 at 12:55:26PM -0400, Dan Mahoney, System Admin wrote:
>> Do any of the above actually use any sort of db/file based storage for the
>> active greylist, or is it all still in main memory?
>
> Nobody have been annoyed enough to change that code.
>
> Well, it's almost true: I did try BDB at once, but I gave it up since I had
> less garantees about integrity in case of crash. The code is available from
> CVS if someone is interested (cvs co -rBDB milter-greylist). It was a fork
> from version 1.3.2 and it has not been updated since. I seriously doubt it
> is usable at all.

(Sorry for the late, late reply, I've been out of the loop in the 
hospital...nasty stuff)

Okay, so would anyone be interested in some sort of "bounty" on this?  I 
realize the lookup time with the list stored in main memory is very fast, 
but how much extra CPU time do you estimate would be used if it were 
mysql?  Sqlite?  BDB?

Did the BDB version have the option to periodically (say, hourly) dump a 
checkpoint file or something similar?

Right now I am only using milter-greylist for a handfull of domains (the 
really really badly spam-affected ones).  I would ideally like to use it 
server-wide (handling 400+ domains on this mail server).  I can't IMAGINE 
the size the process would grow to in that case, but certainly it would be 
one case whereby I'd rather sacrifice the CPU over the RAM.

-Dan

>
>

--

"I love you forever eternally."

-Connaian Expression

--------Dan Mahoney--------
Techie,  Sysadmin,  WebGeek
Gushi on efnet/undernet IRC
ICQ: 13735144   AIM: LarpGM
Site:  http://www.gushi.org
---------------------------

Dan Mahoney, System Admin <danm@...> wrote:

> Did the BDB version have the option to periodically (say, hourly) dump a
> checkpoint file or something similar?

The problem was that it was impossible to be sure no corruption could
occur on the DB file if milter-greylist crashed at a wrong time.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

On Thu, 14 Jun 2007, manu@... wrote:

> Dan Mahoney, System Admin <danm@...> wrote:
>
>> Did the BDB version have the option to periodically (say, hourly) dump a
>> checkpoint file or something similar?
>
> The problem was that it was impossible to be sure no corruption could
> occur on the DB file if milter-greylist crashed at a wrong time.

How is this different from how a crash is handled now?  We're at the mercy 
of the berkeley engine instead of being at the mercy of the dumpfile.  If 
the milter crashes while writing the dumpfile we're still screwed.

I'm not proposing that if one was to start using BDB or whatever, that the 
dumpfile should be done away with, something easy-to-restore and human 
readable in the event of corruption (and with milter-greylist having the 
option to read the dumpfile instead of or in comparison to the BDB file 
(i.e. to corruption check the BDB version).

-Dan

--

"You're not normal!"

-Michael G. Kessler, referring to my modem online time.


--------Dan Mahoney--------
Techie,  Sysadmin,  WebGeek
Gushi on efnet/undernet IRC
ICQ: 13735144   AIM: LarpGM
Site:  http://www.gushi.org
---------------------------

Dan Mahoney, System Admin <danm@...> wrote:

> How is this different from how a crash is handled now?  We're at the mercy
> of the berkeley engine instead of being at the mercy of the dumpfile.  If
> the milter crashes while writing the dumpfile we're still screwed.

You are not: we dump in a new file and rename the new file into the old
one. That means that unless your OS plays the dangerous game of flushing
metadata asynchronously, you are safe whatever happens. On reasonable
systems, rename(2) says something like:

"rename() guarantees that an instance of to will always exist, even if
the system should crash in the middle of the operation."

When I tried to add DBD support, I discovered I could not have such a
garantee anymore, and this is why I gave it up.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

On Fri, 15 Jun 2007, manu@... wrote:

> Dan Mahoney, System Admin <danm@...> wrote:
>
>> How is this different from how a crash is handled now?  We're at the mercy
>> of the berkeley engine instead of being at the mercy of the dumpfile.  If
>> the milter crashes while writing the dumpfile we're still screwed.
>
> You are not: we dump in a new file and rename the new file into the old
> one. That means that unless your OS plays the dangerous game of flushing
> metadata asynchronously, you are safe whatever happens. On reasonable
> systems, rename(2) says something like:
>
> "rename() guarantees that an instance of to will always exist, even if
> the system should crash in the middle of the operation."
>
> When I tried to add DBD support, I discovered I could not have such a
> garantee anymore, and this is why I gave it up.

So then the solution is simple.  Keep the same dumpfile we have now as an 
option.

-Dan

--

"It's three o'clock in the morning.  It's too late for 'oops'.  After
Locate Updates, don't even go there."

-Paul Baecker
  January 3, 2k
  Indeed, sometime after 3AM

--------Dan Mahoney--------
Techie,  Sysadmin,  WebGeek
Gushi on efnet/undernet IRC
ICQ: 13735144   AIM: LarpGM
Site:  http://www.gushi.org
---------------------------

manu@... wrote:
 > Dan Mahoney, System Admin wrote:
 > 
 > > Did the BDB version have the option to periodically (say, hourly) dump a
 > > checkpoint file or something similar?
 > 
 > The problem was that it was impossible to be sure no corruption could
 > occur on the DB file if milter-greylist crashed at a wrong time.

That's why a transactional database should be used.  It
guarantees consistency, no matter if and when the client
application crashes.

Personally I would prefer PostgreSQL which supports
transactions for ages and is rock-stable, but recent
versions of mysql also support transactions in theory
for some of its db formats.

By the way, dumping a "checkpoint" file wouldn't be a
problem at all with a reasonable database.  You would
just make an SQL export once per hour.  The client
application (milter-greylist) would not have to care
about it at all; instead a simple cron job will do.

As a nice side effect, the synchronization protocol
between multiple MX mail servers would not be needed
anymore.  They would simply access the same DB, so they
have the same data.

The processing overhead (CPU time) of the SQL queries
should be negligible, not more than the overhead of a
DNS query.

Best regards
   Oliver

-- 
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606,  Gesch\ufffdftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M\ufffdn-
chen, HRB 125758,  Gesch\ufffdftsf\ufffdhrer: Maik Bachmann, Olaf Erb, Ralf Gebhart

FreeBSD-Dienstleistungen, -Produkte und mehr:  http://www.secnetix.de/bsd

"It combines all the worst aspects of C and Lisp:  a billion different
sublanguages in one monolithic executable.  It combines the power of C
with the readability of PostScript."
        -- Jamie Zawinski, when asked: "What's wrong with perl?"

At Thu, 14 Jun 2007 23:05:30 +0200,
manu@... wrote:
> 
> Dan Mahoney, System Admin <danm@...> wrote:
> 
> > Did the BDB version have the option to periodically (say, hourly) dump a
> > checkpoint file or something similar?
> 
> The problem was that it was impossible to be sure no corruption could
> occur on the DB file if milter-greylist crashed at a wrong time.

Berkeley DB 4.x can guarantee its consistency after system failures
by journaling mechanism just as RDBMSes do. See:

http://www.oracle.com/technology/documentation/berkeley-db/db/gsg_txn/C/introduction.html

Oliver Fromme <olli@...> wrote:

> Personally I would prefer PostgreSQL which supports
> transactions for ages and is rock-stable, but recent
> versions of mysql also support transactions in theory
> for some of its db formats.

The first step in that direction is to create a clean API between the
main logic and the database back-end. Once you access storage through a
well defined set of methods, it's quite easy to implement multiple
back-ends.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...