Milter-greylist

Hi

the following problem ocured while testing milter-greylist. We stopt 
milter-greylist with SIGTERM to add new acls. After restart we lost 
sereval whitelist and greylist entrys because milter-greylist did not 
dump the greylist.db on SIGTERM.

1. is it a known bug?
2. is there a patch or is't planed that milter-greylist will dump the DB 
on SIGTERM?
3. is there a way to reload the conf without loosing the DB in memory


cu

	Michael Menge



-- 
--------------------------------------------------------------------------------
M.Menge                                 Tel.: (49) 7071/29-70316
Universitaet Tuebingen                  Fax.: (49) 7071/29-5912
Zentrum fuer Datenverarbeitung          mail: menge@...-tuebingen.de
Waechterstrasse 76
72074 Tuebingen

On Mon, Dec 19, 2005 at 03:09:57PM +0100, Michael Menge wrote:
> 3. is there a way to reload the conf without loosing the DB in memory

Sure: just let it reload its configuration automatically. No need to 
send signals, milter-greylist monitors the config file changes.

-- 
Emmanuel Dreyfus
manu@...

On Mon, 2005-12-19 at 14:52 +0000, Emmanuel Dreyfus wrote:
> On Mon, Dec 19, 2005 at 03:09:57PM +0100, Michael Menge wrote:
> > 3. is there a way to reload the conf without loosing the DB in memory
> 
> Sure: just let it reload its configuration automatically. No need to 
> send signals, milter-greylist monitors the config file changes.
> 
So,

How do we make to have mg to dump the memory DB?


Raul Dias

Raul Dias wrote on Mon, 19 Dec 2005 14:53:29 -0200:

> How do we make to have mg to dump the memory DB?

I think if you stop it it will do this.

Kai

-- 
Kai Sch\ufffdtzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

Raul Dias <raul@...> wrote:

> How do we make to have mg to dump the memory DB?

It does it on its own every 10 minutes. It's true it could be useful to
have a way of dumping on purpose.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

On Mon, 19 Dec 2005, manu@... wrote:
> Raul Dias <raul@...> wrote:
>> How do we make to have mg to dump the memory DB?
> It does it on its own every 10 minutes. It's true it could be useful to
> have a way of dumping on purpose.

It's another argument to have the db in sql. Then dumps wouldn't be 
required.

-Dan

On Mon, Dec 19, 2005 at 12:40:33PM -0800, goemon@... wrote:
> > It does it on its own every 10 minutes. It's true it could be useful to
> > have a way of dumping on purpose.
> It's another argument to have the db in sql. Then dumps wouldn't be 
> required.

Moving to multiple back-ends support have already been discussed, 
and contributions are welcome. But it seems nobody is unhappy enough 
with the current implementation to move forward...

-- 
Emmanuel Dreyfus
manu@...

On Mon, 2005-12-19 at 21:12 +0100, manu@... wrote:
> Raul Dias <raul@...> wrote:
> 
> > How do we make to have mg to dump the memory DB?
> 
> It does it on its own every 10 minutes. It's true it could be useful to
> have a way of dumping on purpose.
> 

Yep, specially if you need to stop it (e.g. restart host, upgrade,
testing).

Raul Dias

On Mon, 19 Dec 2005, Emmanuel Dreyfus wrote:
> On Mon, Dec 19, 2005 at 12:40:33PM -0800, goemon@... wrote:
>>> It does it on its own every 10 minutes. It's true it could be useful to
>>> have a way of dumping on purpose.
>> It's another argument to have the db in sql. Then dumps wouldn't be
>> required.
> Moving to multiple back-ends support have already been discussed,
> and contributions are welcome. But it seems nobody is unhappy enough
> with the current implementation to move forward...

I haven't time now to write the sql backend. But it's definitely needed 
since milter-greylist keeps failing on me, losing its whitelist settings 
from greylist.conf

-Dan

<goemon@...> wrote:

> I haven't time now to write the sql backend. But it's definitely needed
> since milter-greylist keeps failing on me, losing its whitelist settings
> from greylist.conf

That's a real bug which deserves to be fixed without adding a SQL
backend. Unfortunately, you are the only one wh reported it, and I have
no idea where the problem could come from.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Raul Dias wrote:
> On Mon, 2005-12-19 at 21:12 +0100, manu@... wrote:
>>Raul Dias <raul@...> wrote:
>>
>>>How do we make to have mg to dump the memory DB?
>>It does it on its own every 10 minutes. It's true it could be useful to
>>have a way of dumping on purpose.
>>
> 
> Yep, specially if you need to stop it (e.g. restart host, upgrade,
> testing).
> 
> Raul Dias
> 
> 
I havent done any signal handling in my own programs but there are some 
lines in the code which make me think milter-greylist should dump it's db.

--------- milter-greylist.c --------------
.
.
.
int
main(argc, argv)
.
.
.
	/*
          * Install an atexit() callback to perform
          * a dump when milter-greylist exits.
          */
         if (atexit(*final_dump) != 0) {
                 syslog(LOG_ERR, "atexit() failed: %s", strerror(errno));
                 exit(EX_OSERR);
         }
.
.
.

--------------------------------------------

I did not see the errormessage so milter-greylist should have Installed 
the atexit funktion.

After SIGTERM the program should execute exit() by default, but there 
where no syslog entries form final_dump so it did not get called.





-- 
--------------------------------------------------------------------------------
M.Menge                                 Tel.: (49) 7071/29-70316
Universitaet Tuebingen                  Fax.: (49) 7071/29-5912
Zentrum fuer Datenverarbeitung          mail: menge@...-tuebingen.de
Waechterstrasse 76
72074 Tuebingen

On Tue, Dec 20, 2005 at 10:36:12AM +0100, Michael Menge wrote:
> I havent done any signal handling in my own programs but there are some 
> lines in the code which make me think milter-greylist should dump it's db.

signal handling is forbidden by the milter API. We have a callback on exit,
but AFAIK that does not work if you get a deadly signal.

> After SIGTERM the program should execute exit() by default, but there 
> where no syslog entries form final_dump so it did not get called.

On a fatal signal, the kernel exit the program, by calling the exit system
call in the kernel, but as the libc exit function in userland is not called, 
the exit hook cannot be run. At least that's how I understand the thing.

If we want to remote control milter-greylist dumping, we need to have
a socket control. The easiest way is to reuse the MX sync protocol and 
add a dump command. That would be easy.

-- 
Emmanuel Dreyfus
manu@...

Emmanuel Dreyfus wrote:
> On Tue, Dec 20, 2005 at 10:36:12AM +0100, Michael Menge wrote:
>>I havent done any signal handling in my own programs but there are some 
>>lines in the code which make me think milter-greylist should dump it's db.
> 
> signal handling is forbidden by the milter API. We have a callback on exit,
> but AFAIK that does not work if you get a deadly signal.

Is there a SIGNAL which is not "deadly" and will call the atexit callback?

> 
>>After SIGTERM the program should execute exit() by default, but there 
>>where no syslog entries form final_dump so it did not get called.
> 
> On a fatal signal, the kernel exit the program, by calling the exit system
> call in the kernel, but as the libc exit function in userland is not called, 
> the exit hook cannot be run. At least that's how I understand the thing.
> 
> If we want to remote control milter-greylist dumping, we need to have
> a socket control. The easiest way is to reuse the MX sync protocol and 
> add a dump command. That would be easy.
> 

This may be a work around, but you need a telnet connection to tell 
milter-greylist to dump his db
and then you have to kill the milter with SIGTERM. After it has written 
  his db and before it has got new entries in his db.

With about 180.000 eMails per Day like oure site this may be a problem.
If you add a dump and exit funktion to MX sync there could be some 
security problems.


-- 
--------------------------------------------------------------------------------
M.Menge                                 Tel.: (49) 7071/29-70316
Universitaet Tuebingen                  Fax.: (49) 7071/29-5912
Zentrum fuer Datenverarbeitung          mail: menge@...-tuebingen.de
Waechterstrasse 76
72074 Tuebingen

On Tue, Dec 20, 2005 at 11:44:09AM +0100, Michael Menge wrote:
> >signal handling is forbidden by the milter API. We have a callback on exit,
> >but AFAIK that does not work if you get a deadly signal.
> Is there a SIGNAL which is not "deadly" and will call the atexit callback?

It does not change anything is the signal is not fatal: we cannot catch 
it anyway. The only way to solve that problem is to find a signal that
is catched by libmilter and where the signal handler calls exit().

> >If we want to remote control milter-greylist dumping, we need to have
> >a socket control. The easiest way is to reuse the MX sync protocol and 
> >add a dump command. That would be easy.
> This may be a work around, but you need a telnet connection to tell 
> milter-greylist to dump his db

Not nescessarily, we could have a command line tool that does just that.
something such as:
mgctl 127.0.0.1 dump

> With about 180.000 eMails per Day like oure site this may be a problem.
> If you add a dump and exit funktion to MX sync there could be some 
> security problems.

Oh, you mean a rogue client requesting continous dumps? Then it's time to
use RSA keys...

-- 
Emmanuel Dreyfus
manu@...

On Tue, 20 Dec 2005, manu@... wrote:
> <goemon@...> wrote:
>> I haven't time now to write the sql backend. But it's definitely needed
>> since milter-greylist keeps failing on me, losing its whitelist settings
>> from greylist.conf
> That's a real bug which deserves to be fixed without adding a SQL
> backend. Unfortunately, you are the only one wh reported it, and I have
> no idea where the problem could come from.

Any ideas to debug it?

-Dan

On Tue, Dec 20, 2005 at 03:43:23PM -0800, goemon@... wrote:
> > That's a real bug which deserves to be fixed without adding a SQL
> > backend. Unfortunately, you are the only one wh reported it, and I have
> > no idea where the problem could come from.
> 
> Any ideas to debug it?

I guess the first stop would be to track down the exact situation that
triggers the problem. (not easy, I know)

-- 
Emmanuel Dreyfus
manu@...

Emmanuel Dreyfus wrote:
> On Tue, Dec 20, 2005 at 11:44:09AM +0100, Michael Menge wrote:
>>>signal handling is forbidden by the milter API. We have a callback on exit,
>>>but AFAIK that does not work if you get a deadly signal.
>>Is there a SIGNAL which is not "deadly" and will call the atexit callback?

I havent found any hint that signal handling is forbidden by the milter 
API, AFAIK they say only that lib milter takes care of signal handling

> 
> It does not change anything is the signal is not fatal: we cannot catch 
> it anyway. The only way to solve that problem is to find a signal that
> is catched by libmilter and where the signal handler calls exit().
> 

In the Doc of libmilter they mention only Stop and Abort. If Abort is 
cought xxfi_abort callback is invoked.

>>>If we want to remote control milter-greylist dumping, we need to have
>>>a socket control. The easiest way is to reuse the MX sync protocol and 
>>>add a dump command. That would be easy.
>>This may be a work around, but you need a telnet connection to tell 
>>milter-greylist to dump his db
> 
> Not nescessarily, we could have a command line tool that does just that.
> something such as:
> mgctl 127.0.0.1 dump
> 
>>With about 180.000 eMails per Day like oure site this may be a problem.
>>If you add a dump and exit funktion to MX sync there could be some 
>>security problems.
> 
> Oh, you mean a rogue client requesting continous dumps? Then it's time to
> use RSA keys...
> 

The problem is that at the momen you can only add and remove db entries 
with the MX sync protocol. Even if someone wants to use this they cannot 
do mutch harm.

But if cou can do more things with the MX sync protocol you need 
authentifikation and authorisation. At the moment every user on one of 
the MX server could dump the db ore other features added to the MX sync 
protocol. With IP-Spoothing even more user could do this.

RSA Keys may e one way to solve the problem but i would think thinks 
will get more complicatet with RSA. Therefore i would prefere 
signalhandling, callbacks ore other not remote usable way



-- 
--------------------------------------------------------------------------------
M.Menge                                 Tel.: (49) 7071/29-70316
Universitaet Tuebingen                  Fax.: (49) 7071/29-5912
Zentrum fuer Datenverarbeitung          mail: menge@...-tuebingen.de
Waechterstrasse 76
72074 Tuebingen

On Wed, Dec 21, 2005 at 08:47:26AM +0100, Michael Menge wrote:
> >>>signal handling is forbidden by the milter API. We have a callback on 
> >>>exit,
(snip)
> I havent found any hint that signal handling is forbidden by the milter 
> API, AFAIK they say only that lib milter takes care of signal handling

Which means the milter does not.

> In the Doc of libmilter they mention only Stop and Abort. If Abort is 
> cought xxfi_abort callback is invoked.

It's invoked for each message that is aborted, and if the milter gets
SIGABRT, all messages will be aborted. 

It could make sense to count the number of messages been handled, and 
if the count reach zero in xxfi_abort, make a dump.

> The problem is that at the momen you can only add and remove db entries 
> with the MX sync protocol. Even if someone wants to use this they cannot 
> do mutch harm.

Yes, this is why I did not add any authentication mecanism: an attacker
cannot do anything through MX sync it could not do by simply sending 
an e-mail.

-- 
Emmanuel Dreyfus
manu@...