Milter-greylist

I had noticed that milter-sender has been withdrawn from the FBSD ports a
few days ago.

In the archives, also noted that the callback really needed to depend
heavily on its additional greylisting function.

In reading more, it becomes apparent that the lack of response to the
callback could be many reasons unrelated to spammers/junkers. Callback
sounds like a good idea at first blush.

Also, it was apparently having troubles with corrupted Berkely DB 3+.

Guess I'm glad I never could get it working.....


Happy trails,
Jack L. Stone

System Admin
Sage-american

Jack L. Stone <jacks@...> wrote:

> In reading more, it becomes apparent that the lack of response to the
> callback could be many reasons unrelated to spammers/junkers. Callback
> sounds like a good idea at first blush.

Greylisting can cause callback to tempfail, but the callback scheme
still have some benefits. When you get a permanent failure, I beleive
you have a very strong hint that the sender address is invalid (and thus
that the message should be rejectec)
 
> Also, it was apparently having troubles with corrupted Berkely DB 3+.

I spent a lot of time trying to move milter-greylist database to
Berkeley DB, but I finnally gave up. The only goal of the database dump
is to recover after a crash, and I could not get any garantee of having
the database not corrupted after a crash.

-- 
Emmanuel Dreyfus
Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
le binaire et ceux qui ne le comprennent pas.
manu@...

> I spent a lot of time trying to move milter-greylist database to
> Berkeley DB, but I finnally gave up. The only goal of the database 
dump
> is to recover after a crash, and I could not get any garantee of 
having
> the database not corrupted after a crash.

May be need to use database with transactions? It's help sometimes :)

Do you still want to test my milter-verify under heavily mail traffic?

>Callback
> sounds like a good idea at first blush.

It sounds good always :)
It's a really useful...

> 
> Also, it was apparently having troubles with corrupted Berkely DB 
3+.

I have not get a corrupted Berkley DB 3+ yet with my milter... May be 
later ;) If so, i will be switch to QDBM - it's rule :)

Kind regards,
Eugene

At 08:35 AM 12.7.2004 -0000, eugene_kurmanin wrote:
>
>
>Do you still want to test my milter-verify under heavily mail traffic?
>
....be glad to. But, does the DB 3+ have to be recompiled into sendmail?
If so, I'd have to try it on a test box first.

>>Callback
>> sounds like a good idea at first blush.
>
>It sounds good always :)
>It's a really useful...
>
>> 
>> Also, it was apparently having troubles with corrupted Berkely DB 
>3+.
>
>I have not get a corrupted Berkley DB 3+ yet with my milter... May be 
>later ;) If so, i will be switch to QDBM - it's rule :)
>


>Kind regards,
>Eugene
>
>
>
>
>
>
>
> 
>Yahoo! Groups Links
>
>
>
> 
>
>
>
>


Happy trails,
Jack L. Stone

System Admin
Sage-american

> ....be glad to. But, does the DB 3+ have to be recompiled into 
sendmail?

No. No need to recompile anything.
Do you add users to passwd & aliases only? Or sendmail box just relay 
to another box?

Kind regards,
Eugene

At 03:52 PM 12.7.2004 -0000, eugene_kurmanin wrote:
>
>
>> ....be glad to. But, does the DB 3+ have to be recompiled into 
>sendmail?
>
>No. No need to recompile anything.
>Do you add users to passwd & aliases only?

....yes, this & virtusertable because of numerous vhosts -- then, mail
comes/goes from/to this box through the gateway box.

Or sendmail box just relay 
>to another box?
>

....no

>Kind regards,
>Eugene
>
>
>
>
>
>
>
> 
>Yahoo! Groups Links
>
>
>
> 
>
>
>
>


Happy trails,
Jack L. Stone

System Admin
Sage-american

eugene_kurmanin <ubr@...> wrote:

> > I spent a lot of time trying to move milter-greylist database to
> > Berkeley DB, but I finnally gave up. The only goal of the database 
> dump
> > is to recover after a crash, and I could not get any garantee of 
> having
> > the database not corrupted after a crash.
> 
> May be need to use database with transactions? It's help sometimes :)

Another goal is to preserve the lightweight approach.

In order to scale for much bigger setups, I thought about having
different buckets in the database and dump them to disk in a round robin
fashion. 

-- 
Emmanuel Dreyfus
Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
le binaire et ceux qui ne le comprennent pas.
manu@...

Jack L. Stone <jacks@...> wrote:

> >Do you still want to test my milter-verify under heavily mail traffic?
> >
> ....be glad to. But, does the DB 3+ have to be recompiled into sendmail?
> If so, I'd have to try it on a test box first.

milters operate outsied of sendmail, what DB is inside sendmail does not
really matter.

-- 
Emmanuel Dreyfus
Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
le binaire et ceux qui ne le comprennent pas.
manu@...

At 09:01 PM 12.7.2004 +0100, manu@... wrote:
>
>Jack L. Stone <jacks@...> wrote:
>
>> >Do you still want to test my milter-verify under heavily mail traffic?
>> >
>> ....be glad to. But, does the DB 3+ have to be recompiled into sendmail?
>> If so, I'd have to try it on a test box first.
>
>milters operate outsied of sendmail, what DB is inside sendmail does not
>really matter.
>

Perhaps this statement on the milter-sender developer's website had me
confused:

"...In order to support B/W lists, unknown recipient checks, and a
preserved cache, milter-sender requires Berkeley DB 3 or better. If you do
not require support for Sendmail's access or aliases databases nor a cache,
skip this step ...Note that Sendmail will probably have to be rebuilt to
use Berkeley DB, especially if the library was never installed and/or
Sendmail was built against an older version of Berkeley DB...."

Am I reading that wrong?


Happy trails,
Jack L. Stone

System Admin
Sage-american

--- In milter-greylist@yahoogroups.com, manu@n... wrote:
> eugene_kurmanin <ubr@o...> wrote:
> 
> > > I spent a lot of time trying to move milter-greylist database to
> > > Berkeley DB, but I finnally gave up. The only goal of the database 
> > dump
> > > is to recover after a crash, and I could not get any garantee of 
> > having
> > > the database not corrupted after a crash.
> > 
> > May be need to use database with transactions? It's help sometimes :)
> 
> Another goal is to preserve the lightweight approach.
> 
> In order to scale for much bigger setups, I thought about having
> different buckets in the database and dump them to disk in a round robin
> fashion. 


I have one idea also to implement,  related to berkeley. 

A whitelist database, the milter will only read the database like
sendmail does on access db, get the IP (x1.x2.x3.x4) try to lookup
x1.x2.x3.x4, x1.x2.x3, x1.x2, x1 and when found an OK. Accept the
connection as whitelisted. This make possible to add big databases to
be used with the milter without consuming a lot of memory, and does
not have the read/write problem as they are the administrator
whitelist (I have a friend provider wich has an 100000 IPs whitelist,
which  he made using another milter solution using mysql, and I want
to use his list in my implementation, but the current form with text
file loading in memory isn't a good solution to be used with soo large
lists)

This database can be used also for the config per user/per domain as
specified in site TODO, with keys like :

greylist:domain -> domain greylist override the greylist.conf
greylist:user@domain -> user greylist, override domain and greylist.conf

The same can be done for autowhite and timeout. I'm not an expert in
berkely, but I have used it sometimes, and its very easy to implement
to read only data with fast results, and low memory usage.

ivan_fm <ml@...> wrote:

> I have one idea also to implement,  related to berkeley. 
> 
> A whitelist database, the milter will only read the database like
> sendmail does on access db, get the IP (x1.x2.x3.x4) try to lookup
> x1.x2.x3.x4, x1.x2.x3, x1.x2, x1 and when found an OK.

That's not really a bekreley DB related problem, you can do that with
any backend, can't you?

-- 
Emmanuel Dreyfus
Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
le binaire et ceux qui ne le comprennent pas.
manu@...

> The same can be done for autowhite and timeout. I'm not an expert in
> berkely, but I have used it sometimes, and its very easy to 
implement
> to read only data with fast results, and low memory usage.

QDBM much faster than Berkley DB & and more useful in huge volume 
database.

On Fri, 10 Dec 2004 07:59:24 +0100
manu@... wrote:

MO> 
MO> ivan_fm <ml@...> wrote:
MO> 
MO> > I have one idea also to implement,  related to berkeley. 
MO> > 
MO> > A whitelist database, the milter will only read the database like
MO> > sendmail does on access db, get the IP (x1.x2.x3.x4) try to lookup
MO> > x1.x2.x3.x4, x1.x2.x3, x1.x2, x1 and when found an OK.
MO> 
MO> That's not really a bekreley DB related problem, you can do that with
MO> any backend, can't you?
MO> 

Yes, I'm just thinking about berkeley because as it is used by sendmail, any machine where milter-greylist will be installed will have the 
berkeley libs.



-- 
Ivan F. Martinez

Ivan F. Martinez <ml@...> wrote:

> MO> That's not really a bekreley DB related problem, you can do that with
> MO> any backend, can't you?
>
> Yes, I'm just thinking about berkeley because as it is used by sendmail,
> any machine where milter-greylist will be installed will have the berkeley
> libs.

I'm not sure milter-greylist should use Berkeley DB.

Sendmail's databases are mostly read-only, and generated from text
files. If you crash, it's not a problem: in the worst scenario you
crashed during DB regeneration, and you just have to rebuild it.

milter-greylist is continously modifying its database. If you crash you
can loose the database. The current code is trying hard to avoid
database corruption on crashes, and moving to Berkeley DB will probably
be a step backward on this front.

-- 
Emmanuel Dreyfus
Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
le binaire et ceux qui ne le comprennent pas.
manu@...

On Fri, 10 Dec 2004 22:48:13 +0100
manu@... wrote:

MO> 
MO> Ivan F. Martinez <ml@...> wrote:
MO> 
MO> > MO> That's not really a bekreley DB related problem, you can do
MO> > MO> that with any backend, can't you?
MO> >
MO> > Yes, I'm just thinking about berkeley because as it is used by
MO> > sendmail, any machine where milter-greylist will be installed will
MO> > have the berkeley libs.
MO> 
MO> I'm not sure milter-greylist should use Berkeley DB.
MO> 
MO> Sendmail's databases are mostly read-only, and generated from text
MO> files. If you crash, it's not a problem: in the worst scenario you
MO> crashed during DB regeneration, and you just have to rebuild it.
MO> 
MO> milter-greylist is continously modifying its database. If you crash
MO> you can loose the database. The current code is trying hard to avoid
MO> database corruption on crashes, and moving to Berkeley DB will
MO> probably be a step backward on this front.


My idea is to use Berkeley on the big whitelist, not for dynamic part.
I'm testing a patch to access the db from sendmail and passing data to
milter.

The patch is available :		
http://www.saisp.br/ifm/patches/milter-greylist.c.patch

Also a m4 file to configure milter-greylist, and a sample Rule to get
data from access.db
http://www.saisp.br/ifm/patches/milter-greylist.m4

I'm accepting suggestions to make it better.


--

Ivan F. Martinez <ml@...> wrote:

> My idea is to use Berkeley on the big whitelist, not for dynamic part.
> I'm testing a patch to access the db from sendmail and passing data to
> milter.

Oh, ok, that seems acceptable.
 
> The patch is available :              
> http://www.saisp.br/ifm/patches/milter-greylist.c.patch
> 
> Also a m4 file to configure milter-greylist, and a sample Rule to get
> data from access.db
> http://www.saisp.br/ifm/patches/milter-greylist.m4
> 
> I'm accepting suggestions to make it better.

FWIW, Ivan sent me the patch behind the scene, and I'd like some
feedback from other contributors. Especially from Remy Card for ACL
interraction.

I wonder if there woulnd't be some benefit to expcitely pull various
whitelist/greylist methods from the ACL. Something like this:

acl whitelist sendmaildb 
acl greylist default

That would enable mixing greylist.conf ACL with sendmail DB. The same
trick could be used for SPF. I wonder if it would make sense to do the
same for SMTP auth or authowhitelist, as those would always tend to be
thefirst items in the ACL.

Opinions? Don't hesitate to tell me that it's useless and too
complicated :-)

-- 
Emmanuel Dreyfus
Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
le binaire et ceux qui ne le comprennent pas.
manu@...

On Sat, Dec 11, 2004 at 10:19:14AM +0100, manu@... wrote:
> Ivan F. Martinez <ml@...> wrote:
> 
> > The patch is available :              
> > http://www.saisp.br/ifm/patches/milter-greylist.c.patch
> > 
> > Also a m4 file to configure milter-greylist, and a sample Rule to get
> > data from access.db
> > http://www.saisp.br/ifm/patches/milter-greylist.m4
> > 
> > I'm accepting suggestions to make it better.
> 
> FWIW, Ivan sent me the patch behind the scene, and I'd like some
> feedback from other contributors. Especially from Remy Card for ACL
> interraction.
> 
> I wonder if there woulnd't be some benefit to expcitely pull various
> whitelist/greylist methods from the ACL. Something like this:
> 
> acl whitelist sendmaildb 
> acl greylist default

	This can easily be done but this requires some changes in the ACL 
API (ctx has to be added as a parameter to acl_filter() to enable the use
of sendmail data in this function).

> That would enable mixing greylist.conf ACL with sendmail DB. The same
> trick could be used for SPF. I wonder if it would make sense to do the
> same for SMTP auth or authowhitelist, as those would always tend to be
> thefirst items in the ACL.
> 
> Opinions? Don't hesitate to tell me that it's useless and too
> complicated :-)

	Well, this is certainly not too complicated.  Sendmail DB check, SPF
and SMTP auth can also be moved in the ACL code.  I do not know about
autowhitelist since this feature should not be optional, IMHO.

	BTW, I think that the new ACL scheme should be tested a bit more
before we move existing tests in it.  Any comments from testers?  Is anyone
using the new ACL feature?  Is it working in environments different from
mine?

		R\ufffdmy