Milter-greylist

Has anyone started working on LDAP support?  I need something fairly 
quickly, so if not, I can take care of that.

On Wed, Nov 09, 2005 at 09:39:32AM -0600, Ken Serrine wrote:
> Has anyone started working on LDAP support?  I need something fairly 
> quickly, so if not, I can take care of that.

Nobody works on it. What are you looking for? Storing user-specific 
configuration in LDAP?

Could you discuss here how you will do it?

-- 
Emmanuel Dreyfus
manu@...

On Wed, Nov 09, 2005 at 09:39:32AM -0600, Ken Serrine wrote:

>>Has anyone started working on LDAP support?  I need something fairly 
>>quickly, so if not, I can take care of that.
>>    
>>
>
>Nobody works on it. What are you looking for? Storing user-specific 
>configuration in LDAP?
>
>Could you discuss here how you will do it?
>  
>
Yes.  First and foremost, I need to be able to alter the ACL without 
having to restart (altering it in the greylist.conf causes the milter to 
crash on my system.)  Instead of trying to fix that problem, I'd rather 
use LDAP because I have three servers that all need to be in sync.  
Then, eventually, I will allow my users to administer themselves; ie, 
whitelisting, not being on the greylist, etc., which means I have to use 
LDAP, or some other backend like SQL.  Since I already use LDAP for all 
my mail routing and whitelisting controls, it is the natural fit.

On Wed, 9 Nov 2005, Ken Serrine wrote:
> Has anyone started working on LDAP support?  I need something fairly
> quickly, so if not, I can take care of that.

afaik nobody is doing ldap.

-Dan

On Wed, Nov 09, 2005 at 10:53:37AM -0600, Ken Serrine wrote:
> Then, eventually, I will allow my users to administer themselves; ie, 
> whitelisting, not being on the greylist, etc., which means I have to use 
> LDAP, or some other backend like SQL.  Since I already use LDAP for all 
> my mail routing and whitelisting controls, it is the natural fit.

Do we want the complete ACL to be pulled from an external back-end? Or
do we want to mix file-based ACL and an ACL from an external source?

How would the ACL be stored in LDAP?

-- 
Emmanuel Dreyfus
manu@...

Emmanuel Dreyfus wrote:

>On Wed, Nov 09, 2005 at 10:53:37AM -0600, Ken Serrine wrote:
>  
>
>>Then, eventually, I will allow my users to administer themselves; ie, 
>>whitelisting, not being on the greylist, etc., which means I have to use 
>>LDAP, or some other backend like SQL.  Since I already use LDAP for all 
>>my mail routing and whitelisting controls, it is the natural fit.
>>    
>>
>
>Do we want the complete ACL to be pulled from an external back-end? Or
>do we want to mix file-based ACL and an ACL from an external source?
>
>How would the ACL be stored in LDAP?
>  
>
Ken wrote:
I need to get some time to study the code, but my initial thoughts are 
to have a mix.  So, either or both would be an option.
For the ACL storage, I'm sure it's best to be flexible.  Some may want a 
separate branch, but others may want to just add attributes to existing 
branches.
So, there should probably be additional config options to specify 
exactly what to look up in ldap.
My immediate goals are to let the users choose to be greylisted or not.
Of course, to be useful, the LDAP features would have to be much more 
flexible.

An example for "acl whitelist rcpt nobody@..." would be to add 
the following to the conf file:

LDAPServers      ldap1.abc.example.net   ldap2.abc.example.net   
ldap3.abc.example.net
LDAPBase          ou=greyacl,dc=example,dc=net
LDAPACLKey   mail
LDAPACLAction  action
LDAPACLType     type

so, for example, the equivalent command line query to the greylist 
milter query would be:
 ldapsearch -b "ou=greyacl,dc=example,dc=net" 
"(mail=nobody@...)" action type
The return values would be "action=whitelist" and "type=rcpt", or if the 
recipient wants to be greylisted, then either no record would be found 
or "action=greylist" would be returned.

To be flexible, this would all be more complicated, of course.  We would 
want to minimize the number of LDAP lookups, I think.  So, to cover all 
combinations of ACLs, I think we'd need options to determine if we just 
wanted "rcpt", or "domain", etc. 
In my case, if I only cared about "rcpt", then I wouldn't want the 
milter doing lookups for "domain", "from", etc.
In other words, I wouldn't want all the lookups that are currently 
happening in memory to have to occur across the network to the LDAP 
servers.  If the LDAP server was running on the same box as sendmail, it 
may not be as bad, but mine aren't.  And, I already have over 800 
million hits per day, so any unnecessary lookups would not be appreciated.

On Fri, Nov 11, 2005 at 02:47:27PM -0600, Ken Serrine wrote:
> To be flexible, this would all be more complicated, of course.  We would 
> want to minimize the number of LDAP lookups, I think.  So, to cover all 
> combinations of ACLs, I think we'd need options to determine if we just 
> wanted "rcpt", or "domain", etc. 
> In my case, if I only cared about "rcpt", then I wouldn't want the 
> milter doing lookups for "domain", "from", etc.

It seems a pity to left the ability for users to greylist everything but 
some DNS domains, IP block or sender combinaison. I can understand you
don't want to implement it now because it is of no interest for you yet,
but it would be good to decide of a LDAP directory structure that will
fit the complicated setup later.

I'm not knwoledgable in LDAP, so I cannot be of a lot of help. Is it
possible to query LDAP to retreive a whole branch of the tree? If it is
then we could fetch a set of ACL stored for a recipient.

-- 
Emmanuel Dreyfus
manu@...

On Sat, 12 Nov 2005, Emmanuel Dreyfus wrote:
> On Fri, Nov 11, 2005 at 02:47:27PM -0600, Ken Serrine wrote:
>> To be flexible, this would all be more complicated, of course.  We would
>> want to minimize the number of LDAP lookups, I think.  So, to cover all
>> combinations of ACLs, I think we'd need options to determine if we just
>> wanted "rcpt", or "domain", etc.
>> In my case, if I only cared about "rcpt", then I wouldn't want the
>> milter doing lookups for "domain", "from", etc.
> It seems a pity to left the ability for users to greylist everything but
> some DNS domains, IP block or sender combinaison. I can understand you
> don't want to implement it now because it is of no interest for you yet,
> but it would be good to decide of a LDAP directory structure that will
> fit the complicated setup later.
> I'm not knwoledgable in LDAP, so I cannot be of a lot of help. Is it
> possible to query LDAP to retreive a whole branch of the tree? If it is
> then we could fetch a set of ACL stored for a recipient.

I think something more flexible would be to integrate lua support, so 
people can easily script support for mysql/ldap/abcxyz, with their own 
special site-specific rulesets. They can filter rcpt or domain or helo 
or any complex combination they like.

-Dan

On Sat, Nov 12, 2005 at 01:52:45AM -0800, Dan Hollis wrote:
> I think something more flexible would be to integrate lua support, so 
> people can easily script support for mysql/ldap/abcxyz, with their own 
> special site-specific rulesets. They can filter rcpt or domain or helo 
> or any complex combination they like.

Never heard about this LUA thing. Can you tell more about it?

-- 
Emmanuel Dreyfus
manu@...

On Sat, 12 Nov 2005, Emmanuel Dreyfus wrote:
> On Sat, Nov 12, 2005 at 01:52:45AM -0800, Dan Hollis wrote:
>> I think something more flexible would be to integrate lua support, so
>> people can easily script support for mysql/ldap/abcxyz, with their own
>> special site-specific rulesets. They can filter rcpt or domain or helo
>> or any complex combination they like.
> Never heard about this LUA thing. Can you tell more about it?

http://www.lua.org/

It's a very simple scripting language but relatively powerful, easily 
sandboxed, fully supports threads, 100% pure ansi C, reasonable 
performance, and dead simple to integrate. Complete lua engine with all
standard support libraries compiled in only takes ~100kb.

-Dan

On Fri, Nov 11, 2005 at 02:47:27PM -0600, Ken Serrine wrote:

>To be flexible, this would all be more complicated, of course.  We would
>want to minimize the number of LDAP lookups, I think.  So, to cover all
>combinations of ACLs, I think we'd need options to determine if we just
>wanted "rcpt", or "domain", etc.
>In my case, if I only cared about "rcpt", then I wouldn't want the
>milter doing lookups for "domain", "from", etc.
>
On Sat, 12 Nov 2005, Emmanuel Dreyfus wrote:

>It seems a pity to left the ability for users to greylist everything but
>some DNS domains, IP block or sender combinaison. I can understand you
>don't want to implement it now because it is of no interest for you yet,
>but it would be good to decide of a LDAP directory structure that will
>fit the complicated setup later.
>I'm not knwoledgable in LDAP, so I cannot be of a lot of help. Is it
>possible to query LDAP to retreive a whole branch of the tree? If it is
>then we could fetch a set of ACL stored for a recipient.
>  
>
Ken wrote:

Yes, I agree having a structure in place up front would be the thing to do.
One wouldn't want to retrieve a whole branch because of the 
inefficiency, but more complex queries should be allowed.
In my example, if I was just interested in the recipient, I wouldn't 
want the LDAP servers or the code to do unecessary work.  That isn't to 
say that someone else wouldn't have that need.
So, my query may just be (rcpt=whatever@...), but others may 
need (&(addr=10.0.0.1)(from=pat@...)(rcpt=chris@...)).
But, if one mixes these two into the same directory, then we have to 
account for a whole new level of logic.
I will try to explain better by comparing to the current greylist.conf 
logic.
First, the greylist.conf depends on order.  So, the first match wins.  
There will be no order in LDAP.  Multiple results could be returned for 
a query, and although there is a sort ability in LDAP, I wouldn't think 
we would want ASCII or alphabetical order to determine priority.

So, using greylist.conf, there could be the following entries (not 
necessarily in this order):
acl whitelist domain example.net rcpt pat@...
acl greylist domain example.net
acl whitelist addr 10.0.0.2            # assume that's the IP for 
example.net

Using LDAP, we'd have to have the logic to form a complex query or the 
logic to perform multiple queries and also interpret the results.  It's 
not that this can't be done, but I believe it to be too inefficient for 
a busy site.
The single query would look something like this:
(|((&(addr=10.0.0.2)(type=whitelist))(|(domain=example.net)(type=greylist))(&(domain=example.net)(rcpt=pat@...)(type=whitelist))))
And, remember, the results are not in any specific order, so we can't 
just say "first one wins".  So, we need to decide based on the results 
which rule gets priority.
I'm just trying to illustrate one small example.  I haven't even 
considered all the possible scenerios, but to encompass the needs of all 
greylist users, one can start to see the complexity.

Looking up a user's ACL based on his/her address is not a problem and is 
not complex, but adding the ability to do "from", "domain", and "addr" 
in combination with this will get quite complicated.

One possibility is to let one specify the priority of each type and if 
he/she wants it enabled.  So, for example, one could specify the order as:
addr
domain
from
rcpt
and, then determine if one wants to do each type,
addr (on/off)
domain (on/off)
from (on/off)
rcpt (on/off)

In this example, there would be a maximum of 4 lookups:
if (addr==on) then do ldap query (addr=w.x.y.z);
if ldap match, then perform action;
else if (domain==on) then do ldap query (domain=example.net);
if ldap match, then perform action;
else if (from==on) then do ldap query (from=test@...);
if ldap match, then perform action;
else if (rcpt==on) then do ldap query (rcpt=test@...);
if ldap match, then perform action;

But, this still doesn't account for the cases of combinations such as
acl whitelist from pat@... rcpt pat@...

Dan Hollis wrote:

>I think something more flexible would be to integrate lua support, so 
>people can easily script support for mysql/ldap/abcxyz, with their own 
>special site-specific rulesets. They can filter rcpt or domain or helo 
>or any complex combination they like.
>
>-Dan
>
>
>  
>
Ken wrote:

Lua looks interesting.  Are you thinking of an architecture where we can 
make our own "plug-ins"?

On Mon, Nov 14, 2005 at 10:03:01AM -0600, Ken Serrine wrote:
> So, my query may just be (rcpt=whatever@...), but others may 
> need (&(addr=10.0.0.1)(from=pat@...)(rcpt=chris@...)).
> But, if one mixes these two into the same directory, then we have to 
> account for a whole new level of logic.

I'm not sure I fully understand the query we can send to a LDAP directory.
Would it make sense to have such a greylist.conf:

acl ldap rcpt user@...
acl greylist default

If the recipient matches a acl ldap rcpt line, we query the directory for 
(from, addr, domain) about user@.... We get whitelist or greylist 
(or blacklist if I ever finish implementing it), or no match. if we get
a match, we have the answer, if we don't, we move to next ACL line.

Does that make sense?

Then we could also have acl ldap domain, acl ldap addr and acl ldap from
clauses. It doesn't seem possible to me to mix multiple clauses on an 
acl ldap line.

-- 
Emmanuel Dreyfus
manu@...

On Mon, 14 Nov 2005, Ken Serrine wrote:
> Dan Hollis wrote:
>> I think something more flexible would be to integrate lua support, so
>> people can easily script support for mysql/ldap/abcxyz, with their own
>> special site-specific rulesets. They can filter rcpt or domain or helo
>> or any complex combination they like.
> Lua looks interesting.  Are you thinking of an architecture where we can
> make our own "plug-ins"?

Yes. It means that the ldap can all be done in a lua plugin, and can have 
as simple or as complex rules as you want. And it can be easily made 
site-specific rules without having to touch a single line of 
milter-greylist code.

-Dan