Milter-greylist

After some discussions behind the scene with Remy Card, I repost my last
proposal about an ACL mecanism for milter-greylist, so that he can reply
on this topic and move forward to another proposal. 

I insist on the point that I don't push the syntax presented in this
document rather than another one, but IMO another proposal with another
syntax should propose the equivalent functionnality.


===== My last unanswered proposal about ACL in milter-greylist =====

We would have a first match list of greylist/whitelist keywords, each
using an ACL using a last match basis. Here is a complete proposal, I'd
appreciate if people could comment:

First, a few examples...

# first config, Dan's setup:
acl grandma {
        rcpt grandma@...
        not from friend@...
        not from another.friend@...
}
greylist grandma
whitelist default



# Second config, Remy's setup:
acl foo {
        rcpt .*@...
        not rcpt postmaster@...
}
greylist foo 
whitelist default



# And finaly, Manu's setup:
acl user1 {
        rcpt user1@...
}
acl user2 (
        rcpt user2@...
}
acl user3 {
        rcpt user3@...
}
acl mywhitelist {
        addr 193.54.0.0/16
        domain friendly.com
}
whitelist mywhitelist
greylist user1
greylist user2
greylist user3
whitelist default




default is an internally defined ACL that match anything.

It seems to me that it matches all the needs that have been described
here so far. Now, let's try to be more formal:

For each {greylist|whitelist} keyword {
        for each entry in its ACL {
                if it's addr and the IP doesn't match
                        break out of the ACL loop
                if it's domain and the domain doesn't match
                        break out of  the ACL loop
                if it's from and the sender doesn't match
                        break out of  the ACL loop
                if it's rcpt and the recipient doesn't match
                        break out of  the ACL loop
        }
        if all entries in the ACL matched or if it was void {
                retain the {greylist|whitelist} decision 
                exit the keyword loop
        }
}



The last problem is to convert the older keyword to the new framework so
that we can retain backward compatibility in the config file.

"rcpt X" in test mode will be interpreted as
acl X {
        rcpt X
}
greylist X


"rcpt X" in "normal" mode will be interpreted as
acl X {
        rcpt X
}
whitelist X
 

"{addr|from|domain} X" turns into
acl X {
        {addr|from|domain} X
}
whitelist X


Test mode turns into adding "whitelist default" at the end of the config
Normal mode turns into adding "greylist default" at the end of the
config.


We would tag as obsolete all the old syntax, but we will keep it working
for the sake of backward compatibility (did I ever explained I hated
software that broke my config files? :-) 


-- 
Emmanuel Dreyfus
Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
le binaire et ceux qui ne le comprennent pas.
manu@...

Hi,

On Fri, Nov 12, 2004 at 02:58:52PM +0100, manu@... wrote:
> After some discussions behind the scene with Remy Card, I repost my last
> proposal about an ACL mecanism for milter-greylist, so that he can reply
> on this topic and move forward to another proposal. 

	Thanks, Manu!

> I insist on the point that I don't push the syntax presented in this
> document rather than another one, but IMO another proposal with another
> syntax should propose the equivalent functionnality.

	Well, on my side, I must admit that I really hate the squid-like
syntax.  I think that it introduces an extra complexity that is not needed,
IMHO.  I find squid acl complex to write and to read aand I think that it
would be a pain to handle them internaly in milter-greylist.

	That's why I want to propose a simpler syntax that addresses the
same issues.

> ===== My last unanswered proposal about ACL in milter-greylist =====
> 
> We would have a first match list of greylist/whitelist keywords, each
> using an ACL using a last match basis. Here is a complete proposal, I'd
> appreciate if people could comment:

        I'd like to propose a simpler scheme.  There would be a single
access-list that would be evaluated in sequence.  The evaluation would stop
when encountering the first entry that matches.
   
        Each entry would have this syntax:

acl greylist|whitelist [addr IP/CIDR] [from address|regex]
        [rcpt address|regex] [domain domain|regex]

> First, a few examples...
> 
> # first config, Dan's setup:
> acl grandma {
>         rcpt grandma@...
>         not from friend@...
>         not from another.friend@...
> }
> greylist grandma
> whitelist default

        This could be written:
 
acl whitelist from friend@... rcpt grandma@...
acl whitelist from another.friend@... rcpt grandma@...
acl greylist rcpt grandma@...
acl whitelist rcpt /.*/

> # Second config, Remy's setup:
> acl foo {
>         rcpt .*@...
>         not rcpt postmaster@...
> }
> greylist foo 
> whitelist default

        This could be written:

acl whitelist rcpt postmaster@...
acl greylist rcpt /.*@.../
acl whitelist rcpt /.*/

> # And finaly, Manu's setup:
> acl user1 {
>         rcpt user1@...
> }
> acl user2 (
>         rcpt user2@...
> }
> acl user3 {
>         rcpt user3@...
> }
> acl mywhitelist {
>         addr 193.54.0.0/16
>         domain friendly.com
> }
> whitelist mywhitelist
> greylist user1
> greylist user2
> greylist user3
> whitelist default

        This could be written:

acl whitelist addr 193.54.0.0/16 domain friendly.com
acl greylist rcpt user1@...
acl greylist rcpt user2@...
acl greylist rcpt user3@...
acl whitelist rcpt /.*/

> default is an internally defined ACL that match anything.
> 
> It seems to me that it matches all the needs that have been described
> here so far. Now, let's try to be more formal:
> 
> For each {greylist|whitelist} keyword {
>         for each entry in its ACL {
>                 if it's addr and the IP doesn't match
>                         break out of the ACL loop
>                 if it's domain and the domain doesn't match
>                         break out of  the ACL loop
>                 if it's from and the sender doesn't match
>                         break out of  the ACL loop
>                 if it's rcpt and the recipient doesn't match
>                         break out of  the ACL loop
>         }
>         if all entries in the ACL matched or if it was void {
>                 retain the {greylist|whitelist} decision 
>                 exit the keyword loop
>         }
> }

        With my proposition, I think that the evaluation would be:

match=0
Foreach (entry in the ACL) while (match == 0) {
        match=1
        if the entry contains an IP/CIDR couple and the IP desn't match {
                match=0
                continue
        }
        if the entry contains a domain and the domain doesn't match {
                match=0
                continue
        }
        if the entry contains a from and the sender doesn't match {
                match=0
                continue
        }
        if the entry contains a rcpt and the recipient doesn't match {
                match=0
                continue
        }
}
If (match == 1)
        return the type of the entry (greylist or whitelist)
else
        return the default type

> The last problem is to convert the older keyword to the new framework so
> that we can retain backward compatibility in the config file.
> 
> "rcpt X" in test mode will be interpreted as
> acl X {
>         rcpt X
> }
> greylist X
> 
> 
> "rcpt X" in "normal" mode will be interpreted as
> acl X {
>         rcpt X
> }
> whitelist X
>  
> 
> "{addr|from|domain} X" turns into
> acl X {
>         {addr|from|domain} X
> }
> whitelist X

        With my proposal, conversion is much easier : just add "acl " before
each {addr|domain|from|rcpt} keyword.  A simple sed script could do the work.

> Test mode turns into adding "whitelist default" at the end of the config
> Normal mode turns into adding "greylist default" at the end of the
> config.
> 
> We would tag as obsolete all the old syntax, but we will keep it working
> for the sake of backward compatibility (did I ever explained I hated
> software that broke my config files? :-) 

        I think that it's easy to handle the old syntax, by modifying
conf_yacc.y: just add an ACL entry when parsing the old {addr|domain|from|rcpt}
keywords instead of an exception entry.

	I'm interested in comments on this proposal.  I think that it addresses
the needs that I have been expressed on this list (if I am wrong, please point
out the needs that my proposal does not address).

	Cheers

		R\ufffdmy

Remy Card <Remy.Card@...> wrote:

>         Each entry would have this syntax:
> 
> acl greylist|whitelist [addr IP/CIDR] [from address|regex]
>         [rcpt address|regex] [domain domain|regex]

Some thoughts:
1) It would be nice if the order of the addr, from, rcpt and domain
clauses would not be fixed. i.e.: you could write "from friend@...
rcpt grandma@..." and "rcpt grandma@... from
friend@...", this would give the same result 

2) Is the acl keyword useful at all? We could have greylist and
whitelist keywords without a leading acl.

3) It would be nice to allow a multiline syntax too. Your syntax is nice
because it's lightweight, but having long lines of configuration can be
unconvenient when working on a 25x80 terminal. I'd suggest two
alternative syntaxes:

whitelist from friend@... rcpt grandma@...

or

whitelist {
        from friend@... 
        rcpt grandma@... 
}
 
> > First, a few examples...
> > 
> > # first config, Dan's setup:
> > acl grandma {
> >         rcpt grandma@...
> >         not from friend@...
> >         not from another.friend@...
> > }
> > greylist grandma
> > whitelist default
> 
>         This could be written:
>  
> acl whitelist from friend@... rcpt grandma@...
> acl whitelist from another.friend@... rcpt grandma@...
> acl greylist rcpt grandma@...
> acl whitelist rcpt /.*/

I'd like to head Dan Hollis opinion (it's his setup, after all)

One more thought: you use "acl whitelist rcpt /.*/" as a default rule.
This could also be written "acl whitelist from /.*/" or anything with a
wildcard. A whitelist keyword without any (from, rcpt, domain, addr)
specifier will also do the same thing.   

Maybe a "whitelist default" would be easier to understand for the
newcomer.

 
> > The last problem is to convert the older keyword to the new framework so
> > that we can retain backward compatibility in the config file.
(snip)
>         With my proposal, conversion is much easier : just add "acl " before
> each {addr|domain|from|rcpt} keyword.  A simple sed script could do the work.

You also need to add whitelist or greylist, and the keyword you need
depends if the user was running test mode or not. 

test mode would not affect the way you read the ACL, right?

-- 
Emmanuel Dreyfus
Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
le binaire et ceux qui ne le comprennent pas.
manu@...

On Fri, Nov 12, 2004 at 03:43:45PM +0100, manu@... wrote:
> Remy Card <Remy.Card@...> wrote:
> 
> >         Each entry would have this syntax:
> > 
> > acl greylist|whitelist [addr IP/CIDR] [from address|regex]
> >         [rcpt address|regex] [domain domain|regex]
> 
> Some thoughts:
> 1) It would be nice if the order of the addr, from, rcpt and domain
> clauses would not be fixed. i.e.: you could write "from friend@...
> rcpt grandma@..." and "rcpt grandma@... from
> friend@...", this would give the same result 

	Yep.  Order of clauses in an entry would not be fixed.

> 2) Is the acl keyword useful at all? We could have greylist and
> whitelist keywords without a leading acl.

	I  agree that the acl keyword is useless, but we already have a
greylist keyword:

# How long does a client has to wait before we accept 
# the messages it retries to send. Here, 1 hour.
greylist 30m

	We have to distinguish the two meanings of the greylist keyword.
Maybe we can add a new keyword?

> 3) It would be nice to allow a multiline syntax too. Your syntax is nice
> because it's lightweight, but having long lines of configuration can be
> unconvenient when working on a 25x80 terminal. I'd suggest two
> alternative syntaxes:
> 
> whitelist from friend@... rcpt grandma@...
> 
> or
> 
> whitelist {
>         from friend@... 
>         rcpt grandma@... 
> }

	No problem at all.  I think that yacc can handle this.

> >         This could be written:
> >  
> > acl whitelist from friend@... rcpt grandma@...
> > acl whitelist from another.friend@... rcpt grandma@...
> > acl greylist rcpt grandma@...
> > acl whitelist rcpt /.*/
> 
> I'd like to head Dan Hollis opinion (it's his setup, after all)

	I'm interested in Dan's comments as well.

> One more thought: you use "acl whitelist rcpt /.*/" as a default rule.
> This could also be written "acl whitelist from /.*/" or anything with a
> wildcard. A whitelist keyword without any (from, rcpt, domain, addr)
> specifier will also do the same thing.   
> 
> Maybe a "whitelist default" would be easier to understand for the
> newcomer.

	"acl whitelist rcpt /.*/" was a quick hack.  Actually, I think that
this line could be removed.  The default should be greylisting every mail when
running in normal mode and whitelisting every mail when running in test mode.
Thus, we can rely on this default action and we don't need to use this (ugly)
hack.

	Anyway, we can add a default keyword to be more specific.

> > > The last problem is to convert the older keyword to the new framework so
> > > that we can retain backward compatibility in the config file.
> (snip)
> >         With my proposal, conversion is much easier : just add "acl " before
> > each {addr|domain|from|rcpt} keyword.  A simple sed script could do the work.
> 
> You also need to add whitelist or greylist, and the keyword you need
> depends if the user was running test mode or not. 

	You're right.  I forgot about these keywords.  I guess that the
conversion script will need a parameter to know if milter-greylist runs
in noormal mde or not.

> test mode would not affect the way you read the ACL, right?

	Of course not.  ACL would be interpreted the same way in normal or
in test mode with the exception of the default action.

		R\ufffdmy

Remy Card <Remy.Card@...> wrote:

> > 2) Is the acl keyword useful at all? We could have greylist and
> > whitelist keywords without a leading acl.
> 
>       I  agree that the acl keyword is useless, but we already have a
> greylist keyword:

Oh right. I guess we'll have to live with the acl keyword.
 
>       "acl whitelist rcpt /.*/" was a quick hack.  Actually, I think that
> this line could be removed.  The default should be greylisting every mail when
> running in normal mode and whitelisting every mail when running in test mode.
> Thus, we can rely on this default action and we don't need to use this (ugly)
> hack.
(snip)
> > test mode would not affect the way you read the ACL, right?
> 
>       Of course not.  ACL would be interpreted the same way in normal or
> in test mode with the exception of the default action.

Well my idea is to keep the test mode working as is so that configs
don't get broken at upgrade time, but to deprecate it, because the ACL
setup will allow the same setup with a much more cleaner config. 

I was raising the concern of supporting the old syntax and test mode
with the new scheme. You'll have to "emulate" ACL with the older
keywords. The configuration parser will have to do this conversion while
loading the config:

addr X          ->      acl whitelist addr X
domain X        ->      acl whitelist domain X
from X          ->      acl whitelist from X
rcpt X          ->      acl whitelist rcpt X   if not testmode
rcpt X          ->      acl greylist rcpt X    if testmode      

The result of using testmode and the acl keywords at the same time
should probably be left unspecified: just don't do it, it's not
supported with the new ACL syntax.
 
-- 
Emmanuel Dreyfus
Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
le binaire et ceux qui ne le comprennent pas.
manu@...

On Fri, Nov 12, 2004 at 04:51:08PM +0100, manu@... wrote:
> >       Of course not.  ACL would be interpreted the same way in normal or
> > in test mode with the exception of the default action.
> 
> Well my idea is to keep the test mode working as is so that configs
> don't get broken at upgrade time, but to deprecate it, because the ACL
> setup will allow the same setup with a much more cleaner config. 

	Yes.  Test mode will not be useful any more when ACL are used
by everybody.

> I was raising the concern of supporting the old syntax and test mode
> with the new scheme. You'll have to "emulate" ACL with the older
> keywords. The configuration parser will have to do this conversion while
> loading the config:
> 
> addr X          ->      acl whitelist addr X
> domain X        ->      acl whitelist domain X
> from X          ->      acl whitelist from X
> rcpt X          ->      acl whitelist rcpt X   if not testmode
> rcpt X          ->      acl greylist rcpt X    if testmode      

	Yep.  This is exactly how I intend to handle the current keywords.

> The result of using testmode and the acl keywords at the same time
> should probably be left unspecified: just don't do it, it's not
> supported with the new ACL syntax.

	Well, test mode could still be used to specify the default
b\ufffdhaviour.  In normal mode, the ACL can be implicitly terminated by:
acl greylist default
while test mode will change this implicit last rule to:
acl whitelist default

	IMO, we have to keep test mode until the current syntax is phased
out because the meaning of the rcpt keyword depends on it and we have to
handle the test mode to emulate rcpt with the ACL scheme.

		R\ufffdmy

On Fri, 12 Nov 2004, Emmanuel Dreyfus wrote:
> Remy Card <Remy.Card@...> wrote:
> >         This could be written:
> > acl whitelist from friend@... rcpt grandma@...
> > acl whitelist from another.friend@... rcpt grandma@...
> > acl greylist rcpt grandma@...
> > acl whitelist rcpt /.*/
> I'd like to head Dan Hollis opinion (it's his setup, after all)

IMO syntax should be kept as simple as possible.

Complex multi-level ACL with curly braces should be avoided if you can 
achieve the same effect clearer and easier with simple syntax as above.

1) it's easier for automation (i have scripts rebuilding greylist rules 
for several thousand users on our ISP). current ACLs can be built with 
simple shell script. with multilevel ACL this becomes much more 
cumbersome.
2) it's easier for users to write, and to read.
3) it's easier to parse -> less bugs

-Dan

Dan Hollis <goemon@...> wrote:

> > I'd like to hear Dan Hollis opinion (it's his setup, after all)
> 
> IMO syntax should be kept as simple as possible.
> 
> Complex multi-level ACL with curly braces should be avoided if you can
> achieve the same effect clearer and easier with simple syntax as above.
> 
> 1) it's easier for automation (i have scripts rebuilding greylist rules
> for several thousand users on our ISP). current ACLs can be built with
> simple shell script. with multilevel ACL this becomes much more 
> cumbersome.
> 2) it's easier for users to write, and to read.
> 3) it's easier to parse -> less bugs

I assume that this means you agree with Remy's proposal?

-- 
Emmanuel Dreyfus
Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
le binaire et ceux qui ne le comprennent pas.
manu@...

On Sat, 13 Nov 2004 manu@... wrote:
> Dan Hollis <goemon@...> wrote:
> > > I'd like to hear Dan Hollis opinion (it's his setup, after all)
> > IMO syntax should be kept as simple as possible.
> > Complex multi-level ACL with curly braces should be avoided if you can
> > achieve the same effect clearer and easier with simple syntax as above.
> > 1) it's easier for automation (i have scripts rebuilding greylist rules
> > for several thousand users on our ISP). current ACLs can be built with
> > simple shell script. with multilevel ACL this becomes much more 
> > cumbersome.
> > 2) it's easier for users to write, and to read.
> > 3) it's easier to parse -> less bugs
> I assume that this means you agree with Remy's proposal?

I guess so, if there is a choice then I choose NO to heirarchical config 
with curly braces.

-Dan

Dan Hollis <goemon@...> wrote:

> I guess so, if there is a choice then I choose NO to heirarchical config
> with curly braces.

I proposed to accept both syntax:

acl greylist rcpt a-very-big-addess@... from
/.*@example\.com/

acl greylist {
        rcpt a-very-big-addess@... 
        from /.*@example\.com/
}

Because too long lines are a pain on small terminals. But as both
syntaxes will be available, you will not have to use curly braces.

-- 
Emmanuel Dreyfus
Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
le binaire et ceux qui ne le comprennent pas.
manu@...

On Mon, 15 Nov 2004 manu@... wrote:
> Dan Hollis <goemon@...> wrote:
> > I guess so, if there is a choice then I choose NO to heirarchical config
> > with curly braces.
> I proposed to accept both syntax:
> acl greylist rcpt a-very-big-addess@... from
> /.*@example\.com/
> 
> acl greylist {
>         rcpt a-very-big-addess@... 
>         from /.*@example\.com/
> }
> Because too long lines are a pain on small terminals. But as both
> syntaxes will be available, you will not have to use curly braces.

So really, {} serve no function at all... all of these would be equivalent?

acl greylist rcpt a-very-big-addess@... from /.*@example\.com/

acl greylist \
	rcpt a-very-big-addess@... \
	from /.*@example\.com/

acl greylist {
	rcpt a-very-big-addess@... 
	from /.*@example\.com/
}

-Dan

Dan Hollis <goemon@...> wrote:

> So really, {} serve no function at all... all of these would be equivalent?
> 
> acl greylist rcpt a-very-big-addess@... from
> /.*@example\.com/
> 
> acl greylist \
>       rcpt a-very-big-addess@... \
>       from /.*@example\.com/
> 
> acl greylist {
>       rcpt a-very-big-addess@... 
>       from /.*@example\.com/
> }

Yes, but do you advocate for a third syntax, or for \ instead of {}?

-- 
Emmanuel Dreyfus
Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
le binaire et ceux qui ne le comprennent pas.
manu@...

If there is a choice, I advocate for only one syntax, the first one.

-Dan

On Tue, 16 Nov 2004 manu@... wrote:

> 
> Dan Hollis <goemon@...> wrote:
> 
> > So really, {} serve no function at all... all of these would be equivalent?
> > 
> > acl greylist rcpt a-very-big-addess@... from
> > /.*@example\.com/
> > 
> > acl greylist \
> >       rcpt a-very-big-addess@... \
> >       from /.*@example\.com/
> > 
> > acl greylist {
> >       rcpt a-very-big-addess@... 
> >       from /.*@example\.com/
> > }
> 
> Yes, but do you advocate for a third syntax, or for \ instead of {}?
> 
>

Dan Hollis <goemon@...> wrote:

> If there is a choice, I advocate for only one syntax, the first one.

My idea was to support the one-liner and a multi-line syntax at the same
time. Some admins will prefer the one-liner, some will prefer short
lines.

But should we adopt {} or \ for the multi-line syntax?

-- 
Emmanuel Dreyfus
Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
le binaire et ceux qui ne le comprennent pas.
manu@...

manu@... wrote:

>Dan Hollis <goemon@...> wrote:
>
>  
>
>>If there is a choice, I advocate for only one syntax, the first one.
>>    
>>
>
>My idea was to support the one-liner and a multi-line syntax at the same
>time. Some admins will prefer the one-liner, some will prefer short
>lines.
>
>But should we adopt {} or \ for the multi-line syntax?
>
>  
>
\ has the advantage of precedent (everyone knows '\' at the end of a 
line means continue)...

Graham

On Tue, 16 Nov 2004 manu@... wrote:
> Dan Hollis <goemon@...> wrote:
> > If there is a choice, I advocate for only one syntax, the first one.
> My idea was to support the one-liner and a multi-line syntax at the same
> time. Some admins will prefer the one-liner, some will prefer short
> lines.
> But should we adopt {} or \ for the multi-line syntax?

How about, any line which starts with a tab or space is a continuation of 
the previous line.

acl greylist
	rcpt a-very-big-addess@...
	from /.*@example\.com/

That makes the parsing simplest of all, but still very readable.

-Dan