Milter-greylist

Hi Emmanuel,

I get a lot of spam forwarded from an external server to my server. The 
used SRS return path e.g. looks like this:

   <srs0+uc/j=ux=mailer-service.de=postmaster@...>

To block the forwarding server udag.de in sendmail access is not an 
option, because ham is also forwarded. Sendmail's access file doesn't 
support regular expressions on "From:" filter and I don't see further 
options with sendmail. Is there a way to block those messages with 
milter-greylist using something like this:

-------
list "superspammer" from { \
  *=mailer-service.de=*@... \
}

racl blacklist list superspammer msg "Spam blocked" flushaddr
-------

I'm using milter-greylist 4.3.9-1 on Ubuntu 12.04 LTS, but could build 
backports to newer version if necessary.

Ciao
Marcus

On 2016-09-03 16:08, Marcus Schopen lists-yahoogroups@... 
[milter-greylist] wrote:
> Hi Emmanuel,
> 
> I get a lot of spam forwarded from an external server to my server.
> The
> used SRS return path e.g. looks like this:
> 
> <srs0+uc/j=ux=mailer-service.de=postmaster@...>
> 
> To block the forwarding server udag.de in sendmail access is not an
> option, because ham is also forwarded. Sendmail's access file doesn't
> support regular expressions on "From:" filter and I don't see further
> options with sendmail. Is there a way to block those messages with
> milter-greylist using something like this:
> 
> -------
> list "superspammer" from { \
> *=mailer-service.de=*@... \
> }
> 
> racl blacklist list superspammer msg "Spam blocked" flushaddr
> -------
> 
> I'm using milter-greylist 4.3.9-1 on Ubuntu 12.04 LTS, but could build
> 
> backports to newer version if necessary.

This seems to work

-----------
# Test
list "SUPERSPAMMER" from { \
         /peter.lustig@.../ \
         /.*mailer-service\.de.*@udag\.de$/ \
}

racl blacklist list "SUPERSPAMMER" msg "Bad reputation - %f listed on 
SPAM blacklist %a" flushaddr
-----------

But the line

   /.*mailer-service\.de.*@udag\.de$/ \

never matches on

  <srs0+uc/j=ux=mailer-service.de=postmaster@...>

I've tested different regular expression, but without success. Any ideas 
how the regular expression must look like?

Ciao
Marcus

4 \u0441\u0435\u043d\u0442\u044f\u0431\u0440\u044f 2016�\u0433. 14:03:03 CEST, "Marcus Schopen lists-yahoogroups@... [milter-greylist]" <milter-greylist@yahoogroups.com> \u043f\u0438\u0448\u0435\u0442:
>On 2016-09-03 16:08, Marcus Schopen lists-yahoogroups@... 
>[milter-greylist] wrote:
>> Hi Emmanuel,
>> 
>> I get a lot of spam forwarded from an external server to my server.
>> The
>> used SRS return path e.g. looks like this:
>> 
>> <srs0+uc/j=ux=mailer-service.de=postmaster@...>
>> 
>> To block the forwarding server udag.de in sendmail access is not an
>> option, because ham is also forwarded. Sendmail's access file doesn't
>> support regular expressions on "From:" filter and I don't see further
>> options with sendmail. Is there a way to block those messages with
>> milter-greylist using something like this:
>> 
>> -------
>> list "superspammer" from { \
>> *=mailer-service.de=*@... \
>> }
>> 
>> racl blacklist list superspammer msg "Spam blocked" flushaddr
>> -------
>> 
>> I'm using milter-greylist 4.3.9-1 on Ubuntu 12.04 LTS, but could
>build
>> 
>> backports to newer version if necessary.
>
>This seems to work
>
>-----------
># Test
>list "SUPERSPAMMER" from { \
>         /peter.lustig@.../ \
>         /.*mailer-service\.de.*@udag\.de$/ \
>}
>
>racl blacklist list "SUPERSPAMMER" msg "Bad reputation - %f listed on 
>SPAM blacklist %a" flushaddr
>-----------
>
>But the line
>
>   /.*mailer-service\.de.*@udag\.de$/ \
>
>never matches on
>
>  <srs0+uc/j=ux=mailer-service.de=postmaster@...>
>
>I've tested different regular expression, but without success. Any
>ideas 
>how the regular expression must look like?
>
>Ciao
>Marcus

Just in case, check regexes are on in your build and maybe the domain-matching keyword (don't have docs atm) acts up?
--
Typos courtesy of K-9 Mail on my Samsung Android

Hi Jim,

On 2016-09-04 15:08, Jim Klimov jimklimov@... [milter-greylist] 
wrote:
> Just in case, check regexes are on in your build and maybe the
> domain-matching keyword (don't have docs atm) acts up?

I run a selfbuild backport, because the standard Ubuntu 14.04 LTS 
version doesn't support geoip. My debian/rules file looks like this:

----------
override_dh_auto_configure:
	dh_auto_configure -- \
	--with-user=greylist \
	--sysconfdir=/etc/milter-greylist \
	--with-conffile=/etc/milter-greylist/greylist.conf \
	--with-dumpfile=/var/lib/milter-greylist/greylist.db \
	--with-libspf2=/usr \
	--with-libcurl=/usr \
	--with-libmilter=/usr \
	--with-libopendkim=/usr \
	--with-libGeoIP=/usr \
	--enable-dnsrbl \
	--disable-rpath \
	LDFLAGS=" -Wl,-z,defs -L/usr/lib/libmilter $(LDFLAGS) " \
	CFLAGS=" -fno-strict-aliasing $(CFLAGS) "
----------

extendedregex is enabled in greylist.conf.

Is there anything missing? A configure --help doesn't show any special 
regex options to enable to me.

Ciao!
Marcus

Hi,

a regular expression like

          /.*test123@.../ \

is working and blocks <blabla-test123@...> as expected;  also

         /.*test456.*test789@.../

blocks <blabla-test456-huhu-test789@...>. Good.

But for some strange reasons no effect on

          /.*mailer-service.de.*@.../ \

for sender <srs0+uc/j=ux=mailer-service.de=postmaster@...>.

Tested with milter-greylist 4.5.11.

Ciao!

Hi,

I think there is a problem filtering addresses with equal signs in it 
typically used by SRS addresses. Using this regular expression

         /.*mailer\-service\.de.*@.../ \

works fine on

   <srs0+uc/j-ux-mailer-service.de-postmaster@...>

not NOT on

   <srs0+uc/j=ux=mailer-service.de=postmaster@...>

Any ideas? May be a bug?

Ciao!
Marcus

4 \u0441\u0435\u043d\u0442\u044f\u0431\u0440\u044f 2016�\u0433. 17:05:08 CEST, "Marcus Schopen lists-yahoogroups@... [milter-greylist]" <milter-greylist@yahoogroups.com> \u043f\u0438\u0448\u0435\u0442:
>Hi,
>
>I think there is a problem filtering addresses with equal signs in it 
>typically used by SRS addresses. Using this regular expression
>
>         /.*mailer\-service\.de.*@.../ \
>
>works fine on
>
>   <srs0+uc/j-ux-mailer-service.de-postmaster@...>
>
>not NOT on
>
>   <srs0+uc/j=ux=mailer-service.de=postmaster@...>
>
>Any ideas? May be a bug?
>
>Ciao!
>Marcus

Now I wonder if the code "handles" SRS somehow (maybe the MTA code, not milter-greylist itself), so the regex does not actually get that string at input, but rather the assumed sender email address?

Try adding the log keyword and some format string to dump the rule's progress into your syslog...

Jim
--
Typos courtesy of K-9 Mail on my Samsung Android

Hi Jim,

On 2016-09-04 23:53, Jim Klimov jimklimov@... [milter-greylist] 
wrote:
> 
> Now I wonder if the code "handles" SRS somehow (maybe the MTA code,
> not milter-greylist itself), so the regex does not actually get that
> string at input, but rather the assumed sender email address?
> 
> Try adding the log keyword and some format string to dump the rule's
> progress into your syslog...

I turned on verbose logging.

A test with "mail from: 
<srs0+uc/j=ux=mailer-service.de=postmaster@...>"

ends up in milter-greylist as postmaster@.... All parts before = 
seems to be ignored; same with <blabla=test123@...> which is shown 
as test123@... in greylist logfile. Mimedefang, which runs after 
milter-greylist shows the complete return-path including = sign.

Ciao!
Marcus

5 \u0441\u0435\u043d\u0442\u044f\u0431\u0440\u044f 2016�\u0433. 0:27:34 CEST, "Marcus Schopen lists-yahoogroups@... [milter-greylist]" <milter-greylist@yahoogroups.com> \u043f\u0438\u0448\u0435\u0442:
>Hi Jim,
>
>On 2016-09-04 23:53, Jim Klimov jimklimov@... [milter-greylist] 
>wrote:
>> 
>> Now I wonder if the code "handles" SRS somehow (maybe the MTA code,
>> not milter-greylist itself), so the regex does not actually get that
>> string at input, but rather the assumed sender email address?
>> 
>> Try adding the log keyword and some format string to dump the rule's
>> progress into your syslog...
>
>I turned on verbose logging.
>
>A test with "mail from: 
><srs0+uc/j=ux=mailer-service.de=postmaster@...>"
>
>ends up in milter-greylist as postmaster@.... All parts before = 
>seems to be ignored; same with <blabla=test123@...> which is shown 
>as test123@... in greylist logfile. Mimedefang, which runs after 
>milter-greylist shows the complete return-path including = sign.
>
>Ciao!
>Marcus

Look if both milters use the same or different milter-macros for the email address? If the same, then it is MGL chopping off something, and it is something fixable (or perhaps already tunable?) easily in the scope of the project.
--
Typos courtesy of K-9 Mail on my Samsung Android

Hello

The problem you report stems from this code in milter-greylist.c :

        /*
         * Strip anything before the last '=' in the
         * source address. This avoid problems with
         * mailing lists using a unique sender address
         * for each retry.
         */
        if ((idx = rindex(tmpfrom, '=')) == NULL)
                idx = tmpfrom;

We could add an option to make this operation optional. Or add alternative
from clause (origfrom?)  that let us match on original sender. Let me 
know what you feel is more appropriate.

In the meantime, you can just try commenting the offending code.

-- 
Emmanuel Dreyfus
manu@...

5 \u0441\u0435\u043d\u0442\u044f\u0431\u0440\u044f 2016�\u0433. 5:23:26 CEST, "Emmanuel Dreyfus manu@... [milter-greylist]" <milter-greylist@yahoogroups.com> \u043f\u0438\u0448\u0435\u0442:
>Hello
>
>The problem you report stems from this code in milter-greylist.c :
>
>        /*
>         * Strip anything before the last '=' in the
>         * source address. This avoid problems with
>         * mailing lists using a unique sender address
>         * for each retry.
>         */
>        if ((idx = rindex(tmpfrom, '=')) == NULL)
>                idx = tmpfrom;
>
>We could add an option to make this operation optional. Or add
>alternative
>from clause (origfrom?)  that let us match on original sender. Let me 
>know what you feel is more appropriate.
>
>In the meantime, you can just try commenting the offending code.

I'd say an alternative clause would be more granular in usage, so rules can be done for both types of address in the same config.
--
Typos courtesy of K-9 Mail on my Samsung Android

On Mon, Sep 05, 2016 at 09:32:01AM +0200, Jim Klimov jimklimov@... [milter-greylist] wrote:
> I'd say an alternative clause would be more granular in usage, so rules can be done for both types of address in the same config.

Yes, but we need a name. rawfrom is thebetter I found so far.

-- 
Emmanuel Dreyfus
manu@...

On 2016-09-05 09:39, Emmanuel Dreyfus manu@... [milter-greylist] 
wrote:
> On Mon, Sep 05, 2016 at 09:32:01AM +0200, Jim Klimov jimklimov@...
> [milter-greylist] wrote:
>> I'd say an alternative clause would be more granular in usage, so
> rules can be done for both types of address in the same config.
> 
> Yes, but we need a name. rawfrom is thebetter I found so far.

I like the idea of alternative filtering on from and the original 
envelope from. rawfrom is a good name. :)

Ciao
Marcus

5 \u0441\u0435\u043d\u0442\u044f\u0431\u0440\u044f 2016�\u0433. 9:39:29 CEST, "Emmanuel Dreyfus manu@... [milter-greylist]" <milter-greylist@yahoogroups.com> \u043f\u0438\u0448\u0435\u0442:
>On Mon, Sep 05, 2016 at 09:32:01AM +0200, Jim Klimov jimklimov@...
>[milter-greylist] wrote:
>> I'd say an alternative clause would be more granular in usage, so
>rules can be done for both types of address in the same config.
>
>Yes, but we need a name. rawfrom is thebetter I found so far.

Sounds reasonable and describes (or at least hints to rtfm) the difference in purpose ;)

Jim
--
Typos courtesy of K-9 Mail on my Samsung Android

Hi,

On 2016-09-05 05:23, Emmanuel Dreyfus manu@... [milter-greylist] 
wrote:
> 
> The problem you report stems from this code in milter-greylist.c :
> 
> /*
> * Strip anything before the last '=' in the
> * source address. This avoid problems with
> * mailing lists using a unique sender address
> * for each retry.
> */
> if ((idx = rindex(tmpfrom, '=')) == NULL)
> idx = tmpfrom;

I tried to build a new package just deleting these two lines from 
source:

  if ((idx = rindex(tmpfrom, '=')) == NULL)
  idx = tmpfrom;

Milter is crashing then

  milter_sys_read(greylist): cmd read returned 0, expecting 5
  Milter (greylist): to error state
  Milter (greylist): time command (M), 0

Ciao
Marcus

On Mon, Sep 05, 2016 at 10:20:24AM +0200, Marcus Schopen lists-yahoogroups@... [milter-greylist] wrote:
> I like the idea of alternative filtering on from and the original 
> envelope from. rawfrom is a good name. :)

I have some code that builds, but no time to test. Any volunteers?

-- 
Emmanuel Dreyfus
manu@...

On 2016-09-05 13:39, Marcus Schopen lists-yahoogroups@... 
[milter-greylist] wrote:
> Hi,
> 
> On 2016-09-05 05:23, Emmanuel Dreyfus manu@...
> [milter-greylist]
> wrote:
>> 
>> The problem you report stems from this code in milter-greylist.c :
>> 
>> /*
>> * Strip anything before the last '=' in the
>> * source address. This avoid problems with
>> * mailing lists using a unique sender address
>> * for each retry.
>> */
>> if ((idx = rindex(tmpfrom, '=')) == NULL)
>> idx = tmpfrom;
> 
> I tried to build a new package just deleting these two lines from
> source:
> 
> if ((idx = rindex(tmpfrom, '=')) == NULL)
> idx = tmpfrom;
> 
> Milter is crashing then
> 
> milter_sys_read(greylist): cmd read returned 0, expecting 5
> Milter (greylist): to error state
> Milter (greylist): time command (M), 0

Arrr, too much coffee, too little sleep. Just deleting the line

  if ((idx = rindex(tmpfrom, '=')) == NULL)

"fixes" the problem and the raw SRS mail from address comes up in the 
milter. Nice!

Ciao!
Marcus

On 2016-09-05 14:28, Emmanuel Dreyfus manu@... [milter-greylist] 
wrote:
> On Mon, Sep 05, 2016 at 10:20:24AM +0200, Marcus Schopen
> lists-yahoogroups@... [milter-greylist] wrote:
>> I like the idea of alternative filtering on from and the original
>> envelope from. rawfrom is a good name. :)
> 
> I have some code that builds, but no time to test. Any volunteers?

HERE! :)