Synth-DIY Yahoo! Groups Archives

Thread

Basic rate limit filter

2014-04-18 by Georgi Petrov

Hello all,

Recently milter-greylist was added to Virtualmin GPL as a mail rate limiting option. The default setting is pretty basic and not satisfactory. Therefore I will try to propose more advanced settings/setup.

Milter-greylist is of course way more capable than just limiting the rate of the emails sent, but as this is one really important feature lets start with it.

Rate limiting is important because:

brute foreced/hijacked mail accounts can be used for sending huge amount of spam. I've seen something like 100K emails sent in a night. Of course the IP gets listed as spammer immediately. Rate limiting can slow this down. The admin will have time to react (possibly the admin will be warned).
in a shared environment unfair users can send spam using their own accounts
websites get hacked and scripts that send spam can be installed there
there are certain DoS attacks that can block the email service. For example huge amount of incoming emails to existing users, containing virus. The virus protection (if any) will block them, but the whole service can slow down or completely become blocked.

What we want to achieve with our basic rate limiting:

mail sent from localhost (via PHP, sendmail or webmail) should be limited using the user as key. We should block the user and not the IP as if we block the IP (127.0.0.1), nobody will be able to send anymore. We should probably use the "from" field, as there is no sasl authentication for localhost, but scamming the "from" field is not possible (I should double check this).
mails sent via email clients (Thunderbird etc.) should be limited by the IP or the user. If we use the user it is probably better to use the sasl authentication user, not the "from" field, as it can be scammed. If we use the IP this can be combined with the next rule, but it will also block multiple users using the same IP, one of which is sending too many emails.
mail coming from other mail servers for local users should be limited by the IP. This is for DoS prevention only, the limit here should be high.
all users/IPs should be greylisted if they go over the ratelimit as we want them to be able to send emails after some time and not blacklist them forever.

Here is the original Virtualmin ratelimit rule, which is obviously not good at all (the actual number of emails is controlled by the user via the webinterface):

ratelimit "virtualmin_limit" rcpt 30 / 30m
racl blacklist from /.*/ ratelimit "virtualmin_limit" msg "Message quota exceeded"
racl whitelist default

If somebody sends too many emails from localhost, it gets blacklisted and nobody can send anymore.

Here is my modified version:

ratelimit "virtualmin_limit" rcpt 30 / 30m key "%f"
racl greylist from /.*/ ratelimit "virtualmin_limit" delay 31m autowhite 0m msg "Message quota exceeded"
racl whitelist default

It works as expected limiting per user. If somebody sends too many emails from PHP all the emails wait in the queue until the user drops below the limit. The drawback is that it does not treat local and non local email differently. So it needs improvement.

Here are my specific questions:

- How I can achieve two different rate limit rules for localhost and non localhost (I have tried with some rules but they didn't work)?

- Is there a better way to limit the emails from PHP? They cannot be rejected, so they go in the mail queue and wait there. The queue tries to resend them every few minutes and they are rejected again until the rate drops under the limit.

- Can we achieve some sort admin notification if the ratelimit is blocking somebody? Now I can have warning notifications if the mail queue gets very big.

Thanks a lot for your ideas, comments and time!

Georgi

Re: [milter-greylist] Basic rate limit filter

2014-04-18 by manu@...

Georgi Petrov <georgi.petrov@...> wrote:

> - How I can achieve two different rate limit rules for localhost and non
> localhost (I have tried with some rules but they didn't work)?

I guess you used an ACL whith clause addr 127.0.0.1, but perhaps the
message was sent from ::1 (IPv6) ?

> - Is there a better way to limit the emails from PHP? They cannot be
> rejected, so they go in the mail queue and wait there. The queue tries to
> resend them every few minutes and they are rejected again until the rate
> drops under the limit.

You need your PHP application to talk SMTP instead of using the system
mail/sendmail/whatever command. 

> - Can we achieve some sort admin notification if the ratelimit is blocking
> somebody? Now I can have warning notifications if the mail queue gets very
> big.

You could add a urlcheck clause at the end of your racl blacklist ACL:
if ratelimit does not match it is not evaluated, and if ratelimit
matches, the evaluation continue to the urlcheck clause.

Your urlcheck clause will always evaluate to true but it will send the
signal to a webservice where you can do whatever you want.

An alternative is to log the even, and have another process parsing the
logs. 

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Re: [milter-greylist] Basic rate limit filter

2014-04-22 by Georgi Petrov

Hello Emmanuel,

Thanks a lot for your comments!

Here is what I did try (the limits are very low to let me test without sending too many emails):

ratelimit "virtualmin_limit_local" rcpt 3 / 10m key "%f"

racl greylist addr 127.0.0.1 ratelimit "virtualmin_limit_local" delay 11m autowhite 0m msg "Message quota exceeded"

ratelimit "virtualmin_limit" rcpt 3 / 10m key "%i"

racl greylist from /.*/ ratelimit "virtualmin_limit" delay 11m autowhite 0m msg "Message quota exceeded"

racl whitelist default

The first rate limit rule works fine (the emails indeed come from 127.0.0.1). The problem is that the local emails also hit the second ruleset and eventually 127.0.0.1 gets greylisted if the combined amount of local emails is over the limit. The second ruleset should work for everything except 127.0.0.1 but I didn't find a way to invert the logic there. (Also if some message hits one rule it still propagates down so it can also hit another, the logic is not like in iptables for example). Maybe I can use the regular expression syntax to invert the logic, but in this case I need to use the actual full localhost name which is not as convenient as the IP. Can you advise me on that?

Yes, if the PHP application is able to talk SMTP then it can be much easier, but only few can so we need to keep the mail(); function working. I should investigate if it is possible to fake the "FROM" if you indeed use a PHP appllication that comes from 127.0.0.1 and talks SMTP. In this case we should use the SASL authentication user and create one more rate limit rule.

When adding more keys, for example

ratelimit "virtualmin_limit_local" rcpt 3 / 10m key "%f%i"

is the logic AND or is it OR? From what I see it is just string comparison so the logic is AND. Is there a way to achieve logic OR? For example either the first value is the same or the second? The key value, can it take regular expression?

The admin notification is better if we log the rate_limit event then other scripts will parse the log under some cron schedule and send proper warnings. I think this one is already available in Virtualmin/Webmin

Thanks again for your time!

Georgi

Show quoted textHide quoted text

On Fri, Apr 18, 2014 at 7:41 PM, <manu@netbsd.org> wrote:

Georgi Petrov <georgi.petrov@...> wrote:

> - How I can achieve two different rate limit rules for localhost and non
> localhost (I have tried with some rules but they didn't work)?

I guess you used an ACL whith clause addr 127.0.0.1, but perhaps the
message was sent from ::1 (IPv6) ?

> - Is there a better way to limit the emails from PHP? They cannot be
> rejected, so they go in the mail queue and wait there. The queue tries to
> resend them every few minutes and they are rejected again until the rate
> drops under the limit.

You need your PHP application to talk SMTP instead of using the system
mail/sendmail/whatever command.

> - Can we achieve some sort admin notification if the ratelimit is blocking
> somebody? Now I can have warning notifications if the mail queue gets very
> big.

You could add a urlcheck clause at the end of your racl blacklist ACL:
if ratelimit does not match it is not evaluated, and if ratelimit
matches, the evaluation continue to the urlcheck clause.

Your urlcheck clause will always evaluate to true but it will send the
signal to a webservice where you can do whatever you want.

An alternative is to log the even, and have another process parsing the
logs.

--
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Re: [milter-greylist] Basic rate limit filter

2014-04-22 by Emmanuel Dreyfus

On Tue, Apr 22, 2014 at 12:14:39PM +0300, Georgi Petrov wrote:
> The first rate limit rule works fine (the emails indeed come from
> 127.0.0.1). The problem is that the local emails also hit the second
> ruleset and eventually 127.0.0.1 gets greylisted if the combined amount of
> local emails is over the limit. The second ruleset should work for
> everything except 127.0.0.1 but I didn't find a way to invert the logic
> there.

have a not addr 127.0.0.1 clause in the second ACL, or add a
"racl whitelist addr 127.0.0.1" between the two ACLs.

> ratelimit "virtualmin_limit_local" rcpt 3 / 10m key "%f%i"
> 
> is the logic AND or is it OR? From what I see it is just string comparison
> so the logic is AND. Is there a way to achieve logic OR? For example either
> the first value is the same or the second? The key value, can it take
> regular expression?

Define two ratelimit and use two ACL, that makesa OR.

-- 
Emmanuel Dreyfus
manu@...

Re: [milter-greylist] Basic rate limit filter

2014-04-29 by Georgi Petrov

Hello Emmanuel,

Thanks a lot for your advice! It indeed worked. The rules are now:

ratelimit "virtualmin_limit_local" rcpt 3 / 10m key "%f"

racl greylist addr 127.0.0.1 ratelimit "virtualmin_limit_local" delay 11m autowhite 0m msg "Message quota exceeded"

ratelimit "virtualmin_limit" rcpt 3 / 10m key "%i"

racl greylist not addr 127.0.0.1 ratelimit "virtualmin_limit" delay 11m autowhite 0m msg "Message quota exceeded"

racl whitelist default

I think it should be fine like this.

I will try to read a little more and invent some way of automated outgoing spam checking and blacklisting.

Thanks again!

Georgi

Show quoted textHide quoted text

On Tue, Apr 22, 2014 at 9:47 PM, Emmanuel Dreyfus <manu@...> wrote:

On Tue, Apr 22, 2014 at 12:14:39PM +0300, Georgi Petrov wrote:
> The first rate limit rule works fine (the emails indeed come from
> 127.0.0.1). The problem is that the local emails also hit the second
> ruleset and eventually 127.0.0.1 gets greylisted if the combined amount of
> local emails is over the limit. The second ruleset should work for
> everything except 127.0.0.1 but I didn't find a way to invert the logic
> there.

have a not addr 127.0.0.1 clause in the second ACL, or add a
"racl whitelist addr 127.0.0.1" between the two ACLs.

> ratelimit "virtualmin_limit_local" rcpt 3 / 10m key "%f%i"
>
> is the logic AND or is it OR? From what I see it is just string comparison
> so the logic is AND. Is there a way to achieve logic OR? For example either
> the first value is the same or the second? The key value, can it take
> regular expression?

Define two ratelimit and use two ACL, that makesa OR.

--
Emmanuel Dreyfus
manu@...