2019-05-03 06:17:27 +08:00
|
|
|
# frozen_string_literal: true
|
|
|
|
|
2013-02-06 03:16:51 +08:00
|
|
|
# A redis backed rate limiter.
|
|
|
|
class RateLimiter
|
|
|
|
|
FEATURE: Apply rate limits per user instead of IP for trusted users (#14706)
Currently, Discourse rate limits all incoming requests by the IP address they
originate from regardless of the user making the request. This can be
frustrating if there are multiple users using Discourse simultaneously while
sharing the same IP address (e.g. employees in an office).
This commit implements a new feature to make Discourse apply rate limits by
user id rather than IP address for users at or higher than the configured trust
level (1 is the default).
For example, let's say a Discourse instance is configured to allow 200 requests
per minute per IP address, and we have 10 users at trust level 4 using
Discourse simultaneously from the same IP address. Before this feature, the 10
users could only make a total of 200 requests per minute before they got rate
limited. But with the new feature, each user is allowed to make 200 requests
per minute because the rate limits are applied on user id rather than the IP
address.
The minimum trust level for applying user-id-based rate limits can be
configured by the `skip_per_ip_rate_limit_trust_level` global setting. The
default is 1, but it can be changed by either adding the
`DISCOURSE_SKIP_PER_IP_RATE_LIMIT_TRUST_LEVEL` environment variable with the
desired value to your `app.yml`, or changing the setting's value in the
`discourse.conf` file.
Requests made with API keys are still rate limited by IP address and the
relevant global settings that control API keys rate limits.
Before this commit, Discourse's auth cookie (`_t`) was simply a 32 characters
string that Discourse used to lookup the current user from the database and the
cookie contained no additional information about the user. However, we had to
change the cookie content in this commit so we could identify the user from the
cookie without making a database query before the rate limits logic and avoid
introducing a bottleneck on busy sites.
Besides the 32 characters auth token, the cookie now includes the user id,
trust level and the cookie's generation date, and we encrypt/sign the cookie to
prevent tampering.
Internal ticket number: t54739.
2021-11-18 04:27:30 +08:00
|
|
|
attr_reader :max, :secs, :user, :key, :error_code
|
2015-04-16 07:44:30 +08:00
|
|
|
|
2015-02-03 01:44:21 +08:00
|
|
|
def self.key_prefix
|
2017-12-07 08:48:11 +08:00
|
|
|
"l-rate-limit3:"
|
2015-02-03 01:44:21 +08:00
|
|
|
end
|
2015-01-30 00:44:51 +08:00
|
|
|
|
2013-10-09 12:10:37 +08:00
|
|
|
def self.disable
|
|
|
|
@disabled = true
|
|
|
|
end
|
|
|
|
|
|
|
|
def self.enable
|
|
|
|
@disabled = false
|
|
|
|
end
|
|
|
|
|
2013-02-06 03:16:51 +08:00
|
|
|
# We don't observe rate limits in test mode
|
|
|
|
def self.disabled?
|
2017-12-04 18:23:11 +08:00
|
|
|
@disabled
|
2013-02-06 03:16:51 +08:00
|
|
|
end
|
|
|
|
|
2017-12-04 18:23:11 +08:00
|
|
|
# Only used in test, only clears current namespace, does not clear globals
|
2015-01-30 00:44:51 +08:00
|
|
|
def self.clear_all!
|
2019-12-03 17:05:53 +08:00
|
|
|
Discourse.redis.delete_prefixed(RateLimiter.key_prefix)
|
2015-01-30 00:44:51 +08:00
|
|
|
end
|
|
|
|
|
2017-12-11 14:21:00 +08:00
|
|
|
def self.clear_all_global!
|
2019-12-03 17:05:53 +08:00
|
|
|
Discourse.redis.without_namespace.keys("GLOBAL::#{key_prefix}*").each do |k|
|
|
|
|
Discourse.redis.without_namespace.del k
|
2017-12-11 14:21:00 +08:00
|
|
|
end
|
|
|
|
end
|
|
|
|
|
2015-09-25 01:52:32 +08:00
|
|
|
def build_key(type)
|
|
|
|
"#{RateLimiter.key_prefix}:#{@user && @user.id}:#{type}"
|
|
|
|
end
|
|
|
|
|
FEATURE: Apply rate limits per user instead of IP for trusted users (#14706)
Currently, Discourse rate limits all incoming requests by the IP address they
originate from regardless of the user making the request. This can be
frustrating if there are multiple users using Discourse simultaneously while
sharing the same IP address (e.g. employees in an office).
This commit implements a new feature to make Discourse apply rate limits by
user id rather than IP address for users at or higher than the configured trust
level (1 is the default).
For example, let's say a Discourse instance is configured to allow 200 requests
per minute per IP address, and we have 10 users at trust level 4 using
Discourse simultaneously from the same IP address. Before this feature, the 10
users could only make a total of 200 requests per minute before they got rate
limited. But with the new feature, each user is allowed to make 200 requests
per minute because the rate limits are applied on user id rather than the IP
address.
The minimum trust level for applying user-id-based rate limits can be
configured by the `skip_per_ip_rate_limit_trust_level` global setting. The
default is 1, but it can be changed by either adding the
`DISCOURSE_SKIP_PER_IP_RATE_LIMIT_TRUST_LEVEL` environment variable with the
desired value to your `app.yml`, or changing the setting's value in the
`discourse.conf` file.
Requests made with API keys are still rate limited by IP address and the
relevant global settings that control API keys rate limits.
Before this commit, Discourse's auth cookie (`_t`) was simply a 32 characters
string that Discourse used to lookup the current user from the database and the
cookie contained no additional information about the user. However, we had to
change the cookie content in this commit so we could identify the user from the
cookie without making a database query before the rate limits logic and avoid
introducing a bottleneck on busy sites.
Besides the 32 characters auth token, the cookie now includes the user id,
trust level and the cookie's generation date, and we encrypt/sign the cookie to
prevent tampering.
Internal ticket number: t54739.
2021-11-18 04:27:30 +08:00
|
|
|
def initialize(user, type, max, secs, global: false, aggressive: false, error_code: nil)
|
2013-02-06 03:16:51 +08:00
|
|
|
@user = user
|
2015-09-25 01:52:32 +08:00
|
|
|
@type = type
|
|
|
|
@key = build_key(type)
|
2013-02-06 03:16:51 +08:00
|
|
|
@max = max
|
|
|
|
@secs = secs
|
2017-12-11 14:21:00 +08:00
|
|
|
@global = global
|
2020-11-05 13:36:17 +08:00
|
|
|
@aggressive = aggressive
|
FEATURE: Apply rate limits per user instead of IP for trusted users (#14706)
Currently, Discourse rate limits all incoming requests by the IP address they
originate from regardless of the user making the request. This can be
frustrating if there are multiple users using Discourse simultaneously while
sharing the same IP address (e.g. employees in an office).
This commit implements a new feature to make Discourse apply rate limits by
user id rather than IP address for users at or higher than the configured trust
level (1 is the default).
For example, let's say a Discourse instance is configured to allow 200 requests
per minute per IP address, and we have 10 users at trust level 4 using
Discourse simultaneously from the same IP address. Before this feature, the 10
users could only make a total of 200 requests per minute before they got rate
limited. But with the new feature, each user is allowed to make 200 requests
per minute because the rate limits are applied on user id rather than the IP
address.
The minimum trust level for applying user-id-based rate limits can be
configured by the `skip_per_ip_rate_limit_trust_level` global setting. The
default is 1, but it can be changed by either adding the
`DISCOURSE_SKIP_PER_IP_RATE_LIMIT_TRUST_LEVEL` environment variable with the
desired value to your `app.yml`, or changing the setting's value in the
`discourse.conf` file.
Requests made with API keys are still rate limited by IP address and the
relevant global settings that control API keys rate limits.
Before this commit, Discourse's auth cookie (`_t`) was simply a 32 characters
string that Discourse used to lookup the current user from the database and the
cookie contained no additional information about the user. However, we had to
change the cookie content in this commit so we could identify the user from the
cookie without making a database query before the rate limits logic and avoid
introducing a bottleneck on busy sites.
Besides the 32 characters auth token, the cookie now includes the user id,
trust level and the cookie's generation date, and we encrypt/sign the cookie to
prevent tampering.
Internal ticket number: t54739.
2021-11-18 04:27:30 +08:00
|
|
|
@error_code = error_code
|
2013-02-06 03:16:51 +08:00
|
|
|
end
|
|
|
|
|
|
|
|
def clear!
|
2017-12-04 18:23:11 +08:00
|
|
|
redis.del(prefixed_key)
|
2013-02-06 03:16:51 +08:00
|
|
|
end
|
|
|
|
|
|
|
|
def can_perform?
|
2013-05-24 08:18:59 +08:00
|
|
|
rate_unlimited? || is_under_limit?
|
2013-02-06 03:16:51 +08:00
|
|
|
end
|
|
|
|
|
FEATURE: Apply rate limits per user instead of IP for trusted users (#14706)
Currently, Discourse rate limits all incoming requests by the IP address they
originate from regardless of the user making the request. This can be
frustrating if there are multiple users using Discourse simultaneously while
sharing the same IP address (e.g. employees in an office).
This commit implements a new feature to make Discourse apply rate limits by
user id rather than IP address for users at or higher than the configured trust
level (1 is the default).
For example, let's say a Discourse instance is configured to allow 200 requests
per minute per IP address, and we have 10 users at trust level 4 using
Discourse simultaneously from the same IP address. Before this feature, the 10
users could only make a total of 200 requests per minute before they got rate
limited. But with the new feature, each user is allowed to make 200 requests
per minute because the rate limits are applied on user id rather than the IP
address.
The minimum trust level for applying user-id-based rate limits can be
configured by the `skip_per_ip_rate_limit_trust_level` global setting. The
default is 1, but it can be changed by either adding the
`DISCOURSE_SKIP_PER_IP_RATE_LIMIT_TRUST_LEVEL` environment variable with the
desired value to your `app.yml`, or changing the setting's value in the
`discourse.conf` file.
Requests made with API keys are still rate limited by IP address and the
relevant global settings that control API keys rate limits.
Before this commit, Discourse's auth cookie (`_t`) was simply a 32 characters
string that Discourse used to lookup the current user from the database and the
cookie contained no additional information about the user. However, we had to
change the cookie content in this commit so we could identify the user from the
cookie without making a database query before the rate limits logic and avoid
introducing a bottleneck on busy sites.
Besides the 32 characters auth token, the cookie now includes the user id,
trust level and the cookie's generation date, and we encrypt/sign the cookie to
prevent tampering.
Internal ticket number: t54739.
2021-11-18 04:27:30 +08:00
|
|
|
def seconds_to_wait(now = Time.now.to_i)
|
2021-01-19 17:35:46 +08:00
|
|
|
@secs - age_of_oldest(now)
|
|
|
|
end
|
|
|
|
|
2017-12-04 18:23:11 +08:00
|
|
|
# reloader friendly
|
|
|
|
unless defined? PERFORM_LUA
|
|
|
|
PERFORM_LUA = <<~LUA
|
|
|
|
local now = tonumber(ARGV[1])
|
|
|
|
local secs = tonumber(ARGV[2])
|
|
|
|
local max = tonumber(ARGV[3])
|
2017-12-04 18:19:28 +08:00
|
|
|
|
2017-12-04 18:23:11 +08:00
|
|
|
local key = KEYS[1]
|
2017-12-04 18:19:28 +08:00
|
|
|
|
2017-12-04 18:23:11 +08:00
|
|
|
|
|
|
|
if ((tonumber(redis.call("LLEN", key)) < max) or
|
FIX: sliding window end time in rate limiter (#11691)
If the sliding window size is N seconds, then a moment at the Nth second
should be considered as the moment outside of the sliding window.
Otherwise, if the sliding window is already full, at the Nth second,
a new call wouldn't be allowed, but a time to wait before the next call
would be equal to zero, which is confusing.
In other words, the end of the time range shouldn't be included in the
sliding window.
Let's say we start at the second 0, and the sliding window size is 10
seconds. In the current version of rate limiter, this sliding window will
be considered as a time range [0, 10] (including the end of the range),
which actually is 11 seconds in length.
After this fix, the time range will be considered as [0, 10)
(excluding the end of the range), which is exactly 10 seconds in length.
2021-01-13 02:26:43 +08:00
|
|
|
(now - tonumber(redis.call("LRANGE", key, -1, -1)[1])) >= secs) then
|
2017-12-04 18:23:11 +08:00
|
|
|
redis.call("LPUSH", key, now)
|
|
|
|
redis.call("LTRIM", key, 0, max - 1)
|
|
|
|
redis.call("EXPIRE", key, secs * 2)
|
|
|
|
|
|
|
|
return 1
|
|
|
|
else
|
|
|
|
return 0
|
|
|
|
end
|
|
|
|
LUA
|
|
|
|
|
|
|
|
PERFORM_LUA_SHA = Digest::SHA1.hexdigest(PERFORM_LUA)
|
|
|
|
end
|
|
|
|
|
2020-11-05 13:36:17 +08:00
|
|
|
unless defined? PERFORM_LUA_AGGRESSIVE
|
|
|
|
PERFORM_LUA_AGGRESSIVE = <<~LUA
|
|
|
|
local now = tonumber(ARGV[1])
|
|
|
|
local secs = tonumber(ARGV[2])
|
|
|
|
local max = tonumber(ARGV[3])
|
|
|
|
|
|
|
|
local key = KEYS[1]
|
|
|
|
|
|
|
|
local return_val = 0
|
|
|
|
|
|
|
|
if ((tonumber(redis.call("LLEN", key)) < max) or
|
FIX: sliding window end time in rate limiter (#11691)
If the sliding window size is N seconds, then a moment at the Nth second
should be considered as the moment outside of the sliding window.
Otherwise, if the sliding window is already full, at the Nth second,
a new call wouldn't be allowed, but a time to wait before the next call
would be equal to zero, which is confusing.
In other words, the end of the time range shouldn't be included in the
sliding window.
Let's say we start at the second 0, and the sliding window size is 10
seconds. In the current version of rate limiter, this sliding window will
be considered as a time range [0, 10] (including the end of the range),
which actually is 11 seconds in length.
After this fix, the time range will be considered as [0, 10)
(excluding the end of the range), which is exactly 10 seconds in length.
2021-01-13 02:26:43 +08:00
|
|
|
(now - tonumber(redis.call("LRANGE", key, -1, -1)[1])) >= secs) then
|
2020-11-05 13:36:17 +08:00
|
|
|
return_val = 1
|
|
|
|
else
|
|
|
|
return_val = 0
|
|
|
|
end
|
|
|
|
|
|
|
|
redis.call("LPUSH", key, now)
|
|
|
|
redis.call("LTRIM", key, 0, max - 1)
|
|
|
|
redis.call("EXPIRE", key, secs * 2)
|
|
|
|
|
|
|
|
return return_val
|
|
|
|
LUA
|
|
|
|
|
|
|
|
PERFORM_LUA_AGGRESSIVE_SHA = Digest::SHA1.hexdigest(PERFORM_LUA_AGGRESSIVE)
|
|
|
|
end
|
|
|
|
|
2018-04-18 14:58:40 +08:00
|
|
|
def performed!(raise_error: true)
|
2018-04-25 06:44:07 +08:00
|
|
|
return true if rate_unlimited?
|
2017-12-07 08:48:11 +08:00
|
|
|
now = Time.now.to_i
|
2018-03-01 13:20:42 +08:00
|
|
|
|
2020-11-05 13:36:17 +08:00
|
|
|
if ((max || 0) <= 0) || rate_limiter_allowed?(now)
|
FEATURE: Apply rate limits per user instead of IP for trusted users (#14706)
Currently, Discourse rate limits all incoming requests by the IP address they
originate from regardless of the user making the request. This can be
frustrating if there are multiple users using Discourse simultaneously while
sharing the same IP address (e.g. employees in an office).
This commit implements a new feature to make Discourse apply rate limits by
user id rather than IP address for users at or higher than the configured trust
level (1 is the default).
For example, let's say a Discourse instance is configured to allow 200 requests
per minute per IP address, and we have 10 users at trust level 4 using
Discourse simultaneously from the same IP address. Before this feature, the 10
users could only make a total of 200 requests per minute before they got rate
limited. But with the new feature, each user is allowed to make 200 requests
per minute because the rate limits are applied on user id rather than the IP
address.
The minimum trust level for applying user-id-based rate limits can be
configured by the `skip_per_ip_rate_limit_trust_level` global setting. The
default is 1, but it can be changed by either adding the
`DISCOURSE_SKIP_PER_IP_RATE_LIMIT_TRUST_LEVEL` environment variable with the
desired value to your `app.yml`, or changing the setting's value in the
`discourse.conf` file.
Requests made with API keys are still rate limited by IP address and the
relevant global settings that control API keys rate limits.
Before this commit, Discourse's auth cookie (`_t`) was simply a 32 characters
string that Discourse used to lookup the current user from the database and the
cookie contained no additional information about the user. However, we had to
change the cookie content in this commit so we could identify the user from the
cookie without making a database query before the rate limits logic and avoid
introducing a bottleneck on busy sites.
Besides the 32 characters auth token, the cookie now includes the user id,
trust level and the cookie's generation date, and we encrypt/sign the cookie to
prevent tampering.
Internal ticket number: t54739.
2021-11-18 04:27:30 +08:00
|
|
|
raise RateLimiter::LimitExceeded.new(seconds_to_wait(now), @type, @error_code) if raise_error
|
2018-04-18 14:58:40 +08:00
|
|
|
false
|
|
|
|
else
|
|
|
|
true
|
2017-12-04 15:17:18 +08:00
|
|
|
end
|
2017-12-04 18:23:11 +08:00
|
|
|
rescue Redis::CommandError => e
|
|
|
|
if e.message =~ /READONLY/
|
|
|
|
# TODO,switch to in-memory rate limiter
|
|
|
|
else
|
|
|
|
raise
|
|
|
|
end
|
2013-02-06 03:16:51 +08:00
|
|
|
end
|
|
|
|
|
|
|
|
def rollback!
|
|
|
|
return if RateLimiter.disabled?
|
2017-12-04 18:23:11 +08:00
|
|
|
redis.lpop(prefixed_key)
|
2020-06-11 15:09:12 +08:00
|
|
|
rescue Redis::CommandError => e
|
|
|
|
if e.message =~ /READONLY/
|
|
|
|
# TODO,switch to in-memory rate limiter
|
|
|
|
else
|
|
|
|
raise
|
|
|
|
end
|
2013-02-06 03:16:51 +08:00
|
|
|
end
|
|
|
|
|
2016-03-18 23:17:51 +08:00
|
|
|
def remaining
|
|
|
|
return @max if @user && @user.staff?
|
|
|
|
|
2017-12-04 18:23:11 +08:00
|
|
|
arr = redis.lrange(prefixed_key, 0, @max) || []
|
2017-12-07 08:48:11 +08:00
|
|
|
t0 = Time.now.to_i
|
2016-03-18 23:17:51 +08:00
|
|
|
arr.reject! { |a| (t0 - a.to_i) > @secs }
|
|
|
|
@max - arr.size
|
|
|
|
end
|
|
|
|
|
2013-05-24 08:18:59 +08:00
|
|
|
private
|
|
|
|
|
2020-11-05 13:36:17 +08:00
|
|
|
def rate_limiter_allowed?(now)
|
|
|
|
|
|
|
|
lua, lua_sha = nil
|
|
|
|
if @aggressive
|
|
|
|
lua = PERFORM_LUA_AGGRESSIVE
|
|
|
|
lua_sha = PERFORM_LUA_AGGRESSIVE_SHA
|
|
|
|
else
|
|
|
|
lua = PERFORM_LUA
|
|
|
|
lua_sha = PERFORM_LUA_SHA
|
|
|
|
end
|
|
|
|
|
|
|
|
eval_lua(lua, lua_sha, [prefixed_key], [now, @secs, @max]) == 0
|
|
|
|
end
|
|
|
|
|
2017-12-04 18:23:11 +08:00
|
|
|
def prefixed_key
|
|
|
|
if @global
|
|
|
|
"GLOBAL::#{key}"
|
|
|
|
else
|
2019-12-03 17:05:53 +08:00
|
|
|
Discourse.redis.namespace_key(key)
|
2017-12-04 18:23:11 +08:00
|
|
|
end
|
|
|
|
end
|
|
|
|
|
|
|
|
def redis
|
2019-12-03 17:05:53 +08:00
|
|
|
Discourse.redis.without_namespace
|
2017-12-04 18:23:11 +08:00
|
|
|
end
|
|
|
|
|
2021-01-13 04:09:15 +08:00
|
|
|
def age_of_oldest(now)
|
2013-05-26 03:37:28 +08:00
|
|
|
# age of oldest event in buffer, in seconds
|
2021-01-13 04:09:15 +08:00
|
|
|
now - redis.lrange(prefixed_key, -1, -1).first.to_i
|
2013-05-26 03:37:28 +08:00
|
|
|
end
|
|
|
|
|
2013-05-24 08:18:59 +08:00
|
|
|
def is_under_limit?
|
2021-01-13 04:09:15 +08:00
|
|
|
now = Time.now.to_i
|
|
|
|
|
2015-01-30 00:44:51 +08:00
|
|
|
# number of events in buffer less than max allowed? OR
|
2017-12-04 18:23:11 +08:00
|
|
|
(redis.llen(prefixed_key) < @max) ||
|
2021-05-21 09:43:47 +08:00
|
|
|
# age bigger than sliding window size?
|
2021-01-13 04:09:15 +08:00
|
|
|
(age_of_oldest(now) >= @secs)
|
2013-05-24 08:18:59 +08:00
|
|
|
end
|
|
|
|
|
|
|
|
def rate_unlimited?
|
2014-08-18 08:55:30 +08:00
|
|
|
!!(RateLimiter.disabled? || (@user && @user.staff?))
|
2013-05-24 08:18:59 +08:00
|
|
|
end
|
2017-12-04 18:23:11 +08:00
|
|
|
|
|
|
|
def eval_lua(lua, sha, keys, args)
|
|
|
|
redis.evalsha(sha, keys, args)
|
|
|
|
rescue Redis::CommandError => e
|
|
|
|
if e.to_s =~ /^NOSCRIPT/
|
|
|
|
redis.eval(lua, keys, args)
|
|
|
|
else
|
|
|
|
raise
|
|
|
|
end
|
|
|
|
end
|
2013-02-06 03:16:51 +08:00
|
|
|
end
|