github-mirror/discourse
2
0
Fork 0
mirror of https://github.com/discourse/discourse.git synced 2024-11-22 08:36:42 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

FIX: Use hidden site setting for batch presign rate limit (#16692)

Browse Source 
This was causing issues on some sites, having the const, because this really is heavily
dependent on upload speed. We request 5-10 URLs at a time with this endpoint; for
a 1.5GB upload with 5mb parts this could mean 60 requests to the server to get all
the part URLs. If the user's upload speed is super fast they may request all 60
batches in a minute, if it is slow they may request 5 batches in a minute.

The other external upload endpoints are not hit as often, so they can stay as constant
values for now. This commit also increases the default to 20 requests/minute.
This commit is contained in:
Martin Brennan 2022-05-10 11:14:26 +10:00 committed by GitHub
parent fbcc35b417
commit 244836ddd4
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
3 changed files with 25 additions and 15 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
config/site_settings.yml
View File

@ -1886,6 +1886,9 @@ rate_limits:
max_allowed_secondary_emails:
default: 10
hidden: true
max_batch_presign_multipart_per_minute:
default: 20
hidden: true
developer:
force_hostname:

12
lib/external_upload_helpers.rb
View File

@ -10,7 +10,6 @@ module ExternalUploadHelpers
PRESIGNED_PUT_RATE_LIMIT_PER_MINUTE = 10
CREATE_MULTIPART_RATE_LIMIT_PER_MINUTE = 10
COMPLETE_MULTIPART_RATE_LIMIT_PER_MINUTE = 10
BATCH_PRESIGN_RATE_LIMIT_PER_MINUTE = 10
included do
before_action :external_store_check, only: [
@ -112,8 +111,17 @@ module ExternalUploadHelpers
part_numbers = params.require(:part_numbers)
unique_identifier = params.require(:unique_identifier)
##
# NOTE: This is configurable by hidden site setting because this really is heavily
# dependent on upload speed. We request 5-10 URLs at a time with this endpoint; for
# a 1.5GB upload with 5mb parts this could mean 60 requests to the server to get all
# the part URLs. If the user's upload speed is super fast they may request all 60
# batches in a minute, if it is slow they may request 5 batches in a minute.
#
# The other external upload endpoints are not hit as often, so they can stay as constant
# values for now.
RateLimiter.new(
current_user, "batch-presign", ExternalUploadHelpers::BATCH_PRESIGN_RATE_LIMIT_PER_MINUTE, 1.minute
current_user, "batch-presign", SiteSetting.max_batch_presign_multipart_per_minute, 1.minute
).performed!
part_numbers = part_numbers.map do |part_number|

25
spec/requests/uploads_controller_spec.rb
View File

@ -1056,23 +1056,22 @@ describe UploadsController do
it "rate limits" do
RateLimiter.enable
RateLimiter.clear_all!
SiteSetting.max_batch_presign_multipart_per_minute = 1
stub_const(ExternalUploadHelpers, "BATCH_PRESIGN_RATE_LIMIT_PER_MINUTE", 1) do
stub_list_multipart_request
post "/uploads/batch-presign-multipart-parts.json", params: {
unique_identifier: external_upload_stub.unique_identifier,
part_numbers: [1, 2, 3]
}
stub_list_multipart_request
post "/uploads/batch-presign-multipart-parts.json", params: {
unique_identifier: external_upload_stub.unique_identifier,
part_numbers: [1, 2, 3]
}
expect(response.status).to eq(200)
expect(response.status).to eq(200)
post "/uploads/batch-presign-multipart-parts.json", params: {
unique_identifier: external_upload_stub.unique_identifier,
part_numbers: [1, 2, 3]
}
post "/uploads/batch-presign-multipart-parts.json", params: {
unique_identifier: external_upload_stub.unique_identifier,
part_numbers: [1, 2, 3]
}
expect(response.status).to eq(429)
end
expect(response.status).to eq(429)
end
end