discourse/lib
Martin Brennan e4350bb966
FEATURE: Direct S3 multipart uploads for backups (#14736)
This PR introduces a new `enable_experimental_backup_uploads` site setting (default false and hidden), which when enabled alongside `enable_direct_s3_uploads` will allow for direct S3 multipart uploads of backup .tar.gz files.

To make multipart external uploads work with both the S3BackupStore and the S3Store, I've had to move several methods out of S3Store and into S3Helper, including:

* presigned_url
* create_multipart
* abort_multipart
* complete_multipart
* presign_multipart_part
* list_multipart_parts

Then, S3Store and S3BackupStore either delegate directly to S3Helper or have their own special methods to call S3Helper for these methods. FileStore.temporary_upload_path has also removed its dependence on upload_path, and can now be used interchangeably between the stores. A similar change was made in the frontend as well, moving the multipart related JS code out of ComposerUppyUpload and into a mixin of its own, so it can also be used by UppyUploadMixin.

Some changes to ExternalUploadManager had to be made here as well. The backup direct uploads do not need an Upload record made for them in the database, so they can be moved to their final S3 resting place when completing the multipart upload.

This changeset is not perfect; it introduces some special cases in UploadController to handle backups that was previously in BackupController, because UploadController is where the multipart routes are located. A subsequent pull request will pull these routes into a module or some other sharing pattern, along with hooks, so the backup controller and the upload controller (and any future controllers that may need them) can include these routes in a nicer way.
2021-11-11 08:25:31 +10:00
..
auth DEV: simplify username suggester (#14531) 2021-10-27 14:41:24 +04:00
autospec DEV: Move chrome binary check into a shared lib (#13451) 2021-06-21 13:28:48 +10:00
backup_restore FEATURE: Direct S3 multipart uploads for backups (#14736) 2021-11-11 08:25:31 +10:00
common_passwords DEV: s/\$redis/Discourse\.redis (#8431) 2019-12-03 10:05:53 +01:00
compression FIX: Decompressing lots of small files triggered error 2020-01-09 15:11:31 +01:00
content_security_policy FIX: Set CSP base-uri to self (#13654) 2021-07-07 09:43:48 -04:00
demon DEV: IMAP debugging improvements (#11784) 2021-01-21 11:37:47 +10:00
discourse_dev DEV: only use the records that are auto populated by the task. (#14360) 2021-09-17 09:47:32 +05:30
email FIX: Remove List-Post email header (#14554) 2021-10-11 20:57:42 +03:00
emoji DEV: replaces huge generated emoji list by a simpler regex (#11053) 2021-04-22 08:43:06 +02:00
faker DEV: move discourse_dev gem to the core. (#13360) 2021-06-14 20:34:44 +05:30
file_store FEATURE: Direct S3 multipart uploads for backups (#14736) 2021-11-11 08:25:31 +10:00
freedom_patches FIX: Ensure id sequences are not reset during db:migrate (#14184) 2021-08-30 12:31:22 +01:00
generators/rails DEV: removes plugin generator (#14101) 2021-08-20 11:29:06 +02:00
guardian FIX: Show right message when permanently deleting topic (#14717) 2021-10-26 18:31:15 +03:00
highlight_js
i18n DEV: Fix rubocop issues (#14715) 2021-10-27 11:39:28 +03:00
imap DEV: Use Rails query builder (#14759) 2021-10-28 15:14:23 +03:00
import
import_export FEATURE: include user custom fields in base exporter (#14690) 2021-10-22 10:02:56 -07:00
javascripts DEV: Allow transformed values to be used in all widget hbs statements (#13331) 2021-06-08 16:46:07 +01:00
middleware DEV: Fix rubocop issues (#14715) 2021-10-27 11:39:28 +03:00
migration DEV: Promote old post-deploy migrations to pre-deploy migrations (#13477) 2021-06-22 16:02:24 +01:00
onebox FIX: Display Instagram Oneboxes in an iframe (#14789) 2021-11-02 14:34:51 -04:00
plugin DEV: Plugin instance method for push_notification_filters (#14787) 2021-11-03 12:21:33 -05:00
pretty_text DEV: replaces huge generated emoji list by a simpler regex (#11053) 2021-04-22 08:43:06 +02:00
rate_limiter FEATURE: Allow admins to permanently delete posts and topics (#14406) 2021-10-13 12:53:23 +03:00
reviewable DEV: APIs for plugin to add custom reviewable confirm modal (#12246) 2021-03-02 10:28:27 -06:00
scheduler
search FIX: remove superfluous spaces from CJK blurbs (#12629) 2021-04-12 12:46:42 +10:00
seed_data FIX: Support Ruby 3 keyword arguments 2021-10-05 11:25:00 -04:00
sidekiq DEV: s/\$redis/Discourse\.redis (#8431) 2019-12-03 10:05:53 +01:00
site_settings DEV: Fix rubocop issues (#14715) 2021-10-27 11:39:28 +03:00
stylesheet FIX: Order outputted theme stylesheets (#14133) 2021-08-25 09:37:07 +08:00
svg_sprite DEV: Remove a few unused icons (#14696) 2021-10-22 12:03:58 -04:00
tasks FIX: LocalJumpError in rake tasks (#14874) 2021-11-10 16:53:55 +01:00
theme_store FIX: Remove whitespace from theme git versions (#12857) 2021-04-27 17:45:54 +01:00
topic_query FIX: Exclude PMs that user sent to themselves. (#14496) 2021-10-04 11:55:35 +08:00
turbo_tests DEV: Upgrade Rails to 6.1.3.1 (#12688) 2021-04-21 12:36:32 +03:00
validators FIX: Support Ruby 3 keyword arguments 2021-10-05 11:25:00 -04:00
webauthn SECURITY: 2FA with U2F / TOTP 2020-01-15 11:27:12 +01:00
wizard FEATURE: Enable auto dark mode on new instances (#14208) 2021-09-02 14:55:38 -04:00
admin_confirmation.rb DEV: Upgrade Redis to 4.2.1. 2020-06-15 10:05:22 +08:00
admin_constraint.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
admin_user_index_query.rb DEV: Standardize table sorting verbiage (#9757) 2020-05-14 20:10:59 -06:00
age_words.rb
archetype.rb
auth.rb DEV: remove instagram login site settings and auth classes. (#11073) 2020-10-30 09:09:56 +05:30
backup_restore.rb DEV: Upgrade Rails to 6.1.3.1 (#12688) 2021-04-21 12:36:32 +03:00
badge_posts_view_manager.rb DEV: stop freezing frozen strings 2020-04-30 16:48:53 +10:00
badge_queries.rb FIX: Don't grant sharing badges to users who don't exist (#13851) 2021-07-27 16:32:59 +10:00
base62.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
bookmark_manager.rb FEATURE: Topic-level bookmarks (#14353) 2021-09-21 08:45:47 +10:00
bookmark_query.rb FEATURE: Go to last unread for topic-level bookmark links (#14396) 2021-09-21 13:49:56 +10:00
bookmark_reminder_notification_handler.rb DEV: Ignore reminder_type for bookmarks (#14349) 2021-09-16 09:56:54 +10:00
browser_detection.rb FIX: Detect DiscourseHub user agent. 2019-08-09 11:58:15 +03:00
cache.rb DEV: Fix rubocop issues (#14715) 2021-10-27 11:39:28 +03:00
canonical_url.rb FEATURE: default canonical URL (#9738) 2020-05-12 09:13:20 +10:00
category_badge.rb
chrome_installed_checker.rb DEV: Move chrome binary check into a shared lib (#13451) 2021-06-21 13:28:48 +10:00
comment_migration.rb
composer_messages_finder.rb FEATURE: Make allow_uploaded_avatars accept TL (#14091) 2021-08-24 10:46:28 +03:00
configurable_urls.rb Replace base_uri with base_path (#10879) 2020-10-09 12:51:24 +01:00
content_buffer.rb
content_security_policy.rb PERF: Eager load Theme associations in Stylesheet Manager. 2021-06-21 11:06:58 +08:00
cooked_post_processor.rb FIX: remove 'crawl_images' site setting (#14646) 2021-10-19 17:12:29 +05:30
crawler_detection.rb FEATURE: Implement browser update in crawler view (#12448) 2021-03-22 19:41:42 +02:00
csrf_token_verifier.rb DEV: Provide method for auth plugins to generate a CSRF token 2019-08-13 01:13:08 +01:00
current_user.rb
custom_renderer.rb
custom_setting_providers.rb
db_helper.rb DEV: Upgrade Rails to 6.1.3.1 (#12688) 2021-04-21 12:36:32 +03:00
directory_helper.rb
discourse_cookie_store.rb
discourse_dev.rb DEV: move discourse_dev gem to the core. (#13360) 2021-06-14 20:34:44 +05:30
discourse_diff.rb Escape values of HTML attributes 2021-08-10 10:25:15 -04:00
discourse_event.rb DEV: Plugin API to add directory columns (#13440) 2021-06-22 13:00:04 -05:00
discourse_hub.rb DEV: s/\$redis/Discourse\.redis (#8431) 2019-12-03 10:05:53 +01:00
discourse_ip_info.rb FIX: MaxMind DB file not downloading correctly 2020-01-05 22:08:13 +11:00
discourse_js_processor.rb DEV: Add support for class properties in babel (#13189) 2021-05-27 16:13:14 -04:00
discourse_logstash_logger.rb FIX: Use 'hostname' when Discourse.os_hostname is not available 2020-02-18 13:37:39 +02:00
discourse_plugin_registry.rb DEV: Plugin instance method for push_notification_filters (#14787) 2021-11-03 12:21:33 -05:00
discourse_redis.rb DEV: Pass kwargs to the redis gem when calling methods/commands that we don't wrap (#14530) 2021-10-06 17:42:04 +03:00
discourse_tagging.rb FIX: Show required tags to staff by default and override limit (#13242) 2021-06-02 12:43:34 -04:00
discourse_updates.rb FIX: Regression introduced in #14715 (#14842) 2021-11-09 17:20:09 +11:00
discourse.rb FEATURE: add Unseen view (#13977) 2021-08-10 18:30:34 +04:00
disk_space.rb FIX: correct upload statistics report for external storage 2020-02-20 15:15:53 +11:00
distributed_cache.rb PERF: Defer setting of distributed cache in more spots. 2021-06-04 09:13:18 +08:00
distributed_memoizer.rb DEV: Replace Time.new with Time.now (#9142) 2020-03-09 17:37:49 +01:00
distributed_mutex.rb FIX: Off-by-one error setting the distributed mutex key to expire 2020-02-03 14:54:50 +00:00
edit_rate_limiter.rb FEATURE: Increase daily edit limits proportionally to trust level (#13090) 2021-05-19 13:57:21 +04:00
email_backup_token.rb DEV: s/\$redis/Discourse\.redis (#8431) 2019-12-03 10:05:53 +01:00
email_cook.rb PERF: Avoid lookbehinds when replacing links in imported emails (#11931) 2021-02-02 17:34:00 +01:00
email_updater.rb FEATURE: add maximum limit for secondary emails (#12599) 2021-04-05 20:31:42 +05:30
email.rb FIX: Replace use of regular expression (#12838) 2021-04-27 08:48:51 +03:00
encodings.rb
enum_site_setting.rb
enum.rb
excerpt_parser.rb DEV: Remove dead code 2021-05-31 10:22:50 +08:00
feed_element_installer.rb
feed_item_accessor.rb
file_helper.rb DEV: Remove warnings on console (#14608) 2021-10-14 23:17:47 +02:00
filter_best_posts.rb
final_destination.rb FIX: Canonical URLs may be relative (#14825) 2021-11-05 14:20:14 -03:00
flag_query.rb FIX: use allowlist and blocklist terminology (#10209) 2020-07-27 10:23:54 +10:00
flag_settings.rb
gaps.rb
global_path.rb
guardian.rb FIX: Allow bulk invites to be used with DiscourseConnect (#14862) 2021-11-09 17:43:23 +00:00
has_errors.rb
hijack.rb DEV: Add more debugging context to onebox generation 2020-10-22 12:50:22 +08:00
homepage_constraint.rb
html_prettify.rb DEV: stop freezing frozen strings 2020-04-30 16:48:53 +10:00
html_to_markdown.rb FIX: Hoisting linebreaks shouldn't fail for HTML5 elements (#14364) 2021-09-17 10:41:34 +02:00
http_language_parser.rb FIX: Include resolved locale in anonymous cache key (#10289) 2020-07-22 18:00:07 +01:00
image_sizer.rb
import_export.rb FEATURE: Rake task to export groups (#9450) 2020-04-17 14:59:54 -07:00
inline_oneboxer.rb FEATURE: check blocked_onebox_domains setting for inline oneboxes (#11944) 2021-02-03 21:45:22 +05:30
introduction_updater.rb FIX: replace default welcome topic post with new value from wizard 2020-04-01 15:42:45 -04:00
ip_addr.rb
js_locale_helper.rb DEV: move discourse_dev gem to the core. (#13360) 2021-06-14 20:34:44 +05:30
json_error.rb
letter_avatar.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
markdown_linker.rb
mem_info.rb
message_bus_diags.rb PERF: avoid shelling to get hostname aggressively 2020-02-18 15:13:19 +11:00
method_profiler.rb DEV: Add output_sql_to_stderr! to MethodProfiler (#12445) 2021-03-19 17:48:30 +10:00
mini_sql_multisite_connection.rb DEV: upgrade mini_sql (#12465) 2021-03-24 08:48:04 +11:00
mobile_detection.rb
new_post_manager.rb SECURITY: Escape watched word in error message (#14434) 2021-09-24 11:55:15 +03:00
new_post_result.rb Support for custom messages and redirects when creating posts (#8434) 2019-11-29 09:30:54 -05:00
notification_levels.rb
onebox.rb DEV: Absorb onebox gem into core (#12979) 2021-05-26 15:11:35 +05:30
oneboxer.rb FEATURE: Onebox can match engines based on the content_type (#13876) 2021-07-30 13:36:30 -04:00
onpdiff.rb FIX: Show a correct diff when editing consecutive paragraphs (#8177) 2019-10-11 03:50:37 -04:00
pbkdf2.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
permalink_constraint.rb
pinned_check.rb
plain_text_to_markdown.rb DEV: stop freezing frozen strings 2020-04-30 16:48:53 +10:00
plugin_gem.rb FIX: ensure plugin's gems are in the gem path (#12727) 2021-04-16 10:21:39 +02:00
plugin_initialization_guard.rb DEV: Print backtrace of error when plugin fails to initialize. 2020-06-09 10:25:43 +08:00
post_action_creator.rb PERF: Update like count in visible posts without an extra GET per like (#14869) 2021-11-10 13:22:26 -03:00
post_action_destroyer.rb FIX: Unlike own posts on ownership transfer (#10446) 2020-08-19 09:21:02 -06:00
post_action_result.rb
post_creator.rb FIX: Don't publish PM archive events to acting user. (#14291) 2021-09-10 09:20:50 +08:00
post_destroyer.rb FIX: Skip post validations for system revisions when author deletes post (#14824) 2021-11-08 09:33:41 +08:00
post_jobs_enqueuer.rb FIX: Do not send emails to mailing_list_mode subscribers for PMs (#14159) 2021-08-26 15:16:35 +10:00
post_locker.rb
post_merger.rb FEATURE: TL4 & category moderators can merge posts (#12843) 2021-04-27 18:24:27 +02:00
post_revisor.rb FIX: Make PostRevisor more consistent (#14841) 2021-11-09 16:29:37 +02:00
presence_channel.rb DEV: Make PresenceChannel timeout configurable per-channel (#14697) 2021-10-22 16:24:55 +01:00
pretty_text.rb FIX: Do not check for duplicate links in Onebox (#13345) 2021-06-18 18:55:24 +03:00
promotion.rb FIX: check if BasicBadge is enabled for TL1 welcome message (#13983) 2021-08-11 08:39:25 +10:00
quote_comparer.rb FEATURE: Nokogumbo (#9577) 2020-05-05 13:46:57 +10:00
rake_helpers.rb Try fix upload_spec flakys and remove logging from tasks/uploads_spec 2020-02-18 15:08:58 +10:00
rate_limiter.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
read_only_header.rb
retrieve_title.rb FIX: increase chunk size to fetch title tag correctly (#14144) 2021-09-03 13:15:58 +05:30
route_format.rb
route_matcher.rb REFACTOR: Introduce RouteMatcher class 2020-10-19 10:40:55 +01:00
rtl.rb Check site default locale if Rtl class is initialized without a user (#8417) 2019-11-26 15:01:37 -05:00
s3_cors_rulesets.rb DEV: Improve s3:ensure_cors_rules logging (#14832) 2021-11-08 11:44:12 +10:00
s3_helper.rb FEATURE: Direct S3 multipart uploads for backups (#14736) 2021-11-11 08:25:31 +10:00
s3_inventory.rb FIX: increase inventory lag for s3 to 2 days (#11606) 2020-12-30 16:05:42 +11:00
score_calculator.rb
screening_model.rb
search.rb FIX: Use the same mode for chinese search when indexing and querying. (#14780) 2021-11-01 10:14:47 +08:00
secure_session.rb DEV: s/\$redis/Discourse\.redis (#8431) 2019-12-03 10:05:53 +01:00
shrink_uploaded_image.rb DEV: Improve script/downsize_uploads.rb (#13508) 2021-06-24 00:09:40 +02:00
single_sign_on_provider.rb FEATURE: Rename 'Discourse SSO' to DiscourseConnect (#11978) 2021-02-08 10:04:33 +00:00
single_sign_on.rb FIX: log proper error message when SSO nonce verification fails (#14077) 2021-08-18 18:44:12 +05:30
site_icon_manager.rb PERF: Defer setting of distributed cache in more spots. 2021-06-04 09:13:18 +08:00
site_setting_extension.rb DEV: Sanitize HTML admin inputs (#14681) 2021-10-27 11:33:07 -03:00
slug.rb FIX: Make category slugs lowercase (#11277) 2021-01-12 17:28:33 +02:00
socket_server.rb
spam_handler.rb FIX: use allowlist and blocklist terminology (#10209) 2020-07-27 10:23:54 +10:00
sql_builder.rb
staff_constraint.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
staff_message_format.rb
suggested_topics_builder.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
system_message.rb DEV: Add option to send system message to groups (#12256) 2021-03-02 18:51:50 +01:00
temporary_db.rb DEV: Add annotate rake tasks, and enforce via GitHub actions 2021-07-06 10:11:06 +01:00
temporary_redis.rb DEV: Introduce TemporaryRedis and unset DISCOURSE_* env vars in the themes:isolated_test rake task (#13401) 2021-06-23 07:38:43 +03:00
text_cleaner.rb FEATURE: Correctly convert topic title to uppercase and lowercase for Turkish default locale (#13115) 2021-05-24 18:13:30 +10:00
text_sentinel.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
theme_javascript_compiler.rb FEATURE: Allow theme tests to be run in production (take 2) (#12845) 2021-04-28 23:12:08 +03:00
theme_modifier_helper.rb Code review comments. 2021-06-21 11:06:58 +08:00
theme_settings_manager.rb DEV: use upload id to save in theme setting instead of URL. (#14341) 2021-09-16 07:58:53 +05:30
theme_settings_parser.rb DEV: support json_schema in theme settings (#12294) 2021-03-10 20:15:04 -05:00
theme_translation_manager.rb
theme_translation_parser.rb
timeline_lookup.rb FIX: ensures timeline_lookup includes last tuple (#11829) 2021-01-25 11:30:59 +01:00
topic_creator.rb FIX: include new tags in validation if user can create one. (#14744) 2021-10-28 11:59:46 +05:30
topic_list_responder.rb DEV: Refactor draft attributes for CategoryList and TopicList. 2020-07-24 10:11:30 +08:00
topic_publisher.rb FIX: Use destroy_all instead of delete_all for shared drafts 2020-03-05 11:13:43 -08:00
topic_query_params.rb FIX: Build correct topic list filter (#11473) 2020-12-11 14:20:48 +02:00
topic_query.rb PERF: Use a subquery when excluding a tag from topic query. (#14577) 2021-10-13 09:20:56 +11:00
topic_retriever.rb FEATURE: Fallback to system users when creating new TopicEmbed (#12386) 2021-03-15 11:58:53 -03:00
topic_subtype.rb
topic_upload_security_manager.rb DEV: Add security_last_changed_at and security_last_changed_reason to uploads (#11860) 2021-01-29 09:03:44 +10:00
topic_view.rb FIX: rename action_code_href to action_code_path (#14834) 2021-11-08 14:32:17 +11:00
topics_bulk_action.rb FIX: Don't publish PM archive events to acting user. (#14291) 2021-09-10 09:20:50 +08:00
trust_level.rb FIX: Don't store translated trust level names in anonymous cache (#13224) 2021-06-01 22:11:48 +02:00
turbo_tests.rb FIX: Made turbo_rspec display errors in shared groups correctly 2019-08-29 12:41:14 +01:00
twitter_api.rb DEV: Update rubocop-discourse from 2.3.2 to 2.4.0 (#11079) 2020-10-30 15:04:29 +01:00
unicorn_logstash_patch.rb DEV: Fix lint. 2020-07-21 15:55:03 +08:00
unread.rb FEATURE: Add last visit indication to topic view page. (#13471) 2021-07-05 14:17:31 +08:00
upload_creator.rb FEATURE: Direct S3 multipart uploads for backups (#14736) 2021-11-11 08:25:31 +10:00
upload_fixer.rb
upload_markdown.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
upload_recovery.rb FIX: Support Ruby 3 keyword arguments 2021-10-05 11:25:00 -04:00
upload_security.rb FIX: Do not mark badge image uploads as secure (#13193) 2021-05-28 12:35:52 +10:00
url_helper.rb FEATURE: revert disallowing putting URLs in titles for TL0 users (#13970) 2021-08-06 20:07:42 +04:00
user_lookup.rb REVERT "FIX: do not show private group flair on user avatars" (#13991) 2021-08-10 17:25:11 +05:30
user_name_suggester.rb DEV: simplify username suggester (#14531) 2021-10-27 14:41:24 +04:00
vary_header.rb FIX: Include the Vary:Accept header on all Accept-based responses (#14647) 2021-10-25 12:53:50 +01:00
version.rb Version bump to v2.8.0.beta7 (#14667) 2021-10-20 17:29:41 -04:00
webauthn.rb DEV: stop freezing frozen strings 2020-04-30 16:48:53 +10:00
wizard.rb DEV: Allow plugins to add wizard steps after specific steps (#9315) 2020-04-01 08:36:50 -05:00
zeitwerk_config.rb FIX: Better and more secure validation of periods for TopicQuery 2021-07-23 14:24:44 -04:00