discourse/app/services
Martin Brennan e4350bb966
FEATURE: Direct S3 multipart uploads for backups (#14736)
This PR introduces a new `enable_experimental_backup_uploads` site setting (default false and hidden), which when enabled alongside `enable_direct_s3_uploads` will allow for direct S3 multipart uploads of backup .tar.gz files.

To make multipart external uploads work with both the S3BackupStore and the S3Store, I've had to move several methods out of S3Store and into S3Helper, including:

* presigned_url
* create_multipart
* abort_multipart
* complete_multipart
* presign_multipart_part
* list_multipart_parts

Then, S3Store and S3BackupStore either delegate directly to S3Helper or have their own special methods to call S3Helper for these methods. FileStore.temporary_upload_path has also removed its dependence on upload_path, and can now be used interchangeably between the stores. A similar change was made in the frontend as well, moving the multipart related JS code out of ComposerUppyUpload and into a mixin of its own, so it can also be used by UppyUploadMixin.

Some changes to ExternalUploadManager had to be made here as well. The backup direct uploads do not need an Upload record made for them in the database, so they can be moved to their final S3 resting place when completing the multipart upload.

This changeset is not perfect; it introduces some special cases in UploadController to handle backups that was previously in BackupController, because UploadController is where the multipart routes are located. A subsequent pull request will pull these routes into a module or some other sharing pattern, along with hooks, so the backup controller and the upload controller (and any future controllers that may need them) can include these routes in a nicer way.
2021-11-11 08:25:31 +10:00
..
spam_rule FIX: use allowlist and blocklist terminology (#10209) 2020-07-27 10:23:54 +10:00
anonymous_shadow_creator.rb FIX: ensures shadow has last_posted_at before comparing to site setting (#10374) 2020-08-05 13:20:51 +02:00
badge_granter.rb FEATURE: Add option to grant badge multiple times to users using Bulk Award (#13571) 2021-07-15 05:53:26 +03:00
color_scheme_revisor.rb FEATURE: User selectable color schemes (#10544) 2020-08-28 10:36:52 -04:00
destroy_task.rb FIX: do not send rejection emails to auto-deleted reviewable users (#12160) 2021-02-22 18:37:47 +05:30
email_settings_exception_handler.rb FEATURE: Improve group email settings UI (#13083) 2021-05-28 09:28:18 +10:00
email_settings_validator.rb FEATURE: Improve group email settings UI (#13083) 2021-05-28 09:28:18 +10:00
email_style_updater.rb FEATURE: support SCSS in custom email style 2019-10-23 15:42:37 -04:00
external_upload_manager.rb FEATURE: Direct S3 multipart uploads for backups (#14736) 2021-11-11 08:25:31 +10:00
group_action_logger.rb DEV: remove calls to guardian from GroupActionLogger (#13835) 2021-07-28 15:04:04 +04:00
group_mentions_updater.rb FIX: Mentions updater should work regardless of .notify 2020-02-18 16:02:26 -05:00
group_message.rb FEATURE: remove duplicated messages about new advices (#14319) 2021-09-15 08:59:25 +10:00
handle_chunk_upload.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
heat_settings_updater.rb FIX: round the calculated heat values 2019-06-06 15:44:55 -04:00
inline_uploads.rb PERF: Reduce work when external sources are allowed in InlineUploads (#14594) 2021-10-13 18:59:05 +02:00
notification_consolidator.rb DEV: handle all notification consolidations in new 'NotificationConsolidator' class. 2019-12-05 14:36:06 +05:30
notification_emailer.rb FEATURE: Send an email notification when a post is approved. (#12665) 2021-04-12 12:08:23 -03:00
post_action_notifier.rb FEATURE: Add setting to disable notifications for topic tags edits (#14794) 2021-11-02 13:53:21 -04:00
post_alerter.rb DEV: Plugin instance method for push_notification_filters (#14787) 2021-11-03 12:21:33 -05:00
post_owner_changer.rb FIX: Changing the post owner didn't update the reply_to_user_id of replies (#13862) 2021-07-27 20:49:08 +02:00
push_notification_pusher.rb DEV: Allow tag to be passed in payload to PushNotificationPusher (#14810) 2021-11-03 14:18:48 -05:00
random_topic_selector.rb FIX: Use Discourse.system_user when we need a placeholder admin (#9781) 2020-06-24 15:51:30 +10:00
search_indexer.rb FIX: Reduce input of to_tsvector to follow limits (#13806) 2021-07-28 18:25:14 +03:00
site_settings_task.rb DEV: Fix rubocop issues (#14715) 2021-10-27 11:39:28 +03:00
staff_action_logger.rb FIX: Update only passed custom fields (#14357) 2021-09-17 13:37:56 +03:00
themes_install_task.rb DEV: Add an option to skip a theme update from the themes:install task. (#12905) 2021-04-30 09:31:41 -07:00
topic_status_updater.rb FIX: Auto close topic from category settings based on topic created_at (#12082) 2021-02-17 07:51:39 +10:00
topic_timestamp_changer.rb FIX: when updating timestamps on topic set a correct bump date (#13746) 2021-07-16 11:56:51 +04:00
tracked_topics_updater.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
trust_level_granter.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
user_action_manager.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
user_activator.rb FEATURE: Allow using invites when DiscourseConnect SSO is enabled (#12419) 2021-03-19 10:20:10 +10:00
user_anonymizer.rb FIX: Destroy associated user api keys when making a user anonymous. (#11760) 2021-01-25 11:07:22 -03:00
user_authenticator.rb FEATURE: Allow invites redemption with Omniauth providers. 2021-03-09 09:27:18 +08:00
user_destroyer.rb FIX: perform agree_and_keep action only if possible. (#13967) 2021-09-06 11:41:44 +05:30
user_merger.rb DEV: don't merge email address if target user is not human. (#13915) 2021-08-03 10:04:35 +05:30
user_notification_renderer.rb DEV: Upgrade Rails to 6.1.3.1 (#12688) 2021-04-21 12:36:32 +03:00
user_notification_schedule_processor.rb FEATURE: Create notification schedule to automatically set do not disturb time (#11665) 2021-01-20 10:31:52 -06:00
user_silencer.rb FIX: Skip sending PM email for user silence (#12240) 2021-03-02 09:18:09 +10:00
user_updater.rb FEATURE: save local date to calendar (#14486) 2021-10-06 14:11:52 +11:00
username_changer.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
username_checker_service.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
wildcard_domain_checker.rb SECURITY: vulnerability in WildcardUrlChecker 2019-12-13 09:29:09 -05:00
wildcard_url_checker.rb FIX: Allow any protocol in wildcard url checker (#8651) 2020-01-02 16:03:13 +00:00
word_watcher.rb FIX: Add word boundaries to replace and tag watched words (#13405) 2021-06-18 18:54:06 +03:00