Commit Graph

47 Commits

Author SHA1 Message Date
Loïc Guitaut
78a3efa710 SECURITY: Rate limit the creation of backups 2023-03-16 16:09:08 +01:00
David Taylor
5a003715d3
DEV: Apply syntax_tree formatting to app/* 2023-01-09 14:14:59 +00:00
Martin Brennan
49c49e8ae0
FEATURE: Local chunked uppy backup uploads with a new uploader plugin (#14894)
This takes the uppy chunking algorithm and combines it with some
form submission from resumable.js for parity with the current
backup controller to make local backup uploads work with uppy.
We can then use this to replace the resumable-upload component
and the resumable.js library from our codebase, once stable.

This is disabled by default, so people using local backups will not
be affected. The enable_experimental_backup_uploader site setting
must be enabled for this to work.
2021-11-23 08:45:42 +10:00
Martin Brennan
b96c10a903
DEV: Extract shared external upload routes into controller helper (#14984)
This commit refactors the direct external upload routes (get presigned
put, complete external, create/abort/complete multipart) into a
helper which is then included in both BackupController and the
UploadController. This is done so UploadController doesn't need
strange backup logic added to it, and so each controller implementing
this helper can do their own validation/error handling nicely.

This is a follow up to e4350bb966
2021-11-18 09:17:23 +10:00
Martin Brennan
08e625c446
FIX: Use previous chunk to check if local backup chunk upload complete (#14896)
Uppy and Resumable slice up their chunks differently, which causes a difference
in this algorithm. Let's take a 131.6MB file (137951695 bytes) with a 5MB (5242880 bytes)
chunk size. For resumable, there are 26 chunks, and uppy there are 27. This is
controlled by forceChunkSize in resumable which is false by default. The final
chunk size is 6879695 (chunk size + remainder) whereas in uppy it is 1636815 (just remainder).

This means that the current condition of uploaded_file_size + current_chunk_size >= total_size
is hit twice by uppy, because it uses a more correct number of chunks. This
can be solved for both uppy and resumable by checking the _previous_ chunk
number * chunk_size as the uploaded_file_size.

An example of what is happening before that change, using the current
chunk number to calculate uploaded_file_size.

chunk 26: resumable: uploaded_file_size (26 * 5242880) + current_chunk_size (6879695) = 143194575 >= total_size (137951695) ? YES
chunk 26: uppy: uploaded_file_size (26 * 5242880) + current_chunk_size (5242880) = 141557760 >= total_size (137951695) ? YES
chunk 27: uppy: uploaded_file_size (27 * 5242880) + current_chunk_size (1636815) = 143194575 >= total_size (137951695) ? YES

An example of what this looks like after the change, using the previous
chunk number to calculate uploaded_file_size:

chunk 26: resumable: uploaded_file_size (25 * 5242880) + current_chunk_size (6879695) = 137951695 >= total_size (137951695) ? YES
chunk 26: uppy: uploaded_file_size (25 * 5242880) + current_chunk_size (5242880) = 136314880 >= total_size (137951695) ? NO
chunk 27: uppy: uploaded_file_size (26 * 5242880) + current_chunk_size (1636815) = 137951695 >= total_size (137951695) ? YES
2021-11-15 15:08:21 +10:00
Osama Sayegh
45ccadeeeb
DEV: Upgrade Rails to 6.1.3.1 (#12688)
Rails 6.1.3.1 deprecates a few API and has some internal changes that break our tests suite, so this commit fixes all the deprecations and errors and now Discourse should be fully compatible with Rails 6.1.3.1. We also have a new release of the rails_failover gem that's compatible with Rails 6.1.3.1.
2021-04-21 12:36:32 +03:00
Sam Saffron
64b3512084
DEV: use DiskSpace module for all disk space calculations
This normalizes it so we only carry one place for grabbing disk space size

It also normalizes the command made so it uses Discourse.execute_command
which splits off params in a far cleaner way.
2020-02-18 15:13:19 +11:00
Blake Erickson
b73a133bb5 FIX: Prevent scientific notation in free space check (#8473)
It's possibly that when trying to upload a backup the free space check
will output scientific notation resulting in an incorrect "There is not
enough space on disk" error.

The free space check uses the Linux `print` command which could return a
number using scientific notation like `1.60459e+10` and when ruby
converts it to an integer it will have the value of `1` instead of
`16045879296`. Which means even though you have 16GB of free space you
could not upload a 1GB backup file.

This commit uses the `printf` command instead which allows you to
specify that you do not want scientific notation.

I'm not sure why this hasn't been an issue before, but I was
experiencing it locally in development.
2019-12-11 12:57:03 +11:00
Dan Ungureanu
e4f05a0d15
FIX: Call the right method 2019-12-03 14:16:06 +02:00
Krzysztof Kotlarek
427d54b2b0 DEV: Upgrading Discourse to Zeitwerk (#8098)
Zeitwerk simplifies working with dependencies in dev and makes it easier reloading class chains. 

We no longer need to use Rails "require_dependency" anywhere and instead can just use standard 
Ruby patterns to require files.

This is a far reaching change and we expect some followups here.
2019-10-02 14:01:53 +10:00
Gerhard Schlager
651a5b6e40 SECURITY: Validate backup chunk identifier 2019-07-22 08:43:16 +02:00
Sam Saffron
96360a779f DEV: add a few frozen string literals 2019-05-02 16:57:12 +10:00
Gerhard Schlager
dc961fecb9 FIX: Outgoing emails were not disabled after restoring backup 2019-02-25 16:07:24 +01:00
Gerhard Schlager
6a8007e5fb FEATURE: Improve handling of backup storage errors 2019-02-20 15:16:49 +01:00
Guo Xiang Tan
84d4c81a26 FEATURE: Support backup uploads/downloads directly to/from S3.
This reverts commit 3c59106bac.
2018-10-15 09:43:31 +08:00
Guo Xiang Tan
3c59106bac Revert "FEATURE: Support backup uploads/downloads directly to/from S3."
This reverts commit c29a4dddc1.

We're doing a beta bump soon so un-revert this after that is done.
2018-10-11 11:08:23 +08:00
Gerhard Schlager
c29a4dddc1 FEATURE: Support backup uploads/downloads directly to/from S3. 2018-10-11 10:38:43 +08:00
Arpit Jalan
f9ab3848ed FEATURE: support disabling emails for non-staff users 2018-06-07 18:31:08 +05:30
Arpit Jalan
ed4b845930 FIX: render error message when backup download fails 2018-01-05 19:46:43 +05:30
Robin Ward
69a90f31fb FEATURE: Allow Forums to disable the Backups feature 2017-12-21 15:22:04 -05:00
Guo Xiang Tan
97ceebb570 SECURITY: Don't pass email backup token to sidekiq as a parameter.
* This exposes the token in the Sidekiq dashboard which can be
  viewed by an admin and defeats the purpose of using a token
  in the download backup email ink.
2017-12-18 11:25:22 +08:00
Guo Xiang Tan
77d4c4d8dc Fix all the errors to get our tests green on Rails 5.1. 2017-09-25 13:48:58 +08:00
Blake Erickson
80858bae2c FEATURE: further restrict downloading of backups
- send email to logged in admin when they press the "download" button
- show pop-up that email was sent
- create email template
- require a valid token to download backup
2017-03-01 08:28:34 -07:00
Régis Hanol
fbf9172db8 FIX: log backups download/destroy staff action
FIX: clean up junk left by the specs
RENAME: 'backup_operation' to 'backup_create' to match other backup log types
2017-01-16 19:53:31 +01:00
Guo Xiang Tan
515f50e42e FEATURE: Log admin action when readonly mode is changed. 2017-01-12 09:41:02 +08:00
Guo Xiang Tan
cdd550e947 Use a different Redis key when PG failover sets site to readonly mode. 2017-01-11 16:38:49 +08:00
cpradio
2eddeab66b Escape the hyphen 2016-09-16 19:07:46 -04:00
cpradio
0d2d8797b6 FIX: Backup validation wasn't escaping hyphens 2016-09-16 15:20:42 -04:00
Guo Xiang Tan
512922d776 SECURITY: Add filename validation for backup uploads. 2016-09-16 11:58:14 +08:00
Guo Xiang Tan
64858c10fe
FIX: Set a not expiring key for user enabled readonly mode. 2016-06-29 15:10:01 +08:00
Robin Ward
ccf9b70671 When restoring a backup, disable emails.
This prevents accidental sending of emails after a restore before
the admin has had a chance to review everything.
2016-06-24 17:15:15 -04:00
Régis Hanol
f3c868e7bb run the BackupChunksMerger job in 5 seconds 2016-03-03 12:01:13 +01:00
Arpit Jalan
8f62a0caa8 FEATURE: log backup operation 2016-02-27 23:33:18 +05:30
Régis Hanol
96c23d51a2 FIX: don't break the message bus when restoring a backup 2015-08-27 20:02:13 +02:00
Régis Hanol
babbbc06d1 FIX: add support for .tgz and .gz backup files 2015-04-07 15:26:47 +02:00
Sam
497042ddf2 FIX: don't restrict to local filesystem for df check
FIX: check correct directory when looking at backup limits
2015-01-27 08:25:57 +11:00
Régis Hanol
5504622c1b rename export/import in favor of backup/restore for better consistency 2014-10-10 20:04:07 +02:00
Sam
9428ad779f FIX: send content length with backups 2014-09-23 09:25:53 +10:00
Régis Hanol
8a20d05ba5 FEATURE: backup without uploads 2014-08-20 18:53:58 +02:00
Arpit Jalan
727184641e FEATURE: Bulk Invite 2014-06-09 01:43:39 +05:30
Robin Ward
dc1d6decf5 Support for removal of old backups automatically via a site setting 2014-03-12 12:24:35 -04:00
Régis Hanol
ca9f6e9137 BUGFIX: couldn't upload backup on OSX
--B is not a valid df option on OSX
2014-02-28 15:43:50 +01:00
Régis Hanol
70ca6171d1 BUGFIX: fix math & unit when checking disk space 2014-02-27 16:55:12 +01:00
Régis Hanol
043901ef46 FEATURE: warn the user when there is not enough space on disk to upload a backup 2014-02-26 19:38:06 +01:00
Régis Hanol
68a935c36b FEATURE: upload backups 2014-02-22 01:41:01 +01:00
Régis Hanol
b89d328de2 display/preload the logs of the last/current operation 2014-02-13 13:31:14 -08:00
Régis Hanol
3be1b5569a backups controller & specs 2014-02-13 13:31:13 -08:00