Commit Graph

34 Commits

Author SHA1 Message Date
Matt Palmer
a98d2a8086
FEATURE: allow S3 ACLs to be disabled (#21769)
AWS recommends running buckets without ACLs, and to use resource policies to manage access control instead.
This is not a bad idea, because S3 ACLs are whack, and while resource policies are also whack, they're a more constrained form of whack.
Further, some compliance regimes get antsy if you don't go with the vendor's recommended settings, and arguing that you need to enable ACLs on a bucket just to store images in there is more hassle than it's worth.
The new site setting (s3_use_acls) cannot be disabled when secure
uploads is enabled -- the latter relies on private ACLs for security
at this point in time. We may want to reexamine this in future.
2023-06-06 15:47:40 +10:00
David Taylor
6417173082
DEV: Apply syntax_tree formatting to lib/* 2023-01-09 12:10:19 +00:00
Yasuo Honda
dbbfad7ed0 FIX: Support Ruby 3 keyword arguments 2021-10-05 11:25:00 -04:00
Jarek Radosz
891987a284
DEV: Recover missing files of existing uploads (#10757)
UploadRecovery only worked on missing Upload records. Now it also works with existing ones that have an invalid_etag status.

The specs (first that test the S3 path) are a bit of stub-a-palooza, but that's how much this class interacts with the the outside world. 🤷‍♂️
2020-10-01 14:54:45 +02:00
Jarek Radosz
188d224303
DEV: Restore UploadRecovery specs (#10543)
Originally disabled in 0c0192e. Upload specs now use separate paths for each spec worker.

Fixes an issue in UploadRecovery#recover_from_local – it didn't take into account the testing infix (e.g. test_0) in the uploads/tombstone paths.
2020-08-27 15:57:10 +02:00
Krzysztof Kotlarek
e0d9232259
FIX: use allowlist and blocklist terminology (#10209)
This is a PR of the renaming whitelist to allowlist and blacklist to the blocklist.
2020-07-27 10:23:54 +10:00
Michael Brown
dbe0111822 FEATURE: allow UploadRecovery to be run on a single post (#8094) 2019-10-02 14:57:36 +10:00
Vinoth Kannan
a1fc35fdf7 FIX: make uploads recovery compatible with multisite. 2019-08-02 07:08:21 +05:30
Vinoth Kannan
e44d56e4d2 DEV: raise error only when 'STOP_ON_ERROR' env variable is available. 2019-08-01 23:54:06 +05:30
Sam Saffron
e8799f0ba4 DEV: improve uploads:recover job so it stores a map of old to new sha
Previous attempt created broken images
2019-05-22 15:51:09 +10:00
Sam Saffron
ebcb571de7 FIX: allow upload recovery to recover uploads with sha mismatch
Filename on disk may mismatch sha of file in some old 1X setups. This will
attempt to recover file even if sha1 mismatches. We had an old bug that
caused this.

This also adds `uploads:fix_relative_upload_links` which attempts to replace
urls of the format `/upload/default/...` with `upload://`
2019-05-22 15:24:36 +10:00
Sam Saffron
30990006a9 DEV: enable frozen string literal on all files
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.

Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
Guo Xiang Tan
24347ace10 FIX: Properly associate user_profiles background urls via upload id.
`Upload#url` is more likely and can change from time to time. When it
does changes, we don't want to have to look through multiple tables to
ensure that the URLs are all up to date. Instead, we simply associate
uploads properly to `UserProfile` so that it does not have to replicate
the URLs in the table.
2019-05-02 14:58:24 +08:00
Vinoth Kannan
d0fe42e2ef FIX: should look through posts for image markdown
Downloaded onebox images only included in the cooked HTML content. So we have to check 'post.cooked' instead of 'raw'. bfdd0fe64c
2019-04-10 13:52:35 +05:30
Vinoth Kannan
f5ac01253a FIX: use new changed method name and remove whitespaces 2019-04-09 02:25:26 +05:30
Guo Xiang Tan
6a95d3fded FIX: UploadRecovery should look through posts for img src and bbcode. 2019-04-02 11:41:00 +08:00
Guo Xiang Tan
1f9799c979 FIX: Handle more cases in UploadRecovery. 2019-04-02 11:29:26 +08:00
Guo Xiang Tan
ac52d8d3c7 Fix UploadRecovery from S3 fails with bucket name containing sub-folder. 2018-10-01 19:03:02 +08:00
Guo Xiang Tan
a3dbe3685e Fix the build. 2018-10-01 16:42:38 +08:00
Guo Xiang Tan
de85bb0a39 FIX: Don't update user_profile URLs unless upload is persisted. 2018-10-01 14:21:39 +08:00
Guo Xiang Tan
e262a08350 Add UploadRecovery#recover_user_profile_backgrounds. 2018-10-01 10:51:54 +08:00
Guo Xiang Tan
1a64b3a487 FIX: Don't try to recover an invalid sha1. 2018-09-20 14:21:57 +08:00
Guo Xiang Tan
293cf600f0 FIX: Use a tmp file in UploadRecovery for local store. 2018-09-19 15:47:15 +08:00
Guo Xiang Tan
bc7f58191e FIX: UploadRecovery should look at links too. 2018-09-19 11:52:57 +08:00
Guo Xiang Tan
4a92c5b2d6 UploadRecovery should recover attachments too. 2018-09-19 10:44:36 +08:00
Guo Xiang Tan
529e4018f0 DEV: Avoid using send and make the method public instead. 2018-09-13 16:34:32 +08:00
Guo Xiang Tan
0a06b3d977 Accept custom AR relation for UploadRecovery. 2018-09-13 16:33:14 +08:00
Guo Xiang Tan
5eb65ad612 FIX: Do not try to recover invalid Upload#short_url in UploadRecovery. 2018-09-13 13:59:17 +08:00
Guo Xiang Tan
d99dd840e4 Add basic test case for UploadRecovery. 2018-09-13 13:26:23 +08:00
Guo Xiang Tan
a3b3b0810d Rescue errors when running dry run for UploadRecovery. 2018-09-13 11:57:51 +08:00
Guo Xiang Tan
4dc25ad201 Fix s3 recovery from tombstone in UploadRecovery. 2018-09-13 09:19:45 +08:00
Guo Xiang Tan
24c55bd613 Add dry run option to UploadRecovery. 2018-09-12 21:53:01 +08:00
Guo Xiang Tan
2cc48cfd06 Fix incorrect variable. 2018-09-12 02:06:14 -07:00
Guo Xiang Tan
c053f8ccf6 New rake task uploads:recover. 2018-09-12 01:52:30 -07:00