github-mirror/discourse
2
0
Fork 0
mirror of https://github.com/discourse/discourse.git synced 2024-11-27 12:13:37 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
35,775 Commits 214 Branches 500 Tags 960 MiB
7c32411881
Commit Graph

150 Commits

Author SHA1 Message Date
Guo Xiang Tan
d82876896e FIX: uploads:migrate_to_s3 broken for GlobalSetting using file provider. 2019-03-11 14:21:35 +08:00
Régis Hanol
ad87b0d662
Make "uploads:recover_from_tombstone" call the "uploads:recover" rake task 2019-03-07 14:15:30 +01:00
Guo Xiang Tan
433f07fcb3 Fix confusing ENV in migrate_to_s3 rake task. 2019-02-22 14:44:46 +08:00
Rishabh
e69634ec3a FIX: Use s3_endpoint in migrate_to_s3 when not using S3 
- overrides :region and uses :endpoint when SiteSetting.s3_endpoint is provided
- Now, we can use the new rake task with DigitalOcean Spaces
- I've tested that it's compatible with/without bucket folder path
- I've tested that it's compatible with S3 and it doesn't break S3 for non-default regions
- follow-up on 97e17fe0
 2019-02-21 15:42:48 +05:30
Vinoth Kannan
0472bd4adc FIX: Remove 'backfill_etags' keyword argument from 'uploads:missing' rake task 
And etags backfilling code is optimized
 2019-02-15 00:34:35 +05:30
Vinoth Kannan
7b5931013a Update rake task to backfill etags from s3 inventory 2019-02-14 05:18:06 +05:30
Guo Xiang Tan
0cf2df3028 Fix remap in migrate_to_s3 rake task. 
The current way of doing the remap only allows to run the rake task
once. Running the rake task more than once will end up badly.
 2019-01-23 15:50:44 +08:00
Guo Xiang Tan
07850994d3 Add ENV to skip multisite prefix when migrating to s3. 2019-01-23 15:19:50 +08:00
Guo Xiang Tan
979d03aa68 Remove s3 bucket check in migrate_to_s3 task. 
Bucket creation is expected to be handled by the user. If the bucket
does not exist, the script will fail anyway.
 2019-01-23 15:04:51 +08:00
Guo Xiang Tan
99cd3ff6ee FIX: migrate_to_s3 task not setting the right content_disposition. 2019-01-23 15:04:47 +08:00
Rishabh
97e17fe084 FIX: Use ENV values instead of 'S3Helper.s3_options' in migrate_to_s3 
This commit makes the rake task operational for all regions for s3. If we declare s3_endpoint as https://s3.amazonaws.com while
creating an instance of Aws::S3::Client, head_bucket fails for all s3 regions apart from us-east-1. The commit manually defines all
parameters for Aws::S3::Client apart from s3_endpoint to bypass this problem make this task usable for AWS S3.

Removing s3_endpoint from the payload means that custom endpoints like Minio/DO Spaces for will not work in the meantime and we'll
have to add support for a custom `s3_endpoint` in the future.

This commit follows up on 60790eb0.
 2019-01-20 20:55:27 +05:30
Rishabh
60790eb006 FIX: Use GlobalSetting values instead of ENV variables in migrate_to_s3 
TIL how GlobalSetting works in sync with environment variables
Also fixes a small bug where bucket value was being used when it could have been nil
 2019-01-16 14:40:38 +05:30
Rishabh
ff8f9dc1c9 FIX: prefix should precede folder path (follow-up on 10fbb07e) 2019-01-15 15:58:19 +05:30
Rishabh
10fbb07e1a FIX: include folder name in prefix for listing files on S3 (follow-up on 3ec38f5a) 
Fix the destination url in remap since it's already a part of s3_base_url
 2019-01-15 15:23:55 +05:30
Régis Hanol
3ec38f5a3b Revert "FIX: migrate_to_s3 rake task with folder path" 
This reverts commit 97fd12e8af.
 2019-01-08 19:44:31 +01:00
Régis Hanol
97fd12e8af
FIX: migrate_to_s3 rake task with folder path 2019-01-08 18:56:18 +01:00
Rishabh
f181e9cc08
FIX: Add compatibility for bucket folder paths in migrate_to_s3 task (#6855) 
* FIX: Add compatibility for bucket folder paths in migrate_to_s3 task
* Refactor bucket_name split logic into S3Helper
 2019-01-08 20:04:48 +05:30
Rishabh
efc481d9c0 DEV: Use puts instead of printing newline (follow up on c5b7bda1) 2019-01-05 01:20:00 +05:30
Rishabh
c5b7bda198 DEV: Show migrate_to_s3 output on a new line 2019-01-04 18:09:54 +05:30
Guo Xiang Tan
9e50813252 FIX: Pass all necessary options in migrate_to_s3 rake task. 2019-01-02 09:11:23 +08:00
Régis Hanol
5381096bfd PERF: new 'migrate_to_s3' rake task 2018-12-26 17:34:49 +01:00
Vinoth Kannan
fd272eee44 FEATURE: Make uploads:missing task compatible with s3 uploads 2018-11-27 00:54:51 +05:30
Vinoth Kannan
bc41057949 minor copy edit 2018-11-20 12:07:56 +05:30
Vinoth Kannan
1a9a2bd5c1 DEV: Report the missing uploads count 2018-11-19 12:06:46 +05:30
Rishabh
4a12cfaecb Remove trailing whitespace for Rubocop 2018-11-13 17:19:26 +05:30
Brian Helba
ea94323766 FIX: 'migrate_from_s3' rake task should respect max sizes (#6598) 
Rather than hardcode a maximum size of 20MB for uploads migrated from S3, the task should use site settings for this value.
 2018-11-13 12:27:38 +01:00
Guo Xiang Tan
14ff47f6f1 Fix typo. 2018-11-08 16:42:12 +08:00
Guo Xiang Tan
7290145641 PERF: Speed up migrate_to_s3 rake task. 
* Prioritizes non-image uploads
* Does one remap per upload instead of 3 remaps previously
* Every 100 uploads migrated, do 2 remaps which fixes broken
  URLs
* Exclude email_logs table from remap
 2018-11-08 16:39:56 +08:00
Guo Xiang Tan
0232a3b5e5 PERF: Exclude tables when remapping in migrate_to_s3 rake task. 2018-11-08 12:37:36 +08:00
Guo Xiang Tan
918633aa12 FIX: upload:migrate_to_s3 rake task not remapping properly. 2018-10-10 15:09:21 +08:00
Guo Xiang Tan
24c55bd613 Add dry run option to UploadRecovery. 2018-09-12 21:53:01 +08:00
Guo Xiang Tan
c053f8ccf6 New rake task uploads:recover. 2018-09-12 01:52:30 -07:00
Guo Xiang Tan
6d01e0aa04 DEV: Print the error class in uploads:list_posts_with_broken_images. 2018-09-12 01:06:51 -07:00
Guo Xiang Tan
94ff428571 Pass the right value to rake task. 2018-09-10 20:07:28 +08:00
Guo Xiang Tan
4a966c639d DEV: Update uploads:list_posts_with_broken_images to recover from tombstone. 2018-09-10 17:01:41 +08:00
Guo Xiang Tan
68572b8afc Print error messages on why upload fails to save. 2018-09-10 16:02:13 +08:00
Guo Xiang Tan
0aca80e92a Fixes to uploads:list_posts_with_broken_images. 2018-09-10 15:16:29 +08:00
Guo Xiang Tan
8496537590 Add RECOVER_FROM_S3 to uploads:list_posts_with_broken_images rake task. 2018-09-10 15:14:30 +08:00
Guo Xiang Tan
72834f19ff DEV: Add rake tasks to list posts with broken images. 2018-09-05 16:54:15 +08:00
Guo Xiang Tan
33f4aa2835 Fix migrate_to_s3 rake task, File#close! is not a valid method. 2018-08-16 13:26:58 +08:00
Sam
1fc2597626 better error handling for upload extension fixer 2018-08-10 11:28:22 +10:00
Guo Xiang Tan
0879610ffd Add missing require in uploads:fix_incorrect_extensions. 2018-08-08 15:39:37 +08:00
Guo Xiang Tan
17047806b9 Add a rake task to fix uploads with wrong extension. 2018-08-08 13:15:17 +08:00
Régis Hanol
14bbd5d167 FIX: use the right URL when downloading the file from S3 2018-08-02 10:04:41 +02:00
Régis Hanol
0f1137dafa FIX: 'migrate_from_s3' rake task wasn't handling short urls 2018-08-01 22:58:46 +02:00
Sam
5f64fd0a21 DEV: remove exec_sql and replace with mini_sql 
Introduce new patterns for direct sql that are safe and fast.

MiniSql is not prone to memory bloat that can happen with direct PG usage.
It also has an extremely fast materializer and very a convenient API

- DB.exec(sql, *params) => runs sql returns row count
- DB.query(sql, *params) => runs sql returns usable objects (not a hash)
- DB.query_hash(sql, *params) => runs sql returns an array of hashes
- DB.query_single(sql, *params) => runs sql and returns a flat one dimensional array
- DB.build(sql) => returns a sql builder

See more at: https://github.com/discourse/mini_sql
 2018-06-19 16:13:36 +10:00
Régis Hanol
9e7d5a3cdf FIX: 'uploads:recover_from_tombstone' rake task wasn't restoring attachments 2018-03-28 15:51:47 +02:00
Guo Xiang Tan
5782a23a69 Allow uploads:missing to skip optimized image check. 2018-03-12 20:08:53 +08:00
Régis Hanol
cbf4204ecc FIX: uploads:migrate_from_s3 wasn't destroying records before creating new ones 2018-01-24 13:22:07 +01:00
Régis Hanol
28d432263e rework the uploads:migrate_from_s3 rake task 2018-01-23 18:48:06 +01:00
Sam
83a346cd68 FIX: migrate_from_s3 rake task not working 2018-01-22 11:42:09 +11:00
Régis Hanol
b6c9d7a9d2 move requires to top of file 2018-01-19 17:51:42 +01:00
Régis Hanol
2d49250107 FIX: support recovering images using 'upload://' scheme 2018-01-19 15:40:17 +01:00
Sam
70bb2aa426 FEATURE: allow specifying s3 config via globals 
This refactors handling of s3 so it can be specified via GlobalSetting

This means that in a multisite environment you can configure s3 uploads
without actual sites knowing credentials in s3

It is a critical setting for situations where assets are mirrored to s3.
 2017-10-06 16:20:01 +11:00
Guo Xiang Tan
5012d46cbd Add rubocop to our build. (#5004) 2017-07-28 10:20:09 +09:00
Régis Hanol
b7e6620109 FIX: 'uploads:migrate_from_s3' rake task 2017-06-23 10:15:11 +02:00
Régis Hanol
94cee37edd FIX: uploads:recover_from_tombstone rake task 2017-06-22 17:00:05 +02:00
Robin Ward
cdbe027c1c Refactor FileHelper to use keyword arguments. 2017-05-24 13:54:26 -04:00
Régis Hanol
9641d2413d REFACTOR: upload workflow creation into UploadCreator 
- Automatically convert large-ish PNG/BMP to JPEG
- Updated fast_image to latest version
 2017-05-11 00:16:57 +02:00
Guo Xiang Tan
b5d6d3c326 FIX: Make recover from tombstone script work for multisite. 2016-09-22 12:28:36 +08:00
Guo Xiang Tan
333d6f9f10 More improvements to restore from tombstone script. 2016-09-22 10:39:39 +08:00
Guo Xiang Tan
1d4451db5d Improve restore from tombstone script by not having to loop through all posts. 2016-09-21 18:01:59 +08:00
Guo Xiang Tan
b62b6ba12c Improve restore from tombstone rake task. 2016-09-21 17:47:27 +08:00
Guo Xiang Tan
3af5b19002 Add rake task to restore missing uploads from tombstone. 2016-09-21 16:56:32 +08:00
Guo Xiang Tan
e4b75f604c FIX: Make clean up upload script a safer task to run. 2016-09-05 10:06:02 +08:00
Guo Xiang Tan
df8d24734a Regenerate missing optimized images for all dbs by default. 2016-09-02 13:06:31 +08:00
Guo Xiang Tan
e4883725e6 Script to analyze uploads on a site. 2016-09-02 12:07:06 +08:00
Guo Xiang Tan
82324db71a Allow uploads:missing Rake task to be run for a single site. 2016-09-02 10:22:39 +08:00
Guo Xiang Tan
2349ab5a51 puts not putc. 2016-08-29 10:30:10 +08:00
Guo Xiang Tan
e57ed5223a Fix rake task. 2016-08-26 15:46:42 +08:00
Guo Xiang Tan
2448db4716 Prioritize user avatars when regenerating missing optimized images. 2016-08-25 18:29:52 +08:00
Guo Xiang Tan
f19232646f Rescue all errors when backfilling shas for uploads. 2016-08-23 15:05:37 +08:00
Guo Xiang Tan
574b8fdf41 Allow backfilling of shas to complete. 2016-08-23 10:34:32 +08:00
Guo Xiang Tan
5222022842 Use another rake task to regenerate optimized images for multisite. 2016-08-19 15:52:09 +08:00
Régis Hanol
9971e62254 FIX: migrate_from_s3 rake task 2016-07-25 12:12:10 +02:00
Régis Hanol
557e59c28e check the uploads doesn't already exists before copying it 2016-04-13 16:33:00 +02:00
Régis Hanol
8e4113a468 add multisite support for the 'uploads:gather' rake task 2016-04-12 16:00:25 +02:00
Régis Hanol
71fe66c8c2 'uploads:gather' should only work on local uploads 2016-04-11 21:17:33 +02:00
Régis Hanol
90dfc96206 new 'uploads:gather' rake task 2016-04-11 20:42:40 +02:00
uppfinnarn
ca8c5f7382 migrate_to_s3: check for missing local files 
If an image is already on S3, but not available locally (eg. if they're already migrated), `path` is nil. This crashes the task, because File.exists?(nil) is not valid.
 2015-11-16 11:39:40 +01:00
Régis Hanol
189cb3ff12 FEATURE: move migrate_to_new_scheme into a background job 
- new hidden site setting 'migrate_to_new_scheme' (defaults to false)
- new rake tasks to toggle migration to new scheme
- FIX: migrate_to_new_scheme also works with CDN
- PERF: improve perf of the DbHelper.remap method
- REFACTOR: UrlHelper is now a class
 2015-06-12 12:07:57 +02:00
Régis Hanol
c1cf602de2 FIX: uploads:backfill_shas rake task 2015-06-10 17:19:58 +02:00
Régis Hanol
6000a37ff1 new 'uploads:migrate_to_new_pattern' task 2015-06-05 18:37:34 +02:00
Régis Hanol
a5d93c6705 FIX: undefined method 'max_file_size_kb' 2015-05-26 16:39:41 +02:00
Régis Hanol
bb0c2813ac FEATURE: generate (avatar) thumbnails in a background task 
FIX: keep the "uploading..." indicator until the server replies via the MessageBus
FIX: text was disapearing when uploading an avatar

PERF: always use a region for S3 (defaults to 'us-east-1')
FEATURE: ApplyCDN middleware when using S3
FIX: use the same pattern to store files on S3 and locally
PERF: keep a local cache of uploads when generating thumbnails
FEATURE: migrate_to_s3 rake task
 2015-05-25 17:59:00 +02:00
Régis Hanol
5e527178b9 FEATURE: new 'uploads:migrate_to_new_pattern' rake task 2015-05-19 12:31:51 +02:00
Régis Hanol
54b63bfdbf remove ALL useless 'order by' 2015-05-12 09:37:48 +02:00
Régis Hanol
232b5387d0 remove useless 'order by' 2015-05-12 09:36:46 +02:00
Régis Hanol
b489384b59 FIX: uploads rake tasks to support subfolders 2015-05-12 09:28:43 +02:00
Régis Hanol
95a57ce3e4 properly fix missing uploads by redownloading them 2015-05-11 19:07:39 +02:00
Régis Hanol
48f4589c6f try to fix missing downloads by re-downloading them 2015-05-11 17:03:48 +02:00
Régis Hanol
6d83526bf1 add support for multisite to the 'uploads:regenerate_missing_optimized' rake task 2015-05-11 16:19:16 +02:00
Régis Hanol
27a9b5ee13 only deal with local uploads 2015-05-11 15:41:52 +02:00
Régis Hanol
ca89d16946 FEATURE: new 'uploads:regenerate_missing_optimized' images rake task 2015-05-11 12:59:50 +02:00
Sam
996ab1ae17 FEATURE: task to find all missing local images 2015-05-11 10:30:39 +10:00
Régis Hanol
6b85d5582c FIX: 'uploads:migrate_from_s3' rake task 2015-03-18 18:23:55 +01:00
Régis Hanol
652cc3efba FEATURE: new rake task to clean up uploads & thumbnails 2014-09-29 18:31:53 +02:00
Régis Hanol
87c0c5d6f9 TRIVIAL: we might run the task after disabling s3 storage 2014-06-25 23:16:04 +02:00
Régis Hanol
bdb6032c11 FEATURE: rake task to migrate uploads from S3 to local storage 2014-06-24 15:35:15 +02:00
Régis Hanol
f94e4ffdcb added 'uploads:backfill_shas' rake task 2013-06-17 02:49:33 +02:00