Guo Xiang Tan
eaa7527933
DEV: Switch posts:inline_uploads scope to be more generic.
2019-06-14 11:53:34 +08:00
Guo Xiang Tan
9bce3d5e65
DEV: Multisite support for rake posts:inline_uploads.
2019-06-13 16:13:45 +08:00
Guo Xiang Tan
d49c193289
FIX: posts:inline_uploads rake task needs to bypass bump.
2019-06-12 09:13:31 +08:00
Guo Xiang Tan
864222372f
Remove incorrectly committed byebug.
...
Follow up to 49b921147a
2019-06-11 16:33:49 +08:00
Guo Xiang Tan
49b921147a
FIX: Undefined method in posts:inline_uploads.
2019-06-11 16:28:41 +08:00
Guo Xiang Tan
1cf3bf016c
Make rubocop happy.
2019-06-10 09:49:02 +08:00
Guo Xiang Tan
29259b46ae
DEV: Verbose mode for posts:inline_uploads rake task.
2019-06-10 08:59:11 +08:00
Gerhard Schlager
fb121815a5
DEV: Allow rebaking oncooked posts of single site
2019-06-04 15:47:36 +02:00
Guo Xiang Tan
1991af2abb
DEV: Switch InlineUploads to a regexp based implementation.
2019-06-04 15:54:25 +08:00
Guo Xiang Tan
d93e5fb00d
DEV: Class that converts MD with old attachment links to new MD.
2019-06-04 15:54:25 +08:00
Arpit Jalan
af20d616de
FEATURE: new rake task to invalidate broken images in post
2019-06-04 12:40:51 +05:30
Arpit Jalan
388433ec06
DEV: remove unused method (rake task deleted in 6a274fd)
2019-05-31 11:16:10 +05:30
Guo Xiang Tan
6a274fd2cd
DEV: Remove redundant rake task.
...
Onceoff job created in 028121b95b
2019-05-31 10:05:13 +08:00
Sam Saffron
9a232e1a0a
FEATURE: use GIVE_UP=1 to inform rake posts:missing_uploads you are done
...
We need this give up for cases where uploads can not be recovered
This also improves the recovery routines
2019-05-28 14:44:59 +10:00
Sam Saffron
c780d36ddc
DEV: Correct posts:recover_uploads_from_index
...
Was creating a blank index file by mistake
2019-05-28 12:39:39 +10:00
Sam Saffron
9ce06d53d1
DEV: improve upload recovery rake task
...
- Better error handling
- Automatically generates index file if missing
- Improved output
2019-05-24 13:36:03 +10:00
Guo Xiang Tan
791241e0b2
DEV: Fix rake task to delete old export topics.
2019-05-24 11:20:53 +08:00
Sam Saffron
76f7e90e23
FEATURE: rake posts:recover_uploads_from_index improvements
...
We now recover more smartly based on failed db remaps as well.
2019-05-24 12:38:38 +10:00
Sam Saffron
1059aafc55
FEATURE: rake task for recovery of uploads from an index file
...
`rake posts:recover_uploads_from_index`
Searches through all missing uploads in the cluster, if it finds one it
tries to find it in the "upload index file" and creates a new upload for
it.
2019-05-23 17:08:22 +10:00
Sam Saffron
a5ce9cb470
FEATURE: fix_relative_upload_links now multisite safety
...
This also finds `<img src="/uploads/xyz` HTML images in raw and corrects
them. Also handles some cross multisite recovery and provides better output
2019-05-23 15:09:16 +10:00
Sam Saffron
96e58125a7
FEATURE: rename task to posts:destroy_old_data_exports
...
Previously we were only catching one type of data export, the new job will
catch every csv export we have
Job is pretty safe as it filters on system user id / pm with a particular
slug
2019-05-23 15:02:56 +10:00
Sam Saffron
954293655f
FEATURE: rake posts:destroy_old_user_data_exports
...
Historically we would keep the user data export posts around but delete
the uploads.
This leaves a lot of broken uploads in the system.
This rake task allows us to clean up old mess.
2019-05-23 11:11:37 +10:00
Sam Saffron
a9af77d650
PERF: ensure priority for rake rebake tasks is ultra low
...
Flooding the default queue with lots of jobs is never a good thing. This
reduces the risk of sidekiq going into a bad state post rebake
2019-05-22 10:31:49 +10:00
Sam Saffron
d6daa60d0a
FEATURE: multisite support for posts:rebake_uncooked_posts
2019-05-22 10:04:54 +10:00
Sam Saffron
73f178a634
FEATURE: posts:rebake_uncooked_posts to look at mismatching baked_version
...
also amends flagging onebox updates to set baked_version to nil
2019-05-22 09:43:31 +10:00
Sam Saffron
4f296608da
FEATURE: add uploads:s3_migration_status for looking at current status
...
Also a few minor cleanups and better progress reporting
2019-05-22 09:00:32 +10:00
Vinoth Kannan
338b927f95
DEV: include local uploads while finding missing uploads
...
in rake task 'posts:missing_uploads'
2019-05-21 22:07:08 +05:30
Sam Saffron
f4d4f7871e
FEATURE: make posts:missing_uploads multisite friendly
...
Previously this rake job would only run on a single site which is a bit
misleading
This also adds `VERBOSE=1 rake posts:missing_uploads` that will provide a
full report of missing uploads
2019-05-21 12:45:51 +10:00
Vinoth Kannan
9f43852609
FIX: remove the unwanted code.
2019-05-20 19:10:21 +05:30
Sam Saffron
30990006a9
DEV: enable frozen string literal on all files
...
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.
Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
Vinoth Kannan
87cd4701b8
FEATURE: option to skip posts with ignored missing uploads
2019-05-09 05:11:15 +05:30
Vinoth Kannan
e8fafbc123
List and restore missing post uploads from S3 inventory.
2019-05-04 01:16:20 +05:30
Vinoth Kannan
c448e014fe
Recover missing old scheme uploads from tombstone directory too.
2019-04-29 11:33:57 +05:30
Vinoth Kannan
8698a10920
Don't raise error if a record already exists.
2019-04-25 21:48:39 +05:30
Vinoth Kannan
a65ce81e25
Don't check same upload url twice in the rake task
2019-04-25 20:06:10 +05:30
Vinoth Kannan
1724c27713
FIX: reload the 'post' model to retrive raw field value.
2019-04-25 02:09:27 +05:30
Vinoth Kannan
aed683390c
FIX: Don't treat 'upload_patterns' as constant to make the rake task compatible with multisite
2019-04-25 02:06:20 +05:30
Vinoth Kannan
b0bb1d95a5
trying to recover old scheme uploads on 'posts:missing_uploads' rake task
2019-04-25 01:16:18 +05:30
Vinoth Kannan
3dae12dec1
reduce unlinked count from missing post upload count
2019-04-19 21:53:00 +05:30
Vinoth Kannan
4fcefc3615
add unlinked missing post upload count in stat
2019-04-19 20:39:21 +05:30
Vinoth Kannan
ba6369edc5
Add more missing upload stats in rake task
2019-04-19 20:14:40 +05:30
Vinoth Kannan
48cedf3ebe
fix: retrieve posts in order by id.
...
so we can find starting and ending id of the batch.
2019-04-19 18:30:59 +05:30
Vinoth Kannan
ce8d30f667
PERF: use 'delete_all' method instead of 'destroy_all'
2019-04-19 18:00:01 +05:30
Vinoth Kannan
5de483a1b9
PERF: Very post upload's existence by preloaded upload sha1s array
2019-04-19 17:27:16 +05:30
Vinoth Kannan
8d5c900142
DEV: add unique missing uploads index in post custom fields
...
https://review.discourse.org/t/feature-send-missing-post-uploads-stat-to-prometheus/2609/6?u=vinothkannans
2019-04-10 18:09:35 +05:30
Vinoth Kannan
d0fe42e2ef
FIX: should look through posts for image markdown
...
Downloaded onebox images only included in the cooked HTML content. So we have to check 'post.cooked' instead of 'raw'. bfdd0fe64c
2019-04-10 13:52:35 +05:30
Vinoth Kannan
4878ee95e5
DEV: optimize sql query to narrow down the filtering of post with uploads
2019-04-10 12:58:10 +05:30
Vinoth Kannan
bfdd0fe64c
FIX: empty the missing list on each post loop
2019-04-10 11:20:26 +05:30
Vinoth Kannan
914ada1c74
DEV: convert scheduled job EnsurePostUploadsExistence into a rake task
2019-04-09 02:07:35 +05:30
Gerhard Schlager
341836eb42
Fix the rake task and importer instead
2018-10-17 16:48:09 +02:00
