109 Commits

Author	SHA1	Message	Date
Sam Saffron	7632fe0b58	PERF: rake posts:rebake_uncooked_posts runs inline ... Running this inline makes more sense otherwise there is extreme risk in saturating sidekiq queue. This also reworks ordering and selection so we double check if a post needs rebaking prior to rebaking, this unlocks the ability to run this rake task from multiple consoles.	2019-08-13 10:28:42 +10:00
Guo Xiang Tan	a1a61fcd2d	DEV: Use File#file? in posts:missing_uploads rake task.	2019-08-01 10:59:34 +08:00
Rishabh	dcb47d902b	REFACTOR: Rename SiteSetting.disable_edit_notifications to disable_system_edit_notifications (#7958) ... * REFACTOR: Rename SiteSetting.disable_edit_notifications to disable_system_edit_notifications - The older name could cause some confusion because the setting does not disable all edit notifications, only system ones. * FIX: Add frozen_string_literal: true in the migration * DEV: Deprecate 'disable_edit_notifications'	2019-07-31 20:20:41 +05:30
Guo Xiang Tan	8a64b0c8e8	Revert "DEV: Remove unused kwarg and properly check for local missing uploads." ... This reverts commit 97769f3d02. The code is confusing but this change is quite risky. Defer for now until we can look at it properly.	2019-07-29 14:35:34 +08:00
Guo Xiang Tan	97769f3d02	DEV: Remove unused kwarg and properly check for local missing uploads.	2019-07-29 14:21:06 +08:00
Guo Xiang Tan	e9dc48ccae	DEV: Fix multisite support for posts:missing_uploads.	2019-07-29 10:56:11 +08:00
Sam Saffron	b9a2a4d904	DEV: Avoid emoji for symbols ... Rake task needs to work across a wide variety of terminal emulators. Requiring an emoji font is superflous.	2019-07-05 09:41:15 +10:00
Guo Xiang Tan	2b19e2acc8	Fix typo in a0aeabbb94.	2019-06-25 15:18:57 +08:00
Guo Xiang Tan	a0aeabbb94	DEV: Safer replacement for old scheme uploads for posts:missing_uploads.	2019-06-25 11:34:26 +08:00
Guo Xiang Tan	f3c13d80c9	DEV: Don't copy file when trying to recover missing uploads.	2019-06-24 16:32:59 +08:00
Guo Xiang Tan	2cc5561504	DEV: Correct description in posts:missing_uploads rake task.	2019-06-24 14:57:43 +08:00
Guo Xiang Tan	eaa7527933	DEV: Switch posts:inline_uploads scope to be more generic.	2019-06-14 11:53:34 +08:00
Guo Xiang Tan	9bce3d5e65	DEV: Multisite support for rake posts:inline_uploads.	2019-06-13 16:13:45 +08:00
Guo Xiang Tan	d49c193289	FIX: posts:inline_uploads rake task needs to bypass bump.	2019-06-12 09:13:31 +08:00
Guo Xiang Tan	864222372f	Remove incorrectly committed byebug. ... Follow up to 49b921147a.	2019-06-11 16:33:49 +08:00
Guo Xiang Tan	49b921147a	FIX: Undefined method in posts:inline_uploads.	2019-06-11 16:28:41 +08:00
Guo Xiang Tan	1cf3bf016c	Make rubocop happy.	2019-06-10 09:49:02 +08:00
Guo Xiang Tan	29259b46ae	DEV: Verbose mode for posts:inline_uploads rake task.	2019-06-10 08:59:11 +08:00
Gerhard Schlager	fb121815a5	DEV: Allow rebaking oncooked posts of single site	2019-06-04 15:47:36 +02:00
Guo Xiang Tan	1991af2abb	DEV: Switch InlineUploads to a regexp based implementation.	2019-06-04 15:54:25 +08:00
Guo Xiang Tan	d93e5fb00d	DEV: Class that converts MD with old attachment links to new MD.	2019-06-04 15:54:25 +08:00
Arpit Jalan	af20d616de	FEATURE: new rake task to invalidate broken images in post	2019-06-04 12:40:51 +05:30
Arpit Jalan	388433ec06	DEV: remove unused method (rake task deleted in 6a274fd)	2019-05-31 11:16:10 +05:30
Guo Xiang Tan	6a274fd2cd	DEV: Remove redundant rake task. ... Onceoff job created in 028121b95b	2019-05-31 10:05:13 +08:00
Sam Saffron	9a232e1a0a	FEATURE: use GIVE_UP=1 to inform rake posts:missing_uploads you are done ... We need this give up for cases where uploads can not be recovered This also improves the recovery routines	2019-05-28 14:44:59 +10:00
Sam Saffron	c780d36ddc	DEV: Correct posts:recover_uploads_from_index ... Was creating a blank index file by mistake	2019-05-28 12:39:39 +10:00
Sam Saffron	9ce06d53d1	DEV: improve upload recovery rake task ... - Better error handling - Automatically generates index file if missing - Improved output	2019-05-24 13:36:03 +10:00
Guo Xiang Tan	791241e0b2	DEV: Fix rake task to delete old export topics.	2019-05-24 11:20:53 +08:00
Sam Saffron	76f7e90e23	FEATURE: rake posts:recover_uploads_from_index improvements ... We now recover more smartly based on failed db remaps as well.	2019-05-24 12:38:38 +10:00
Sam Saffron	1059aafc55	FEATURE: rake task for recovery of uploads from an index file ... `rake posts:recover_uploads_from_index` Searches through all missing uploads in the cluster, if it finds one it tries to find it in the "upload index file" and creates a new upload for it.	2019-05-23 17:08:22 +10:00
Sam Saffron	a5ce9cb470	FEATURE: fix_relative_upload_links now multisite safety ... This also finds `<img src="/uploads/xyz` HTML images in raw and corrects them. Also handles some cross multisite recovery and provides better output	2019-05-23 15:09:16 +10:00
Sam Saffron	96e58125a7	FEATURE: rename task to posts:destroy_old_data_exports ... Previously we were only catching one type of data export, the new job will catch every csv export we have Job is pretty safe as it filters on system user id / pm with a particular slug	2019-05-23 15:02:56 +10:00
Sam Saffron	954293655f	FEATURE: rake posts:destroy_old_user_data_exports ... Historically we would keep the user data export posts around but delete the uploads. This leaves a lot of broken uploads in the system. This rake task allows us to clean up old mess.	2019-05-23 11:11:37 +10:00
Sam Saffron	a9af77d650	PERF: ensure priority for rake rebake tasks is ultra low ... Flooding the default queue with lots of jobs is never a good thing. This reduces the risk of sidekiq going into a bad state post rebake	2019-05-22 10:31:49 +10:00
Sam Saffron	d6daa60d0a	FEATURE: multisite support for posts:rebake_uncooked_posts	2019-05-22 10:04:54 +10:00
Sam Saffron	73f178a634	FEATURE: posts:rebake_uncooked_posts to look at mismatching baked_version ... also amends flagging onebox updates to set baked_version to nil	2019-05-22 09:43:31 +10:00
Sam Saffron	4f296608da	FEATURE: add uploads:s3_migration_status for looking at current status ... Also a few minor cleanups and better progress reporting	2019-05-22 09:00:32 +10:00
Vinoth Kannan	338b927f95	DEV: include local uploads while finding missing uploads ... in rake task 'posts:missing_uploads'	2019-05-21 22:07:08 +05:30
Sam Saffron	f4d4f7871e	FEATURE: make posts:missing_uploads multisite friendly ... Previously this rake job would only run on a single site which is a bit misleading This also adds `VERBOSE=1 rake posts:missing_uploads` that will provide a full report of missing uploads	2019-05-21 12:45:51 +10:00
Vinoth Kannan	9f43852609	FIX: remove the unwanted code.	2019-05-20 19:10:21 +05:30
Sam Saffron	30990006a9	DEV: enable frozen string literal on all files ... This reduces chances of errors where consumers of strings mutate inputs and reduces memory usage of the app. Test suite passes now, but there may be some stuff left, so we will run a few sites on a branch prior to merging	2019-05-13 09:31:32 +08:00
Vinoth Kannan	87cd4701b8	FEATURE: option to skip posts with ignored missing uploads	2019-05-09 05:11:15 +05:30
Vinoth Kannan	e8fafbc123	List and restore missing post uploads from S3 inventory.	2019-05-04 01:16:20 +05:30
Vinoth Kannan	c448e014fe	Recover missing old scheme uploads from tombstone directory too.	2019-04-29 11:33:57 +05:30
Vinoth Kannan	8698a10920	Don't raise error if a record already exists.	2019-04-25 21:48:39 +05:30
Vinoth Kannan	a65ce81e25	Don't check same upload url twice in the rake task	2019-04-25 20:06:10 +05:30
Vinoth Kannan	1724c27713	FIX: reload the 'post' model to retrive raw field value.	2019-04-25 02:09:27 +05:30
Vinoth Kannan	aed683390c	FIX: Don't treat 'upload_patterns' as constant to make the rake task compatible with multisite	2019-04-25 02:06:20 +05:30
Vinoth Kannan	b0bb1d95a5	trying to recover old scheme uploads on 'posts:missing_uploads' rake task	2019-04-25 01:16:18 +05:30
Vinoth Kannan	3dae12dec1	reduce unlinked count from missing post upload count	2019-04-19 21:53:00 +05:30