github-mirror/discourse
2
0
Fork 0
mirror of https://github.com/discourse/discourse.git synced 2024-11-26 23:13:48 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
33,441 Commits 214 Branches 500 Tags 960 MiB
dd1a034e7b
Commit Graph

763 Commits

Author SHA1 Message Date
Gerhard Schlager
f7a2648694 FEATURE: Migrate uploads to S3 during restore 2019-06-04 15:47:36 +02:00
Gerhard Schlager
fb121815a5 DEV: Allow rebaking oncooked posts of single site 2019-06-04 15:47:36 +02:00
Guo Xiang Tan
1991af2abb DEV: Switch InlineUploads to a regexp based implementation. 2019-06-04 15:54:25 +08:00
Guo Xiang Tan
d93e5fb00d DEV: Class that converts MD with old attachment links to new MD. 2019-06-04 15:54:25 +08:00
Arpit Jalan
af20d616de FEATURE: new rake task to invalidate broken images in post 2019-06-04 12:40:51 +05:30
David Taylor
81dcadf788
DEV: Stop compiling themes during DB migration. Recompile on cdn change. (#7676) 
This is an improved implementation for bc8b7b13
 2019-06-03 16:38:02 +01:00
Rafael dos Santos Silva
725588f835 FIX: migrate_to_s3 wasn't IAM profile aware 2019-06-01 12:09:46 -03:00
Arpit Jalan
388433ec06 DEV: remove unused method (rake task deleted in 6a274fd) 2019-05-31 11:16:10 +05:30
Arpit Jalan
5c526e5abb FEATURE: new rake task to watch all replied topics 2019-05-31 11:08:00 +05:30
Guo Xiang Tan
6a274fd2cd DEV: Remove redundant rake task. 
Onceoff job created in 028121b95b
 2019-05-31 10:05:13 +08:00
Joffrey JAFFEUX
630e9814bc
datetime is not available at this point (#7630) 2019-05-29 14:06:32 +02:00
Joffrey JAFFEUX
6439004161
DEV: do not use STDERR to print tests timestamps (#7629) 2019-05-29 13:28:02 +02:00
Joffrey JAFFEUX
5d47d1833b
DEV: prints timestamp at every docker:test commands (#7628) 2019-05-29 12:20:33 +02:00
Vinoth Kannan
e12ae453e9 FIX: verify the exitence of s3_object properly without db name 2019-05-29 15:10:36 +05:30
Vinoth Kannan
9a9a06e34b DEV: add option to skip etag verification on 'migrate_to_s3' rake task 2019-05-29 14:16:36 +05:30
Vinoth Kannan
b3779dc377 DEV: rename 'uploads:missing' rake task into 'uploads:missing_files'. 
To improve the readability.
 2019-05-28 23:30:43 +05:30
Ralph Rooding
04abf18194 Fixed typo in S3 rake task 2019-05-28 16:57:55 +08:00
Sam Saffron
9a232e1a0a FEATURE: use GIVE_UP=1 to inform rake posts:missing_uploads you are done 
We need this give up for cases where uploads can not be recovered

This also improves the recovery routines
 2019-05-28 14:44:59 +10:00
Sam Saffron
c780d36ddc DEV: Correct posts:recover_uploads_from_index 
Was creating a blank index file by mistake
 2019-05-28 12:39:39 +10:00
Sam Saffron
3de8c58134 DEV: clean up backup maxmind path 
This follows up on 6580025a

reuses a list and removes a map that was erronuously there.
 2019-05-28 11:47:58 +10:00
Sam Saffron
7429700389 FIX: ensure we can download maxmind without redis or db config 
This also corrects FileHelper.download so it supports "follow_redirect"
correctly (it used to always follow 1 redirect) and adds a `validate_url`
param that will bypass all uri validation if set to false (default is true)
 2019-05-28 10:28:57 +10:00
Sam Saffron
e4e2acf148 FIX: rake maxminddb:get broke cause it required environment 2019-05-28 08:59:28 +10:00
Sam Saffron
6580025af9 FEATURE: add backup directory for mmdb files 
This new `DISCOURSE_MAXMIND_BACKUP_PATH` can be used a secondary location
for maxmind db. That way a build machine, for example can cache it on the
host and reuse between builds.

Also per 5bfeef77 added proper error raising for download fails from
dedicated rake task

This also moves "refresh_maxmind_db_during_precompile_days" to a global
setting, it did not make sense in a site setting
 2019-05-27 16:51:24 +10:00
Régis Hanol
2f5938c3bf DEV: use FileHelper to download MaxMindDB 2019-05-24 18:11:39 +02:00
Régis Hanol
8c9ffee3af TEMP: log more information when failing to download db 2019-05-24 17:33:52 +02:00
Régis Hanol
9ed2c54e27 TEMP: log information when failing to download db 2019-05-24 17:26:04 +02:00
Régis Hanol
0e24cb0f78
DEV: don't skip discourse-calendar plugin specs 2019-05-24 11:35:22 +02:00
Sam Saffron
9ce06d53d1 DEV: improve upload recovery rake task 
- Better error handling
- Automatically generates index file if missing
- Improved output
 2019-05-24 13:36:03 +10:00
Guo Xiang Tan
791241e0b2 DEV: Fix rake task to delete old export topics. 2019-05-24 11:20:53 +08:00
Sam Saffron
76f7e90e23 FEATURE: rake posts:recover_uploads_from_index improvements 
We now recover more smartly based on failed db remaps as well.
 2019-05-24 12:38:38 +10:00
Sam Saffron
1059aafc55 FEATURE: rake task for recovery of uploads from an index file 
`rake posts:recover_uploads_from_index`

Searches through all missing uploads in the cluster, if it finds one it
tries to find it in the "upload index file" and creates a new upload for
it.
 2019-05-23 17:08:22 +10:00
Sam Saffron
7ce58df7bf lint the file 
somehow I did not notice this on save
 2019-05-23 15:28:41 +10:00
Sam Saffron
a5ce9cb470 FEATURE: fix_relative_upload_links now multisite safety 
This also finds `<img src="/uploads/xyz` HTML images in raw and corrects
them. Also handles some cross multisite recovery and provides better output
 2019-05-23 15:09:16 +10:00
Sam Saffron
96e58125a7 FEATURE: rename task to posts:destroy_old_data_exports 
Previously we were only catching one type of data export, the new job will
catch every csv export we have

Job is pretty safe as it filters on system user id / pm with a particular
slug
 2019-05-23 15:02:56 +10:00
Sam Saffron
954293655f FEATURE: rake posts:destroy_old_user_data_exports 
Historically we would keep the user data export posts around but delete
the uploads.

This leaves a lot of broken uploads in the system.

This rake task allows us to clean up old mess.
 2019-05-23 11:11:37 +10:00
Sam Saffron
e8799f0ba4 DEV: improve uploads:recover job so it stores a map of old to new sha 
Previous attempt created broken images
 2019-05-22 15:51:09 +10:00
Sam Saffron
ebcb571de7 FIX: allow upload recovery to recover uploads with sha mismatch 
Filename on disk may mismatch sha of file in some old 1X setups. This will
attempt to recover file even if sha1 mismatches. We had an old bug that
caused this.

This also adds `uploads:fix_relative_upload_links` which attempts to replace
urls of the format `/upload/default/...` with `upload://`
 2019-05-22 15:24:36 +10:00
Sam Saffron
f772ecc597 DEV: Correct missing output detailing failure 
uploads:s3_migration_status was failing but not returning proper output
 2019-05-22 12:58:54 +10:00
Guo Xiang Tan
5429c9b5e9 DEV: Fix incorrectly hardcoded value in rake task. 2019-05-22 09:01:25 +08:00
Sam Saffron
a9af77d650 PERF: ensure priority for rake rebake tasks is ultra low 
Flooding the default queue with lots of jobs is never a good thing. This
reduces the risk of sidekiq going into a bad state post rebake
 2019-05-22 10:31:49 +10:00
Sam Saffron
d6daa60d0a FEATURE: multisite support for posts:rebake_uncooked_posts 2019-05-22 10:04:54 +10:00
Sam Saffron
a2428bd862 FEATURE: warn about sidekiq overload prior to migrating 
Also makes pre-flight check ensure there is no giant backlog of posts that
need to be cooked
 2019-05-22 10:04:33 +10:00
Sam Saffron
5fdc7b7ca2 Correct 59012fc0 
Logic was flipped here by mistake, oops
 2019-05-22 09:48:03 +10:00
Sam Saffron
73f178a634 FEATURE: posts:rebake_uncooked_posts to look at mismatching baked_version 
also amends flagging onebox updates to set baked_version to nil
 2019-05-22 09:43:31 +10:00
Sam Saffron
4f296608da FEATURE: add uploads:s3_migration_status for looking at current status 
Also a few minor cleanups and better progress reporting
 2019-05-22 09:00:32 +10:00
Sam Saffron
59012fc0f7 PERF: mark posts for rebake vs forcing a rebake inline when migrating to s3 
Rebaking posts can be expensive instead of blocking here simply mark posts
for rebake.

We can then work through them faster in other jobs, plus this should not
hold of a datacenter migration.
 2019-05-22 08:39:25 +10:00
Vinoth Kannan
338b927f95 DEV: include local uploads while finding missing uploads 
in rake task 'posts:missing_uploads'
 2019-05-21 22:07:08 +05:30
Vinoth Kannan
7b82850f66 FIX: migrate_to_s3 task should remap attachment links too. 2019-05-21 21:58:11 +05:30
Sam Saffron
8360415453 FEATURE: big warning for uploads missing which can be very very slow on S3 2019-05-21 16:11:56 +10:00
Sam Saffron
cb86d8279a FEATURE: add toggle for uploads:missing so it can skip external 
Validating s3 uploads in uploads:missing can be very expensive, allow to
bypass.
 2019-05-21 16:11:56 +10:00