github-mirror/discourse
2
0
Fork 0
mirror of https://github.com/discourse/discourse.git synced 2024-11-26 22:53:43 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
33,207 Commits 214 Branches 500 Tags 960 MiB
e20c30987c
Commit Graph

6251 Commits

Author SHA1 Message Date
Osama Sayegh
e20c30987c
FEATURE: detect theme errors and catch them (#7589) 
* FEATURE: detect theme errors and catch them

* Bump COMPILER_VERSION

* Feedback

* Override eslint no console for one line

* Can't use our ajax method

* remove emoji from translation file
 2019-05-24 17:25:55 +03:00
Régis Hanol
a51e2271af DEV: remove temporary rescue when downloading MaxMindDB 
follow-up 8c8c925d1b
 2019-05-24 15:13:19 +02:00
Gerhard Schlager
c1e9a70d59 FIX: Fallback locale was not available for extra translations 
Translations from fallback locales were not sent to the client
for admin_js and wizard_js.
 2019-05-24 11:38:26 +02:00
Régis Hanol
0e24cb0f78
DEV: don't skip discourse-calendar plugin specs 2019-05-24 11:35:22 +02:00
Sam Saffron
9ce06d53d1 DEV: improve upload recovery rake task 
- Better error handling
- Automatically generates index file if missing
- Improved output
 2019-05-24 13:36:03 +10:00
Guo Xiang Tan
791241e0b2 DEV: Fix rake task to delete old export topics. 2019-05-24 11:20:53 +08:00
Sam Saffron
76f7e90e23 FEATURE: rake posts:recover_uploads_from_index improvements 
We now recover more smartly based on failed db remaps as well.
 2019-05-24 12:38:38 +10:00
Régis Hanol
f9bb8dde3c TEMP: output the error to STDERR 2019-05-23 16:55:04 +02:00
Régis Hanol
8c8c925d1b TEMP: catch exception when failing to download MaxMindDB 2019-05-23 16:30:21 +02:00
Régis Hanol
cb3ddeca40 DEV: remove unnecessary requires 2019-05-23 16:01:40 +02:00
Sam Saffron
1059aafc55 FEATURE: rake task for recovery of uploads from an index file 
`rake posts:recover_uploads_from_index`

Searches through all missing uploads in the cluster, if it finds one it
tries to find it in the "upload index file" and creates a new upload for
it.
 2019-05-23 17:08:22 +10:00
Sam Saffron
7ce58df7bf lint the file 
somehow I did not notice this on save
 2019-05-23 15:28:41 +10:00
Sam Saffron
a5ce9cb470 FEATURE: fix_relative_upload_links now multisite safety 
This also finds `<img src="/uploads/xyz` HTML images in raw and corrects
them. Also handles some cross multisite recovery and provides better output
 2019-05-23 15:09:16 +10:00
Sam Saffron
96e58125a7 FEATURE: rename task to posts:destroy_old_data_exports 
Previously we were only catching one type of data export, the new job will
catch every csv export we have

Job is pretty safe as it filters on system user id / pm with a particular
slug
 2019-05-23 15:02:56 +10:00
Sam Saffron
954293655f FEATURE: rake posts:destroy_old_user_data_exports 
Historically we would keep the user data export posts around but delete
the uploads.

This leaves a lot of broken uploads in the system.

This rake task allows us to clean up old mess.
 2019-05-23 11:11:37 +10:00
Bianca Nenciu
631b16ef86 FIX: Fix build. 
Follow-up to 25453eb.
 2019-05-22 17:39:44 +03:00
Bianca Nenciu
25453eb922 FIX: Avoid logging a warn for 404s. 2019-05-22 15:18:38 +03:00
Gerhard Schlager
e224100023 FIX: Don't convert :) into Emoji when emojis or emoji shurtcuts are disabled 2019-05-22 11:44:39 +02:00
Sam Saffron
e8799f0ba4 DEV: improve uploads:recover job so it stores a map of old to new sha 
Previous attempt created broken images
 2019-05-22 15:51:09 +10:00
Sam Saffron
ebcb571de7 FIX: allow upload recovery to recover uploads with sha mismatch 
Filename on disk may mismatch sha of file in some old 1X setups. This will
attempt to recover file even if sha1 mismatches. We had an old bug that
caused this.

This also adds `uploads:fix_relative_upload_links` which attempts to replace
urls of the format `/upload/default/...` with `upload://`
 2019-05-22 15:24:36 +10:00
Sam Saffron
f772ecc597 DEV: Correct missing output detailing failure 
uploads:s3_migration_status was failing but not returning proper output
 2019-05-22 12:58:54 +10:00
Guo Xiang Tan
5429c9b5e9 DEV: Fix incorrectly hardcoded value in rake task. 2019-05-22 09:01:25 +08:00
Sam Saffron
a9af77d650 PERF: ensure priority for rake rebake tasks is ultra low 
Flooding the default queue with lots of jobs is never a good thing. This
reduces the risk of sidekiq going into a bad state post rebake
 2019-05-22 10:31:49 +10:00
Sam Saffron
d6daa60d0a FEATURE: multisite support for posts:rebake_uncooked_posts 2019-05-22 10:04:54 +10:00
Sam Saffron
a2428bd862 FEATURE: warn about sidekiq overload prior to migrating 
Also makes pre-flight check ensure there is no giant backlog of posts that
need to be cooked
 2019-05-22 10:04:33 +10:00
Sam Saffron
5fdc7b7ca2 Correct 59012fc0 
Logic was flipped here by mistake, oops
 2019-05-22 09:48:03 +10:00
Sam Saffron
73f178a634 FEATURE: posts:rebake_uncooked_posts to look at mismatching baked_version 
also amends flagging onebox updates to set baked_version to nil
 2019-05-22 09:43:31 +10:00
Sam Saffron
4f296608da FEATURE: add uploads:s3_migration_status for looking at current status 
Also a few minor cleanups and better progress reporting
 2019-05-22 09:00:32 +10:00
Sam Saffron
59012fc0f7 PERF: mark posts for rebake vs forcing a rebake inline when migrating to s3 
Rebaking posts can be expensive instead of blocking here simply mark posts
for rebake.

We can then work through them faster in other jobs, plus this should not
hold of a datacenter migration.
 2019-05-22 08:39:25 +10:00
Vinoth Kannan
338b927f95 DEV: include local uploads while finding missing uploads 
in rake task 'posts:missing_uploads'
 2019-05-21 22:07:08 +05:30
Vinoth Kannan
7b82850f66 FIX: migrate_to_s3 task should remap attachment links too. 2019-05-21 21:58:11 +05:30
Sam Saffron
8360415453 FEATURE: big warning for uploads missing which can be very very slow on S3 2019-05-21 16:11:56 +10:00
Sam Saffron
cb86d8279a FEATURE: add toggle for uploads:missing so it can skip external 
Validating s3 uploads in uploads:missing can be very expensive, allow to
bypass.
 2019-05-21 16:11:56 +10:00
Sam Saffron
f4d4f7871e FEATURE: make posts:missing_uploads multisite friendly 
Previously this rake job would only run on a single site which is a bit
misleading

This also adds `VERBOSE=1 rake posts:missing_uploads` that will provide a
full report of missing uploads
 2019-05-21 12:45:51 +10:00
Sam Saffron
0f76e3090c FIX: use tar -xzvf to extract maxmind vs Ruby wrappers 
The tar/gz extractor in Ruby is somewhat buggy, instead rely on battle
tested `tar` and `gzip` Linux commands
 2019-05-21 10:48:18 +10:00
Vinoth Kannan
2941c77abc FIX: skip upload recovery if file not found in s3 2019-05-21 00:06:36 +05:30
David Taylor
0fbff66d97 DEV: Correct rake task syntax error 2019-05-20 17:43:30 +01:00
David Taylor
31902159af DEV: Allow skipping failed migrations when running S3 migration 
Use the SKIP_FAILED environment variable to skip failed sites. Use with caution - make sure you go back and re-run the failed migrations afterwards.
 2019-05-20 17:25:56 +01:00
David Taylor
a15cca9a0f DEV: Improve error message for posts:missing_uploads during S3 migration 2019-05-20 16:09:22 +01:00
Vinoth Kannan
2bfc0cf145 FIX: skip old scheme upload URLs while validating s3 uploads remap 2019-05-20 19:13:41 +05:30
Vinoth Kannan
9f43852609 FIX: remove the unwanted code. 2019-05-20 19:10:21 +05:30
David Taylor
41bc90dd3e DEV: Add progress indicator for post rebake during s3 migration 
Now that we run sidekiq jobs synchronously, this is important
 2019-05-20 14:19:58 +01:00
David Taylor
77a06209cb DEV: Skip S3 migration if all uploads are already migrated 
This makes the task resumable in a multisite context
 2019-05-20 14:17:37 +01:00
Gerhard Schlager
b788948985 FEATURE: English locale with international date formats 
Makes en_US the new default locale
 2019-05-20 13:47:20 +02:00
Vinoth Kannan
bc0c4b7253 FIX: should not migrate the system uploads to s3 2019-05-20 14:27:34 +05:30
Vinoth Kannan
be3fb85a04 DEV: add post migration checks and raise error if failed. 2019-05-20 14:18:28 +05:30
Sam Saffron
50e6ee55a9 DEV: add "WAIT_FOR_URL" env to smoke test 
This allows you to wait up to N seconds for the smoke test url to come up
in some cases you want to kick off the smoke test prior to having the smoke
test env ready to accept connections
 2019-05-20 17:23:09 +10:00
Sam Saffron
08371db0cc FIX: ensure we don't queue any jobs during s3 migration 
Previously we could flood sidekiq with jobs which is not ideal.

This ensures we are 100% done when the job is done.
 2019-05-20 16:28:50 +10:00
Vinoth Kannan
2a7065c505 FIX: skip uploads without etag in s3 inventory check. 2019-05-20 00:09:52 +05:30
Régis Hanol
7696b92c8c FIX: handle clicks counters in full quotes 2019-05-17 14:17:29 +02:00