Commit Graph

1234 Commits

Author SHA1 Message Date
Penar Musaraj
ec2943c5bc
DEV: Update jquery.fileupload and dependencies (#9466) 2020-04-28 10:39:29 -04:00
Sam Saffron
6b62d75c50
FIX: remove unsuppored rake task
rebuilding user_actions is not something that should be done.

Plugins such as solved and assigned extend it, there are tons of
little rules that were not captured in `user_actions:rebuild`
2020-04-28 16:27:42 +10:00
Rafael dos Santos Silva
b810fc282d FEATURE: Digital Ocean Spaces compatibility for Object Storage
DO does not implement tagging support for S3 objects. Removing our default
empty tag fixes compatibility.

The expire_missing_assets rake task can't be used with that service still,
but this patch allows normal operation.
2020-04-22 13:57:14 -03:00
Martin Brennan
628ba9d1e2
FEATURE: Promote bookmarks with reminders to core functionality (#9369)
The main thrust of this PR is to take all the conditional checks based on the `enable_bookmarks_with_reminders` away and only keep the code from the `true` path, making bookmarks with reminders the core bookmarks feature. There is also a migration to create `Bookmark` records out of `PostAction` bookmarks for a site.

### Summary

* Remove logic based on whether enable_bookmarks_with_reminders is true. This site setting is now obsolete, the old bookmark functionality is being removed. Retain the setting and set the value to `true` in a migration.
* Use the code from the rake task to create a database migration that creates bookmarks from post actions.
* Change the bookmark report to read from the new table.
* Get rid of old endpoints for bookmarks
* Link to the new bookmarks list from the user summary page
2020-04-22 13:44:19 +10:00
tshenry
e41f1e6ec2
FEATURE: Rake task to export groups (#9450) 2020-04-17 14:59:54 -07:00
Martin Brennan
51672b9121
FIX: Minor bookmark with reminder issue cleanup (#9436)
* Count user summary bookmarks from new Bookmark table if bookmarks with reminders enabled
* Update topic user bookmarked column when new topic bookmark changed
* Make in:bookmarks search work with new bookmarks
* Fix batch inserts for bookmark rake task (and thus migration). We were only inserting one bookmark at a time, completely defeating the purpose of batching!
2020-04-16 11:32:21 +10:00
Mark VanLandingham
f07c4a781c
DEV: Add options to theme install rake task - more options (#9394) 2020-04-14 09:17:00 -05:00
Sam Saffron
e2284cf739
Revert "We have had errors reported due to migrations breaking and are reverting"
This reverts commit 8b46f14744.

It corrects the reason for the revert:

We rely on SafeMigrate existing cause we call it from migrations,
Zeitwerk will autoload it.

Instead of previous pattern we explicitly bypass all the hacks in
production mode.

We need to disable SafeMigrate cause it is not thread safe.

A thread safe implementation is possible but not worth the effort,
we catch the issues in dev and test.
2020-04-14 11:31:07 +10:00
Robin Ward
8b46f14744 We have had errors reported due to migrations breaking and are reverting
these series of commits.

See:
https://meta.discourse.org/t/new-installation-fails-on-migration/147425

This reverts commit 80e832662b.
This reverts commit de5f2d3308.
This reverts commit 6b192d29fa.
This reverts commit a5b582f686.
This reverts commit 708dd97dfd.
2020-04-09 14:18:56 -04:00
Sam Saffron
80e832662b
DEV: fake stdout must respond to close
If it does not respond to close Logger will refuse to use it
2020-04-09 14:36:01 +10:00
Sam Saffron
a5b582f686
DEV: demux stdout when running multisite migrate
This avoids mixing up output in such a way that we can not tell which
site ran which migrations

Avoids threads all fighting for output
2020-04-09 12:50:14 +10:00
Sam Saffron
708dd97dfd
PERF: speed up migrations on multisite
Previously we were migrating multisites serially, this is extremely slow
especially when 200 dbs are involved.

The new implementation defaults to running 20 migrations concurrently, leading
to a 20x speedup.

We also amended it so errors are printed out last, something that makes
debugging failures easier.

This is code specific to Discourse cause we integrate SeedFu with our
migrations and can not include this in the multisite gem.
2020-04-09 11:58:20 +10:00
Sam Saffron
f062ebf274
DEV: reduce noise in logs regarding migration
If a migration performs no changes it should not output stuff.

Previously we would output information about seeds which was very noisy.

On multisite this was particularly bad
2020-04-09 10:05:22 +10:00
Sam Saffron
8a50ab45aa
DEV: use REPORT_REQUESTS=1 to find all requests
We can use this to profile our pretender and ensure nothing is superfluous
and nothing is missing
2020-04-02 16:01:49 +11:00
Martin Brennan
6f978bc95c
FIX: First pass to improve efficiency of secure uploads rake task (#9284)
Get rid of harmful each loop over uploads to update. Instead we put all the unique access control posts for the uploads into a map for fast access (vs using the slow .find through array) and look up the post when it is needed when looping through the uploads in batches.

On a Discourse instance with ~93k uploads, a simplified version of the old method takes > 1 minute, and a simplified version of the new method takes ~18s and uses a lot less memory.
2020-03-26 15:59:57 +10:00
Martin Brennan
efd5fb665b
DEV: Fix flaky time sensitive uploads.rake specs (#9283)
Also fix issues in spec where certain uploads were not considered secure
2020-03-26 13:31:39 +10:00
Martin Brennan
097851c135
FIX: Change secure media to encompass attachments as well (#9271)
If the “secure media” site setting is enabled then ALL files uploaded to Discourse (images, video, audio, pdf, txt, zip etc. etc.) will follow the secure media rules. The “prevent anons from downloading files” setting will no longer have any bearing on upload security. Basically, the feature will more appropriately be called “secure uploads” instead of “secure media”.

This is being done because there are communities out there that would like all attachments and media to be secure based on category rules but still allow anonymous users to download attachments in public places, which is not possible in the current arrangement.
2020-03-26 07:16:02 +10:00
Joffrey JAFFEUX
86eb03af85
DEV: allows to run up/down migrations of a plugin (#9241)
This is mostly useful while developing a plugin, to avoid manual actions of deleting tables and schema_migrations rows.

Usage:
bundle exec rake plugin:migrate:down[discourse-calendar]
2020-03-19 19:30:08 +01:00
Martin Brennan
16799da580
FIX: Improve bookmark modal on mobile and bookmark sync rake task (#9221)
* Improve the bookmark mobile on modal so it doesn't go all the way to the edge and the custom datetime input is easier to use
* Improve the rake task for syncing so it does not error for topics that no longer exist and batches 2000 inserts at a time, clearing the array each time
2020-03-17 15:15:22 +10:00
Sam Saffron
a14313e9d0
FIX: RANDOM_PASSWORD not working rake admin:create
We regressed behavior of this env var which enable admin account
creation with a random password
2020-03-15 17:24:11 +11:00
Robin Ward
c9d1577087 Let's not log the username/password
This could easily be seen by someone who shouldn't.
2020-03-11 12:48:10 -04:00
Gerhard Schlager
8fa8bab9ff FIX: Don't optimize icons during db:migrate when restoring backup
Uploads are extracted after the DB migration, so this could lead to a failure during the restore. Site icons get optimized after extracting uploads.
2020-03-04 16:59:49 +01:00
Martin Brennan
0388653a4d
DEV: Upload and secure media retroactive rake task improvements (#9027)
* Add uploads:sync_s3_acls rake task to ensure the ACLs in S3 are the correct (public-read or private) setting based on upload security

* Improved uploads:disable_secure_media to be more efficient and provide better messages to the user.

* Rename uploads:ensure_correct_acl task to uploads:secure_upload_analyse_and_update as it does more than check the ACL

* Many improvements to uploads:secure_upload_analyse_and_update

* Make sure that upload.access_control_post is unscoped so deleted posts are still fetched, because they still affect the security of the upload.

* Add escape hatch for capture_stdout in the form of RAILS_ENABLE_TEST_STDOUT. If provided the capture_stdout code will be ignored, so you can see the output if you need.
2020-03-03 10:03:58 +11:00
Martin Brennan
cfd56e9159 Include access control post when loading uploads in rake task
* to avoid N+1 query
2020-02-18 10:35:15 +10:00
Martin Brennan
9dcc454a07
FIX: Improvements and fixes for update_upload_acl rake task (#8980)
The rake task was broken, because the addition of the
UploadSecurity check returned true/false instead of the
upload ID to determine which uploads to set secure.
Also it was rebaking the posts in the wrong place and
pretty inefficiently at that. Also it was rebaking before
the upload was being changed to secure in the DB.
This also updates the task to set the access_control_post_id
for all uploads. the first post the upload is linked to is used
for the access control. if the upload doesn't get changed to
secure this doesn't affect anything.
Added a spec for the rake task to cover common cases.
2020-02-17 14:21:43 +10:00
Martin Brennan
e1e74abd4f
FEATURE: Improving bookmarks part 2 -- Topic Bookmarking (#8954)
### UI Changes

If `SiteSetting.enable_bookmarks_with_reminders` is enabled:

* Clicking "Bookmark" on a topic will create a new Bookmark record instead of a post + user action
* Clicking "Clear Bookmarks" on a topic will delete all the new Bookmark records on a topic
* The topic bookmark buttons control the post bookmark flags correctly and vice-versa
Disabled selecting the "reminder type" for bookmarks in the UI because the backend functionality is not done yet (of sending users notifications etc.)

### Other Changes

* Added delete bookmark route (but no UI yet)
* Added a rake task to sync the old PostAction bookmarks to the new Bookmark table, which can be run as many times as we want for a site (it will not create duplicates).
2020-02-13 16:26:02 +10:00
Joffrey JAFFEUX
87e92da085
DEV: makes popper.js part of javascript rake task (#8847) 2020-02-04 15:34:46 +01:00
Gerhard Schlager
4e8be6f18b FIX: uploads:s3_migration_status rake task was broken 2020-01-28 22:10:25 +01:00
Roman Rizzi
8eb2147f1f
FIX: Ensure sourcemap's source is correct. Uses the full assets path this time. (#8774) 2020-01-23 14:44:00 -03:00
romanrizzi
1de5b4ad64 Revert "FIX: Ensure sourcemap's source is correct"
This reverts commit 4553a0f6e0.
2020-01-22 16:31:34 -03:00
Roman Rizzi
4553a0f6e0
FIX: Ensure sourcemap's source is correct 2020-01-22 12:57:28 -03:00
Gerhard Schlager
ab07b945c2
Merge pull request #8736 from gschlager/rename_reply_id_column
REFACTOR: Rename `post_replies.reply_id` column to `post_replies.reply_post_id`
2020-01-17 17:24:49 +01:00
Martin Brennan
7c32411881
FEATURE: Secure media allowing duplicated uploads with category-level privacy and post-based access rules (#8664)
### General Changes and Duplication

* We now consider a post `with_secure_media?` if it is in a read-restricted category.
* When uploading we now set an upload's secure status straight away.
* When uploading if `SiteSetting.secure_media` is enabled, we do not check to see if the upload already exists using the `sha1` digest of the upload. The `sha1` column of the upload is filled with a `SecureRandom.hex(20)` value which is the same length as `Upload::SHA1_LENGTH`. The `original_sha1` column is filled with the _real_ sha1 digest of the file. 
* Whether an upload `should_be_secure?` is now determined by whether the `access_control_post` is `with_secure_media?` (if there is no access control post then we leave the secure status as is).
* When serializing the upload, we now cook the URL if the upload is secure. This is so it shows up correctly in the composer preview, because we set secure status on upload.

### Viewing Secure Media

* The secure-media-upload URL will take the post that the upload is attached to into account via `Guardian.can_see?` for access permissions
* If there is no `access_control_post` then we just deliver the media. This should be a rare occurrance and shouldn't cause issues as the `access_control_post` is set when `link_post_uploads` is called via `CookedPostProcessor`

### Removed

We no longer do any of these because we do not reuse uploads by sha1 if secure media is enabled.

* We no longer have a way to prevent cross-posting of a secure upload from a private context to a public context.
* We no longer have to set `secure: false` for uploads when uploading for a theme component.
2020-01-16 13:50:27 +10:00
Gerhard Schlager
e474cda321 REFACTOR: Restoring of backups and migration of uploads to S3 2020-01-14 11:41:35 +01:00
David Taylor
c8d438cc63
DEV: Allow CSP to be enabled during QUnit tests (#8668)
The QUnit rake task starts a server in test mode. We need a tweak to allow dynamic CSP hostnames in test mode. This tweak is already present in development mode.

To allow CSP to work, the browser host/port must match what the server sees. Therefore we need to disable the enforce_hostname middleware in test mode. To keep rspec and production as similar as possible, we skip enforce_hostname using an environment variable.

Also move the qunit rake task to use unicorn, for consistency with development and production.
2020-01-07 12:22:58 +00:00
Martin Brennan
abca91cc4d
FEATURE: Add rake task to disable secure media (#8669)
* Add a rake task to disable secure media. This sets all uploads to `secure: false`, changes the upload ACL to public, and rebakes all the posts using the uploads to make sure they point to the correct URLs. This is in a transaction for each upload with the upload being updated the last step, so if the task fails it can be resumed.
* Also allow viewing media via the secure url if secure media is disabled, redirecting to the normal CDN url, because otherwise media links will be broken while we go and rebake all the posts + update ACLs
2020-01-07 12:27:24 +10:00
Blake Erickson
b7b85f9ade
FEATURE: Turn csp on by default (#8665)
* turn csp on by default

* remove csp migration for new sites now that is is on by default

* Ensure CSP is off before starting qunit
2020-01-06 13:42:21 -07:00
Blake Erickson
fc6b093dce FIX: Ensure CSP is off for qunit
If CSP is turned on qunit won't run at all, not even via the CLI, this
is causing the js tests to fail.

Follow up to: 3193b0f6e6
2020-01-03 18:43:19 -07:00
Sam Saffron
412e1ebbe2 DEV: correct parallel specs rake tasks
This used to work due to side effects.

`rake parallel:migrate` used to work very inconsistently and would only migrate
some of the databases.

This introduces the recommended change to db.yml so the correct database is
found based off TEST_ENV_NUMBER if for some reason we did not set it using
RAILS_DB

Also avoids a bunch of schema dumping which is not needed when migrating
parallel specs



DB number 1 is very odd cause for whatever reason parallel spec is not
setting it.
2019-12-31 14:07:55 +11:00
Robin Ward
9fc1a754fe DEV: Error message if multisite DB can't be created 2019-12-19 11:52:27 -05:00
Robin Ward
480e2ec523 FIX: If we run db:migrate on its own, it should load the environment 2019-12-16 14:46:33 -05:00
Hector Bustillos
42758379b9 FEATURE: Adds a message when the passwords doesn't match in rake admin:create (#8543) 2019-12-13 11:40:33 +11:00
David Taylor
4c9ca24ccf
FEATURE: Hash API keys in the database (#8438)
API keys are now only visible when first created. After that, only the first four characters are stored in the database for identification, along with an sha256 hash of the full key. This makes key usage easier to audit, and ensures attackers would not have access to the live site in the event of a database leak.

This makes the merge lower risk, because we have some time to revert if needed. Once the change is confirmed to be working, we will add a second commit to drop the `key` column.
2019-12-12 11:45:00 +00:00
David Taylor
67605b55d5 DEV: Prevent plugin loading when running core db migrations in tests
This is required because bin/rake automatically loads plugins when migrating. In our continuous integration, we don't want plugins to break the core build. They should only be loaded for the plugin build.
2019-12-10 16:11:49 +00:00
Neil Lalonde
891b71bdf5 Fix release notes rake task
It breaks when a commit message is "FIX:"
2019-12-05 11:53:10 -05:00
romanrizzi
b13f427450 DEV: Drop uglifyJS V2 support 2019-12-04 15:58:38 -03:00
David Taylor
46d8fd3831 FIX: Allow for nil upload record when migrating to S3 2019-12-04 15:13:39 +00:00
Joffrey JAFFEUX
0d3d2c43a0
DEV: s/\$redis/Discourse\.redis (#8431)
This commit also adds a rubocop rule to prevent global variables.
2019-12-03 10:05:53 +01:00
Penar Musaraj
0a6b29a942 DEV: Allow skipping --fail-fast rspec option for plugin specs 2019-11-22 13:27:47 -05:00
Jeff Wong
c6d8dbd4a9 Revert "FEATURE: Normalize the service worker route (#8359)"
This reverts commit 9799a651b6.
2019-11-20 14:10:17 -08:00
Jeff Wong
9799a651b6
FEATURE: Normalize the service worker route (#8359)
* FEATURE: Normalize the service worker route

Update cache headers so they are not immutable outside of the rails app

Add the ability to purge the service worker cache from localhost

Rails -> nginx will pass immutable flags so the file is cached until reloaded.
In most cases, nginx will have its cache flushed on rebuild (new image)

For those needing dynamic re-caching (such as upgrading via the UI),
a rake task for flushing the service worker script is provided
through `assets:flush_sw`
2019-11-20 11:33:41 -08:00
Penar Musaraj
102909edb3 FEATURE: Add support for secure media (#7888)
This PR introduces a new secure media setting. When enabled, it prevent unathorized access to media uploads (files of type image, video and audio). When the `login_required` setting is enabled, then all media uploads will be protected from unauthorized (anonymous) access. When `login_required`is disabled, only media in private messages will be protected from unauthorized access. 

A few notes: 

- the `prevent_anons_from_downloading_files` setting no longer applies to audio and video uploads
- the `secure_media` setting can only be enabled if S3 uploads are already enabled and configured
- upload records have a new column, `secure`, which is a boolean `true/false` of the upload's secure status
- when creating a public post with an upload that has already been uploaded and is marked as secure, the post creator will raise an error
- when enabling or disabling the setting on a site with existing uploads, the rake task `uploads:ensure_correct_acl` should be used to update all uploads' secure status and their ACL on S3
2019-11-18 11:25:42 +10:00
Robin Ward
397852a9a6 FIX: Support for rake db:rollback with plugins
For this to work we need to overwrite `db:rollback` in our Rakefile like
we do for migrate, so that it removes the load_config dependency. This
allows our custom migration paths to work.
2019-11-15 16:38:46 -05:00
romanrizzi
0ba0aa00e7 DEV: Use a bigger suite for warming up the tmp folder 2019-11-13 18:34:10 -03:00
Simon Cossar
544a1379b6
Reference posts table for deleted_at and user_id (#8332) 2019-11-11 11:03:57 -05:00
Jay Pfaffman
63bd07492e FIX: Update api_key rake task for recent changes
New usage: 

```
rake api_key:get_or_create_master["Onboarding Key"]
```
2019-11-08 23:34:15 +00:00
Sam Saffron
d8f7f363cd FEATURE: stop updating last_posted_at on users for messages and whispers
This ensures we only update last_posted_at which is user facing for non messages
and non whispers.

We still update this date for secure categories, we do not revert it for
deleted posts.
2019-10-31 09:01:26 +11:00
Krzysztof Kotlarek
c32bd8ae48 FEATURE: Remove attachments and truncate raw field for incoming emails (#8253)
Adds the settings: 

raw_email_max_length, raw_rejected_email_max_length, delete_rejected_email_after_days. 

These settings control retention of the "raw" emails logs.

raw_email_max_length ensures that if we get incoming email that is huge we will truncate it removing uploads from the raw log.

raw_rejected_email_max_length introduces an even more aggressive truncation for rejected incoming mail. 

delete_rejected_email_after_days controls how many days we will keep rejected emails for (default 90)
2019-10-30 16:54:35 +11:00
romanrizzi
197ad8cd6e DEV: Run a faster test when warming up 2019-10-28 17:19:55 -03:00
romanrizzi
d76d0e75ec DEV: Move warmup inside docker rake task 2019-10-25 16:31:05 -03:00
Gerhard Schlager
452d541b74 DEV: Add env variable to allow easier debugging of smoke tests
Set `DEBUG_NODE=1` when running `rake smoke:test` and use your favorite tool to debug the smoke tests. See https://nodejs.org/en/docs/guides/debugging-getting-started/ for more information.

The debugger will break at the beginning of the smoke tests when the env variable is set.
2019-10-24 17:48:15 +02:00
Krzysztof Kotlarek
f34a0141c7 FIX: Correct path to ImportExport module (#8227)
During the move from Classic autoloader to Zeitwerk import_export module was moved to correct file name convention.
427d54b2b0 (diff-d896ec33b95afb7fae9f8bfe73d0580b)

Problem is that export/import is still using old path to require that module

Meta: https://meta.discourse.org/t/topic-and-category-export-import/38930/40
2019-10-23 17:27:14 +11:00
Daniel Waterworth
55a1394342 DEV: pluck_first
Doing .pluck(:column).first is a very common pattern in Discourse and in
most cases, a limit cause isn't being added. Instead of adding a limit
clause to all these callsites, this commit adds two new methods to
ActiveRecord::Relation:

pluck_first, equivalent to limit(1).pluck(*columns).first

and pluck_first! which, like other finder methods, raises an exception
when no record is found
2019-10-21 12:08:20 +01:00
Roman Rizzi
10478cb395
DEV: Bump uglifyjs (#7834)
* Rewrite uglifyjs command to work with 3.x

* Use ES5 syntax in plain JS files

* Use the older command if uglifyJS V2.x is installed
2019-10-09 10:02:49 -03:00
Sam Saffron
6471fbd453 FIX: maxminddb:get task no longer working
Following our zeitwerk upgrade we needed to be more explicit about the
require
2019-10-04 09:47:33 +10:00
Krzysztof Kotlarek
427d54b2b0 DEV: Upgrading Discourse to Zeitwerk (#8098)
Zeitwerk simplifies working with dependencies in dev and makes it easier reloading class chains. 

We no longer need to use Rails "require_dependency" anywhere and instead can just use standard 
Ruby patterns to require files.

This is a far reaching change and we expect some followups here.
2019-10-02 14:01:53 +10:00
Joffrey JAFFEUX
cb8fa46970
DEV: prevents populate task to crash (#8111)
Generated emails/names/sentences were crashing with the following error:

```
Can not transliterate strings with ASCII-8BIT encoding
```
2019-09-22 05:18:48 -07:00
Sam Saffron
1ca257be79 DEV: db:migrate no longer works after db:schema:load
In Rails 6 due to internal changes, the following sequence no longer works:

```
RAILS_ENV=test bin/rake db:migrate
RAILS_ENV=test bin/rake db:schema:dump
dropdb discourse_test
createdb discourse_test
RAILS_ENV=test bin/rake db:schema:load
RAILS_ENV=test bin/rake db:migrate
```

What appears to be happening is that our tracking of plugin migrations is
being missed on schema:dump or load.

A more comprehensive fix restoring schema:dump / load support will be
investigated.
2019-09-18 13:17:49 +10:00
Sam Saffron
025d4ee91f FIX: Rails 6 multisite migrations and plugin migrations
Prior to this change plugin migrations were not working and multisite
migrations not working.

Rails internals changed so we need to account for it.

Specifically semantics of `db:migrate` in rails changed so it is sort of
a "multisite:migrate".
2019-09-13 09:17:54 +10:00
Roman Rizzi
568232052e
DEV: Remove FlagQuery class and old code (#8064) 2019-09-12 13:21:33 -03:00
Gerhard Schlager
56c6e8e961 FIX: POP3 doesn't work with TLS 1.3 2019-09-11 18:43:19 +02:00
Michael Brown
503a11cc88 FIX: inline_uploads and subfolder (#8076)
* FIX: inline_uploads and subfolder

* if subfolder, also look for images with a path containing
  cdn_url + relative_url_root

* FIX: migrate_to_s3 task and subfolder
2019-09-11 11:50:48 +10:00
Gerhard Schlager
874985db71 DEV: Ignore duplicate prefixes when generating release notes 2019-09-04 15:45:48 +02:00
Arpit Jalan
75f37ac16a Rename lazyYT plugin directory name to lazy-yt 2019-08-21 14:35:14 +05:30
Sam Saffron
7632fe0b58 PERF: rake posts:rebake_uncooked_posts runs inline
Running this inline makes more sense otherwise there is extreme risk in
saturating sidekiq queue.

This also reworks ordering and selection so we double check if a post needs
rebaking prior to rebaking, this unlocks the ability to run this rake task
from multiple consoles.
2019-08-13 10:28:42 +10:00
Rafael dos Santos Silva
606c0ed14d
FIX: S3 uploads were missing a cache-control header (#7902)
Admins still need to run the rake task to fix the files who where uploaded previously.
2019-08-06 14:55:17 -03:00
Sam Saffron
4c6a0313f2 FEATURE: improve stats provided by rake db:stats
New stats include index size and sort by index + table size

Eg:

```
table_name                  | row_estimate | table_size | index_size | total_size
--------------------------------------------------------------------------
topic_views                 | 18260776     | 796 MB     | 2015 MB    | 2811 MB
post_timings                | 15813837     | 668 MB     | 1153 MB    | 1821 MB
posts                       | 386890       | 1532 MB    | 215 MB     | 1747 MB
post_search_data            | 317575       | 580 MB     | 162 MB     | 742 MB
incoming_emails             | 34412        | 734 MB     | 7160 kB    | 741 MB
incoming_links              | 5169302      | 306 MB     | 377 MB     | 683 MB
email_logs                  | 5            | 32 kB      | 418 MB     | 418 MB
topic_users                 | 2410166      | 217 MB     | 155 MB     | 372 MB
user_actions                | 1361253      | 101 MB     | 266 MB     | 366 MB
...

```
2019-08-02 17:25:42 +10:00
Vinoth Kannan
e44d56e4d2 DEV: raise error only when 'STOP_ON_ERROR' env variable is available. 2019-08-01 23:54:06 +05:30
Daniel Waterworth
20bc4a38a5
FIX: DistributedMutex (#7953) 2019-08-01 09:12:05 +01:00
Guo Xiang Tan
a1a61fcd2d DEV: Use File#file? in posts:missing_uploads rake task. 2019-08-01 10:59:34 +08:00
Rishabh
dcb47d902b
REFACTOR: Rename SiteSetting.disable_edit_notifications to disable_system_edit_notifications (#7958)
* REFACTOR: Rename SiteSetting.disable_edit_notifications to disable_system_edit_notifications

- The older name could cause some confusion because the setting does not disable all edit notifications, only system ones.

* FIX: Add frozen_string_literal: true in the migration

* DEV: Deprecate 'disable_edit_notifications'
2019-07-31 20:20:41 +05:30
Penar Musaraj
a1fb718340 DEV: Tweak plugin rake task console message
Show "Allowing write to all repos!" message only once.
2019-07-30 09:53:56 -04:00
Guo Xiang Tan
8a64b0c8e8 Revert "DEV: Remove unused kwarg and properly check for local missing uploads."
This reverts commit 97769f3d02.

The code is confusing but this change is quite risky. Defer for now
until we can look at it properly.
2019-07-29 14:35:34 +08:00
Guo Xiang Tan
97769f3d02 DEV: Remove unused kwarg and properly check for local missing uploads. 2019-07-29 14:21:06 +08:00
Guo Xiang Tan
e9dc48ccae DEV: Fix multisite support for posts:missing_uploads. 2019-07-29 10:56:11 +08:00
Blake Erickson
d26aa6e71e REFACTOR: Cleanup rake tasks based on feedback
Follow up to: [FEATURE: Create a rake task for destroying categories][1]

- `Discourse.system_user` is my friend
- Remove puts statements from rake tasks that don't return anything
- `for_each` is also my friend
- Use `human_users` to also exclude discobot
- Sort/format categories:list

[1]: 092eeb5ca3
2019-07-18 19:15:01 -06:00
Blake Erickson
092eeb5ca3 FEATURE: Create a rake task for destroying categories
Created a rake task for destroying multiple categories along with any
subcategories and topics the belong to those categories.

Also created a rake task for listing all of your categories.

Refactored existing destroy rake tasks to use new logging method, that
allows for puts output in the console but prevents it from showing in
the specs.
2019-07-17 12:44:14 -06:00
Rafael dos Santos Silva
1221d34284
FEATURE: Make Discourse work offline with WorkboxJS (#7870) 2019-07-15 13:05:55 -03:00
Gerhard Schlager
ce8e099639 FEATURE: Use configured quotation marks in fancy topic title 2019-07-12 21:10:10 +02:00
Joffrey JAFFEUX
550e811652
DEV: allows lodash to be updated with rake javascript:update (#7881) 2019-07-11 16:57:03 +02:00
Penar Musaraj
befcf67c90 DEV: run db:create and db:migrate on turbo specs
This ensures multisite specs run correctly when running ./bin/turbo_rspec
2019-07-08 14:27:51 -04:00
Sam Saffron
b9a2a4d904 DEV: Avoid emoji for symbols
Rake task needs to work across a wide variety of terminal emulators.
Requiring an emoji font is superflous.
2019-07-05 09:41:15 +10:00
Ralph Rooding
1318e0b288 FEATURE: Rake themes installer (#7848)
* Delete remote_theme when deleting the theme

* Install themes and theme components through rake

* Removed unnecessary test
2019-07-04 14:33:05 -04:00
Joffrey JAFFEUX
e9bb13c630
DEV: updates jquery-color to prevent it to pull old jquery version (#7857) 2019-07-04 15:03:21 +02:00
Daniel Waterworth
d6aa92e98e DEV: Add a verbose option to ./bin/turbo_rspec 2019-06-27 15:49:21 +01:00
Daniel Waterworth
b3ec3fac93 FIX: turbo_rspec doesn't accept these options 2019-06-26 17:05:27 +01:00
Daniel Waterworth
68d54d818a DEV: Add USE_TURBO option to docker rake task (#7811) 2019-06-26 11:54:39 -04:00
Guo Xiang Tan
d21594f4f7 Revert changes added by mistake in 2b19e2acc8. 2019-06-25 15:25:12 +08:00
Guo Xiang Tan
2b19e2acc8 Fix typo in a0aeabbb94. 2019-06-25 15:18:57 +08:00
Guo Xiang Tan
a0aeabbb94 DEV: Safer replacement for old scheme uploads for posts:missing_uploads. 2019-06-25 11:34:26 +08:00
Guo Xiang Tan
f3c13d80c9 DEV: Don't copy file when trying to recover missing uploads. 2019-06-24 16:32:59 +08:00
Guo Xiang Tan
2cc5561504 DEV: Correct description in posts:missing_uploads rake task. 2019-06-24 14:57:43 +08:00
Sam Saffron
5bc92296be DEV: lint a bunch of files we missed 2019-06-21 11:33:41 +10:00
Daniel Waterworth
e18ce56f4b DEV: Add a new way to run specs in parallel with better output (#7778)
* DEV: Add a new way to run specs in parallel with better output

This commit:

 1. adds a new executable, `bin/interleaved_rspec` which works much like
    `rspec`, but runs the tests in parallel.

 2. adds a rake task, `rake interleaved:spec` which runs the whole test
    suite.

 3. makes autospec use this new wrapper by default. You can disable this
    by running `PARALLEL_SPEC=0 rake autospec`.

It works much like the `parallel_tests` gem (and relies on it), but
makes each subprocess use a machine-readable formatter and parses this
output in order to provide a better overall summary.

(It's called interleaved, because parallel was taken and naming is
hard).

* Make popen3 invocation safer

* Use FileUtils instead of shelling out

* DRY up reporter

* Moved summary logic into Reporter

* s/interleaved/turbo/g

* Move Reporter into its own file

* Moved run into its own class

* Moved Runner into its own file

* Move JsonRowsFormatter under TurboTests

* Join on threads at the end

* Acted on feedback from eviltrout
2019-06-21 10:59:01 +10:00
Penar Musaraj
4bbb43cb2b DEV: pull plugin repos before linting them 2019-06-14 13:50:15 -04:00
Sam Saffron
ecb2fd8222 DEV: improve db:create to catch all cases
- No RAILS_ENV - create multisite / dev / test
- `test` RAILS_ENV - create multisite / test
2019-06-14 15:47:05 +10:00
Sam Saffron
a01488ae67 DEV: improve on rake db:create
Followup on 3af00a65 which broke build
2019-06-14 15:06:07 +10:00
Sam Saffron
3af00a65e6 FIX: site settings loading default values when no db
This fixes a condition where an intermittent db connection could cause
invalid site settings to be stored

It also removes a catch all we had.

Somewhere around Rails 5 `db:create` started wanting full environment
this is a problem for Discourse since it needs to boot up data from the
db.

This removes the catch all and surgically adds a db / redis bypass to
db:create task.
2019-06-14 14:21:07 +10:00
Guo Xiang Tan
eaa7527933 DEV: Switch posts:inline_uploads scope to be more generic. 2019-06-14 11:53:34 +08:00
Penar Musaraj
66b15b9d87 DEV: support installing plugin gems in test images
- adds plugin:install_all_gems rake task

- adds UPDATE_ALL_PLUGINS env variable in docker:test
2019-06-13 09:58:57 -04:00
Guo Xiang Tan
9bce3d5e65 DEV: Multisite support for rake posts:inline_uploads. 2019-06-13 16:13:45 +08:00
Sam Saffron
215216df48 FIX: stop re-downloading backup maxmind
We omitted updating the mmdb_time which changed causing a constant
re-download of the maxmind db
2019-06-13 17:42:26 +10:00
Penar Musaraj
e47ca6633b DEV: Fix plugin:update_all task
Reenable subtask so it loops through all plugins.

Only run for plugins that are git repositories.
2019-06-12 17:21:56 -04:00
Guo Xiang Tan
d49c193289 FIX: posts:inline_uploads rake task needs to bypass bump. 2019-06-12 09:13:31 +08:00
Guo Xiang Tan
864222372f Remove incorrectly committed byebug.
Follow up to 49b921147a.
2019-06-11 16:33:49 +08:00
Guo Xiang Tan
49b921147a FIX: Undefined method in posts:inline_uploads. 2019-06-11 16:28:41 +08:00
Guo Xiang Tan
1cf3bf016c Make rubocop happy. 2019-06-10 09:49:02 +08:00
Guo Xiang Tan
29259b46ae DEV: Verbose mode for posts:inline_uploads rake task. 2019-06-10 08:59:11 +08:00
Joffrey JAFFEUX
8cd13b22a1
DEV: allows smoke test to run on macOS (#7735) 2019-06-07 18:20:48 +02:00
Bianca Nenciu
f63b8bb79d FIX: Periodically ensure consistency of categories. (#7663) 2019-06-06 11:30:52 +02:00
Penar Musaraj
f00275ded3 FEATURE: Support private attachments when using S3 storage (#7677)
* Support private uploads in S3
* Use localStore for local avatars
* Add job to update private upload ACL on S3
* Test multisite paths
* update ACL for private uploads in migrate_to_s3 task
2019-06-06 13:27:24 +10:00
Gerhard Schlager
f7a2648694 FEATURE: Migrate uploads to S3 during restore 2019-06-04 15:47:36 +02:00
Gerhard Schlager
fb121815a5 DEV: Allow rebaking oncooked posts of single site 2019-06-04 15:47:36 +02:00
Guo Xiang Tan
1991af2abb DEV: Switch InlineUploads to a regexp based implementation. 2019-06-04 15:54:25 +08:00
Guo Xiang Tan
d93e5fb00d DEV: Class that converts MD with old attachment links to new MD. 2019-06-04 15:54:25 +08:00
Arpit Jalan
af20d616de FEATURE: new rake task to invalidate broken images in post 2019-06-04 12:40:51 +05:30
David Taylor
81dcadf788
DEV: Stop compiling themes during DB migration. Recompile on cdn change. (#7676)
This is an improved implementation for bc8b7b13
2019-06-03 16:38:02 +01:00
Rafael dos Santos Silva
725588f835 FIX: migrate_to_s3 wasn't IAM profile aware 2019-06-01 12:09:46 -03:00
Arpit Jalan
388433ec06 DEV: remove unused method (rake task deleted in 6a274fd) 2019-05-31 11:16:10 +05:30
Arpit Jalan
5c526e5abb FEATURE: new rake task to watch all replied topics 2019-05-31 11:08:00 +05:30
Guo Xiang Tan
6a274fd2cd DEV: Remove redundant rake task.
Onceoff job created in 028121b95b
2019-05-31 10:05:13 +08:00
Joffrey JAFFEUX
630e9814bc
datetime is not available at this point (#7630) 2019-05-29 14:06:32 +02:00
Joffrey JAFFEUX
6439004161
DEV: do not use STDERR to print tests timestamps (#7629) 2019-05-29 13:28:02 +02:00
Joffrey JAFFEUX
5d47d1833b
DEV: prints timestamp at every docker:test commands (#7628) 2019-05-29 12:20:33 +02:00
Vinoth Kannan
e12ae453e9 FIX: verify the exitence of s3_object properly without db name 2019-05-29 15:10:36 +05:30
Vinoth Kannan
9a9a06e34b DEV: add option to skip etag verification on 'migrate_to_s3' rake task 2019-05-29 14:16:36 +05:30
Vinoth Kannan
b3779dc377 DEV: rename 'uploads:missing' rake task into 'uploads:missing_files'.
To improve the readability.
2019-05-28 23:30:43 +05:30
Ralph Rooding
04abf18194 Fixed typo in S3 rake task 2019-05-28 16:57:55 +08:00
Sam Saffron
9a232e1a0a FEATURE: use GIVE_UP=1 to inform rake posts:missing_uploads you are done
We need this give up for cases where uploads can not be recovered

This also improves the recovery routines
2019-05-28 14:44:59 +10:00
Sam Saffron
c780d36ddc DEV: Correct posts:recover_uploads_from_index
Was creating a blank index file by mistake
2019-05-28 12:39:39 +10:00
Sam Saffron
3de8c58134 DEV: clean up backup maxmind path
This follows up on 6580025a

reuses a list and removes a map that was erronuously there.
2019-05-28 11:47:58 +10:00
Sam Saffron
7429700389 FIX: ensure we can download maxmind without redis or db config
This also corrects FileHelper.download so it supports "follow_redirect"
correctly (it used to always follow 1 redirect) and adds a `validate_url`
param that will bypass all uri validation if set to false (default is true)
2019-05-28 10:28:57 +10:00
Sam Saffron
e4e2acf148 FIX: rake maxminddb:get broke cause it required environment 2019-05-28 08:59:28 +10:00
Sam Saffron
6580025af9 FEATURE: add backup directory for mmdb files
This new `DISCOURSE_MAXMIND_BACKUP_PATH` can be used a secondary location
for maxmind db. That way a build machine, for example can cache it on the
host and reuse between builds.

Also per 5bfeef77 added proper error raising for download fails from
dedicated rake task

This also moves "refresh_maxmind_db_during_precompile_days" to a global
setting, it did not make sense in a site setting
2019-05-27 16:51:24 +10:00
Régis Hanol
2f5938c3bf DEV: use FileHelper to download MaxMindDB 2019-05-24 18:11:39 +02:00
Régis Hanol
8c9ffee3af TEMP: log more information when failing to download db 2019-05-24 17:33:52 +02:00
Régis Hanol
9ed2c54e27 TEMP: log information when failing to download db 2019-05-24 17:26:04 +02:00
Régis Hanol
0e24cb0f78
DEV: don't skip discourse-calendar plugin specs 2019-05-24 11:35:22 +02:00
Sam Saffron
9ce06d53d1 DEV: improve upload recovery rake task
- Better error handling
- Automatically generates index file if missing
- Improved output
2019-05-24 13:36:03 +10:00
Guo Xiang Tan
791241e0b2 DEV: Fix rake task to delete old export topics. 2019-05-24 11:20:53 +08:00
Sam Saffron
76f7e90e23 FEATURE: rake posts:recover_uploads_from_index improvements
We now recover more smartly based on failed db remaps as well.
2019-05-24 12:38:38 +10:00
Sam Saffron
1059aafc55 FEATURE: rake task for recovery of uploads from an index file
`rake posts:recover_uploads_from_index`

Searches through all missing uploads in the cluster, if it finds one it
tries to find it in the "upload index file" and creates a new upload for
it.
2019-05-23 17:08:22 +10:00
Sam Saffron
7ce58df7bf lint the file
somehow I did not notice this on save
2019-05-23 15:28:41 +10:00
Sam Saffron
a5ce9cb470 FEATURE: fix_relative_upload_links now multisite safety
This also finds `<img src="/uploads/xyz` HTML images in raw and corrects
them. Also handles some cross multisite recovery and provides better output
2019-05-23 15:09:16 +10:00
Sam Saffron
96e58125a7 FEATURE: rename task to posts:destroy_old_data_exports
Previously we were only catching one type of data export, the new job will
catch every csv export we have

Job is pretty safe as it filters on system user id / pm with a particular
slug
2019-05-23 15:02:56 +10:00
Sam Saffron
954293655f FEATURE: rake posts:destroy_old_user_data_exports
Historically we would keep the user data export posts around but delete
the uploads.

This leaves a lot of broken uploads in the system.

This rake task allows us to clean up old mess.
2019-05-23 11:11:37 +10:00
Sam Saffron
e8799f0ba4 DEV: improve uploads:recover job so it stores a map of old to new sha
Previous attempt created broken images
2019-05-22 15:51:09 +10:00
Sam Saffron
ebcb571de7 FIX: allow upload recovery to recover uploads with sha mismatch
Filename on disk may mismatch sha of file in some old 1X setups. This will
attempt to recover file even if sha1 mismatches. We had an old bug that
caused this.

This also adds `uploads:fix_relative_upload_links` which attempts to replace
urls of the format `/upload/default/...` with `upload://`
2019-05-22 15:24:36 +10:00
Sam Saffron
f772ecc597 DEV: Correct missing output detailing failure
uploads:s3_migration_status was failing but not returning proper output
2019-05-22 12:58:54 +10:00
Guo Xiang Tan
5429c9b5e9 DEV: Fix incorrectly hardcoded value in rake task. 2019-05-22 09:01:25 +08:00
Sam Saffron
a9af77d650 PERF: ensure priority for rake rebake tasks is ultra low
Flooding the default queue with lots of jobs is never a good thing. This
reduces the risk of sidekiq going into a bad state post rebake
2019-05-22 10:31:49 +10:00
Sam Saffron
d6daa60d0a FEATURE: multisite support for posts:rebake_uncooked_posts 2019-05-22 10:04:54 +10:00
Sam Saffron
a2428bd862 FEATURE: warn about sidekiq overload prior to migrating
Also makes pre-flight check ensure there is no giant backlog of posts that
need to be cooked
2019-05-22 10:04:33 +10:00
Sam Saffron
5fdc7b7ca2 Correct 59012fc0
Logic was flipped here by mistake, oops
2019-05-22 09:48:03 +10:00
Sam Saffron
73f178a634 FEATURE: posts:rebake_uncooked_posts to look at mismatching baked_version
also amends flagging onebox updates to set baked_version to nil
2019-05-22 09:43:31 +10:00
Sam Saffron
4f296608da FEATURE: add uploads:s3_migration_status for looking at current status
Also a few minor cleanups and better progress reporting
2019-05-22 09:00:32 +10:00
Sam Saffron
59012fc0f7 PERF: mark posts for rebake vs forcing a rebake inline when migrating to s3
Rebaking posts can be expensive instead of blocking here simply mark posts
for rebake.

We can then work through them faster in other jobs, plus this should not
hold of a datacenter migration.
2019-05-22 08:39:25 +10:00
Vinoth Kannan
338b927f95 DEV: include local uploads while finding missing uploads
in rake task 'posts:missing_uploads'
2019-05-21 22:07:08 +05:30
Vinoth Kannan
7b82850f66 FIX: migrate_to_s3 task should remap attachment links too. 2019-05-21 21:58:11 +05:30
Sam Saffron
8360415453 FEATURE: big warning for uploads missing which can be very very slow on S3 2019-05-21 16:11:56 +10:00
Sam Saffron
cb86d8279a FEATURE: add toggle for uploads:missing so it can skip external
Validating s3 uploads in uploads:missing can be very expensive, allow to
bypass.
2019-05-21 16:11:56 +10:00
Sam Saffron
f4d4f7871e FEATURE: make posts:missing_uploads multisite friendly
Previously this rake job would only run on a single site which is a bit
misleading

This also adds `VERBOSE=1 rake posts:missing_uploads` that will provide a
full report of missing uploads
2019-05-21 12:45:51 +10:00
David Taylor
0fbff66d97 DEV: Correct rake task syntax error 2019-05-20 17:43:30 +01:00
David Taylor
31902159af DEV: Allow skipping failed migrations when running S3 migration
Use the SKIP_FAILED environment variable to skip failed sites. Use with caution - make sure you go back and re-run the failed migrations afterwards.
2019-05-20 17:25:56 +01:00
David Taylor
a15cca9a0f DEV: Improve error message for posts:missing_uploads during S3 migration 2019-05-20 16:09:22 +01:00
Vinoth Kannan
2bfc0cf145 FIX: skip old scheme upload URLs while validating s3 uploads remap 2019-05-20 19:13:41 +05:30
Vinoth Kannan
9f43852609 FIX: remove the unwanted code. 2019-05-20 19:10:21 +05:30
David Taylor
41bc90dd3e DEV: Add progress indicator for post rebake during s3 migration
Now that we run sidekiq jobs synchronously, this is important
2019-05-20 14:19:58 +01:00
David Taylor
77a06209cb DEV: Skip S3 migration if all uploads are already migrated
This makes the task resumable in a multisite context
2019-05-20 14:17:37 +01:00
Vinoth Kannan
bc0c4b7253 FIX: should not migrate the system uploads to s3 2019-05-20 14:27:34 +05:30
Vinoth Kannan
be3fb85a04 DEV: add post migration checks and raise error if failed. 2019-05-20 14:18:28 +05:30
Sam Saffron
50e6ee55a9 DEV: add "WAIT_FOR_URL" env to smoke test
This allows you to wait up to N seconds for the smoke test url to come up
in some cases you want to kick off the smoke test prior to having the smoke
test env ready to accept connections
2019-05-20 17:23:09 +10:00
Sam Saffron
08371db0cc FIX: ensure we don't queue any jobs during s3 migration
Previously we could flood sidekiq with jobs which is not ideal.

This ensures we are 100% done when the job is done.
2019-05-20 16:28:50 +10:00
Joffrey JAFFEUX
9ce9d72e71
DEV: makes hidepassed default when running qunit (#7558)
Mostly useful when not running headless, but I endup doing it a lot when debugging, one less thing to check.
2019-05-16 14:37:01 +02:00
Joffrey JAFFEUX
4a9756ff3f
DEV: sets rack server to test env when using rake qunit:test (#7554) 2019-05-16 10:44:29 +02:00
David Taylor
5605dba85c DEV: Automatically annotate plugin models alongside core models 2019-05-13 16:37:47 +01:00
David Taylor
750c125707 DEV: Include foreign keys in model annotations 2019-05-13 15:53:42 +01:00
Guo Xiang Tan
c00dab89e4 Fix the build take 2. 2019-05-13 11:22:48 +08:00
Guo Xiang Tan
8c4caac2e4 Fix frozen string error in plugin:install_all_official. 2019-05-13 10:19:02 +08:00
Guo Xiang Tan
8165ceb320 Make rubocop happy. 2019-05-13 09:55:44 +08:00
Sam Saffron
30990006a9 DEV: enable frozen string literal on all files
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.

Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
David Taylor
20daa76829 DEV: Change priority to ultra_low for post-s3-migration rebake 2019-05-10 18:37:45 +08:00
Guo Xiang Tan
626808e100 Fix incorrect condition in groups:grant_badge rake task.
`#find` raises an error if the id given to it is invalid. As a result,
the conditional to check whether a `group` or `badge` is `present?` will
not be executed if any of the ids are invalid.

Follow up to
6ba914033c.
2019-05-09 14:53:13 +08:00
Vinoth Kannan
87cd4701b8 FEATURE: option to skip posts with ignored missing uploads 2019-05-09 05:11:15 +05:30
Penar Musaraj
a92c925378 Use faster Brotli compression for non-default locale files 2019-05-08 14:55:24 -04:00
Penar Musaraj
d9ab722d05 Revert "Revert "PERF: Skip compressing locales for faster rebuilds (#7501)" (#7502)"
This reverts commit 6ea811c923.
2019-05-08 14:55:06 -04:00
Joffrey JAFFEUX
6ea811c923
Revert "PERF: Skip compressing locales for faster rebuilds (#7501)" (#7502) 2019-05-08 07:54:21 +02:00
Penar Musaraj
22d6203eb7 DEV: Use moment-timezone with 10-year range data
We do not need to use the full timezone dataset (it is a 950kb file!), only previous/next 5 years should suffice.
2019-05-07 23:42:49 -04:00
Sam Saffron
5c0bb29461 DEV: re-enable compression for node asset compilation
#b9d82818 makes enormous improvements to our bootstrap time, however going
to still keep compress for now despite the cost and watch it for a few weeks
2019-05-08 10:24:57 +10:00
Penar Musaraj
b9d8281818 PERF: Skip compressing locales for faster rebuilds (#7501)
* Do not brotli all locales in precompile

* Try without gzip

* uglify without compressing, always gzip

* skip uglify for unused locales

* FIX: Uglifier needs harmony for ES6 compatibility

* Use node uglifier if available

* Minor refactor
2019-05-08 10:22:19 +10:00
Guo Xiang Tan
2adbec1b3c PERF: Speed up migrate_to_s3 rake task by deleting optimized image record.
No point moving all optimized image files to tombstone when the store is
changing. Also, `destroy_all` can easily blow memory since we are no
loading in batches.
2019-05-07 16:10:32 +08:00
Sam Saffron
9be70a22cd DEV: introduce new API to look up dynamic site setting
This removes all uses of both `send` and `public_send` from consumers of
SiteSetting and instead introduces a `get` helper for dynamic lookup

This leads to much cleaner and safer code long term as we are always explicit
to test that a site setting is really there before sending an arbitrary
string to the class

It also removes a couple of risky stubs from the auth provider test
2019-05-07 11:00:30 +10:00
Arpit Jalan
6ba914033c FEATURE: rake task to bulk grant badge to members of a group 2019-05-06 12:41:50 +05:30
Vinoth Kannan
e8fafbc123 List and restore missing post uploads from S3 inventory. 2019-05-04 01:16:20 +05:30
Penar Musaraj
a5b804fec4 DEV: Add --profile and --fail-fast when running plugin specs, tweak parallel test balancing 2019-05-01 22:47:18 -04:00
David Taylor
0e303c7f5d
FEATURE: Automatically generate optimized site metadata icons (#7372)
This change automatically resizes icons for various purposes. Admins can now upload `logo` and `logo_small`, and everything else will be auto-generated. Specific icons can still be uploaded separately if required.

## Core

- Adds an SiteIconManager module which manages automatic resizing and fallback

- Icons are looked up in the OptimizedImage table at runtime, and then cached in Redis. If the resized version is missing for some reason, then most icons will fall back to the original files. Some icons (e.g. PWA Manifest) will return `nil` (because an incorrectly sized icon is worse than a missing icon). 

- `SiteSetting.site_large_icon_url` will return the optimized version, including any fallback. `SiteSetting.large_icon` continues to return the upload object. This means that (almost) no changes are required in core/plugins to support this new system.

- Icons are resized whenever a relevant site setting is changed, and during post-deploy migrations

## Wizard

- Allows `requiresRefresh` wizard steps to reload data via AJAX instead of a full page reload

- Add placeholders to the **icons** step of the wizard, which automatically update from the "Square Logo"

- Various copy updates to support the changes

- Remove the "upload-time" resizing for `large_icon`. This is no longer required.

## Site Settings UX

- Move logo/icon settings under a new "Branding" tab

- Various copy changes to support the changes

- Adds placeholder support to the `image-uploader` component

- Automatically reloads site settings after saving. This allows setting placeholders to change based on changes to other settings

- Upload site settings will be assigned a placeholder if SiteIconManager `responds_to?` an icon of the same name

## Dashboard Warnings

- Remove PWA icon and PWA title warnings. Both are now handled automatically.

## Bonus

- Updated the sketch logos to use @awesomerobot's new high-res designs
2019-05-01 14:44:45 +01:00
David Taylor
bc8b7b13ef DEV: Recompile theme fields after db:migrate 2019-04-30 11:52:30 +01:00
Vinoth Kannan
c448e014fe Recover missing old scheme uploads from tombstone directory too. 2019-04-29 11:33:57 +05:30
Vinoth Kannan
8698a10920 Don't raise error if a record already exists. 2019-04-25 21:48:39 +05:30
Arpit Jalan
5a70367a05 fix the build 2019-04-25 20:19:52 +05:30
Vinoth Kannan
a65ce81e25 Don't check same upload url twice in the rake task 2019-04-25 20:06:10 +05:30
Arpit Jalan
0d331d33c3 DEV: do not use find_each for avatars:clean task
https://review.discourse.org/t/fix-show-available-details-when-rescuing-error/2865
2019-04-25 18:46:34 +05:30
Vinoth Kannan
1724c27713 FIX: reload the 'post' model to retrive raw field value. 2019-04-25 02:09:27 +05:30
Vinoth Kannan
aed683390c FIX: Don't treat 'upload_patterns' as constant to make the rake task compatible with multisite 2019-04-25 02:06:20 +05:30
Vinoth Kannan
b0bb1d95a5 trying to recover old scheme uploads on 'posts:missing_uploads' rake task 2019-04-25 01:16:18 +05:30
Vinoth Kannan
3dae12dec1 reduce unlinked count from missing post upload count 2019-04-19 21:53:00 +05:30
Vinoth Kannan
4fcefc3615 add unlinked missing post upload count in stat 2019-04-19 20:39:21 +05:30
Vinoth Kannan
ba6369edc5 Add more missing upload stats in rake task 2019-04-19 20:14:40 +05:30
Vinoth Kannan
48cedf3ebe fix: retrieve posts in order by id.
so we can find starting and ending id of the batch.
2019-04-19 18:30:59 +05:30
Vinoth Kannan
ce8d30f667 PERF: use 'delete_all' method instead of 'destroy_all' 2019-04-19 18:00:01 +05:30
Vinoth Kannan
5de483a1b9 PERF: Very post upload's existence by preloaded upload sha1s array 2019-04-19 17:27:16 +05:30
Arpit Jalan
989325ea7c FIX: show available details when rescuing error 2019-04-15 14:09:52 +05:30
Arpit Jalan
07b961ab13 DEV: rescue avatar rake tasks with error backtrace 2019-04-15 14:06:29 +05:30
Arpit Jalan
a423e9b14a FIX: rescue avatars:refresh rake task 2019-04-14 14:26:15 +05:30
David Taylor
24ef4f7b2b Remove support for disable_jump_reply user setting (#7359) 2019-04-12 09:03:06 +10:00
Sam Saffron
84f12ac3f7 DEV: No need to carry two brotli implementations around
old version of brotli binary is long gone, require brotli 1.0.0 and up
2019-04-11 16:53:29 +10:00
Arpit Jalan
a20f58554b IMPORT: create category definitions in import:ensure_consistency task 2019-04-11 12:06:37 +05:30
Sam Saffron
baa7a9836c FEATURE: remove "COMPRESS_BROTLI" optional behavior
The compress brotli functionality is no longer optional, this has worked
well for years. The name of the ENV var is also confusing cause it does
not have a `DISCOURSE_` prefix which caused issues with the web upgrader

Brotli support is now unconditionally on
2019-04-11 12:36:18 +10:00
Vinoth Kannan
8d5c900142 DEV: add unique missing uploads index in post custom fields
https://review.discourse.org/t/feature-send-missing-post-uploads-stat-to-prometheus/2609/6?u=vinothkannans
2019-04-10 18:09:35 +05:30
Bianca Nenciu
c523a12187 FIX: Fix assets:precompile rake task. 2019-04-10 14:03:26 +03:00
Bianca Nenciu
4555d0c598 FEATURE: Refresh MaxmindDb during assets:precompile. (#7340) 2019-04-10 11:37:29 +02:00
Bianca Nenciu
b5008586c5 DEV: Remove HTML parser from Tautologistics. (#7344) 2019-04-10 11:21:22 +02:00
Vinoth Kannan
d0fe42e2ef FIX: should look through posts for image markdown
Downloaded onebox images only included in the cooked HTML content. So we have to check 'post.cooked' instead of 'raw'. bfdd0fe64c
2019-04-10 13:52:35 +05:30
Vinoth Kannan
4878ee95e5 DEV: optimize sql query to narrow down the filtering of post with uploads 2019-04-10 12:58:10 +05:30
Vinoth Kannan
bfdd0fe64c FIX: empty the missing list on each post loop 2019-04-10 11:20:26 +05:30
Vinoth Kannan
914ada1c74 DEV: convert scheduled job EnsurePostUploadsExistence into a rake task 2019-04-09 02:07:35 +05:30
Robin Ward
b58867b6e9 FEATURE: New 'Reviewable' model to make reviewable items generic
Includes support for flags, reviewable users and queued posts, with REST API
backwards compatibility.

Co-Authored-By: romanrizzi <romanalejandro@gmail.com>
Co-Authored-By: jjaffeux <j.jaffeux@gmail.com>
2019-03-28 12:45:10 -04:00
Guo Xiang Tan
71e431de9e DEV: Fix hardcoded value introduced in cc2bac86e9. 2019-03-26 07:45:21 +08:00
Gerhard Schlager
cd4fd447ca Make Rubocop happy 2019-03-25 17:04:45 +01:00
Guo Xiang Tan
cc2bac86e9 FIX: Dry run broken for uploads:migrate_to_s3. 2019-03-25 22:38:24 +08:00
Guo Xiang Tan
19c3c25db1 FIX: Handle BBCode in migrate_to_s3 task as well. 2019-03-22 16:47:06 +08:00
Guo Xiang Tan
2d34be24be FIX: Rebake lightbox and use short upload urls in migrate_to_s3. 2019-03-22 13:09:59 +08:00
Joffrey JAFFEUX
c18a6069c2
DEV: upgrades bootstrap-modal (#7235) 2019-03-21 14:18:12 +01:00
Guo Xiang Tan
4e594f2b2b FIX: Destroy optimized images in upload:migrate_to_s3 rake task.
`OptimizedImage` are currently not renegerated when the image changes
store.
2019-03-21 16:50:15 +08:00
Guo Xiang Tan
8e5e5d7d35 DEV: Make setting up of multisite DB in test env clearer. 2019-03-21 09:58:07 +08:00
Sam Saffron
600313373b DEV: skip creating multisite db when running JS tests only 2019-03-21 12:19:12 +11:00
Penar Musaraj
ed74774001 DEV: rebalance parallel tests (again) 2019-03-19 10:21:50 -04:00
Penar Musaraj
60b6c67c1c DEV: rebalance parallel tests (take 2) 2019-03-19 09:59:03 -04:00
Penar Musaraj
c8e1d3c2e7 DEV: rebalance parallel tests 2019-03-19 09:23:10 -04:00
Gerhard Schlager
8a99d59422 FIX: seed data classes couldn't always be found 2019-03-18 23:13:47 +01:00
Gerhard Schlager
3fd04df781
FEATURE: Locale support for seeded categories and topics (#7110) 2019-03-18 21:09:13 +01:00
Joffrey JAFFEUX
d91b47064e
FIX: windows has no country flags (#7199) 2019-03-18 19:28:49 +01:00
Gerhard Schlager
81108ee783 DEV: Increase timeout for JS tests and add env variable 2019-03-18 17:39:53 +01:00
Régis Hanol
9dbca41152 FIX: don't check system uploads in migrate_to_s3 rake task 2019-03-18 11:01:18 +01:00
Joffrey JAFFEUX
e1f5620b04
do not skip until we re-merge ember 3.7 (#7183) 2019-03-15 18:31:24 +01:00
Penar Musaraj
b43f1687ee FIX: Update user options in rake task 2019-03-15 11:41:16 -04:00
Joffrey JAFFEUX
aa4b84925a FEATURE: updates emoji to latest 12.0 version (#7095)
* improved emoji support

- always optimize images as part of the task
- use the unicode standard ordering/naming for sections

* UX: more height for when there are recently used
2019-03-15 16:12:07 +01:00
Régis Hanol
a6a52d1523 FIX: images:compress rake task
Optimized a few unoptimized PNGs.
2019-03-15 15:23:57 +01:00
Kelvin
45a97ad82f Fix SMTP connection test
This test did not support 'no auth' use case and other auth methods except 'login'. I fixed it by simply making the call to start() in the right way. 
As shown in the source code of Net::SMTP (https://github.com/ruby/ruby/blob/ruby_2_5/lib/net/smtp.rb#L452), the start() function does accept the 'user' and 'secret' arguments. Also, in do_start() function (https://github.com/ruby/ruby/blob/ruby_2_5/lib/net/smtp.rb#L542), it automatically checks the auth method and args, skips the authentication if 'user' is not provided, and selects the right auth method from 'plain', 'login' or 'cram_md5'. This is exactly all of what we should do in a connection test and the odd 'auth_login' call in the previous code makes problems. 
BTW, I am using 'localhost' as the third argument, which is the same as the default value in start(). This parameter is the domain address sent along with the 'ehlo' command in SMTP protocol. I have seen some documents, e.g. https://github.com/tpn/msmtp/blob/master/doc/msmtp.1#L455, saying that 'localhost' is fine. It works for me.
2019-03-15 09:50:51 -04:00
Joffrey JAFFEUX
9a5f08ee61
DEV: skip discourse-chat-integration tests (#7164)
Suspected of causing timeout issues in our test suite with Ember 3.7
2019-03-13 16:50:56 +01:00
Penar Musaraj
49cdb072d7 DEV: Use --profile and --fail-fast in CI only 2019-03-11 22:04:47 -04:00
Guo Xiang Tan
d82876896e FIX: uploads:migrate_to_s3 broken for GlobalSetting using file provider. 2019-03-11 14:21:35 +08:00
David Taylor
bc52437153 FIX: Add additional rescue for assets:precompile
This error can be caused if assets are precompiled before the database has been migrated
2019-03-07 14:03:05 +00:00
Régis Hanol
ad87b0d662
Make "uploads:recover_from_tombstone" call the "uploads:recover" rake task 2019-03-07 14:15:30 +01:00
Penar Musaraj
edffd0097f
DEV: Support running Ruby tests in parallel (#7087) 2019-02-28 16:36:22 -05:00
Penar Musaraj
9bf11a7c02
FEATURE: Add localized timezone names (#7004)
Uses github.com/discourse/moment-timezone-names-translations to translate timezone names.
Plugins can also provide their own timezone name translations.
2019-02-25 14:40:02 -05:00
Guo Xiang Tan
433f07fcb3 Fix confusing ENV in migrate_to_s3 rake task. 2019-02-22 14:44:46 +08:00
Rishabh
e69634ec3a FIX: Use s3_endpoint in migrate_to_s3 when not using S3
- overrides :region and uses :endpoint when SiteSetting.s3_endpoint is provided
- Now, we can use the new rake task with DigitalOcean Spaces
- I've tested that it's compatible with/without bucket folder path
- I've tested that it's compatible with S3 and it doesn't break S3 for non-default regions
- follow-up on 97e17fe0
2019-02-21 15:42:48 +05:30
Vinoth Kannan
0472bd4adc FIX: Remove 'backfill_etags' keyword argument from 'uploads:missing' rake task
And etags backfilling code is optimized
2019-02-15 00:34:35 +05:30
Penar Musaraj
e55c19d8ce DEV: update ACE Editor to 1.4.2
Remove duplicate "src-min" folder and fix rake task

Remove unused "snippets" folder
2019-02-14 11:05:28 -05:00
Penar Musaraj
090e9c8432
FIX: make composer full screen shortcut work when inputs have focus (#6907)
- Uses a Mousetrap plugin for global shortcuts
- Implemented for search `ctrl+alt+f` and composer fullscreen `shift+f11` shortcuts
2019-02-14 00:19:27 -05:00
Vinoth Kannan
7b5931013a Update rake task to backfill etags from s3 inventory 2019-02-14 05:18:06 +05:30
Penar Musaraj
5cef5b34ef DEV: Yarn-manage moment and moment-timezone libraries 2019-02-12 13:57:52 -05:00
Joffrey JAFFEUX
831d7d2f87
run tests on discourse-code-review (#6988) 2019-02-08 11:39:45 +01:00
Penar Musaraj
995c87a91e Don't run specs on code-review again 2019-02-07 16:57:05 -05:00
Penar Musaraj
d639cadb7a
DEV: Plugin scss errors should break precompile (#6974) 2019-02-07 09:27:42 -05:00
David Taylor
68173cd234 Revert "DEV: Allow rake plugin:spec to traverse symlinks"
This reverts commit dfd63b185f.

https://meta.discourse.org/t/108110/11
2019-02-01 12:40:29 +00:00
David Taylor
dfd63b185f DEV: Allow rake plugin:spec to traverse symlinks 2019-02-01 10:40:41 +00:00
Arpit Jalan
82b7795f36 Revert "don't run specs on code-review for now"
This reverts commit cb493b66e0.
2019-01-29 11:49:42 +05:30
Joshua Rosenfeld
cb493b66e0
don't run specs on code-review for now
code-review is currently causing failing builds. Remove from testing until it is fixed
2019-01-24 11:48:19 -05:00
Guo Xiang Tan
0cf2df3028 Fix remap in migrate_to_s3 rake task.
The current way of doing the remap only allows to run the rake task
once. Running the rake task more than once will end up badly.
2019-01-23 15:50:44 +08:00
Guo Xiang Tan
07850994d3 Add ENV to skip multisite prefix when migrating to s3. 2019-01-23 15:19:50 +08:00
Guo Xiang Tan
979d03aa68 Remove s3 bucket check in migrate_to_s3 task.
Bucket creation is expected to be handled by the user. If the bucket
does not exist, the script will fail anyway.
2019-01-23 15:04:51 +08:00
Guo Xiang Tan
99cd3ff6ee FIX: migrate_to_s3 task not setting the right content_disposition. 2019-01-23 15:04:47 +08:00
Rishabh
97e17fe084 FIX: Use ENV values instead of 'S3Helper.s3_options' in migrate_to_s3
This commit makes the rake task operational for all regions for s3. If we declare s3_endpoint as https://s3.amazonaws.com while
creating an instance of Aws::S3::Client, head_bucket fails for all s3 regions apart from us-east-1. The commit manually defines all
parameters for Aws::S3::Client apart from s3_endpoint to bypass this problem make this task usable for AWS S3.

Removing s3_endpoint from the payload means that custom endpoints like Minio/DO Spaces for will not work in the meantime and we'll
have to add support for a custom `s3_endpoint` in the future.

This commit follows up on 60790eb0.
2019-01-20 20:55:27 +05:30
Penar Musaraj
020e0d623a DEV: run "yarn install" before lint tests 2019-01-17 14:17:44 -05:00
Michael Brown
664c282400 FIX: improve rake emails:test task
* FIX: rake emails:test to bypass the "STARTTLS required" message

* FIX: too much j

* hint meta.discourse.org to people with unknown errors from the mail test
2019-01-17 13:41:10 +11:00
Rishabh
60790eb006 FIX: Use GlobalSetting values instead of ENV variables in migrate_to_s3
TIL how GlobalSetting works in sync with environment variables
Also fixes a small bug where bucket value was being used when it could have been nil
2019-01-16 14:40:38 +05:30
Rishabh
ff8f9dc1c9 FIX: prefix should precede folder path (follow-up on 10fbb07e) 2019-01-15 15:58:19 +05:30
Rishabh
10fbb07e1a FIX: include folder name in prefix for listing files on S3 (follow-up on 3ec38f5a)
Fix the destination url in remap since it's already a part of s3_base_url
2019-01-15 15:23:55 +05:30
Penar Musaraj
b28d68efbf DEV: Use yarn to manage Highlight.js dependency
Moves Highlight.js files to vendor/assets/javascripts

Adds Highlight.js in yarn package management

Removes old rake task and reliance on NPM to build Highlight.js

Highlight.js is now integrated in the "javascript:update" rake task
2019-01-14 12:46:35 -05:00
Florian Heidenreich
39983f9d95 FIX: list staff users within the last month 2019-01-14 10:29:00 -05:00
Régis Hanol
3ec38f5a3b Revert "FIX: migrate_to_s3 rake task with folder path"
This reverts commit 97fd12e8af.
2019-01-08 19:44:31 +01:00
Régis Hanol
97fd12e8af
FIX: migrate_to_s3 rake task with folder path 2019-01-08 18:56:18 +01:00
Rishabh
f181e9cc08
FIX: Add compatibility for bucket folder paths in migrate_to_s3 task (#6855)
* FIX: Add compatibility for bucket folder paths in migrate_to_s3 task
* Refactor bucket_name split logic into S3Helper
2019-01-08 20:04:48 +05:30
Rishabh
efc481d9c0 DEV: Use puts instead of printing newline (follow up on c5b7bda1) 2019-01-05 01:20:00 +05:30
Rishabh
c5b7bda198 DEV: Show migrate_to_s3 output on a new line 2019-01-04 18:09:54 +05:30
Vinoth Kannan
902f535111 FIX: upload method in S3Helper will expect a file object param 2019-01-04 15:30:45 +05:30
Sam
e2dca641c6 handle exceptions in s3:correct_acl task
We need to handle arbitrary exceptions in this task, especially since the
task is not easily resumable.

Simply output problem uploads as you hit them for now.
2019-01-04 08:32:09 +11:00
Sam
05a3e3670f FEATURE: add rake task that resets ACL on every object in S3
Some previous migrations to S3 may have bad ACLs set on objects. This
introduces a new rake task (`rake s3:correct_acl`) that will reset ACL on
every S3 object.

Vast majority of users will never have to run it, but if you have ACL issues
this is the atomic solution.
2019-01-04 08:13:43 +11:00
Gerhard Schlager
a474bf966c DEV: Skip prettier in single plugin test if patterns aren't found
This reverts d06ca90c94
It didn't work because yarn doesn't return the original exit code of the failed command.
2019-01-02 14:36:09 +01:00
Gerhard Schlager
d06ca90c94 DEV: Ignore prettier error when there are no ES6 files 2019-01-02 11:30:09 +01:00
Guo Xiang Tan
9e50813252 FIX: Pass all necessary options in migrate_to_s3 rake task. 2019-01-02 09:11:23 +08:00
Régis Hanol
5381096bfd PERF: new 'migrate_to_s3' rake task 2018-12-26 17:34:49 +01:00
Sam Saffron
abbbcb2622 DEV: allow creating users with random passwords via rake
Use: `RANDOM_PASSWORD=1 bin/rake admin:create`

Handy in conjunction with dev mode /session/username/become.
2018-12-18 11:43:16 +08:00
David Taylor
77fe57f923 FEATURE: Rake task to list users which have been staff in the past month 2018-12-14 16:29:12 +00:00
Sam
0ca61242b8 FEATURE: polyfill intersection observer for IE11 / iOS Safari
This feature is used for defer loading of images and in future for post cloaking

This gives us a polyfill so we can safely use the feature in problem browsers

The polyfill supports "polling" but it does not appear we need it yet.

If we discover anything odd here, consider setting poll interval per:

https://github.com/w3c/IntersectionObserver/tree/master/polyfill

```
var io = new IntersectionObserver(callback);
io.POLL_INTERVAL = 100; // Time in milliseconds.
```

Keeping the mutation observer cause we often mutate the DOM
2018-12-12 15:36:08 +11:00
Rishabh
314c084e5d DEV: nuke code dependent comments from scheduler.rake 2018-12-10 16:16:19 +05:30
Rishabh
4767a5a26e
DEV: Update incorrect Job frequency 2018-12-10 13:26:19 +05:30
Régis Hanol
3c9c95ac83 Update Rubocop to 0.60 2018-12-04 10:48:16 +01:00
David Taylor
404acef6e3 DEV: Move run-qunit.js out of the vendor directory 2018-12-03 16:16:37 +00:00
Penar Musaraj
03deda2147
Upgrade to FontAwesome 5 (take two) (#6673)
* Add missing icons to set

* Revert FA5 revert

 This reverts commit 42572ff

* use new SVG syntax in locales

* Noscript page changes (remove login button, center "powered by" footer text)

* Cast wider net for SVG icons in settings

- include any _icon setting for SVG registry (offers better support for plugin settings)

- let themes store multiple pipe-delimited icons in a setting

- also replaces broken onebox image icon with SVG reference in cooked post processor

* interpolate icons in locales

* Fix composer whisper icon alignment

* Add support for stacked icons

* SECURITY: enforce hostname to match discourse hostname

This ensures that the hostname rails uses for various helpers always matches
the Discourse hostname

* load SVG sprite with pre-initializers

* FIX: enable caching on SVG sprites

* PERF: use JSONP for SVG sprites so they are served from CDN

This avoids needing to deal with CORS for loading of the SVG

Note, added the svg- prefix to the filename so we can quickly tell in
dev tools what the file is

* Add missing SVG sprite JSONP script to CSP

* Upgrade to FA 5.5.0

* Add support for all FA4.7 icons

- adds complete frontend and backend for renamed FA4.7 icons

- improves performance of SvgSprite.bundle and SvgSprite.all_icons

* Fix group avatar flair preview

- adds an endpoint at /svg-sprites/search/:keyword

- adds frontend ajax call that pulls icon in avatar flair preview even when it is not in subset

* Remove FA 4.7 font files
2018-11-26 16:49:57 -05:00
Vinoth Kannan
fd272eee44 FEATURE: Make uploads:missing task compatible with s3 uploads 2018-11-27 00:54:51 +05:30
Kyle E. Mitchell
15e793fd3b FEATURE: Terms of Service v1.0.0
Co-authored-by: Gerhard Schlager <mail@gerhard-schlager.at>
2018-11-21 00:45:16 +01:00
Vinoth Kannan
bc41057949 minor copy edit 2018-11-20 12:07:56 +05:30
Vinoth Kannan
1a9a2bd5c1 DEV: Report the missing uploads count 2018-11-19 12:06:46 +05:30
Guo Xiang Tan
a895626d10 DB based S3 settings fixes for s3:upload_assets. 2018-11-15 15:48:40 +08:00
Guo Xiang Tan
68b640f6a9 Allow s3:upload_assets to use site settings from DB take 2. 2018-11-15 15:10:39 +08:00
Guo Xiang Tan
c56d682f45 Allow s3:upload_assets to use site settings from DB. 2018-11-15 15:06:19 +08:00
Rishabh
4a12cfaecb Remove trailing whitespace for Rubocop 2018-11-13 17:19:26 +05:30
Brian Helba
ea94323766 FIX: 'migrate_from_s3' rake task should respect max sizes (#6598)
Rather than hardcode a maximum size of 20MB for uploads migrated from S3, the task should use site settings for this value.
2018-11-13 12:27:38 +01:00
Gerhard Schlager
24e5be3f0c FIX: Relative links in translations should work with subfolder 2018-11-08 23:31:05 +00:00
Guo Xiang Tan
14ff47f6f1 Fix typo. 2018-11-08 16:42:12 +08:00
Guo Xiang Tan
7290145641 PERF: Speed up migrate_to_s3 rake task.
* Prioritizes non-image uploads
* Does one remap per upload instead of 3 remaps previously
* Every 100 uploads migrated, do 2 remaps which fixes broken
  URLs
* Exclude email_logs table from remap
2018-11-08 16:39:56 +08:00
Sam
42572ff138 Revert font awesome 5 changes
We are still pushing ahead on this 100% just need a bit longer to prepare
all plugins
2018-11-08 16:12:18 +11:00
Guo Xiang Tan
0232a3b5e5 PERF: Exclude tables when remapping in migrate_to_s3 rake task. 2018-11-08 12:37:36 +08:00
Penar Musaraj
005e1ecb9b
FEATURE: Update Font Awesome to v5.4.1 and SVGs (#6557)
* First take on subsetting svg icons

* FontAwesome 5 svg subset WIP

* Include icons from plugins/badges into svg sprite subset

* add svg icon support to themes

* Add spec for SvgSprite

* Misc. SVG icon fixes

* Use FA5 svgs in local-dates plugin

* CSS adjustments, fix SVG icons in group flair

* Use SVG icons in poll plugin

* Add SVG icons to /wizard
2018-11-07 13:05:43 -05:00
Bianca Nenciu
e1e392f15b DEV: Use DiscourseIpInfo for all IP queries. (#6482)
* DEV: Use DiscourseIpInfo for all IP queries.

* UX: Use latitude and longitude for more precision.
2018-10-30 22:08:57 +00:00
David Taylor
bafe3cd99a Revert "DEV: Lint official plugins in CI (#6519)"
This reverts commit 093cab2db0.
2018-10-22 22:30:33 +01:00
David Taylor
093cab2db0
DEV: Lint official plugins in CI (#6519) 2018-10-22 21:28:38 +01:00
Gerhard Schlager
341836eb42 Fix the rake task and importer instead 2018-10-17 16:48:09 +02:00
Matt Palmer
b1d7582abe Run specs on discourse-prometheus-alert-receiver
Sam wants to watch the world burn.
2018-10-17 15:32:57 +11:00
Vinoth Kannan
4c2331260e run specs on discourse-calendar plugin 2018-10-16 01:26:24 +05:30
Arpit Jalan
d76658ff8c FEATURE: new rake task to anonymize all users 2018-10-15 16:19:25 +05:30
Guo Xiang Tan
84d4c81a26 FEATURE: Support backup uploads/downloads directly to/from S3.
This reverts commit 3c59106bac.
2018-10-15 09:43:31 +08:00
Joshua Rosenfeld
fd58ca1903
remove manually mapping of discourse-logster-rate-limit-checker, repo renamed to match 2018-10-12 15:56:23 -04:00
Joshua Rosenfeld
43a7b08a48
don't run specs on prometheus-alert-receiver for now either 2018-10-12 12:20:49 -04:00
Joshua Rosenfeld
ca71815bb2
don't run specs on calendar for now 2018-10-12 12:12:21 -04:00
Joshua Rosenfeld
9d4916012a
manually map discourse-perspective too 2018-10-12 10:32:07 -04:00
Joshua Rosenfeld
2cb176f1ed
manually map discourse-logster-rate-limit-checker
also remove obsolete plugins
2018-10-12 10:27:24 -04:00
Chris Butler
ddbed82bca Exit with an error if there are errors with settings import (#6475)
If the site_settings import has any errors or settings that are not found, this
will cause the task to exit with a non-zero exit code.

This is useful when using this task as part of automated configuration deployment,
where you may not want to continue with the process if a setting fails to
import.
2018-10-11 15:57:57 +08:00
Guo Xiang Tan
3c59106bac Revert "FEATURE: Support backup uploads/downloads directly to/from S3."
This reverts commit c29a4dddc1.

We're doing a beta bump soon so un-revert this after that is done.
2018-10-11 11:08:23 +08:00
Gerhard Schlager
c29a4dddc1 FEATURE: Support backup uploads/downloads directly to/from S3. 2018-10-11 10:38:43 +08:00
Guo Xiang Tan
918633aa12 FIX: upload:migrate_to_s3 rake task not remapping properly. 2018-10-10 15:09:21 +08:00
Sam
9b01e2b855 FIX: maxmind download task does not need environment 2018-10-10 11:16:28 +11:00
Bianca Nenciu
1d26a473e7 FEATURE: Show "Recently used devices" in user preferences (#6335)
* FEATURE: Added MaxMindDb to resolve IP information.

* FEATURE: Added browser detection based on user agent.

* FEATURE: Added recently used devices in user preferences.

* DEV: Added acceptance test for recently used devices.

* UX: Do not show 'Show more' button if there aren't more tokens.

* DEV: Fix unit tests.

* DEV: Make changes after code review.

* Add more detailed unit tests.

* Improve logging messages.

* Minor coding style fixes.

* DEV: Use DropdownSelectBoxComponent and run Prettier.

* DEV: Fix unit tests.
2018-10-09 22:21:41 +08:00
Guo Xiang Tan
77c293f82d Prefer Process.clock_gettime over Time.now. 2018-10-05 10:25:52 +08:00
Penar Musaraj
dc8a16b43b
Use Yarn to manage dependencies, cleanup unused bootstrap libs (#6436) 2018-10-03 10:15:43 -04:00
Guo Xiang Tan
13dde44c72 FIX: Limit concurrent assets:precompile to number of available cores. 2018-10-03 16:46:21 +08:00
David Taylor
b8c0e77908 DEV: apply plugin migrations when testing in Travis and Docker
In `RAILS_ENV=test`, plugins are not loaded by default. Therefore we need to explicitly specify `LOAD_PLUGINS=1` when we want to apply plugin migrations.
2018-09-13 12:39:54 +01:00
David Taylor
9461c99210 DEV: Add travis_fold to termination stage 2018-09-12 18:59:48 +01:00
David Taylor
26bd67a865 DEV: Add travis_fold statements to docker_test 2018-09-12 17:52:58 +01:00
Guo Xiang Tan
24c55bd613 Add dry run option to UploadRecovery. 2018-09-12 21:53:01 +08:00
Guo Xiang Tan
c053f8ccf6 New rake task uploads:recover. 2018-09-12 01:52:30 -07:00
Guo Xiang Tan
6d01e0aa04 DEV: Print the error class in uploads:list_posts_with_broken_images. 2018-09-12 01:06:51 -07:00
Blake Erickson
1d41f3c3fb
Merge pull request #6380 from discourse/rake-destroy-sub-category
FIX: Allow `rake destroy:topics` to delete topics in sub-categories
2018-09-10 10:26:04 -06:00
Guo Xiang Tan
94ff428571 Pass the right value to rake task. 2018-09-10 20:07:28 +08:00
David Taylor
84fc7abb73 FIX: Allow rake destroy:topics to delete topics in sub-categories 2018-09-10 12:52:14 +01:00
Guo Xiang Tan
4a966c639d DEV: Update uploads:list_posts_with_broken_images to recover from tombstone. 2018-09-10 17:01:41 +08:00
Guo Xiang Tan
68572b8afc Print error messages on why upload fails to save. 2018-09-10 16:02:13 +08:00
Guo Xiang Tan
0aca80e92a Fixes to uploads:list_posts_with_broken_images. 2018-09-10 15:16:29 +08:00
Guo Xiang Tan
8496537590 Add RECOVER_FROM_S3 to uploads:list_posts_with_broken_images rake task. 2018-09-10 15:14:30 +08:00
Gerhard Schlager
f13c34aaed Adds a check for invalid message formats to rake i18n:check 2018-09-06 17:27:17 +02:00
Gerhard Schlager
3c09026fe4 Minor improvements to rake i18n:check 2018-09-06 16:54:30 +02:00
Guo Xiang Tan
72834f19ff DEV: Add rake tasks to list posts with broken images. 2018-09-05 16:54:15 +08:00
Gerhard Schlager
f4956f79a5 Make Rubocop happy 2018-09-05 01:04:43 +02:00
Gerhard Schlager
6658a6601c Remove unused rake task 2018-09-05 00:47:39 +02:00
Gerhard Schlager
0d8c72d8c4 DEV: Add rake task to check locale files for errors 2018-09-05 00:47:39 +02:00
Guo Xiang Tan
8dc1463ab3 Enable Lint/ShadowingOuterLocalVariable for Rubocop. 2018-09-04 10:16:42 +08:00
Guo Xiang Tan
492b6dc306 DEV: yarn exec doesn't do what I think it does. 2018-08-30 15:55:36 +08:00
Guo Xiang Tan
720726ea68 Use yarn exec when running eslint and prettier. 2018-08-29 11:03:42 +08:00
Gerhard Schlager
6588a45880 Make Rubocop happy 2018-08-23 15:09:20 +02:00
Gerhard Schlager
409ee66839 Add optional "ignore_case" parameter to posts:remap rake task 2018-08-23 14:49:17 +02:00
Guo Xiang Tan
0f07494b12
Merge pull request #6259 from pfaffman/patch-2
FIX: disable_2fa fix method selection
2018-08-21 17:29:42 +08:00
Gerhard Schlager
802ad0dc61 Add rake task for updating static pages 2018-08-16 19:51:20 +02:00
Joffrey JAFFEUX
3fbf7b164d
FEATURE: emoji 11.0 "recently added" update (#6275)
Adds support for emojis in the http://www.unicode.org/emoji/charts-11.0/emoji-released.html list
2018-08-16 11:59:45 +02:00
Arpit Jalan
8e3b685aa4 make rubocop happy 2018-08-16 14:03:46 +05:30
Arpit Jalan
df55e5bac4 optimize categories:create_definition task 2018-08-16 14:00:07 +05:30
Arpit Jalan
78e14abb32 new rake task to bulk tag all topics in a category 2018-08-16 13:56:57 +05:30
Arpit Jalan
777055f960 new rake task to create category definition 2018-08-16 13:50:52 +05:30
Guo Xiang Tan
33f4aa2835 Fix migrate_to_s3 rake task, File#close! is not a valid method. 2018-08-16 13:26:58 +08:00
Jay Pfaffman
71a1d75d7e
FIX: disable_2fa fix method selection
The previous code resulted in

    NameError: undefined local variable or method `totp' for main:Object

I now understand what @tgxworld meant about we should only disable totp when I submitted this before.

This is the kind of Ruby stuff that I still don't understand well,(perhaps this isn't the most Ruby way to do this?) but this does what I think is supposed to happen. And it worked just now.
2018-08-10 14:45:40 -07:00
Sam
1fc2597626 better error handling for upload extension fixer 2018-08-10 11:28:22 +10:00
Guo Xiang Tan
0879610ffd Add missing require in uploads:fix_incorrect_extensions. 2018-08-08 15:39:37 +08:00
Guo Xiang Tan
17047806b9 Add a rake task to fix uploads with wrong extension. 2018-08-08 13:15:17 +08:00
Régis Hanol
14bbd5d167 FIX: use the right URL when downloading the file from S3 2018-08-02 10:04:41 +02:00
Régis Hanol
0f1137dafa FIX: 'migrate_from_s3' rake task wasn't handling short urls 2018-08-01 22:58:46 +02:00
Neil Lalonde
b829452c75
Merge pull request #6209 from discourse/mini_scheduler
REFACTOR: extract scheduler to the mini_scheduler gem
2018-08-01 10:28:24 -04:00
Neil Lalonde
4ad7ce70ce REFACTOR: extract scheduler to the mini_scheduler gem 2018-07-31 17:12:55 -04:00
Sam
f0c203a5cf clean up previous commit 2018-07-31 14:50:02 +10:00
Jay Pfaffman
b55d9e63a0 Rake mail test debugging (#6171)
* FEATURE: rake emails:test add debugging

* Oops! Remove safety code

* more language tweaks
2018-07-31 14:45:59 +10:00
Gerhard Schlager
e42038eae2 Rake task called method with wrong arguments 2018-07-24 00:10:09 +02:00
Jay Pfaffman
0ed2834c2d FEATURE: Add users:disable_2factor rake task
https://meta.discourse.org/t/admin-locked-out-of-2fa/92156/2?u=pfaffman
2018-07-16 09:56:55 +08:00
Arpit Jalan
536cef86f4 PERF: do not carry post ids in memory when rebaking all posts 2018-07-11 14:34:33 +05:30
Guo Xiang Tan
59a2767de8 Ensure that we restore the site setting in posts:rebake rake task. 2018-07-06 16:22:54 +08:00
Neil Lalonde
92000bc8a0 FEATURE: add a rake task to recalculate user stats post_count and topic_count 2018-07-05 10:38:46 -04:00
Sam
d1b21aa73b add timings to asset precompile 2018-07-04 09:42:38 +10:00
Gerhard Schlager
23acddc9fa Add descriptions for rake tasks 2018-06-27 20:24:46 +02:00
Joffrey JAFFEUX
2f7960bd2a
DEV: updates prettier and displays linters/prettifiers version in CI 2018-06-20 18:34:49 +02:00
Joffrey JAFFEUX
8126b603e4
fix prettier 2018-06-20 18:26:43 +02:00