Commit Graph

1117 Commits

Author SHA1 Message Date
David Taylor
bd8c2b4615
DEV: Correct theme:qunit help text (#13104)
- Task name is themes:qunit, not themes:unit
- Some shells try to expand the square brackets. The whole thing should be enclosed in quotes to avoid this
2021-05-20 22:30:12 +01:00
Arpit Jalan
626b8465ba
FIX: do not validate uploads when running uploads:fix_missing_s3 task (#13096) 2021-05-20 15:29:09 +10:00
Arpit Jalan
130160537c
FEATURE: add support for "skip_validations" option in UploadCreator (#13094)
FIX: do not validate uploads when running `uploads:fix_missing_s3` task
2021-05-19 20:54:52 +05:30
Martin Brennan
3d2cace94f
DEV: Add service to validate email settings (#13021)
We have a few places in the code where we need to validate various email related settings, and will have another soon with the improved group email settings UI. This PR introduces a class which can validate POP3, IMAP, and SMTP credentials and also provide a friendly error message for issues if they must be presented to an end user.

This PR does not change any existing code to use the new service. I have added a TODO to change POP3 validation and the email test rake task to use the new validator post-release.
2021-05-13 15:11:23 +10:00
Blake Erickson
37812b8e35
Revert "DEV: Drop old IE11 intersection-observer references" (#13017)
This reverts commit 7360a0f70f.

iOS still wants this sometimes. Probably best to revert for now and we can
always remove this again later.

See: https://meta.discourse.org/t/189799/11?u=blake
2021-05-10 17:47:09 -06:00
Blake Erickson
1b02dce594
DEV: Drop old IE11 intersection-observer references (#12942)
It's been awhile since we have supported IE11 so it should be safe to remove
IntersectionObserver now.

From a TODO task in this repo:
> drop when we eventually drop IE11

Announcement of when we removed IE11 support:

https://meta.discourse.org/t/137984/40?u=blake
2021-05-04 17:54:54 -06:00
Roman Rizzi
5794787300
FIX: Make the docker:test task install dependencies before calling yarn ember test. (#12943) 2021-05-04 16:30:35 -03:00
Jeff Wong
656b0ae39e
DEV: Add an option to skip a theme update from the themes:install task. (#12905)
A theme can now specify `skip_update: true` in the yml config for
update allowing for the theme to be installed only if it does not already
exist.
2021-04-30 09:31:41 -07:00
David Taylor
ad8c7714c8
DEV: Use filesystem-based SchemaCache in development (#12901)
In development we regularly restart/reload Rails, which wipes out the schema cache. This then has to be regenerated using DDL queries on the database.

Instead, we can make use of the `rake db:schema:cache:dump` command. This will dump the schema cache to a YAML file, and then load it when needed. This is significantly faster than rebuilding the cache from DDL queries every time.
2021-04-30 10:54:49 +01:00
Roman Rizzi
7423943ffe
DEV: The docker:test task can run JS tests using the Ember CLI. (#12879)
Set the EMBER_CLI env var to 1 to include these tests.
2021-04-29 14:51:34 -03:00
Osama Sayegh
4f88f2eb15
FEATURE: Allow theme tests to be run in production (take 2) (#12845)
This commit allows site admins to run theme tests in production via a new `/theme-qunit` route. When you visit `/theme-qunit`, you'll see a list of the themes/components installed on your site that have tests, and from there you can select a theme or component that you run its tests.

We also have a new rake task `themes:install_and_test` that can be used to install a list of themes/components on a temporary database and run the tests of the themes/components that are installed. This rake task can be useful when upgrading/deploying a Discourse instance to make sure that the installed themes/components are compatible with the new Discourse version being deployed, and if the tests fail you can abort the build/deploy process so you don't end up with a broken site.
2021-04-28 23:12:08 +03:00
Osama Sayegh
a169dc6832
Revert "FEATURE: Allow theme tests to be run in production (#12815)" (#12840)
This reverts commit 7217dcb67a.

https://meta.discourse.org/t/failed-to-bootstrap-due-to-out-of-memory-killer/188141/18?u=osama

Precompiling test_helper.js is so expensive that it can make bootstrap
fail on servers with limited resources (2GB RAM). We will find another
way that doesn't require much resources.
2021-04-26 23:05:58 +03:00
Osama Sayegh
7217dcb67a
FEATURE: Allow theme tests to be run in production (#12815)
This commit allows site admins to run theme tests in production via a new `/theme-qunit` route. When you visit `/theme-qunit`, you'll see a list of the themes/components installed on your site that have tests, and from there you can select a theme or component that you run its tests.

We also have a new rake task `themes:install_and_test` that can be used to install a list of themes/components on a temporary database and run the tests of the themes/components that are installed. This rake task can be useful when upgrading/deploying a Discourse instance to make sure that the installed themes/components are compatible with the new Discourse version being deployed, and if the tests fail you can abort the build/deploy process so you don't end up with a broken site.
2021-04-26 12:56:45 +03:00
Osama Sayegh
cd24eff5d9
FEATURE: Introduce theme/component QUnit tests (take 2) (#12661)
This commit allows themes and theme components to have QUnit tests. To add tests to your theme/component, create a top-level directory in your theme and name it `test`, and Discourse will save all the files in that directory (and its sub-directories) as "tests files" in the database. While tests files/directories are not required to be organized in a specific way, we recommend that you follow Discourse core's tests [structure](https://github.com/discourse/discourse/tree/master/app/assets/javascripts/discourse/tests).

Writing theme tests should be identical to writing plugins or core tests; all the `import` statements and APIs that you see in core (or plugins) to define/setup tests should just work in themes.

You do need a working Discourse install to run theme tests, and you have 2 ways to run theme tests:

* In the browser at the `/qunit` route. `/qunit` will run tests of all active themes/components as well as core and plugins. The `/qunit` now accepts a `theme_name` or `theme_url` params that you can use to run tests of a specific theme/component like so: `/qunit?theme_name=<your_theme_name>`.

* In the command line using the `themes:qunit` rake task. This take is meant to run tests of a single theme/component so you need to provide it with a theme name or URL like so: `bundle exec rake themes:qunit[name=<theme_name>]` or `bundle exec rake themes:qunit[url=<theme_url>]`.

There are some refactors to how Discourse processes JavaScript that comes with themes/components, and these refactors may break your JS customizations; see https://meta.discourse.org/t/upcoming-core-changes-that-may-break-some-themes-components-april-12/186252?u=osama for details on how you can check if your themes/components are affected and what you need to do to fix them.

This commit also improves theme error handling in Discourse. We will now be able to catch errors that occur when theme initializers are run and prevent them from breaking the site and other themes/components.
2021-04-12 15:02:58 +03:00
Penar Musaraj
e77b9dfd45
DEV: Use Terser for JS minification/compression if available (#12656) 2021-04-09 08:51:49 -04:00
Osama Sayegh
2b9ab3a0d9
Revert "FEATURE: Introduce theme/component QUnit tests (#12517)" (#12632)
This reverts commit a53d8d3e61 and 105634435f.

Reverted because the change broke some components. Will be added back in a few days.
2021-04-07 17:45:49 +03:00
Osama Sayegh
a53d8d3e61
FEATURE: Introduce theme/component QUnit tests (#12517)
This commit allows themes and theme components to have QUnit tests. To add tests to your theme/component, create a top-level directory in your theme and name it `test`, and Discourse will save all the files in that directory (and its sub-directories) as "tests files" in the database. While tests files/directories are not required to be organized in a specific way, we recommend that you follow Discourse core's tests [structure](https://github.com/discourse/discourse/tree/master/app/assets/javascripts/discourse/tests).

Writing theme tests should be identical to writing plugins or core tests; all the `import` statements and APIs that you see in core (or plugins) to define/setup tests should just work in themes.

You do need a working Discourse install to run theme tests, and you have 2 ways to run theme tests:

* In the browser at the `/qunit` route. `/qunit` will run tests of all active themes/components as well as core and plugins. The `/qunit` now accepts a `theme_name` or `theme_url` params that you can use to run tests of a specific theme/component like so: `/qunit?theme_name=<your_theme_name>`.

* In the command line using the `themes:qunit` rake task. This take is meant to run tests of a single theme/component so you need to provide it with a theme name or URL like so: `bundle exec rake themes:qunit[name=<theme_name>]` or `bundle exec rake themes:qunit[url=<theme_url>]`.

There are some refactors to internal code that's responsible for processing themes/components in Discourse, most notably:

* `<script type="text/discourse-plugin">` tags are automatically converted to modules.

* The `theme-settings` service is removed in favor of a simple `lib` file responsible for managing theme settings. This was done to allow us to register/lookup theme settings very early in our Ember app lifecycle and because there was no reason for it to be an Ember service.

These refactors should 100% backward compatible and invisible to theme developers.
2021-04-07 10:39:57 +03:00
Jeff Wong
3a22c654e2
FIX: rake themes:update should fail if a theme update fails (#12608) 2021-04-05 15:53:34 -07:00
Gerhard Schlager
acc73f2989
FIX: uploads:fix_missing_s3 rake task used wrong SHA1 (#12495) 2021-03-25 11:35:29 +01:00
Sam
dc6b547ed8
FIX: rake db:validate_indexes was broken (#12463)
A file was moved, but zeitwerk can not find it due to custom inflection.

Renamed so it can be properly found.
2021-03-24 08:53:01 +11:00
Penar Musaraj
aee7ef0dc9
DEV: Fix build due to highlight.js branch issue (#12441)
Highlight.js changed their default branch from master to main. This switches to the @highlightjs/cdn-assets package, thus sidestepping the problem. It's a slightly cleaner integration though (no need to build locally anymore).
2021-03-18 18:21:23 -04:00
Daniel Waterworth
e7ac906f21
DEV: Move TemporaryDB class to its own file (#12383)
rake tasks get repeated reloaded during the tests and it causes
redefinition warnings.
2021-03-12 11:27:13 -06:00
Sam
10001e4e8d
DEV: add db:validate_indexes task (#12273)
Added a new task to test if indexes are coherent with a blank database

This allows us to detect for cases where somehow indexes are out of sync

FIX_INDEXES=1 or `rake db:validate_indexes[fix]` to correct the issues it finds. 

Detects: 

- Badly named indexes that need to be renamed
- Missing indexes
- Extra indexes

Can correct all 3 with the fix option
2021-03-10 10:59:20 +11:00
Penar Musaraj
6809cccd88
DEV: Add experimental json_scheme site setting type (#12226) 2021-03-01 09:15:17 -05:00
Penar Musaraj
45c5fd2172
DEV: Remove JoyPixels emoji option (#12197)
- removes the option from site settings
- deletes the site setting on existing sites that have it
- marks posts using emojis as requiring a rebake

Note that the actual image files are not removed here, the plan is to
remove them in a few weeks/months (when presumably the rebaking of old
posts has been completed).
2021-02-26 07:44:52 -05:00
Rafael dos Santos Silva
01584c5b22
FIX: emails.rake-use domain when sending mail test (#12188)
`rake emails:test` task was always sending `localhost` as the domain name rather than using `smtp[:domain]` (aka `DISCOURSE_SMTP_DOMAIN`. `discourse-setup` has recently been modified to always set `DISCOURSE_SMTP_DOMAIN`, so it's important that this test rake task actually use the value.

I tested this change on a standard production install, and it's working as expected. Hopefully I managed not to bungle the copy/paste of the single line here in the github edit window.

Co-authored-by: Jay Pfaffman <jay@literatecomputing.com>
2021-02-23 14:16:51 -03:00
Joshua Rosenfeld
b79c06f3b7
Include UI commits when generating release notes. (#12127)
`UX` is the officially supported prefix per https://meta.discourse.org/t/19392, but sometimes `UI` is used instead. We should still include those commits.
2021-02-18 10:53:23 -05:00
Bianca Nenciu
08acf51be0
FEATURE: Use diffhtml to update composer preview (#11237)
Displaying videos, animated GIFs or any kind of rich content in preview
used to refresh on every keystroke, which could cause performance
problems.
2021-02-18 16:07:26 +02:00
Jeff Wong
f17e745fe3
DEV: log rake plugin:update_all plugin_path on error (#12100)
* DEV: log rake plugin:update_all plugin_path on error

Allows a dev to know which plugin was failing to update
2021-02-16 11:50:27 -08:00
Jarek Radosz
abe688beb3 DEV: Use discourse/discourse_test in CI workflows
Using our testing Docker image (`discourse/discourse_test:release`) allows us to drop "Update imagemagick" step which shaves ~10 minutes from all runs.
2021-02-11 15:24:15 +01:00
Penar Musaraj
a1955b0542
Dev: --dev option does nothing in yarn (#12032) 2021-02-10 19:19:47 -05:00
Jarek Radosz
43948f6a10
DEV: Support for main branches in plugin:update (#12027)
Based on https://github.com/discourse/docker_manager/pull/94
2021-02-10 11:34:17 -05:00
T
1a159de36f
tasks/emails.rake: catch exception and puts message (#12024)
* tasks/emails.rake: catch exception and puts message

* add username and auth
2021-02-10 10:42:21 -05:00
Penar Musaraj
3da2e85855
FIX: Save updated remote theme, only update enabled themes/components (#11960) 2021-02-04 14:44:07 +11:00
Vinoth Kannan
9d2eaec88f
DEV: enable CORS to all CDN get requests from workbox. (#11896)
To prevent opaque cache files, now all the CDN files will be requested in 'cors' mode if the cdn_cors_enabled global setting is enabled. Before enabling the setting, should enable the cors in the CDN server by adding the response header `access-control-allow-origin: *` or `access-control-allow-origin: https://discourse.example.com.`

And other external file requests other than CDN will not be cached if the response type is opaque.
2021-02-02 11:38:29 +05:30
Martin Brennan
f49e3e5731
DEV: Add security_last_changed_at and security_last_changed_reason to uploads (#11860)
This PR adds security_last_changed_at and security_last_changed_reason to uploads. This has been done to make it easier to track down why an upload's secure column has changed and when. This necessitated a refactor of the UploadSecurity class to provide reasons why the upload security would have changed.

As well as this, a source is now provided from the location which called for the upload's security status to be updated as they are several (e.g. post creator, topic security updater, rake tasks, manual change).
2021-01-29 09:03:44 +10:00
Joshua Rosenfeld
b70e0d88f7
Include accessibility commits when generating release notes (#11884) 2021-01-28 12:58:04 -05:00
Penar Musaraj
a9dfc5a547
DEV: Use absolute path for popper.js sourcemap (#11762) 2021-01-20 08:59:34 -05:00
Jarek Radosz
650e3d18c4
DEV: Add a note to S3 migration task (#11738)
A followup to #11703
2021-01-18 17:12:47 +01:00
Michael K Johnson
2a23e54632
FIX: remove migrate_from_s3 task that silently corrupts data (#11703)
Transient errors in migration are ignored, silently corrupting
data, and the migration is incomplete and misses many sources of
uploads, which will lead to an incorrect expectation of independence
from the remote object storage after announcing that the migration
was successful, regardles of whether transient errors permanently
corrupted the data.

Remove this migration until such time as it is re-written to
follow the same pattern has the migration to s3, moving the
core logic out of the task.
2021-01-17 22:33:29 +01:00
Penar Musaraj
820b3e672a
DEV: Remove jquery.ba-resize (#11457) 2020-12-11 11:36:32 -05:00
Neil Lalonde
aa09d83453 DEV: remove PR numbers from release_note:generate output 2020-11-30 17:46:05 -05:00
Robin Ward
059e9cb1d2 REFACTOR: Remove .erb from emoji-picker
We can't use erb in ember-cli, and it seems the emoji groups rarely
change anyway. This commit migrates the ERB to pre-rendered javascript
that is updated via the `rake javascript:update_constants` task.
2020-11-25 11:23:02 -05:00
Sam
c540bf5e2b
FIX: correct cdn path (#11324)
This was a typo in a118ec13
2020-11-23 13:03:49 +11:00
Sam
a118ec13bc
FIX: stop including GlobalPath in default context (#11323)
We do not want these method names to clash, instead encapsulate the helpers
so we do not add methods to Kernel

Correct a but exposed by Ruby 2.7
2020-11-23 12:59:45 +11:00
Robin Ward
75e92e1bd7 REFACTOR: The Favcount library needs global variables
This moves the library into our lib folder, and refactored it to more
modern Javascript. I've kept the MIT license at the top of the file.

Doing this allows us to import it as a library in Ember CLI and ditch
yet another global variable.
2020-11-17 15:04:17 -05:00
Dan Ungureanu
bc8423a1bf
FEATURE: Add auto update field to themes (#11102)
Themes marked for auto update will be automatically updated when
Discourse is updated. This is triggered by discourse_docker or
docker_manager running Rake task 'themes:update'.
2020-11-16 14:44:09 +02:00
Gerhard Schlager
5c662128d3
DEV: Rake task for creating admin should ask for full name if required (#11096) 2020-11-02 11:12:42 +01:00
Vinoth Kannan
af4938baf1
Revert "DEV: enable cors to all cdn get requests from workbox. (#10684)" (#11076)
This reverts commit e3de45359f.

We need to improve out strategy by adding a cache breaker with this change ... some assets on CDNs and clients may have incorrect CORS headers which can cause stuff to break.
2020-10-30 16:05:35 +11:00
Vinoth Kannan
e3de45359f
DEV: enable cors to all cdn get requests from workbox. (#10685)
Now all external requests from the service worker will be in CORS mode without credentials.
2020-10-28 23:36:19 +05:30
Neil Lalonde
d77e30a011
DEV: generated release notes shouldn't include squashed commits (#10936)
Limit git log output to the first line of the commit message
when generating the list of commits in a release. Some commit messages
are including the names of all commits that were squashed, resulting
in duplicate and confusing lines in the release notes.
2020-10-15 16:20:32 -04:00
Justin DiRose
f4f2e081d7
FIX: Resolve issues running qunit via rake (#10922)
discourse-perspective-api was not successfully running tests via the
qunit:test rake task due to inconsistent naming between core and the
repo. As a result we no longer need the mapping in the plugin rake task, too.
2020-10-14 14:05:10 -05:00
Penar Musaraj
caa58a4bd1
DEV: Retry when Net::HTTP throws EOFError (#10921)
Might fix an inconsistent issue when running tests in CI.
2020-10-14 11:55:26 -04:00
Robin Ward
4a8f5dbfbd REFACTOR: Remove .erb file from javascript tests
We can't use erb in Ember CLI (since it does not have Ruby) so this has
been ported to use our `javascript:update_constants` rake test instead.

Note we don't have to run this every time a notification type as it's
only used by fixtures to fill in some specific types we test against.
2020-10-08 18:36:09 -04:00
Robin Ward
23f24bfb51 REFACTOR: Move javascript tests inside discourse app
This is where they should be as far as ember is concerned. Note this is
a huge commit and we should be really careful everything continues to
work properly.
2020-10-02 11:29:36 -04:00
jbrw
8dac8abdb4
DEV - write JS files to versioned subdirectories (#10707)
* DEV - write JS files to versioned subdirectories

* Cleaner JS in cacheBuster
2020-09-21 15:06:56 -04:00
Neil Lalonde
f1743ff69c
FIX: error "unknown attribute verified" in uploads rake tasks
The new column is verification_status.
verified: nil is now verification_status: unchecked
2020-09-21 10:47:52 -04:00
Martin Brennan
80268357e7
DEV: Change upload verified column to be integer (#10643)
Per review https://review.discourse.org/t/dev-add-verified-to-uploads-and-fill-in-s3-inventory-10406/14180

Change the verified column for Upload to a verified_status integer column, to avoid having NULL as a weird implicit status.
2020-09-17 13:35:29 +10:00
jbrw
033cebf978
DEV - versions of JS files written to a JS file to be included by loa… (#10649)
* DEV - versions of JS files written to a JS file to be included by load-script and appended as params to URLs

* Formatting

* Incorporate feedback from PR

* Update filename of public-js-versions
2020-09-11 13:53:56 -04:00
Joffrey JAFFEUX
a60ba75ac2
FIX: prevents eslint failure on unmatched patterns in plugins (#10648)
The recent upgrade of eslint fails if no files is found for the defined patterns.
2020-09-10 20:17:40 +02:00
Penar Musaraj
7a0d1cadc5
DEV: Add option to precompile default multisite stylesheets 2020-09-03 13:00:09 -04:00
Daniel Waterworth
5bb30d8016 FIX: whoops, linting 2020-09-03 12:10:11 +01:00
Daniel Waterworth
6518cac5cf DEV: Add log rake task
Now, if you do `rake log db:migrate`, it'll run migrations, but with
logging enabled, so that you can see what is actually happening.
2020-09-03 11:43:30 +01:00
Joffrey JAFFEUX
02f0637892
FIX: updates highlightjs to latest version to avoid a freeze (#10578)
Example repro of the freeze:

```
```css
[
```
2020-09-02 15:14:48 +02:00
Jay Pfaffman
5e66d7d082
FIX: emails.rake use authentication (#10572)
It looks like support got added for DISCOURSE_SMTP_AUTHENTICATION in Discourse, but not in this rake task (or perhaps it's always been broken?).

N.B.: I have not tested this code, but it reportedly works and looks safe to me.

https://meta.discourse.org/t/smtp-unrecognized-authentication-type-with-office-365/145830/6?u=pfaffman
2020-09-01 11:08:18 -04:00
Daniel Waterworth
a3580fa292 DEV: Show plugin versions in CI 2020-09-01 16:05:07 +01:00
Jarek Radosz
7290ca1f4a DEV: Remove jquery.cookie
A follow up to c172f2068d

cc: @eviltrout
2020-08-31 11:53:01 +02:00
Sam Saffron
80e17f5f9e
DEV: Remove staff bypass on fix missing
After thinking about it, I worry that this will potentially leave a site
setting set when people hit ctrl-c ... feels a tiny bit risky, so leaving
it out.
2020-08-28 12:35:35 +10:00
Sam Saffron
ed9323043f
DEV: Allow all uploads when fixing missing s3
If we do not do this we can not properly re-download files
2020-08-28 12:28:59 +10:00
Sam Saffron
f7314f8ab4
DEV: Improve s3 upload image analysis
- Introduces uploads:delete_missing_s3 which can be used to "give up" and
delete broken records from the database

- Fixes a bug in fix_missing_s3 - crashing on deleted posts

- Adds more info to analyze_missing_s3
2020-08-27 11:50:07 +10:00
Sam Saffron
b60c39fdf0
DEV: search more carefully for missing uploads
rake uploads:analyze_missing_s3 was not looking at all places, amend it so
it looks in all the places where uploads could exist.
2020-08-26 17:48:42 +10:00
Gerhard Schlager
208d85aaff DEV: Ensure prettier uses the same patterns everywhere 2020-08-20 16:27:32 +02:00
Sam Saffron
2a7490149c
DEV: don't fail if in uploads:fix_missing_s3 when fix fails
Previously a single error on a file like invalid extension could fail the
entire rake task
2020-08-18 17:55:49 +10:00
Sam Saffron
24fe08230f
FEATURE: ensure posts are rebaked when missing is fixed
This ensures any corrupt optimized images are removed and re-created
2020-08-18 15:37:24 +10:00
Vinoth Kannan
476d26159a
FEATURE: add new user option skip_new_user_tips. (#10437)
And add new site setting `default_other_skip_new_user_tips` in user preferences category.
2020-08-14 19:10:56 +05:30
Penar Musaraj
eae8b0465c
DEV: Upgrade ACE Editor to 1.4.12, remove extra ACE files (#10431) 2020-08-13 21:53:27 -04:00
Sam Saffron
5be26c7d24
DEV: add error handling in case download fails 2020-08-13 13:48:23 +10:00
Jeff Wong
787ad7d84d FIX: allow plugin pinning to fetch missing commits
Add update for fetching git commits if they do not exist, eg with
clone --depth 1 - only can fetch via git fetch --depth 1 {remote} {ref}
the ref needs to be a full, non-ambiguous reference.
2020-08-12 20:03:37 -07:00
Sam Saffron
5011435ec7
DEV: do not correct sha when correctly uploads 2020-08-13 11:52:57 +10:00
Sam Saffron
f48fa30ecd
DEV: fix_missing_s3 attempts to re-download if missing
unverified uploads get re-downloaded and corrected if they exist in the task
2020-08-13 11:22:14 +10:00
David Taylor
451b9b245f
DEV: Rename new upload rake tasks
These tasks are s3-specific, so update the names to make that very clear
2020-08-12 23:26:13 +01:00
Sam Saffron
ec173a72d9
DEV: add more counts to analyze missing 2020-08-12 17:28:58 +10:00
Sam Saffron
97fe68e980
DEV: look for avatars when analyzing missing uploads 2020-08-12 14:04:21 +10:00
Sam Saffron
990cd8c2e4
FEATURE: introduce tasks for dealing with legacy broken uploads
`rake uploads:analyze_missing` can be used get rich information regarding
uploads missing from s3 (where verified is false)

`rake uploads:fix_missing` is a work in progress task for automatically
correcting certain historic issues. At the moment it simply rebakes all
posts with missing uploads, but it will improve over time
2020-08-12 13:33:04 +10:00
Daniel Waterworth
abebc4e05e DEV: Use verbose mode when running tests in CI 2020-08-07 17:04:55 +01:00
Jarek Radosz
cd4f251891
FEATURE: Poll breakdown 2.0 (#10345)
The poll breakdown modal replaces the grouped pie charts feature.

Includes:

* MODAL: Untangle `onSelectPanel`
Previously modal-tab component would call on click the onSelectPanel callback with itself (modal-tab) as `this` which severely limited its usefulness. Now showModal binds the callback to its controller.

"The PR includes a fix/change to d-modal (b7f6ec6) that hasn't been extracted to a separate PR because it's not currently possible to test a change like this in abstract, i.e. with dynamically created controllers/components in tests. The percentage/count toggle test for the poll breakdown feature is essentially a test for that d-modal modification."
2020-08-06 17:57:06 +02:00
Guo Xiang Tan
850d5696f2
FIX: Disable concurrent migration for multisite:migrate.
Rails migration is not thread safe.
2020-08-05 15:12:56 +08:00
Guo Xiang Tan
1d281353b6
FIX: Don't run seeds if multisite migration fails. 2020-08-05 15:12:56 +08:00
Sam Saffron
e8328968bb
DEV: allow qunit:test to run concurrently
Previously we were fighting on pid file which makes it hard to run multiple
2020-08-05 16:57:12 +10:00
Régis Hanol
ef92b72ff6 DEV: ensure 'posts:reorder_posts' rake task work on single topic
when the topic_id argument is provided.
2020-08-04 14:57:49 +02:00
Régis Hanol
712ab33ff8 DEV: fix 'posts:reorder' rake task
Running the reorder rake task was triggering the following error:

PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "post_timings_unique"

I re-worked the queries and refactored to use the same couple of queries for all similar tables/columns.
2020-08-04 11:07:07 +02:00
Krzysztof Kotlarek
e0d9232259
FIX: use allowlist and blocklist terminology (#10209)
This is a PR of the renaming whitelist to allowlist and blacklist to the blocklist.
2020-07-27 10:23:54 +10:00
Robin Ward
5e5973cabe FIX: Our test build of highlight.js was broken
Some definitions rely on others, in particular the c/cpp/c-like ones,
and we were appending the bundle of all files in the folder.

Instead for testing I've limited us to just three definitions. This has
the benefit of being a lot smaller to download/parse in test mode too.
2020-07-20 15:34:24 -04:00
Robin Ward
9889b7277f FIX: Silence route-recognizer source map errors in development mode 2020-07-15 15:42:04 -04:00
Robin Ward
e8bf304f05 FIX: Add popper sourcemap 2020-07-15 15:29:12 -04:00
Jeff Wong
339549d14a
Support plugin and Theme compatibility version manifests (#9995)
Adds a new rake task `plugin:checkout_compatible_all` and
`plugin:checkout_compatible[plugin-name]` that check out compatible plugin
versions.

Supports a .discourse-compatibility file in the root of plugins and themes that
list out a plugin's compatibility with certain discourse versions:

eg: .discourse-compatibility
```
2.5.0.beta6: some-git-hash
2.4.4.beta4: some-git-tag
2.2.0: git-reference
```

This ensures older Discourse installs are able to find and install older
versions of plugins without intervention, through the manifest only.

It iterates through the versions in descending order. If the current Discourse
version matches an item in the manifest, it checks out the listed plugin target.
If the Discourse version is greater than an item in the manifest, it checks out
the next highest version listed in the manifest.

If no versions match, it makes no change.
2020-07-06 14:48:00 -07:00
Blake Erickson
69803599a9 DEV: Refactor seed data filter
Added a small helper class to for seed data because we need to add the
same filter to multisite:migrate as we have in db:migrate. Having this
filter in both places means we can get rid of the SKIP_SEED flag.
2020-06-26 14:36:50 -06:00
Guo Xiang Tan
c16ad39f8e
DEV: Run seeds irregardless of post deploy migration flag.
Follow up to 01937b2d
2020-06-26 11:04:34 +08:00
Guo Xiang Tan
01937b2de2
Revert "FIX: Seed needs to run before optimizing site icons."
This reverts commit 715ddf3861.
2020-06-26 11:03:47 +08:00
Guo Xiang Tan
715ddf3861
FIX: Seed needs to run before optimizing site icons. 2020-06-26 08:58:53 +08:00
Blake Erickson
01b6349a67
DEV: Add skip seed flag (#10116)
* add a flag to skip seed

* only seed when running post deployment migrations
2020-06-25 10:14:58 -06:00
Guo Xiang Tan
0384b6d910
FIX: multisite:migrate failing to properly seed data. 2020-06-23 09:10:02 +08:00
Dan Ungureanu
2e1efbde52
FIX: Pass local scope variable to inner function 2020-06-22 19:23:59 +03:00
David Taylor
e29afa200a
FIX: Cleanup migrations with timestamps in the future
A future-dated migration was accidently introduced by me in 45c399f0. This was removed in b9762afc, but other migrations had already been generated based on its incorrect date. This commit removes the offending data in the schema_migrations table, and corrects the version in the published_pages migration.

This commit also adds a check to db:migrate which raises an error when invalid migration timestamps are used.
2020-06-17 15:58:22 +01:00
Guo Xiang Tan
45eb97c202
FIX: Thread safety issues with multisite:migrate and SeedFu. 2020-06-17 16:15:43 +08:00
Jarek Radosz
669c940ec3 Revert "DEV: Remove the remaining ENV["TRAVIS"] usage (#10041)"
This reverts commit 78aff841e3.

See https://review.discourse.org/t/dev-remove-the-remaining-env-travis-usage-10041/12737/4?u=cvx
2020-06-16 19:42:00 +02:00
Jarek Radosz
78aff841e3
DEV: Remove the remaining ENV["TRAVIS"] usage (#10041) 2020-06-16 17:41:15 +02:00
Blake Erickson
a2713578dd DEV: Allow plugins to exclude seed data
This allows plugins to specify if they would like to filter out any seed
data files from running during migrations.
2020-06-15 15:30:25 -06:00
Kane York
5f3fd23fa8
DEV: Add stub flush_sw task to prevent rebuild errors (#10021) 2020-06-10 13:07:37 -07:00
Joffrey JAFFEUX
d4caf69ed7
DEV: makes SKIP_INSTALL_PLUGINS called last and global (#9990) 2020-06-05 17:59:23 +02:00
Sam Saffron
1c48853ede
DEV: allow skipping of a single plugin when installing
SKIP_INSTALL_PLUGINS can be used to tell plugin:install_all_official
to skip a plugin.

Comma seperated list
2020-06-05 18:26:25 +10:00
Joshua Rosenfeld
76af25f753
FEATURE: Support append when bulk tagging via rake (#9978)
New argument option for the `tags:bulk_tag_category` rake task.
Backwards compatible - defaults to false (no append) if no argument given.
2020-06-04 09:33:48 -04:00
Arpit Jalan
07774894a6 DEV: notify when the rake ask is completed 2020-06-04 12:00:06 +05:30
Arpit Jalan
a93d24501c FIX: base import script was not updating first_post_created_at column
FEATURE: new rake task to update first_post_created_at column

The not-equal operator (`<>`) in PostgreSQL does not compare values
with NULL. We should instead use `IS DISTINCT FROM` when comparing
values with NULL.
2020-06-04 11:26:40 +05:30
Michael K Johnson
81e6bc7a0f
FEATURE: Add uploads:batch_migrate_from_s3 task to limit total posts migrated at once (#9933)
Allow limiting the number of migrations to do at once, both to do migrations that
have impact limited to multiple off-peak usage hours to reduce user impact from
a migration, and to allow tests that do only a very small number for test
purposes. ("Give me a ping, Vasili. One ping only, please.")
2020-06-04 09:48:11 +10:00
Gerhard Schlager
f683c5d0e0 DEV: Check English locale for errors in CI
Moves the most important checks into a linter. It gets executed by Lefthook as well as the docker rake task and Github actions. Doing those checks in rspec takes too long and it produces errors when the discourse:test Docker image contains old, invalid locale files.
2020-06-03 21:54:58 +02:00
Sam Saffron
57a3d4e0d2
FEATURE: whitelist theme repo mode (experimental)
In some restricted setups all JS payloads need tight control.

This setting bans admins from making changes to JS on the site and
requires all themes be whitelisted to be used.

There are edge cases we still need to work through in this mode
hence this is still not supported in production and experimental.

Use an example like this to enable:

`DISCOURSE_WHITELISTED_THEME_REPOS="https://repo.com/repo.git,https://repo.com/repo2.git"`

By default this feature is not enabled and no changes are made.

One exception is that default theme id was missing a security check
this was added for correctness.
2020-06-03 13:19:57 +10:00
Aman Gupta Karmani
6358e79f0f
FIX: catch NoMethodError during precompile_css too (#9940)
Ran into this on heroku:

    NoMethodError: undefined method `enabled?' for #<Theme:0x0000558f069e7718>
    vendor/bundle/ruby/2.6.0/gems/activemodel-6.0.1/lib/active_model/attribute_methods.rb:431:in `method_missing'
    app/models/theme.rb:155:in `block (2 levels) in transform_ids'
    vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/delegation.rb:85:in `each'
    vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/delegation.rb:85:in `each'
    vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/query_methods.rb:260:in `select'
    vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/query_methods.rb:260:in `select'
    app/models/theme.rb:155:in `block in transform_ids'
    app/models/theme.rb:105:in `get_set_cache'
    app/models/theme.rb:140:in `transform_ids'
    app/models/theme.rb:321:in `list_baked_fields'
    app/models/theme.rb:317:in `resolve_baked_field'
    lib/stylesheet/manager.rb:289:in `theme_digest'
2020-06-02 15:18:03 +10:00
Joffrey JAFFEUX
7635c18a14
DEV: ensures highlightjs is correctly tested (#9923) 2020-05-29 13:05:44 +02:00
Michael Brown
d9a02d1336
Revert "Revert "Merge branch 'master' of https://github.com/discourse/discourse""
This reverts commit 20780a1eee.

* SECURITY: re-adds accidentally reverted commit:
  03d26cd6: ensure embed_url contains valid http(s) uri
* when the merge commit e62a85cf was reverted, git chose the 2660c2e2 parent to land on
  instead of the 03d26cd6 parent (which contains security fixes)
2020-05-23 00:56:13 -04:00
Jeff Atwood
20780a1eee Revert "Merge branch 'master' of https://github.com/discourse/discourse"
This reverts commit e62a85cf6f, reversing
changes made to 2660c2e21d.
2020-05-22 20:25:56 -07:00
Martin Brennan
779dc30d2e FIX: Join bookmarks migration on users to avoid missing user records
* the post_actions table has no FK to users, so if a user has been
  deleted we may end up with dangling post_action records, which then
  interferes with the bookmarks migration because bookmarks DO have
  an FK to users
2020-05-19 16:31:05 +10:00
Kane York
869f9b20a2
PERF: Dematerialize topic_reply_count (#9769)
* PERF: Dematerialize topic_reply_count

It's only ever used for trust level promotions that run daily, or compared to 0. We don't need to track it on every post creation.

* UX: Add symbol in TL3 report if topic reply count is capped

* DEV: Drop user_stats.topic_reply_count column
2020-05-14 15:42:00 -07:00
Robin Ward
42b300b9be DEV: Skip I18n in plugins until they can be updated 2020-05-14 10:40:53 -04:00
Rafael dos Santos Silva
1c0c425f38
DEV: Detect correct data dir using ENV 2020-05-12 13:10:57 -03:00
Robin Ward
8311374a5a DEV: Replace version.js.erb with pre generated file 2020-05-11 15:43:09 -04:00
Robin Ward
d2b16a7618
DEV: Generate emoji/data on demand (#9744)
These emoji rarely change and can be re-generated when we make changes
to them.
2020-05-11 15:29:46 -04:00
Robin Ward
34e5f807ec DEV: User UPPER_CASE for constants 2020-05-08 15:50:55 -04:00
Robin Ward
7f373e8b93 DEV: Don't use js.erb for constants
Adds a new rake task to auto generate a constants.js file with the
constants present. This makes migrating to Ember CLI easier, but also
slightly speeds up asset compilation by having to do less work.

If the constants change you need to run:
`rake javascripts:update_constants`
2020-05-08 14:14:01 -04:00
David Taylor
72ad701df0
DEV: Stub #flush in StdOutDemux for multisite:migrate
https://meta.discourse.org/t/multisite-migrate-error/150579/2
2020-05-06 11:58:35 +01:00
Krzysztof Kotlarek
9bff0882c3
FEATURE: Nokogumbo (#9577)
* FEATURE: Nokogumbo

Use Nokogumbo HTML parser.
2020-05-05 13:46:57 +10:00
Robin Ward
08fbf199ad FIX: S3 rake task can ignore yarn.lock 2020-04-29 13:14:04 -04:00
Robin Ward
9ec908950d DEV: Better error handling for s3 task 2020-04-29 12:54:39 -04:00
Penar Musaraj
ec2943c5bc
DEV: Update jquery.fileupload and dependencies (#9466) 2020-04-28 10:39:29 -04:00
Sam Saffron
6b62d75c50
FIX: remove unsuppored rake task
rebuilding user_actions is not something that should be done.

Plugins such as solved and assigned extend it, there are tons of
little rules that were not captured in `user_actions:rebuild`
2020-04-28 16:27:42 +10:00
Rafael dos Santos Silva
b810fc282d FEATURE: Digital Ocean Spaces compatibility for Object Storage
DO does not implement tagging support for S3 objects. Removing our default
empty tag fixes compatibility.

The expire_missing_assets rake task can't be used with that service still,
but this patch allows normal operation.
2020-04-22 13:57:14 -03:00
Martin Brennan
628ba9d1e2
FEATURE: Promote bookmarks with reminders to core functionality (#9369)
The main thrust of this PR is to take all the conditional checks based on the `enable_bookmarks_with_reminders` away and only keep the code from the `true` path, making bookmarks with reminders the core bookmarks feature. There is also a migration to create `Bookmark` records out of `PostAction` bookmarks for a site.

### Summary

* Remove logic based on whether enable_bookmarks_with_reminders is true. This site setting is now obsolete, the old bookmark functionality is being removed. Retain the setting and set the value to `true` in a migration.
* Use the code from the rake task to create a database migration that creates bookmarks from post actions.
* Change the bookmark report to read from the new table.
* Get rid of old endpoints for bookmarks
* Link to the new bookmarks list from the user summary page
2020-04-22 13:44:19 +10:00
tshenry
e41f1e6ec2
FEATURE: Rake task to export groups (#9450) 2020-04-17 14:59:54 -07:00
Martin Brennan
51672b9121
FIX: Minor bookmark with reminder issue cleanup (#9436)
* Count user summary bookmarks from new Bookmark table if bookmarks with reminders enabled
* Update topic user bookmarked column when new topic bookmark changed
* Make in:bookmarks search work with new bookmarks
* Fix batch inserts for bookmark rake task (and thus migration). We were only inserting one bookmark at a time, completely defeating the purpose of batching!
2020-04-16 11:32:21 +10:00
Mark VanLandingham
f07c4a781c
DEV: Add options to theme install rake task - more options (#9394) 2020-04-14 09:17:00 -05:00
Sam Saffron
e2284cf739
Revert "We have had errors reported due to migrations breaking and are reverting"
This reverts commit 8b46f14744.

It corrects the reason for the revert:

We rely on SafeMigrate existing cause we call it from migrations,
Zeitwerk will autoload it.

Instead of previous pattern we explicitly bypass all the hacks in
production mode.

We need to disable SafeMigrate cause it is not thread safe.

A thread safe implementation is possible but not worth the effort,
we catch the issues in dev and test.
2020-04-14 11:31:07 +10:00
Robin Ward
8b46f14744 We have had errors reported due to migrations breaking and are reverting
these series of commits.

See:
https://meta.discourse.org/t/new-installation-fails-on-migration/147425

This reverts commit 80e832662b.
This reverts commit de5f2d3308.
This reverts commit 6b192d29fa.
This reverts commit a5b582f686.
This reverts commit 708dd97dfd.
2020-04-09 14:18:56 -04:00
Sam Saffron
80e832662b
DEV: fake stdout must respond to close
If it does not respond to close Logger will refuse to use it
2020-04-09 14:36:01 +10:00
Sam Saffron
a5b582f686
DEV: demux stdout when running multisite migrate
This avoids mixing up output in such a way that we can not tell which
site ran which migrations

Avoids threads all fighting for output
2020-04-09 12:50:14 +10:00
Sam Saffron
708dd97dfd
PERF: speed up migrations on multisite
Previously we were migrating multisites serially, this is extremely slow
especially when 200 dbs are involved.

The new implementation defaults to running 20 migrations concurrently, leading
to a 20x speedup.

We also amended it so errors are printed out last, something that makes
debugging failures easier.

This is code specific to Discourse cause we integrate SeedFu with our
migrations and can not include this in the multisite gem.
2020-04-09 11:58:20 +10:00
Sam Saffron
f062ebf274
DEV: reduce noise in logs regarding migration
If a migration performs no changes it should not output stuff.

Previously we would output information about seeds which was very noisy.

On multisite this was particularly bad
2020-04-09 10:05:22 +10:00
Sam Saffron
8a50ab45aa
DEV: use REPORT_REQUESTS=1 to find all requests
We can use this to profile our pretender and ensure nothing is superfluous
and nothing is missing
2020-04-02 16:01:49 +11:00
Martin Brennan
6f978bc95c
FIX: First pass to improve efficiency of secure uploads rake task (#9284)
Get rid of harmful each loop over uploads to update. Instead we put all the unique access control posts for the uploads into a map for fast access (vs using the slow .find through array) and look up the post when it is needed when looping through the uploads in batches.

On a Discourse instance with ~93k uploads, a simplified version of the old method takes > 1 minute, and a simplified version of the new method takes ~18s and uses a lot less memory.
2020-03-26 15:59:57 +10:00
Martin Brennan
efd5fb665b
DEV: Fix flaky time sensitive uploads.rake specs (#9283)
Also fix issues in spec where certain uploads were not considered secure
2020-03-26 13:31:39 +10:00
Martin Brennan
097851c135
FIX: Change secure media to encompass attachments as well (#9271)
If the “secure media” site setting is enabled then ALL files uploaded to Discourse (images, video, audio, pdf, txt, zip etc. etc.) will follow the secure media rules. The “prevent anons from downloading files” setting will no longer have any bearing on upload security. Basically, the feature will more appropriately be called “secure uploads” instead of “secure media”.

This is being done because there are communities out there that would like all attachments and media to be secure based on category rules but still allow anonymous users to download attachments in public places, which is not possible in the current arrangement.
2020-03-26 07:16:02 +10:00