Why this change?
Similar to d0117ff6e3, `plugins:update_all` spends most of its time waiting
on the network. On my local machine, this takes up to 2 mins when I have
all the official plugins installed. On a 32 cores machine, the total
time is cut down to 4 seconds.
What does this change do?
1. Move the logic in the `plugin:update` Rake task into a method.
2. Updates the `plugin:update` and `plugin:update_all` to rely on the
new method.
3. Wraps the method call to update a plugin in `plugin:update_all` in a
`Concurrent::Promise`
This change also adds the `--quiet` option to the `git pull` option
since the `git pull` output is just noise for 99% of the time.
Followup to e37fb3042d
Some plugins like discourse-ai and discourse-saml do not
nicely change from kebab-case to Title Case (e.g. Ai, Saml),
and anyway this method of getting the plugin name is not
translated either.
Better to use the plugin setting category if it exists,
since that is written by a human and is translated.
* DEV: Convert approve_new_topics_unless_trust_level to groups
This change converts the `approve_new_topics_unless_trust_level` site
setting to `approve_new_topics_unless_allowed_groups`.
See: https://meta.discourse.org/t/283408
- Hides the old setting
- Adds the new site setting
- Add a deprecation warning
- Updates to use the new setting
- Adds a migration to fill in the new setting if the old setting was
changed
- Adds an entry to the site_setting.keywords section
- Updates tests to account for the new change
After a couple of months we will remove the
`approve_new_topics_unless_trust_level` setting entirely.
Internal ref: /t/115696
* add missing translation
* Add keyword entry
* Add migration
Why this change?
`plugin:install_all_official` is quite slow at the moment taking roughly
1 minute and 51 seconds on my machine. Since most of the time is spent
waiting on the network, we can actually speed up the Rake task
significantly by executing the cloning concurrently. With a 8 cores
machine, cloning all plugins will only take 15 seconds.
What does this change do?
This change wraps the `git clone` operation in the
`plugin:install_all_official` Rake task in a `Concurrent::Promise` which
basically runs the `git clone` operation in a Thread. The `--quiet`
option has also been added to `git clone` since running stuff
concurrently messes up the output. That could be fixed but it has been
determined to be not worth it since the output from `git clone` is
meaningless to us.
Why this change?
There are instances where we would like to customize what the
`docker:test:setup` Rake task does.
What does this change do?
Adds a bunch of env variables that could be set to customize what the
`docker:test:setup` Rake test does.
We were throwing ArgumentError in UrlHelper.normalised_encode,
but it was incorrect -- we were passing ArgumentError.new
2 arguments which is not supported. Fix this and have a hint
of which URL is causing the issue for debugging.
Why this change?
We support a `USE_TURBO` environment variable which tells the
`docker:test` rake task to run rspec tests in parallel. However, this
currently does not apply to system tests.
What does this change do?
This commit runs system specs for both core and plugins using
`./bin/turbo_rspec` when the `USE_TURBO` environment is present. Note
that when running system specs, we will only spawn X number of test
processes where X is half the number of available CPU cores. This is
done because we have to leave CPU resources for the chrome processes
that will be created.
This change converts the `approve_unless_trust_level` site setting to
`approve_unless_allowed_groups`.
See: https://meta.discourse.org/t/283408
- Adds the new site setting
- Adds a deprecation warning
- Updates core to use the new settings.
- Adds a migration to fill in the new setting of the old setting was
changed
- Adds an entry to the site_setting.keywords section
- Updates many tests to account for the new change
After a couple of months we will remove the `approve_unless_trust_level`
setting entirely.
Internal ref: /t/115696
* Remove checkmark for official plugins
* Add author for plugin, which is By Discourse for all discourse
and discourse-org github plugins
* Link to meta topic instead of github repo
* Add experimental flag for plugin metadata and show this as a
badge on the plugin list if present
---------
Co-authored-by: chapoi <101828855+chapoi@users.noreply.github.com>
This commit adds a new `search_default_sort_order` site setting,
set to "relevance" by default, that controls the default sort order
for the full page /search route.
If the user changes the order in the dropdown on that page, we remember
their preference automatically, and it takes precedence over the site
setting as a default from then on. This way people who prefer e.g.
Latest Post as their default can make it so.
Why this change?
As the number of themes which the Discourse team supports officially
grows, we want to ensure that changes made to Discourse core do not
break the plugins. As such, we are adding a step to our Github actions
test job to run the QUnit tests for all official themes.
What does this change do?
This change adds a new job to our tests Github actions workflow to run the QUnit
tests for all official plugins. This is achieved with the following
changes:
1. Update `testem.js` to rely on the `THEME_TEST_PAGES` env variable to set the
`test_page` option when running theme QUnit tests with testem. The
`test_page` option [allows an array to be specified](https://github.com/testem/testem#multiple-test-pages) such that tests for
multiple pages can be run at the same time. We are relying on a ENV variable
because the `testem` CLI does not support passing a list of pages
to the `--test_page` option.
2. Support a `/testem-theme-qunit/:testem_id/theme-qunit` Rails route in the development environment. This
is done because testem prefixes the path with a unique ID to the configured `test_page` URL.
This is problematic for us because we proxy all testem requests to the
Rails server and testem's proxy configuration option does not allow us
to easily rewrite the URL to remove the prefix. Therefore, we configure a proxy in testem to prefix `theme-qunit` requests with
`/testem-theme-qunit` which can then be easily identified by the Rails server and routed accordingly.
3. Update `qunit:test` to support a `THEME_IDS` environment variable
which will allow it to run QUnit tests for multiple themes at the
same time.
4. Support `bin/rake themes:qunit[ids,"<theme_id>|<theme_id>"]` to run
the QUnit tests for multiple themes at the same time.
5. Adds a `themes:qunit_all_official` Rake task which runs the QUnit
tests for all the official themes.
Why this change?
By default the `db:create` Rake task in activerecord creates the
databases for both the development and test environment. This while
seemingly odd is by design from Rails. In order to avoid creating the
test database, Rails supports the `SKIP_TEST_DATABASE` environment
variable which we should respect when creating the multisite test
database.
When we started using NumberField for integer site settings
in e113eff663, we did not end up
passing down a min/max value for the integer to the field, which
meant that for some fields where negative numbers were allowed
we were not accepting that as valid input.
This commit passes down the min/max options from the server for
integer settings then in turn passes them down to NumberField.
c.f. https://meta.discourse.org/t/delete-user-self-max-post-count-not-accepting-1-to-disable/285162
Why this change?
As the number of themes which the Discourse team supports officially
grows, we want to ensure that changes made to Discourse core do not
break the plugins. As such, we are adding a step to our Github actions
test job to run the system tests for all official themes.
What does this change do?
This change adds a step to our Github actions test job to run the system
tests for all official plugins. This is achieved by the introduction of
the `themes:install_all_official` Rake task which installs all the
themes that are officially supported by the Discourse team.
- Remove vendored copy
- Update Rails implementation to look for language definitions in node_modules
- Use webpack-based dynamic import for hljs core
- Use browser-native dynamic import for site-specific language bundle (and fallback to webpack-based dynamic import in tests)
- Simplify markdown implementation to allow all languages into the `lang-{blah}` className
- Now that all languages are passed through, resolve aliases at runtime to avoid the need for the pre-built `highlightjs-aliases` index
Previously, the app HTML served by the Ember-CLI proxy was generated based on a 'bootstrap json' payload generated by Rails. This inevitably leads to differences between the Rails HTML and the Ember-CLI HTML.
This commit overhauls our proxying strategy. Now, we totally ignore the ember-cli `index.html` file. Instead, we take the full HTML from Rails and surgically replace script URLs based on a `data-discourse-entrypoint` attribute. This should be faster (only one request to Rails), more robust, and less confusing for developers.
Followup to fe05fdae24
For consistency with other S3 settings, make the global setting
the same name as the site setting and use SiteSetting.Upload
too so it reads from the correct place.
This adds the ability to collect stats without exposing them
among other stats via API.
The most important thing I wanted to achieve is to provide
an API where stats are not exposed by default, and a developer
has to explicitly specify that they should be
exposed (`expose_via_api: true`). Implementing an opposite
solution would be simpler, but that's less safe in terms of
potential security issues.
When working on this, I had to refactor the current solution.
I would go even further with the refactoring, but the next steps
seem to be going too far in changing the solution we have,
and that would also take more time. Two things that can be
improved in the future:
1. Data structures for holding stats can be further improved
2. Core stats are hard-coded in the About template (it's hard
to fix it without correcting data structures first, see point 1):
63a0700d45/app/views/about/index.html.erb (L61-L101)
The most significant refactorings are:
1. Introducing the `Stat` model
2. Aligning the way the core and the plugin stats' are registered
There was a registry for preloaded site categories and a new one has
been introduced recently for categories serialized through a
CategoryList.
Having two registries created a lot of friction for developers and this
commit merges them into a single one, providing a unified API.
There is an edge case where the following occurs:
1. The user sets a bookmark reminder on a post/topic
2. The post/topic is changed to a PM before or after the reminder
fires, and the notification remains unread by the user
3. The user opens their bookmark reminder notification list
and they can still see the notification even though they cannot
access the topic anymore
There is a very low chance for information leaking here, since
the only thing that could be exposed is the topic title if it
changes to something sensitive.
This commit filters the bookmark unread notifications by using
the bookmarkable can_see? methods and also prevents sending
reminder notifications for bookmarks the user can no longer see.
Followup to 5fc1586abf
There are certain cases where the tos_url and privacy_policy_url
can end up with a "nil" value in the Discourse.urls_cache.
The cause of this is unclear, but it seems to behave differently
between doing this caching in the rails console and the running
server.
To avoid this we can just not store anything that looks like nil
in the cache; we can delete the cache keys entirely if we don't
need them anymore.
When quoting a chat message in a post, if that message contains a mention,
that mention should be ignored. But we've been detecting them and sending
notifications to users. This PR fixes the problem. Since this fix is for
the chat plugin, I had to introduce a new API for plugins:
# We strip posts before detecting mentions, oneboxes, attachments etc.
# We strip those elements that shouldn't be detected. For example,
# a mention inside a quote should be ignored, so we strip it off.
# Using this API plugins can register their own post strippers.
def register_post_stripper(&block)
end
Previously, we were parsing webpack JS chunk filenames from the HTML files which ember-cli generates. This worked ok for simple entrypoints, but falls apart once we start using async imports(), which are not included in the HTML.
This commit uses the stats plugin to generate an assets.json file, and updates Rails to parse it instead of the HTML. Caching on the Rails side is also improved to avoid reading from the filesystem multiple times per request in develoment.
Co-authored-by: Godfrey Chan <godfreykfc@gmail.com>
This commit adds an `enable_s3_transfer_acceleration` site setting,
which is hidden to begin with. We are adding this because in certain
regions, using https://aws.amazon.com/s3/transfer-acceleration/ can
drastically speed up uploads, sometimes as much as 70% in certain
regions depending on the target bucket region. This is important for
us because we have direct S3 multipart uploads enabled everywhere
on our hosting.
To start, we only want this on the uploads bucket, not the backup one.
Also, this will accelerate both uploads **and** downloads, depending
on whether a presigned URL is used for downloading. This is the case
when secure uploads is enabled, not anywhere else at this time. To
enable the S3 acceleration on downloads more generally would be a
more in-depth change, since we currently store S3 Upload record URLs
like this:
```
url: "//test.s3.dualstack.us-east-2.amazonaws.com/original/2X/6/123456.png"
```
For acceleration, `s3.dualstack` would need to be changed to `s3-accelerate.dualstack`
here.
Note that for this to have any effect, Transfer Acceleration must be enabled
on the S3 bucket used for uploads per https://docs.aws.amazon.com/AmazonS3/latest/userguide/transfer-acceleration-examples.html.
With Embroider, we can rely on async `import()` to do the splitting
for us.
This commit extracts from `pretty-text` all the parts that are
meant to be loaded async into a new `discourse-markdown-it` package
that is also a V2 addon (meaning that all files are presumed unused
until they are imported, aka "static").
Mostly I tried to keep the very discourse specific stuff (accessing
site settings and loading plugin features) inside discourse proper,
while the new package aims to have some resembalance of a general
purpose library, a MarkdownIt++ if you will. It is far from perfect
because of how all the "options" stuff work but I think it's a good
start for more refactorings (clearing up the interfaces) to happen
later.
With this, pretty-text and app/lib/text are mostly a kitchen sink
of loosely related text processing utilities.
After the refactor, a lot more code related to setting up the
engine are now loaded lazily, which should be a pretty nice win. I
also noticed that we are currently pulling in the `xss` library at
initial load to power the "sanitize" stuff, but I suspect with a
similar refactoring effort those usages can be removed too. (See
also #23790).
This PR does not attempt to fix the sanitize issue, but I think it
sets things up on the right trajectory for that to happen later.
Co-authored-by: David Taylor <david@taylorhq.com>
When Discourse first introduced brotli support, reverse-proxy/CDN support for passing through the accept-encoding header to our NGINX server was very poor. Therefore, a separate `/brotli_assets/...` path was introduced to serve the brotli assets. This worked well, but introduces additional complexity and inconsistencies.
Nowadays, Brotli encoding is well supported, so we don't need the separate paths any more. Requests can be routed to the asset `.js` URLs, and NGINX will serve the brotli/gzip version of the asset automatically.
The motivation of this PR is to remove our dependence on Ember's 'named outlets', which are removed in Ember 4+.
At a high-level, the changes can be summarized as:
- The top-level `discovery` route is totally emptied of all logic. The HTML structure of the template is moved into the `<Discovery::Layout />` component for use by child routes.
- `AbstractTopicRoute` and `AbstractCategoryRoute` routes now both lean on the `DiscoverySortableController` and associated template. This controller is where most of the logic from the old top-level `discovery` controller has ended up.
- All navigation controllers/templates have been replaced with components. `navigation/categories`, `navigation/category` and `navigation/default` were very similar, and so they've all been combined into `<Navigation::Default>`. `navigation/filter` gets its own component.
- The `discovery/topics` controller/template have been moved into a new `<Discovery::Topics>` component.
Various other parts of the app have been tweaked to support these changes, but I've tried to keep that to a minimum.
Anything from `<TopicList>` down is untouched, which should hopefully mean that a large proportion of topic-list-customizing themes are unaffected.
For more information, see https://meta.discourse.org/t/282816
For deprecated site settings, we log out a warning when
the old setting is used. However when we convert all the client
settings to JSON, we are creating a lot of log noise like this:
> Deprecation notice: `SiteSetting.anonymous_posting_min_trust_level` has been deprecated.
We don't need to do this because we are just dumping the JSON.
We updated scheduled admin checks to run concurrently in their own jobs. The main reason for this was so that we can implement re-check functionality for especially flaky checks (e.g. group e-mail credentials check.)
This works in the following way:
1. The check declares its retry policy using class methods.
2. A block can be yielded to if there are problems, but before they are committed to Redis.
3. The job uses this block to either a) schedule a retry if there are any remaining or b) do nothing and let the check commit.
This commit introduces a new feature that allows theme developers to manage the transformation of theme settings over time. Similar to Rails migrations, the theme settings migration system enables developers to write and execute migrations for theme settings, ensuring a smooth transition when changes are required in the format or structure of setting values.
Example use cases for the theme settings migration system:
1. Renaming a theme setting.
2. Changing the data type of a theme setting (e.g., transforming a string setting containing comma-separated values into a proper list setting).
3. Altering the format of data stored in a theme setting.
All of these use cases and more are now possible while preserving theme setting values for sites that have already modified their theme settings.
Usage:
1. Create a top-level directory called `migrations` in your theme/component, and then within the `migrations` directory create another directory called `settings`.
2. Inside the `migrations/settings` directory, create a JavaScript file using the format `XXXX-some-name.js`, where `XXXX` is a unique 4-digit number, and `some-name` is a descriptor of your choice that describes the migration.
3. Within the JavaScript file, define and export (as the default) a function called `migrate`. This function will receive a `Map` object and must also return a `Map` object (it's acceptable to return the same `Map` object that the function received).
4. The `Map` object received by the `migrate` function will include settings that have been overridden or changed by site administrators. Settings that have never been changed from the default will not be included.
5. The keys and values contained in the `Map` object that the `migrate` function returns will replace all the currently changed settings of the theme.
6. Migrations are executed in numerical order based on the XXXX segment in the migration filenames. For instance, `0001-some-migration.js` will be executed before `0002-another-migration.js`.
Here's a complete example migration script that renames a setting from `setting_with_old_name` to `setting_with_new_name`:
```js
// File name: 0001-rename-setting.js
export default function migrate(settings) {
if (settings.has("setting_with_old_name")) {
settings.set("setting_with_new_name", settings.get("setting_with_old_name"));
}
return settings;
}
```
Internal topic: t/109980
Followup to b53449eac9, we cannot
generate the links to plugin admin pages in this way because it
depends on which plugins are installed; we would need to somehow
do it at runtime. Leaving it out for now, for people who need to
find these admin routes the Ember Inspector extension for Chrome
can be used in the meantime.
NOTE: Most of this is experimental and will be removed at a later
time, which is why things like translations have not been added.
The new /admin-revamp UI uses a sidebar for admin nav. This initial
step adds a script to generate a map of all the current admin nav
into a format the sidebar to read. Then, people can experiment
with different changes to this structure.
The structure can then be edited from `/admin-revamp/config/sidebar-experiment`,
and it is saved to local storage so people can visually experiment with different ways
of showing the admin sidebar links.
