We have all these calls to Group.refresh_automatic_groups! littered throughout the tests. Including tests that are seemingly unrelated to groups. This is because automatic group memberships aren't fabricated when making a vanilla user. There are two places where you'd want to use this:
You have fabricated a user that needs a certain trust level (which is now based on group membership.)
You need the system user to have a certain trust level.
In the first case, we can pass refresh_auto_groups: true to the fabricator instead. This is a more lightweight operation that only considers a single user, instead of all users in all groups.
The second case is no longer a thing after #25400.
We want to exclude the system user from group user counts, since intuitively admins wouldn't include them.
Originally this was accomplished by booting said system user from the groups, but this is causing problems, because the system user needs TL group membership to perform certain tasks.
After this PR, system user is still in the TL groups, but excluded when refreshing the user count.
Some preparatory refactoring as we're working on TL groups for the system user. On User we have a scope #human_users to exclude the system user, DiscoBot, etc. This PR adds the same scope (delegated to User) on Group.
This is a temporary fix to address an issue where the
system user is losing its automatic groups when the server
is running. If any auto groups are provided, and the user is
a system user, then we return true. The system user is admin,
moderator, and TL4, so they usually have all auto groups.
We can remove this when we get to the bottom of why the auto
groups are being deleted.
We're changing the implementation of trust levels to use groups. Part of this is to have site settings that reference trust levels use groups instead. It converts the min_trust_to_post_links site setting to post_links_allowed_groups.
This isn't used by any of our plugins or themes, so very little fallout.
The old strategy used to load 25 categories at a time, including the
subcategories. The new strategy loads 20 parent categories and at most
5 subcategories for each parent category, for a maximum of 120
categories in total.
- Decrease gravity, we come in too hot prioritizing too many new topics
- Remove all muted topics / categories and tags from the hot list
- Punish topics with zero likes in algorithm
This introduces a new experimental hot sort ordering.
It attempts to float top conversations by first prioritizing a topics with lots of recent activity (likes and users responding)
The schedule that updates hot topics is disabled unless the hidden site setting: `experimental_hot_topics` is enabled.
You can control "decay" with `hot_topic_gravity` and `recency` with `hot_topics_recent_days`
Data is stored in the new `topic_hot_scores` table and you can check it out on the `/hot` route once
enabled.
---------
Co-authored-by: Penar Musaraj <pmusaraj@gmail.com>
Why this change?
Currently, is it hard to iteratively write a theme settings migrations
because our theme migrations system does not rollback. Therefore, we
want to allow theme developers to be able to write QUnit tests for their
theme migrations files enabling them to iteratively write their theme
migrations.
What does this change do?
1. Update `Theme#baked_js_tests_with_digest` to include all `ThemeField`
records of `ThemeField#target` equal to `migrations`. Note that we do
not include the `settings` and `themePrefix` variables for migration files.
2. Don't minify JavaScript test files becasue it makes debugging in
development hard.
We're changing the implementation of trust levels to use groups. Part of this is to have site settings that reference trust levels use groups instead. It converts the min_trust_level_to_tag_topics site setting to tag_topic_allowed_groups.
* FIX: respect creation date when paginating group activity posts
There are scenarios where the chronological order of posts doesn't match the order of their IDs. For instance, when moving the first post from one topic or PM to another, a new post (with a higher ID) will be created, but it will retain the original creation time.
This PR changes the group activity page and endpoint to paginate posts using created_at instead of relying on ID ordering.
Why this change?
Importing theme with the `bundle` params is used mainly by
`discourse_theme` CLI in the development environment. However, we do not
want migrations to automatically run in the development environment
and instead want the developer to be intentional about running theme
migrations. As such, this commit adds support for a
`skip_migrations` param when importing a theme with the `bundle` params.
This commit also adds a `migrated` attribute for migrations theme fields
to indicate whether a migrations theme field has been migrated or not.
* FEATURE: Cache embed contents in the database
This will be useful for features that rely on the semantic content of topics, like the many AI features
Co-authored-by: Roman Rizzi <rizziromanalejandro@gmail.com>
We're changing the implementation of trust levels to use groups. Part of this is to have site settings that reference trust levels use groups instead. It converts the min_trust_level_to_create_tag site setting to create_tag_allowed_groups.
This PR maintains backwards compatibility until we can update plugins and themes using this.
This fixes an issue where any string for an enum site setting
(such as TrustLevelSetting) would be converted to an integer
if the default value for the enum was an integer. This is an
issue because things like "admin" and "staff" would get silently
converted to 0 which is "valid" because it's TrustLevel[0],
but it's unexpected behaviour. It's best to just let the site
setting validator catch this broken value.
Currently, the reviewable queue includes ReviewableFlaggedPost with posts that have already been hidden. This allows for such hidden posts to be cleared up by the auto-tool.
Categories will no longer be preloaded when `lazy_load_categories` is
enabled through PreloadStore.
Instead, the list of site categories will continue to be populated
by `Site.updateCategory` as more and more categories are being loaded
from different sources (topic lists, category selectors, etc).
When setting an old TL based site setting in the console e.g.:
SiteSetting.min_trust_level_to_allow_ignore = TrustLevel[3]
We will silently convert this to the corresponding Group::AUTO_GROUP. And vice-versa, when we read the value on the old setting, we will automatically get the lowest trust level corresponding to the lowest auto group for the new setting in the database.
This commit refactor CategoryList to remove usage of EmberObject,
hopefully make the code more readable and fixes various edge cases with
lazy loaded categories (third level subcategories not being visible,
subcategories not being visible on category page, requesting for more
pages even if the last one did not return any results, etc).
The problems have always been here, but were not visible because a lot
of the processing was handled by the server and then the result was
serialized. With more of these being moved to the client side for the
lazy category loading, the problems became more obvious.
We're changing the implementation of trust levels to use groups. Part of this is to have site settings that reference trust levels use groups instead. It converts the min_trust_level_to_allow_invite site setting to invite_allowed_groups.
Nothing much of note. This is used in one place and there's no fallout.
Using min_trust_to_create_topic and create_topic_allowed_groups together was part of #24740
Now, when plugins specs are fixed, we can safely remove that part of logic.
This is v0 of admin sidebar navigation, which moves
all of the top-level admin nav from the top of the page
into a sidebar. This is hidden behind a enable_admin_sidebar_navigation
site setting, and is opt-in for now.
This sidebar is dynamically shown whenever the user enters an
admin route in the UI, and is hidden and replaced with either
the:
* Main forum sidebar
* Chat sidebar
Depending on where they navigate to. For now, custom sections
are not supported in the admin sidebar.
This commit removes the experimental admin sidebar generation rake
task but keeps the experimental sidebar UI for now for further
testing; it just uses the real nav as the default now.
We're changing the implementation of trust levels to use groups. Part of this is to have site settings that reference trust levels use groups instead. It converts the min_trust_to_flag_posts site setting to flag_post_allowed_groups.
Note: In the original setting, "posts" is plural. I have changed this to "post" singular in the new setting to match others.
This change converts the min_trust_to_create_topic site setting to
create_topic_allowed_groups.
See: https://meta.discourse.org/t/283408
- Hides the old setting
- Adds the new site setting
- Add a deprecation warning
- Updates to use the new setting
- Adds a migration to fill in the new setting if the old setting was
changed
- Adds an entry to the site_setting.keywords section
- Updates tests to account for the new change
- After a couple of months, we will remove the min_trust_to_create_topicsetting entirely.
Internal ref: /t/117248
A lot of work has been put in the select kits used for selecting
categories: CategorySelector, CategoryChooser, CategoryDrop, however
they still do not work as expected when these selectors already have
values set, because the category were still looked up in the list of
categories stored on the client-side Categrories.list().
This PR fixes that by looking up the categories when the selector is
initialized. This required altering the /categories/find.json endpoint
to accept a list of IDs that need to be looked up. The API is called
using Category.asyncFindByIds on the client-side.
CategorySelector was also updated to receive a list of category IDs as
attribute, instead of the list of categories, because the list of
categories may have not been loaded.
During this development, I noticed that SiteCategorySerializer did not
serializer all fields (such as permission and notification_level)
which are not a property of category, but a property of the relationship
between users and categories. To make this more efficient, the
preload_user_fields! method was implemented that can be used to
preload these attributes for a user and a list of categories.
When rebaking and in various other places for posts, we
run through the uploads and call `update_secure_status` on
each of them.
However, if the secure status didn't change, we were still
calling S3 to change the ACL, which would have been a noop
in many cases and takes ~1 second per call, slowing things
down a lot.
Also, we didn't account for the s3_acls_enabled site setting
being false here, and in the specs doing an assertion
that `Discourse.store.update_ACL` is not called doesn't
work; `Discourse.store` isn't a singleton, it re-initializes
`FileStore::S3Store.new` every single time.
Why was the problem?
ActiveRecord's query cache for the connection pool wasn't disabled after the
`with a fake provider runs 'other_phase' for enabled auth methods` test
in `omniauth_callbacks_controller_spec.rb` was run. This was because the
Rack response body in `FakeAuthenticator::Strategy::other_phase` did not
adhere to the expected Rack body format which is "typically an Array of
String instances". Because this expectation was broken, it cascaded the
problem down where it resulted in the ActiveRecord's query cache for the
connection pool not being disabled as it normally should when the
response body is closed.
When the query cache is left enabled, common assertions pattern in RSpec
like `expect { something }.to change { Group.count }` will fail since
the query cache is enabled and the call first call to `Group.count` will
cache the result to be reused later on.
To see the bug in action, one can run the following command:
`bundle exec rspec --seed 44747
spec/requests/omniauth_callbacks_controller_spec.rb:1150
spec/models/group_spec.rb:283`
The category drop was rerendered after every category async change
because it updated the categories list. This is not necessary and
categories can be referenced indirectly by ID instead.
When we check upload security, one of the checks is to
run `access_control_post.with_secure_uploads?`. The problem
here is that the `topic` for the post could be deleted,
which would make the check return `nil` sometimes instead
of false because of safe navigation. We just need to be
more explicit.
Followup to 2443446e62
We introduced video placeholders which prevent preloading
metadata for videos in posts. The structure looks like this
in HTML when the post is cooked:
```
<div class="video-placeholder-container" data-video-src="http://some-url.com/video.mp4" dir="ltr" style="cursor: pointer;">
<div class="video-placeholder-wrapper">
<div class="video-placeholder-overlay">
<svg class="fa d-icon d-icon-play svg-icon svg-string" xmlns="http://www.w3.org/2000/svg">
<use href="#play"></use>
</svg>
</div>
</div>
</div>
```
However, we did not update the code that links post uploads
to the post via UploadReference, so any videos uploaded since
this change are essentially dangling and liable to be deleted.
This also causes some uploads to be marked secure when they
shouldn't be, because they are not picked up and analysed in the
CookedPostProcessor flow.
This improves the implementation of #18993
1. Error message displayed to user is clearer
2. open_db will also be called, even if license key is blank, as it was previously
3. This in turn means no need to keep stubbing 'maxmind_license_key'
Why this change?
The test was randomly failing in
https://github.com/discourse/discourse/actions/runs/6936264158/job/18868087113
with the following failure:
```
expect do user.update_ip_address!("127.0.0.1") end.to change {
UserIpAddressHistory.where(user_id: user.id).count
}.by(1)
expected `UserIpAddressHistory.where(user_id: user.id).count` to have changed by 1, but was changed by 0
```
This is due to the fact that ActiveRecord will actually cache the result
of `UserIpAddressHistory.where(user_id: user.id).count`. However,
`User.update_ip_address!` relies on mini_sql and does not go through
ActiveRecord. As a result, the query cache is not cleared and hence the
flakiness.
What does this change do?
This change uses the `uncached` method provided by ActiveRecord when
we are fetching the count.
Previously we would only recompile a theme locale when its own data changes. However, the output also includes fallback data from other locales, so we need to invalidate all locales when fallback locale data is changed. Building a list of dependent locales is tricky, so let's just invalidate them all.
The most common thing that we do with fab! is:
fab!(:thing) { Fabricate(:thing) }
This commit adds a shorthand for this which is just simply:
fab!(:thing)
i.e. If you omit the block, then, by default, you'll get a `Fabricate`d object using the fabricator of the same name.
This adds the ability to collect stats without exposing them
among other stats via API.
The most important thing I wanted to achieve is to provide
an API where stats are not exposed by default, and a developer
has to explicitly specify that they should be
exposed (`expose_via_api: true`). Implementing an opposite
solution would be simpler, but that's less safe in terms of
potential security issues.
When working on this, I had to refactor the current solution.
I would go even further with the refactoring, but the next steps
seem to be going too far in changing the solution we have,
and that would also take more time. Two things that can be
improved in the future:
1. Data structures for holding stats can be further improved
2. Core stats are hard-coded in the About template (it's hard
to fix it without correcting data structures first, see point 1):
63a0700d45/app/views/about/index.html.erb (L61-L101)
The most significant refactorings are:
1. Introducing the `Stat` model
2. Aligning the way the core and the plugin stats' are registered
There was a registry for preloaded site categories and a new one has
been introduced recently for categories serialized through a
CategoryList.
Having two registries created a lot of friction for developers and this
commit merges them into a single one, providing a unified API.
When quoting a chat message in a post, if that message contains a mention,
that mention should be ignored. But we've been detecting them and sending
notifications to users. This PR fixes the problem. Since this fix is for
the chat plugin, I had to introduce a new API for plugins:
# We strip posts before detecting mentions, oneboxes, attachments etc.
# We strip those elements that shouldn't be detected. For example,
# a mention inside a quote should be ignored, so we strip it off.
# Using this API plugins can register their own post strippers.
def register_post_stripper(&block)
end
This PR does some preparatory refactoring of scheduled admin checks in order for us to be able to do custom retry strategies for some of them.
Instead of running all checks in sequence inside a single, scheduled job, the scheduled job spawns one new job per check.
In order to be concurrency-safe, we need to change the existing Redis data structure from a string (of serialized JSON) to a list of strings (of serialized JSON).
This commit introduces a new feature that allows theme developers to manage the transformation of theme settings over time. Similar to Rails migrations, the theme settings migration system enables developers to write and execute migrations for theme settings, ensuring a smooth transition when changes are required in the format or structure of setting values.
Example use cases for the theme settings migration system:
1. Renaming a theme setting.
2. Changing the data type of a theme setting (e.g., transforming a string setting containing comma-separated values into a proper list setting).
3. Altering the format of data stored in a theme setting.
All of these use cases and more are now possible while preserving theme setting values for sites that have already modified their theme settings.
Usage:
1. Create a top-level directory called `migrations` in your theme/component, and then within the `migrations` directory create another directory called `settings`.
2. Inside the `migrations/settings` directory, create a JavaScript file using the format `XXXX-some-name.js`, where `XXXX` is a unique 4-digit number, and `some-name` is a descriptor of your choice that describes the migration.
3. Within the JavaScript file, define and export (as the default) a function called `migrate`. This function will receive a `Map` object and must also return a `Map` object (it's acceptable to return the same `Map` object that the function received).
4. The `Map` object received by the `migrate` function will include settings that have been overridden or changed by site administrators. Settings that have never been changed from the default will not be included.
5. The keys and values contained in the `Map` object that the `migrate` function returns will replace all the currently changed settings of the theme.
6. Migrations are executed in numerical order based on the XXXX segment in the migration filenames. For instance, `0001-some-migration.js` will be executed before `0002-another-migration.js`.
Here's a complete example migration script that renames a setting from `setting_with_old_name` to `setting_with_new_name`:
```js
// File name: 0001-rename-setting.js
export default function migrate(settings) {
if (settings.has("setting_with_old_name")) {
settings.set("setting_with_new_name", settings.get("setting_with_old_name"));
}
return settings;
}
```
Internal topic: t/109980
Followup to 9762e65758. This
original commit did not take into account the fact that
new topics can end up in the approval queue as a
ReviewableQueuedPost, and so there was a 500 error raised
when accessing `self.topic` when sending a PM to the user.
Why this change?
Currently, we do not have a method to easily retrieve a theme setting's
value on the server side. Such a method can be useful in the test
environment where we need to retrieve the theme's setting and use its
value in assertions.
What does this change do?
This change introduces the `Theme#get_setting` instance method.
This change adds a new event trigger (new_post_moved) when the first post in a topic is moved to a new topic.
Plugins that listen for the new_post_moved event now have an easy way to update old data based on the post id.
This commit adds a new admin UI under the route `/admin-revamp`, which is
only accessible if the user is in a group defined by the new `enable_experimental_admin_ui_groups` site setting. It
also adds a special `admin` sidebar panel that is shown instead of the `main`
forum one when the admin is in this area.
We also add an "Admin Revamp" sidebar link to the community section, which
will only appear if the user is in the setting group:
Within this there are subroutes defined like `/admin-revamp/config/:area`,
these areas could contain any UI imaginable, this is just laying down an
initial idea of the structure and how the sidebar will work. Sidebar links are
currently hardcoded.
Some other changes:
* Changed the `main` and `chat` panels sidebar panel keys to use exported const values for reuse
* Allowed custom sidebar sections to hide their headers with the `hideSectionHeader` option
* Add a `groupSettingArray` setting on `this.siteSettings` in JS, which accepts a group site setting name
and splits it by `|` then converts the items in the array to integers, similar to the `_map` magic for ruby
group site settings
* Adds a `hidden` option for sidebar panels which prevents them from showing in separated mode and prevents
the switch button from being shown
---------
Co-authored-by: Krzysztof Kotlarek <kotlarek.krzysztof@gmail.com>
* FIX: Secure upload post processing race condition
This commit fixes a couple of issues.
A little background -- when uploads are created in the composer
for posts, regardless of whether the upload will eventually be
marked secure or not, if secure_uploads is enabled we always mark
the upload secure at first. This is so the upload is by default
protected, regardless of post type (regular or PM) or category.
This was causing issues in some rare occasions though because
of the order of operations of our post creation and processing
pipeline. When creating a post, we enqueue a sidekiq job to
post-process the post which does various things including
converting images to lightboxes. We were also enqueuing a job
to update the secure status for all uploads in that post.
Sometimes the secure status job would run before the post process
job, marking uploads as _not secure_ in the background and changing
their ACL before the post processor ran, which meant the users
would see a broken image in their posts. This commit fixes that issue
by always running the upload security changes inline _within_ the
cooked_post_processor job.
The other issue was that the lightbox wrapper link for images in
the post would end up with a URL like this:
```
href="/secure-uploads/original/2X/4/4e1f00a40b6c952198bbdacae383ba77932fc542.jpeg"
```
Since we weren't actually using the `upload.url` to pass to
`UrlHelper.cook_url` here, we weren't converting this href to the CDN
URL if the post was not in a secure context (the UrlHelper does not
know how to convert a secure-uploads URL to a CDN one). Now we
always end up with the correct lightbox href. This was less of an issue
than the other one, since the secure-uploads URL works even when the
upload has become non-secure, but it was a good inconsistency to fix
anyway.
This reverts commit 5f0bc4557f.
Through extensive internal discussion we have decided to revert
this change, as it significantly impacted moderation flow for
some Discourse site moderators, especially around "something else"
flags. We need to re-approach how flags are counted holistically,
so to that end this change is being reverted.
Site data is preloaded on the first page load, which includes categories
data. For sites with many categories, site data takes a long time to
serialize and to transfer.
In the future, preloaded category data will be completely removed.
The category style site setting is being deprecated. This commit will
show a warning on the admin dashboard if a site isn't using the default
category style (bullet).
There are cases where a user can copy image markdown from a public
post (such as via the discourse-templates plugin) into a PM which
is then sent via an email. Since a PM is a secure context (via the
.with_secure_uploads? check on Post), the image will get a secure
URL in the PM post even though the backing upload is not secure.
This fixes the bug in that case where the image would be stripped
from the email (since it had a /secure-uploads/ URL) but not re-attached
further down the line using the secure_uploads_allow_embed_images_in_emails
setting because the upload itself was not secure.
The flow in Email::Sender for doing this is still not ideal, but
there are chicken and egg problems around when to strip the images,
how to fit in with other attachments and email size limits, and
when to apply the images inline via Email::Styles. It's convoluted,
but at least this fixes the Template use case for now.
Why this change?
The `PostsController#create` action allows arbitrary topic custom fields
to be set by any user that can create a topic. Without any restrictions,
this opens us up to potential security issues where plugins may be using
topic custom fields in security sensitive areas.
What does this change do?
1. This change introduces the `register_editable_topic_custom_field` plugin
API which allows plugins to register topic custom fields that are
editable either by staff users only or all users. The registered
editable topic custom fields are stored in `DiscoursePluginRegistry` and
is called by a new method `Topic#editable_custom_fields` which is then
used in the `PostsController#create` controller action. When an unpermitted custom fields is present in the `meta_data` params,
a 400 response code is returned.
2. Removes all reference to `meta_data` on a topic as it is confusing
since we actually mean topic custom fields instead.
This commit adds a new Revise... action that can be taken
for queued post reviewables. This will open a modal where
the user can select a Reason from a preconfigured list
(or by choosing Other..., a custom reason) and provide feedback
to the user about their post.
The post will be rejected still, but a PM will also be sent to
the user so they have an opportunity to improve their post when
they resubmit it.
When a user creates or edits a post, we already were updating
the security of uploads in the post based on site settings and
their access control post, which is important since these uploads
may be switched from secure/not secure based on configuration.
The `with_secure_uploads?` method on a post is used to determine
whether to use the secure-uploads URL for all uploads in the post,
regardless of their individual security, so if this is false and
some of the posts are still secure when rebaking, we end up with
broken URLs.
This commit just makes it so rebaking via the UI also re-evaluates
upload security so that when the post is loaded again after processing,
all of the uploads have the correct security.
Why this change?
Back in May 17 2023 along with the release of Discourse 3.1, we announced
on meta that the legacy hamburger dropdown navigation menu is
deprecated and will be dropped in Discourse 3.2. This is the link to the announcement
on meta: https://meta.discourse.org/t/removing-the-legacy-hamburger-navigation-menu-option/265274
## What does this change do?
This change removes the `legacy` option from the `navigation_menu` site
setting and migrates existing sites on the `legacy` option to the
`header dropdown` option.
All references to the `legacy` option in code and tests have been
removed as well.
Meta topic: https://meta.discourse.org/t/suppress-these-tags-from-summary-emails-settings-is-not-working-in-preview-digest-email/279196?u=osama
Follow-up to 477a5dd371
The `digest_suppress_tags` setting is designed to be a list of pipe-delimited tag names, but the tag-based topic suppression logic assumes (incorrectly) that the setting contains pipe-delimited tag IDs. This mismatch in expectations led to the setting not working as expected.
This PR adds a step that converts the list of tag names in the setting to their corresponding IDs, which is then used to suppress topics tagged with those specific tags.
This commit adds limits to themes and theme components on the:
- file size of about.json and .discourse-compatibility
- file size of theme assets
- number of files in a theme
The hidden site setting max_drafts_per_user defaults to 10_000 drafts per user.
The longest key should be "topic_<MAX_BIG_INT>" which is 25 characters.
In #20135 we prevented invalid inputs from being accepted in category setting form fields on the front-end. We didn't do anything on the back-end at that time, because we were still discussing which path we wanted to take. Eventually we decided we want to move this to a new CategorySetting model.
This PR moves the require_topic_approval and require_reply_approval from custom fields to the new CategorySetting model.
This PR is nearly identical to #20580, which migrated num_auto_bump_daily, but since these are slightly more sensitive, they are moved after the previous one is verified.
Why this change?
Currently, we do not have an easy way to test themes and theme components
using Rails system tests. While we support QUnit acceptance tests for
themes and theme components, QUnit acceptance tests stubs out the server
and setting up the fixtures for server responses is difficult and can lead to a
frustrating experience. System tests on the other hand allow authors to
set up the test fixtures using our fabricator system which is much
easier to use.
What does this change do?
In order for us to allow authors to run system tests with their themes
installed, we are adding a `upload_theme` helper that is made available
when writing system tests. The `upload_theme` helper requires a single
`directory` parameter where `directory` is the directory of the theme
locally and returns a `Theme` record.
Currently, if the review queue has both a flagged post and a flagged chat message, one of the two will have some of the labels of their actions replaced by those of the other. In other words, the labels are getting mixed up. For example, a flagged chat message might show up with an action labelled "Delete post".
This is happening because when using bundles, we are sending along the actions in a separate part of the response, so they can be shared by many reviewables. The bundles then index into this bag of actions by their ID, which is something generic describing the server action, e.g. "agree_and_delete".
The problem here is the same action can have different labels depending on the type of reviewable. Now that the bag of actions contains multiple actions with the same ID, which one is chosen is arbitrary. I.e. it doesn't distinguish based on the type of the reviewable.
This change adds an additional field to the actions, server_action, which now contains what used to be the ID. Meanwhile, the ID has been turned into a concatenation of the reviewable type and the server action, e.g. post-agree_and_delete.
This still provides the upside of denormalizing the actions while allowing for different reviewable types to have different labels and descriptions.
At first I thought I would prepend the reviewable type to the ID, but this doesn't work well because the ID is used on the server-side to determine which actions are possible, and these need to be shared between different reviewables. Hence the introduction of server_action, which now serves that purpose.
I also thought about changing the way that the bundle indexes into the bag of actions, but this is happening through some EmberJS mechanism, so we don't own that code.
This adds a new secure_uploads_pm_only site setting. When secure_uploads
is true with this setting, only uploads created in PMs will be marked
secure; no uploads in secure categories will be marked as secure, and
the login_required site setting has no bearing on upload security
either.
This is meant to be a stopgap solution to prevent secure uploads
in a single place (private messages) for sensitive admin data exports.
Ideally we would want a more comprehensive way of saying that certain
upload types get secured which is a hybrid/mixed mode secure uploads,
but for now this will do the trick.
Manipulating theme module paths means that the paths you author are not the ones used at runtime. This can lead to some very unexpected behavior and potential module name clashes. It also meant that the refactor in 16c6ab8661 was unable to correctly match up theme connector js/templates.
While this could technically be a breaking change, I think it is reasonably safe because:
1. Themes are already forced to use relative paths when referencing their own modules (since they're namespaced based on the site-specific id). The only time this might be problematic is when theme tests reference modules in the theme's main `javascripts` directory
2. For things like components/services/controllers/etc. our custom Ember resolver works backwards from the end of the path, so adding `discourse/` in the middle will not affect resolution.
When hiding a post (essentially updating hidden, hidden_at, and hidden_reason_id) our callbacks are running the whole battery of post validations. This can cause the hiding to fail in a number of edge cases. The issue is similar to the one fixed in #11680, but applies to all post validations, none of which should apply when hiding a post.
After some code reading and discussion, none of the validations in PostValidator seem to be relevant when hiding posts, so instead of just skipping unique check, we skip all post validator checks.
So we have to order by calling `find_each(order: :desc)`.
Note that that will order rows by Id, not by `last_match_at`
as we tried before (though that didn't work).
* scrub non-a html tags from tag descriptions on create, strips all tags from tag description when displayed in tag hover
* test for tag description links
* UX: basic render-tag test
* UX: fix linting
* UX: fix linting
* fix broken tests
* Update spec/models/tag_spec.rb
Co-authored-by: Penar Musaraj <pmusaraj@gmail.com>
* UX: use has_sanitizable_fields instead of has_scrubbable_fields to ensafen tag.description
---------
Co-authored-by: Penar Musaraj <pmusaraj@gmail.com>
This commit removes any logic in the app and in specs around
enable_experimental_hashtag_autocomplete and deletes some
old category hashtag code that is no longer necessary.
It also adds a `slug_ref` category instance method, which
will generate a reference like `parent:child` for a category,
with an optional depth, which hashtags use. Also refactors
PostRevisor which was using CategoryHashtagDataSource directly
which is a no-no.
Deletes the old hashtag markdown rule as well.
This brings the theme development experience (via the discourse_theme cli) closer to the experience of making javascript changes in Discourse core/plugins via Ember CLI. Whenever a change is made to a non-css theme field, all clients will be instructed to immediately refresh via message-bus.
In #20135 we prevented invalid inputs from being accepted in category setting form fields on the front-end. We didn't do anything on the back-end at that time, because we were still discussing which path we wanted to take. Eventually we decided we want to move this to a new CategorySetting model.
This PR moves the num_auto_bump_daily from custom fields to the new CategorySetting model.
In addition it sets the default value to 0, which exhibits the same behaviour as when the value is NULL.
FEATURE: Only approved flags for post counters
* Why was this change necessary?
The counters for flagged posts in the user's profile and user index from
the admin view include flags that were rejected, ignored or pending
review. This introduces unnecessary noise. Also the flagged posts
counter in the user's profile includes custom flags which add further
noise to this signal.
* How does it address the problem?
* Modifying User#flags_received_count to return posts with only approved
standard flags
* Refactoring User#number_of_flagged_posts to alias to
User#flags_received_count
* Updating the flagged post staff counter hyperlink to navigate to a
filtered view of that user's approved flagged posts to maintain
consistency with the counter
* Adding system tests for the profile page to cover the flagged posts
staff counter
We currently are accumulating orphaned upload references whenever drafts are deleted.
This change deals with future cases by adding a dependent strategy of delete_all on the Draft#upload_references association. (We don't really need destroy strategy here, since UploadReference is a simple data bag and there are no validations or callbacks on the model.)
It deals with existing cases through a migration that deletes all existing, orphaned draft upload references.
A previous change updated `ReviewableQueuedPost`'s `created_by`
to be consistent with other reviewable types. It assigns
the the creator of the post being queued to `target_created_by` and sets
the `created_by` to the creator of the reviewable itself.
This fix updates some of the `created_by` references missed during the
intial fix.
What is the problem here?
In multiple controllers, we are accepting a `limit` params but do not
impose any upper bound on the values being accepted. Without an upper
bound, we may be allowing arbituary users from generating DB queries
which may end up exhausing the resources on the server.
What is the fix here?
A new `fetch_limit_from_params` helper method is introduced in
`ApplicationController` that can be used by controller actions to safely
get the limit from the params as a default limit and maximum limit has
to be set. When an invalid limit params is encountered, the server will
respond with the 400 response code.
Context of this change:
There are two site settings which an admin can configured to set the
default categories and tags that are shown for a new user. `default_navigation_menu_categories`
is used to determine the default categories while
`default_navigation_menu_tags` is used to determine the default tags.
Prior to this change when seeding the defaults, we will filter out the
categories/tags that the user do not have permission to see. However,
this means that when the user does eventually gain permission down the
line, the default categories and tags do not appear.
What does this change do?
With this commit, we have changed it such that all the categories and tags
configured in the `default_navigation_menu_categories` and
`default_navigation_menu_tags` site settings are seeded regardless of
whether the user's visibility of the categories or tags. During
serialization, we will then filter out the categories and tags which the
user does not have visibility of.
Performing a `Delete User`/`Delete and Block User` reviewable actions for a
queued post reviewable from the `review.show` route results in an error
popup even if the action completes successfully.
This happens because unlike other reviewable types, a user delete action
on a queued post reviewable results in the deletion of the reviewable
itself. A subsequent attempt to reload the reviewable record results in
404. The deletion happens as part of the call to `UserDestroyer` which
includes a step for destroying reviewables created by the user being
destroyed. At the root of this is the creator of the queued post
being set as the creator of the reviewable as instead of the system
user.
This change assigns the creator of the reviewable to the system user and
uses the more approapriate `target_created_by` column for the creator of the
post being queued.
Why this change?
The `legacy` navigation menu option for the `navigation_menu` site
setting will be removed shortly after the release of Discourse 3.1 in
the first beta release of Discourse 3.2. Therefore, we're adding an
admin dashboard warning to give sites on the `legacy` navigation menu a
heads up.
We recently introduced this advice to admins when some translation overrides are outdated or using unknown interpolation keys:
However we missed the case where the original translation key has been renamed or altogether removed. When this happens they are no longer visible in the admin interface, leading to the confusing situation where we say there are outdated translations, but none are shown.
Because we don't explicitly handle this case, some deleted translations were incorrectly marked as having unknown interpolation keys. (This is because I18n.t will return a string like "Translation missing: foo", which obviously has no interpolation keys inside.)
This change adds an additional status, deprecated for TranslationOverride, and the job that checks them will check for this status first, taking precedence over invalid_interpolation_keys. Since the advice only checks for the outdated and invalid_interpolation_keys statuses, this fixes the problem.
Why this change?
Prior to this change, dismissing unreads posts did not publish the
changes across clients for the same user. As a result, users can end up
seeing an unread count being present but saw no topics being loaded when
visiting the `/unread` route.
This PR adds a feature to help admins stay up-to-date with their translations. We already have protections preventing admins from problems when they update their overrides. This change adds some protection in the other direction (where translations change in core due to an upgrade) by creating a notice for admins when defaults have changed.
Terms:
- In the case where Discourse core changes the default translation, the translation override is considered "outdated".
- In the case above where interpolation keys were changed from the ones the override is using, it is considered "invalid".
- If none of the above applies, the override is considered "up to date".
How does it work?
There are a few pieces that makes this work:
- When an admin creates or updates a translation override, we store the original translation at the time of write. (This is used to detect changes later on.)
- There is a background job that runs once every day and checks for outdated and invalid overrides, and marks them as such.
- When there are any outdated or invalid overrides, a notice is shown in admin dashboard with a link to the text customization page.
Known limitations
The link from the dashboard links to the default locale text customization page. Given there might be invalid overrides in multiple languages, I'm not sure what we could do here. Consideration for future improvement.
Follow-up to b27e12445d
This commit adds 2 new site settings `default_sidebar_link_to_filtered_list` and `default_sidebar_show_count_of_new_items` to control the default values for the navigation menu preferences that were added in the linked commit (`sidebar_link_to_filtered_list` and `sidebar_show_count_of_new_items` respectively).
Recently, we added the option for watched tag/categories to take precedence over muted tag/categories. Therefore, `remove_muted_tags` is using `category_users` to check if categories are not watched. There was missing join in CategoryList which was causing an error.
New setting which allow admin to define behavior when topic is in watched category and muted topic and vice versa.
If watched_precedence_over_muted setting is true, that topic is still visible in list of topics and notification is created.
If watched_precedence_over_muted setting is false, that topic is not still visible in list of topics and notification is skipped as well.
This change adds support retroactively updating display names in the new quote format when the user's name is changed. It happens through a background job that is triggered by a callback when a user is saved with a new name.
This method is a huge footgun in production, since it calls
the Redis KEYS command. From the Redis documentation at
https://redis.io/commands/keys/:
> Warning: consider KEYS as a command that should only be used in
production environments with extreme care. It may ruin performance when
it is executed against large databases. This command is intended for
debugging and special operations, such as changing your keyspace layout.
Don't use KEYS in your regular application code.
Since we were only using `delete_prefixed` in specs (now that we
removed the usage in production in 24ec06ff85)
we can remove this and instead rely on `use_redis_snapshotting` on the
particular tests that need this kind of clearing functionality.
Communities can use sidebar or header dropdown, therefore navigation menu is a better name settings in 2 places:
- Old user sidebar preferences;
- Site setting about default tags and categories.
Currently, groups owned by moderators are not visible to them on the
groups page. This happens because, the group visibility queries don't
account for non-admin staff user group ownership.
This change updates the group visibility scope queries to account for a
moderator(non-admin staff user) group ownership.
This patch sets some limits on custom fields:
- an entity can’t have more than 100 custom fields defined on it
- a custom field can’t hold a value greater than 10,000,000 characters
The current implementation of custom fields is relatively complex and
does an upsert in SQL at some point, thus preventing to simply add an
`ActiveRecord` validation on the custom field model without having to
rewrite a part of the existing logic.
That’s one of the reasons this patch is implementing validations in the
`HasCustomField` module adding them to the model including the module.
* FEATURE: reduce avatar sizes to 6 from 20
This PR introduces 3 changes:
1. SiteSetting.avatar_sizes, now does what is says on the tin.
previously it would introduce a large number of extra sizes, to allow for
various DPIs. Instead we now trust the admin with the size list.
2. When `avatar_sizes` changes, we ensure consistency and remove resized
avatars that are not longer allowed per site setting. This happens on the
12 hourly job and limited out of the box to 20k cleanups per cycle, given
this may reach out to AWS 20k times to remove things.
3.Our default avatar sizes are now "24|48|72|96|144|288" these sizes were
very specifically picked to limit amount of bluriness introduced by webkit.
Our avatars are already blurry due to 1px border, so this corrects old blur.
This change heavily reduces storage required by forums which simplifies
site moves and more.
Co-authored-by: David Taylor <david@taylorhq.com>
This commit adds modifiers that allow plugins to change how categories and groups are prefetched into the application and listed in the respective controllers.
Possible use cases:
- prevent some categories/groups from being prefetched when the application loads for performance reasons.
- prevent some categories/groups from being listed in their respective index pages.
When a user chooses to move a topic/message to an existing topic/message, they can now opt to merge the posts chronologically (using a checkbox in the UI).
What is this change required?
I noticed that actions in `SidebarSectionsController` resulted in
lots of N+1 queries problem and I wanted a solution to
prevent such problems without having to write N+1 queries tests. I have
also used strict loading for `SidebarSection` queries in performance
sensitive spots.
Note that in this commit, I have also set `config.active_record.action_on_strict_loading_violation = :log`
for the production environment so that we have more visibility of
potential N+1 queries problem in the logs. In development and test
environment, we're sticking with the default of raising an error.
This patch is a followup of
https://github.com/discourse/discourse/pull/21504 where limits on custom
message for an invite were introduced.
This had a side effect of making some existing invites invalid and with
the current code, they can’t be invalidated anymore.
This patch takes the approach of skipping the validations when invites
are invalidated since the important thing here is to mark the invite as
invalidated regardless of its actual state in the DB. (no other
attributes are updated at the same time anyway)
Comparing arrays without an explicit order or sort is usually a bad idea and leads to flakiness. It also replaces `#sort` calls in a couple of specs with array specific matchers like `contain_exactly` and `match_array`.
In addition to that it switches the arguments of some expectations around, because it should be `expect(actual).to eq(expected)` instead of `expect(expected).to eq(actual)`
* FIX: Video thumbnails can have duplicates
It's possible that a duplicate video or even a very similar video could
generate the same video thumbnail. Because video thumbnails are mapped
to their corresponding video by using the video sha1 in the thumbnail
filename we need to allow for duplicate thumbnails otherwise even when a
thumbnail has been generated for a topic it will not be mapped
correctly.
This will also allow you to re-upload a video on the same topic to
regenerate the thumbnail.
* fix typo
The old method updated only existing records, without considering that
new tags might have been created or some tags might not exist anymore.
This was usually not a problem because the stats were also updated by
other code paths.
However, the ensure consistency job should be more solid and help when
other code paths fail or after importing data.
Also, update category tag stats too should happen when updating other
category stats as well.
- Update welcome topic copy
- Edit the welcome topic automatically when the title or description changes
- Remove “Create your Welcome Topic” banner/CTA
- Add "edit welcome topic" user tip
### Background
When SSRF detection fails, the exception bubbles all the way up, causing a log alert. This isn't actionable, and should instead be ignored. The existing `rescue` does already ignore network errors, but fails to account for SSRF exceptions coming from `FinalDestination`.
### What is this change?
This PR does two things.
---
Firstly, it introduces a common root exception class, `FinalDestination::SSRFError` for SSRF errors. This serves two functions: 1) it makes it easier to rescue both errors at once, which is generally what one wants to do and 2) prevents having to dig deep into the class hierarchy for the constant.
This change is fully backwards compatible thanks to how inheritance and exception handling works.
---
Secondly, it rescues this new exception in `UserAvatar.import_url_for_user`, which is causing sporadic errors to be logged in production. After this SSRF errors are handled the same as network errors.
This fixes a bug in the create invite API where if you passed in an
integer for the group_ids field it would fail to add the user to the
specified group.
The current limit (250 characters) is too low, as we have some
translations used for our badge descriptions that result in a
description length of 264 characters.
To be on the safe side, the limit is now set to 500 characters.
This amends it so our cached counting reliant specs run in synchronize mode
When running async there are situations where data is left over in the table
after a transactional test. This means that repeat runs of the test suite
fail.
* DEV: move sidebar community section to database
Before, community section was hard-coded. In the future, we are planning to allow admins to edit it. Therefore, it has to be moved to database to `custom_sections` table.
Few steps and simplifications has to be made:
- custom section was hidden behind `enable_custom_sidebar_sections` feature flag. It has to be deleted so all forums, see community section;
- migration to add `section_type` column to sidebar section to show it is a special type;
- migration to add `segment` column to sidebar links to determine if link should be displayed in primary section or in more section;
- simplify more section to have one level only (secondary section links are merged);
- ensure that links like `everything` are correctly tracking state;
- make user an anonymous links position consistence. For example, from now on `faq` link for user and anonymous is visible in more tab;
- delete old community-section template.
SearchIndexer is only automatically disabled in `before_all` and `before` blocks which means at the start
of test runs. Enabling the SearchIndexer in one `fab!` block will affect
all other `fab!` blocks which is not ideal as we may be indexing stuff
for search when we don't need to.
Many blog posts use these to illustrate and images were previously omitted
Additionally strip superfluous HTML and BODY tags from embed HTML.
This was incorrectly returned from server.
Group user event webhooks filtered by group fail silently
because the `group_ids` job arg wasn't being passed into the job.
This change add's `group_ids` to the `EmitWebHookEvent` jobs queued for
`user_added_to_group` and `user_removed_from_group` events.
Currently, only user badge grants emit webhook events. This change
extends the `user_badge` webhook to emit user badge revocation events.
A new `user_badge_revoked` event has been introduced instead of relying
on the existing `user_badge_removed` event. `user_badge_removed` emitted
just the `badge_id` and `user_id` which aren't helpful for generating a
meaningful webhook payload for revoked(deleted) user badges.
The new event emits the user badge object.
* FIX: Do not overwrite existing thumbnails
When auto generating video thumbnails they should not overwrite any
existing topic thumbnails.
This also addresses an issue with capitalized file extensions like .MOV
that were being excluded.
* Update app/models/post.rb
Remove comment
Co-authored-by: Penar Musaraj <pmusaraj@gmail.com>
---------
Co-authored-by: Penar Musaraj <pmusaraj@gmail.com>
This fixes a 500 error that occurs when adding a tag to a category's
restricted tag list if the category's restricted tags already included a
synonym tag.
This feature will allow sites to define which emoji are not allowed. Emoji in this list should be excluded from the set we show in the core emoji picker used in the composer for posts when emoji are enabled. And they should not be allowed to be chosen to be added to messages or as reactions in chat.
This feature prevents denied emoji from appearing in the following scenarios:
- topic title and page title
- private messages (topic title and body)
- inserting emojis into a chat
- reacting to chat messages
- using the emoji picker (composer, user status etc)
- using search within emoji picker
It also takes into account the various ways that emojis can be accessed, such as:
- emoji autocomplete suggestions
- emoji favourites (auto populates when adding to emoji deny list for example)
- emoji inline translations
- emoji skintones (ie. for certain hand gestures)
Previously, Discourse's password hashing was hard-coded to a specific algorithm and parameters. Any changes to the algorithm or parameters would essentially invalidate all existing user passwords.
This commit introduces a new `password_algorithm` column on the `users` table. This persists the algorithm/parameters which were use to generate the hash for a given user. All existing rows in the users table are assumed to be using Discourse's current algorithm/parameters. With this data stored per-user in the database, we'll be able to keep existing passwords working while adjusting the algorithm/parameters for newly hashed passwords.
Passwords which were hashed with an old algorithm will be automatically re-hashed with the new algorithm when the user next logs in.
Values in the `password_algorithm` column are based on the PHC string format (https://github.com/P-H-C/phc-string-format/blob/master/phc-sf-spec.md). Discourse's existing algorithm is described by the string `$pbkdf2-sha256$i=64000,l=32$`
To introduce a new algorithm and start using it, make sure it's implemented in the `PasswordHasher` library, then update `User::TARGET_PASSWORD_ALGORITHM`.
This commit adds support for filtering for topics in specific
subcategories via the categories filter query language.
For example: `category:documentation:admins` will filter for topics and
subcategory topics in
the category with slug "admins" whose parent category has the slug
"documentation".
The `=` prefix can also be used such that
`=category:documentation:admins` will exclude subcategory topics of the
category with slug "admins" whose parent category has the slug
"documentation".
`Rails.application.routes.recognize_path(value)` was not working for /admin paths because StaffConstraint.new requires user to check permission.
This validation is not bringing much value, and the easiest way is to drop it. In the worse case scenario, a user will have an incorrect link in their sidebar.
Bug reported: https://meta.discourse.org/t/custom-sidebar-sections-being-tested-on-meta/255303/66
`default_categories_*` site settings will update the category preferences on user creation. But it shouldn't update the user's category preference if a group's setting already updated it for that user.
There was a lot of duplication in the svg parsing and coercion code. This reduces that duplication and causes svg sprite parsing to happen earlier so that more computation is cached.
If you happen to delete the general category before editing the welcome
topic, the banner will still display. This fix adds a after destroy hook
that will clear the entries for the welcome topic banner in the redis
cache.
Currently the auto-bump cooldown is hard-coded to 24 hours.
This change makes the highlighted 24 hours part configurable (defaulting to 24 hours), and the rest of the process remains the same.
This uses the new CategorySetting model associated with Category. We decided to add this because we want to move away from custom fields due to the lack of type casting and validations, but we want to keep the loading of these optional as they are not needed for almost all of the flows.
Category settings will be back-filled to all categories as part of this change, and creating a new category will now also create a category setting.
* FEATURE: Generate thumbnail images for uploaded videos
Topics in Discourse have a topic thumbnail feature which allows themes
to show a preview image before viewing the actual Topic.
This PR allows for the ability to generate a thumbnail image from an
uploaded video that can be use for the topic preview.
Currently, when a suspended user belongs to a group PM (private message
with more than two people in it) and a staff member sends a message to
this group PM, then the suspended user will receive an email.
This happens because a suspended user can only receive emails from staff
members. But in this case, this can be seen as a bug as the expected
behavior would be instead to not send any email to the suspended user. A
staff member can participate in active discussions like any other
member and so their messages in this context shouldn’t be treated
differently than the ones from regular users.
This patch addresses this issue by checking if a suspended user receives
a message from a group PM or not. If that’s the case then an email won’t
be sent no matter if the post originated from a staff member or not.
That column is obsolete since we added the `granted_title_badge_id` column in 2019 (56d3e29a69). Having both columns can lead to inconsistencies (mostly due to old data from before 2019).
For example, `BadgeGranter.revoke_ungranted_titles!` doesn't work correctly if `badge_granted_title` is `false` while `granted_title_badge_id` points to the badge that is used as title.
Similar spirit to e195e6f614,
this moves the Bookmarkable registration to DiscoursePluginRegistry
so plugins which are not enabled do not register additional
bookmarkable classes.
* UX: add type tag and design update
* UX: clarify status copy in reviewQ
* DEV: switch to selectKit
* UX: color approve/reject buttons in RQ
* DEV: regroup actions
* UX: add type tag and design update
* UX: clarify status copy in reviewQ
* Join questions for flagged post with "or" with new I18n function
* Move ReviewableScores component out of context
* Add CSS classes to reviewable-item based on human type
* UX: add table header for scoring
* UX: don't display % score
* UX: prefix modifier class with dash
* UX: reviewQ flag table styling
* UX: consistent use of ignore icon
* DEV: only show context question on pending status
* UX: only show table headers on pending status
* DEV: reviewQ regroup actions for hidden posts
* UX: reviewQ > approve/reject buttons
* UX: reviewQ add fadeout
* UX: reviewQ styling
* DEV: move scores back into component
* UX: reviewQ mobile styling
* UX: score table on mobile
* UX: reviewQ > move meta info outside table
* UX: reviewQ > score layout fixes
* DEV: readd `agree_and_keep` and fix the spec tests.
* Fix the spec tests
* fix the quint test
* DEV: readd deleting replies
* UX: reviewQ copy tweaks
* DEV: readd test for ignore + delete replies
* Remove old
* FIX: Add perform_ignore back in for backwards compat
* DEV: add an action alias `ignore` for `ignore_and_do_nothing`.
---------
Co-authored-by: Martin Brennan <martin@discourse.org>
Co-authored-by: Vinoth Kannan <svkn.87@gmail.com>
Fixes a small issue where allowed user removes themselves from a private message before the post activity (small action) is created.
I also added some test coverage to prevent regression.
/t/92811
Follow up to 098ab29d41. Since
we just used a `cattr_reader` on `About` this was not safe
for multisite, since some sites could have the chat plugin
enabled and some may not. Using `DiscoursePluginRegistry` gets
around this issue, and makes it so the chat stats only show
for a site if `chat_enabled` is true.
