Instead of having to remember every time, just always wait until the
current transaction (if it exists) has committed before clearing any
DistributedCache.
The only exception to this is caches that aren't caching things from
postgres.
This means we have to do the test setup after setting the test
transaction, because doing the test setup involves clearing caches.
Reapplying this - it now doesn't use after_commit if skip_db is set
* FEATURE: Inline topic summary. Cached version accessible to everyone.
Anons and non-members of the `custom_summarization_allowed_groups_map` groups can see cached summaries for any accessible topic. After the first 12 hours and if the posts to summarize have changed, allowed users clicking on the button will automatically re-generate it.
* Ensure chat summaries work and prevent model hallucinations when there are no messages.
These avatar-related helper functions are used in pretty-text, which currently means we load the entire `discourse/lib/utilities` module into the mini-racer when running pretty-text on the server side. This stops us adding any logic or imports to discourse/lib/utilities which may depend on other `discourse/` namespace features.
This commit moves the avatar-related utils into a dedicated module in the `discourse-common` namespace, adds backwards-compatibility shims, and updates the pretty-text config accordingly.
This adds an option to allow non-image s3 files to be downloaded through CDN URL.
Addresses the issues in:
* meta.discourse.org/t/s3-cdn-url-not-being-used-on-non-image-uploads/175332
* meta.discourse.org/t/s3-uploads-using-cdn-for-pdfs/213218
Instead of having to remember every time, just always wait until the
current transaction (if it exists) has committed before clearing any
DistributedCache.
The only exception to this is caches that aren't caching things from
postgres.
This means we have to do the test setup after setting the test
transaction, because doing the test setup involves clearing caches.
Follow-up to b27e12445d
This commit adds 2 new site settings `default_sidebar_link_to_filtered_list` and `default_sidebar_show_count_of_new_items` to control the default values for the navigation menu preferences that were added in the linked commit (`sidebar_link_to_filtered_list` and `sidebar_show_count_of_new_items` respectively).
Why this change?
The process's pid is useful when we're trying to link output from
different processes together. In this case, we want to be able to link
the Rails server logs to the right rspec process.
Before:
[2] Viewing sidebar mobile collapses the sidebar when clicking outside of it
After:
[2] (#176342) Viewing sidebar mobile collapses the sidebar when clicking outside of it
Recently, site setting watched_precedence_over_muted was introduced - https://github.com/discourse/discourse/pull/22252
In this PR, we are allowing users to override it. The option is only displayed when the user has watched categories and muted tags, or vice versa.
This will be used when we move the channel creation for DMs
to happen when we first send a message in a DM channel to avoid
a double-request. For now we can just have a new API endpoint
for creating this that the existing frontend code can use,
that uses the new service pattern.
This also uses the new policy pattern for services where the policy
can be defined in a class so a more dynamic reason for the policy
failing can be sent to the controller.
Co-authored-by: Loïc Guitaut <loic@discourse.org>
Twitter is now redirecting anonymous users (with a browser-like user agent, which FinalDestination uses) to the login page. Skipping redirect-following for twitter.com will allow us to continue oneboxing tweets via the OpenGraph data and the API (when credentials are present).
https://meta.discourse.org/t/269371/17
Updates the interface for implementing summarization strategies and adds a cache layer to summarize topics once.
The cache stores the final summary and each chunk used to build it, which will be useful when we have to extend or rebuild it.
New setting which allow admin to define behavior when topic is in watched category and muted topic and vice versa.
If watched_precedence_over_muted setting is true, that topic is still visible in list of topics and notification is created.
If watched_precedence_over_muted setting is false, that topic is not still visible in list of topics and notification is skipped as well.
While we are unable to support OAUTH2 with pop3 (due to upstream dependency ruby/net-pop#16), we are adding the support for mail pollers plugin. Doing so, it would be possible to write a plugin which then uses other ways (microsoft graph sdk for example) to poll emails from a mailbox.
The idea is that a plugin would define a class which inherits from Email::Poller and defines a poll_mailbox static method which returns an array of strings. Then the plugin could call register_mail_poller(<class_name>) to have it registered. All the configuration (oauth2 tokens, email, etc) could be managed by sitesettings defined in the plugin.
This change adds support retroactively updating display names in the new quote format when the user's name is changed. It happens through a background job that is triggered by a callback when a user is saved with a new name.
Under exceptional cases people may need to resize the notification table.
This only happens on forums with a total of more than 2.5 billion notifications.
This rake task can be used to convert all the notification columns to
bigint to make more room.
https://meta.discourse.org/t/markdown-preview-and-result-differ/263878
The result of this markdown had different results in the composer preview and the post. This is solved by updating Loofah to the latest version and using html5 fragments like our user had reported. While the change was only needed in cooked_post_processor.rb for this fix, other areas also had to be updated due to various side effects.
Upstream added a capital 'T' to the 'Translation missing' message in https://github.com/ruby-i18n/i18n/commit/c5c6e753f3. This caused our translate accelerator patch to diverge, and the change in case affected a number of our specs. This commit updates the translate accelerator to match the upstream casing, and introduces a spec to detect future divergence.
This method is a huge footgun in production, since it calls
the Redis KEYS command. From the Redis documentation at
https://redis.io/commands/keys/:
> Warning: consider KEYS as a command that should only be used in
production environments with extreme care. It may ruin performance when
it is executed against large databases. This command is intended for
debugging and special operations, such as changing your keyspace layout.
Don't use KEYS in your regular application code.
Since we were only using `delete_prefixed` in specs (now that we
removed the usage in production in 24ec06ff85)
we can remove this and instead rely on `use_redis_snapshotting` on the
particular tests that need this kind of clearing functionality.
Added a new modifier hook to allow plugins to modify the @mentions
extracted from a cooked text.
Use case: Some plugins may change how the mentions are cooked to prevent
them from being confused with user or group mentions and display the user
card.
This modifier hook allows the plugin to filter the mentions detected or add new ways
to add mentions into cooked text.
Communities can use sidebar or header dropdown, therefore navigation menu is a better name settings in 2 places:
- Old user sidebar preferences;
- Site setting about default tags and categories.
When we introduced the new quote format with full-name display name:
```
[quote="Ted Johansson, post:1, topic:2, username:ted"]
we overlooked the code responsible for rewriting quotes when a user's name is changed.
```
The functional part of this change adds support for the new quote format in the code that updates quotes when a user's username changes. See the test case in `spec/services/username_changer_spec.rb` for the details.
In addition, this change adds a regression test for PrettyText to cover the new quote format, and extracts the code responsible for rewriting raw and cooked quotes into its own `QuoteRewriter` class. The functionality of the latter is tested through the tests in `spec/services/username_changer_spec.rb`.
* FEATURE: Content custom summarization strategies.
This PR establishes a pattern for plugins to register alternative ways of summarizing content by extending a class that defines an interface.
Core controls which strategy we'll use and who has access to it through the `summarization_strategy` and `custom_summarization_allowed_groups`. It also defines the UI for summarizing topics.
Other plugins can access this summarization mechanism and implement their features, removing cross-plugin customizations, as it currently happens between chat and the discourse-ai plugin.
* Group membership validation and rate limiting
* Work with objects instead of classes
* Port summarization feature from discourse-ai to chat
* Rename available summaries to 'Top Replies' and 'Summary'
Previously workbox JS was vendored into our git repository, and would be loaded from the `public/javascripts` directory with a 1 day cache lifetime. The main aim of this commit is to add 'cachebuster' to the workbox URL so that the cache lifetime can be increased.
- Remove vendored copies of workbox.
- Use ember-cli/broccoli to collect workbox files from node_modules into assets/workbox-{digest}
- Add assets to sprockets manifest so that they're collected from the ember-cli output directory (and uploaded to s3 when configured)
Some of the sprockets-related changes in this commit are not ideal, but we hope to remove sprockets in the not-too-distant future.
Usually, when a user is promoted to TL2 two messages are sent. The
first one is a system message 'tl2_promotion_message' which triggers a
'system_message_sent' Discourse event.
When the event is fired and if Discourse Narrative Bot is enabled, then
a second message is sent to the recipient of the first message. The
recipients was determined by looking at the list of users that can
access that topic and pick the last one. This method does not work if
'site_contact_group_name' site setting is set because it adds the group
in the list of recipients.
A solution to this problem would have been to select the last user in
the list of 'topic_allowed_users', but an even better solution is to
pass the name of the recipients when the 'system_message_sent'
Discourse event is fired.
Adds a new `[grid]` tag that can arrange images (or other media) into a grid in posts.
The grid defaults to a 3-column with a few exceptions:
- if there are only 2 or 4 items, it defaults to a 2-column grid (because it generally looks better)
- on mobile, it defaults to a 2-column grid
- if there is only one item, the grid has no effect
AWS recommends running buckets without ACLs, and to use resource policies to manage access control instead.
This is not a bad idea, because S3 ACLs are whack, and while resource policies are also whack, they're a more constrained form of whack.
Further, some compliance regimes get antsy if you don't go with the vendor's recommended settings, and arguing that you need to enable ACLs on a bucket just to store images in there is more hassle than it's worth.
The new site setting (s3_use_acls) cannot be disabled when secure
uploads is enabled -- the latter relies on private ACLs for security
at this point in time. We may want to reexamine this in future.
Not all revisions involve changes to the actual post/topic content. We
may want to know if a revisions includes the topic title or post raw.
Specifically introducing these for use in the Akismet plugin to
conditionally queue checks.
What does this change do?
Suggested topics by default are ordered in the following way:
1. Unread topics in current category of topic that is being viewed
2. Unread topics in other categories
3. New topics in current category of topics that is being viewed
4. New topics in other categories
5. Random topics
With the experimental new new view, we want to remove the concept of
read and new so that new order is as such:
1. Topics created by the current user with posts that the user has not
read ordered by topic's bumped date
2. Topics in current category of topic with posts that the user has not
read ordered by topic's bumped date
3. Topics in other categories with posts that the user has not read
ordered by topic's bumped date
4. Random topics ordered by topic's bumped date
When a topic already has multiple synonym tags of a target tag, if we try to update the "`tag_id`" column to target tag id then it will raise a unique violation error since there are multiple synonyms present in the topic. So before doing that action, we must delete the problematic tags so the topic has only one synonym tag to update.
This is not an issue when the topic has a target tag already along with synonyms.
`DiscourseIpInfo` expects zeitwerk auto-loading to be available, so we need to ensure the rake task loads the full rails environment. Normally we run this task as part of assets:precompile, so the app is already initialized. This commit only affects the case where the maxmind task is run directly.
Test were sometimes failing with similar error to the following:
```
1) UsernameChanger#override when unicode_usernames is off overrides the username if a new name has different case
Failure/Error:
protect { v8.eval(<<~JS) }
__paths = #{paths_json};
__utils.avatarImg({size: #{size.inspect}, avatarTemplate: #{avatar_template.inspect}}, __getURL);
JS
MiniRacer::RuntimeError:
ReferenceError: __optInput is not defined
# JavaScript at exports.helperContext (<anonymous>:21:17)
# JavaScript at getRawAvatarSize (<anonymous>:108:49)
# JavaScript at avatarUrl (<anonymous>:102:21)
# JavaScript at Object.avatarImg (<anonymous>:129:15)
# JavaScript at <anonymous>:2:9
# ./lib/pretty_text.rb:259:in `block in avatar_img'
# ./lib/pretty_text.rb:661:in `block in protect'
# ./lib/pretty_text.rb:661:in `synchronize'
# ./lib/pretty_text.rb:661:in `protect'
# ./lib/pretty_text.rb:259:in `avatar_img'
# ./app/jobs/regular/update_username.rb:14:in `execute'
```
This should not be needed as it should already have been initialised but that should stop the flakey-ness for now while being a safe change.
Prior to this commit, we didn't have RTL versions of our admin and plugins CSS bundles and we always served LTR versions of those bundles even when users used an RTL locale, causing admin and plugins UI elements to never look as good as when an LTR locale was used. Example of UI issues prior to this commit were: missing margins, borders on the wrong side and buttons too close to each other etc.
This commit creates an RTL version for the admin CSS bundle as well as RTL bundles for all the installed plugins and serves those RTL bundles to users/sites who use RTL locales.
* FEATURE: reduce avatar sizes to 6 from 20
This PR introduces 3 changes:
1. SiteSetting.avatar_sizes, now does what is says on the tin.
previously it would introduce a large number of extra sizes, to allow for
various DPIs. Instead we now trust the admin with the size list.
2. When `avatar_sizes` changes, we ensure consistency and remove resized
avatars that are not longer allowed per site setting. This happens on the
12 hourly job and limited out of the box to 20k cleanups per cycle, given
this may reach out to AWS 20k times to remove things.
3.Our default avatar sizes are now "24|48|72|96|144|288" these sizes were
very specifically picked to limit amount of bluriness introduced by webkit.
Our avatars are already blurry due to 1px border, so this corrects old blur.
This change heavily reduces storage required by forums which simplifies
site moves and more.
Co-authored-by: David Taylor <david@taylorhq.com>
This commit adds modifiers that allow plugins to change how categories and groups are prefetched into the application and listed in the respective controllers.
Possible use cases:
- prevent some categories/groups from being prefetched when the application loads for performance reasons.
- prevent some categories/groups from being listed in their respective index pages.
Rescuing them still makes timing-out tests fail but doesn't break `after` spec cleanup (which could trigger more errors) Using custom error class to avoid any other possible timeout-catching code.
Also:
* remove an unnecessary `.select { |x| x.size > 0 }`
* fix a typo in a test title
Privacy Policy and Terms of Service topics are no longer created by
default for communities that have not set a company name. For this
reason, some URLs were pointing to 404 page.
Why is this change required?
By default, `RSpec` comes with a `--profile=[COUNT]` option as well but
enabling that option means that the entire test suite needs to be
executed. This does not work so well for `turbo_rspec` which splits our
test files into various "buckets" for the tests to be executed in
multiple processes. Therefore, this commit adds a similar
`--profile=[COUNT]` option to `turbo_rspec` but will only profile the
tests being executed. Examples:
`LOAD_PLUGINS=1 bin/turbo_rspec --profile plugins/*/spec/system`
or
`LOAD_PLUGINS=1 bin/turbo_rspec --profile=20 plugins/*/spec/system`
What is the problem?
In the test environement, we were calling `SiteSetting.setting` directly
to introduce new site settings. However, this leads to changes in state of the SiteSettings
hash that is stored in memory as test runs. Changing or leaking states
when running tests is one of the major contributors of test flakiness.
An example of how this resulted in test flakiness is our `spec/integrity/i18n_spec.rb` spec file which
had a test case that would fail because a new "plugin_setting" site
setting was registered in another test case but the site setting did not
have translations for the site setting set.
What is the fix?
There are a couple of changes being introduced in this commit:
1. Make `SiteSetting.setting` a private method as it is not safe to be
exposed as a public method of the `SiteSetting` class
2. Change test cases to use existing site settings in Discourse instead
of creating custom site settings. Existing site settings are not
removed often so we don't really need to dynamically add new site
settings in test cases. Even if the site settings being used in test
cases are removed, updating the test cases to rely on other site
settings is a very easy change.
3. Set up a plugin instance in the test environment as a "fixture"
instead of having each test create its own plugin instance.
Legal topics, such as the Terms of Service and Privacy Policy topics
do not make sense if the entity creating the community is not a company.
These topics will be created and updated only when the company name is
present and deleted when it is not.
In some cases reverse chronological can be very important.
- Oldest post by sam
- Oldest topic by sam
Prior to these new filters we had no way of searching for them.
Now the 2 new orders `order:oldest` and `order:oldest_topic` can be used
to find oldest topics and posts
* Update spec/lib/search_spec.rb
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
* Update spec/lib/search_spec.rb
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
---------
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
* FIX: Video thumbnails can have duplicates
It's possible that a duplicate video or even a very similar video could
generate the same video thumbnail. Because video thumbnails are mapped
to their corresponding video by using the video sha1 in the thumbnail
filename we need to allow for duplicate thumbnails otherwise even when a
thumbnail has been generated for a topic it will not be mapped
correctly.
This will also allow you to re-upload a video on the same topic to
regenerate the thumbnail.
* fix typo
This commit makes some fundamental changes to how hashtag cooking and
icon generation works in the new experimental hashtag autocomplete mode.
Previously we cooked the appropriate SVG icon with the cooked hashtag,
though this has proved inflexible especially for theming purposes.
Instead, we now cook a data-ID attribute with the hashtag and add a new
span as an icon placeholder. This is replaced on the client side with an
icon (or a square span in the case of categories) on the client side via
the decorateCooked API for posts and chat messages.
This client side logic uses the generated hashtag, category, and channel
CSS classes added in a previous commit.
This is missing changes to the sidebar to use the new generated CSS
classes and also colors and the split square for categories in the
hashtag autocomplete menu -- I will tackle this in a separate PR so it
is clearer.
Regressed in eec10efc3d. It means that backend plugin spec failures in CI were not failing the spec suite.
Fixes recent regressions and skips two of them - to be handled next week.
---------
Co-authored-by: Andrei Prigorshnev <a.prigorshnev@gmail.com>
https://meta.discourse.org/t/improving-mailman-email-parsing/253041
When mirroring a public mailling list which uses mailman, there were some cases where the incoming email was not associated to the proper user.
As it happens, for various (undertermined) reasons, the email from the sender is often not in the `From` header but can be in any of the following headers: `Reply-To`, `CC`, `X-Original-From`, `X-MailFrom`.
It might be in other headers as well, but those were the ones we found the most reliable.
The old method updated only existing records, without considering that
new tags might have been created or some tags might not exist anymore.
This was usually not a problem because the stats were also updated by
other code paths.
However, the ensure consistency job should be more solid and help when
other code paths fail or after importing data.
Also, update category tag stats too should happen when updating other
category stats as well.
Currently processing emails that are blank or have a nil value for the mail will cause several errors.
This update allows emails with blank body or missing sender to log the blank email error to the mail logs rather than throwing an error.
In #21498, we split `BaseStore#download` into a "safe" version which returns nil on errors, and an "unsafe" version which raises an exception, which was the old behaviour of `#download`.
This change updates call sites that used the old `#download`, which raised exceptions, to use the new `#download!` to preserve behaviour (and silence deprecation warnings.)
It also silences the deprecation warning in tests.
* DEV: add new thread icon
* FIX: Use new thread icon, fix typo in SVG
UX: move the thread list icon to the right of
the collapse button
---------
Co-authored-by: Martin Brennan <martin@discourse.org>
* FIX: Displaying the wrong number of minimum tags in the composer
When the minimum number of tags set for the category is larger than the minimum number of tags
set in the category tag-groups, the composer was displaying the wrong value.
This commit fixes the value displayed in the composer to show the max value between the required
for the category and the tag-groups set for the category.
This bug was reported on Meta in https://meta.discourse.org/t/tags-from-multiple-tag-groups-required-only-suggest-select-at-least-one-tag/263817
* FIX: Limiting tags in categories not working as expected
When a category was restricted to a tag group A, which was set to only allow
one tag from the group per topic, selecting a tag belonging only to A returned
other tags from A that also belonged to other group/s (if any).
Example:
Tag group A: alpha, beta, gamma, epsilon, delta
Tag group B: alpha, beta, gamma
Both tag groups set to only allow one tag from the group per topic.
If Category 1 was set to only allow tags from the tag group A, and the first tag
selected was epsilon, then, because they also belonged to tag group B, the tags
alpha, beta, and gamma were still returned as valid options when they should not be.
This commit ensures that once a tag from a tag group that restricts its tags to
one per topic is selected, no other tag from this group is returned.
This bug was reported on Meta in https://meta.discourse.org/t/limiting-tags-to-categories-not-working-as-expected/263143.
* FIX: Moving topics does not prompt to add required tag for new category
When a topic moved from a category to another, the tag requirements
of the new category were not being checked.
This allowed a topic to be created and moved to a category:
- that limited the tags to a tag group, with the topic containing tags
not allowed.
- that required N tags from a tag group, with the topic not containing
the required tags.
This bug was reported on Meta in https://meta.discourse.org/t/moving-tagged-topics-does-not-prompt-to-add-required-tag-for-new-category/264138.
* FIX: Editing topics with tag groups from parents allows incorrect tagging
When there was a combination between parent tags defined in a tag group
set to allow only one tag from the group per topic, and other tag groups
relying on this restriction to combine the children tag types with the
parent tag, editing a topic could allow the user to insert an invalid
combination of these tags.
Example:
Automakers tag group: landhover, toyota
- group set to limit one tag from the group per topic
Toyota models group: land-cruiser, hilux, corolla
Landhover models group: evoque, defender, discovery
If a topic was initially set up with the tags toyota, land-cruiser it was
possible to edit it by removing the tag toyota and adding the tag landhover
and other landhover model tags like evoque for example.
In this case, the topic would end up with the tags toyota, land-cruiser,
landhover, evoque because Discourse will automatically insert the
missing parent tag toyota when it detects the tag land-cruiser.
This combination of tags would violate the restriction specified in
the Automakers tag group resulting in an invalid combination of tags.
This commit enforces that the "one tag from the group per topic"
restriction is verified before updating the topic tags and also
make sure the verification checks the compatibility of parent tags that
would be automatically inserted.
After the changes, the user will receive an error similar to:
The tags land-cruiser, landhover cannot be used simultaneously.
Please include only one of them.
Watched words were converted to regular expressions containing \W, which
handled only ASCII characters. Using [^[:word]] instead ensures that
UTF-8 characters are also handled correctly.
* Color for turbo_rspec in CI (`progress` and `documentation` formats)
* Show "DONE" only when `documentation` formatter is used
* Fix formatting
* Collapse RSpec commands
* Add line wrapping to the `progress` formatter (to mitigate GH Actions issue)
- Update welcome topic copy
- Edit the welcome topic automatically when the title or description changes
- Remove “Create your Welcome Topic” banner/CTA
- Add "edit welcome topic" user tip
### Background
When SSRF detection fails, the exception bubbles all the way up, causing a log alert. This isn't actionable, and should instead be ignored. The existing `rescue` does already ignore network errors, but fails to account for SSRF exceptions coming from `FinalDestination`.
### What is this change?
This PR does two things.
---
Firstly, it introduces a common root exception class, `FinalDestination::SSRFError` for SSRF errors. This serves two functions: 1) it makes it easier to rescue both errors at once, which is generally what one wants to do and 2) prevents having to dig deep into the class hierarchy for the constant.
This change is fully backwards compatible thanks to how inheritance and exception handling works.
---
Secondly, it rescues this new exception in `UserAvatar.import_url_for_user`, which is causing sporadic errors to be logged in production. After this SSRF errors are handled the same as network errors.
After this change, in order to join a chat channel, a user needs to be in a group with at least “Reply” permission for the category. If the user only has “See” permission, they are able to preview the channel, but not join it or send messages. The auto-join function also follows this new restriction.
---------
Co-authored-by: Martin Brennan <martin@discourse.org>
