Commit Graph

10338 Commits

Author SHA1 Message Date
Leonardo Mosquera
a86590ffd6
UX: Add a rake task to monitor progress for long rebakes (#27517)
* UX: Add a rake task to monitor progress for long rebakes

When doing mass rebaking, this task will print progress, speed and
expected time to completion (ETC) on a loop. It is meant for rebakes
that take several hours or days.

It will calculate a 10m moving average over the past 6 hours, and print
ETC accordingly.

It also shows Sidekiq stats in case Sidekiq jobs for rebaking were
enqueued separately, instead of using the rebake_posts tasks in this
file (which are 100% synchronous and do not touch Sidekiq at all).

NOTE: only the currently unbaked count at task start time is considered;
this is useful in live communities with lots of traffic, where new posts
might otherwise change the goal posts continuously.

* Satisfy stree
2024-07-03 07:52:47 +10:00
Keegan George
ea58140032
DEV: Remove summarization code (#27373) 2024-07-02 08:51:47 -07:00
Jan Cernik
6599b85a75
DEV: Block accidental serialization of entire AR models (#27668) 2024-07-01 17:08:48 -03:00
Martin Brennan
ffc99253fa
DEV: Resolve TODO comments for martin-brennan
I am changing many of these to notes or resolving them as is,
most of these I have not actively worked on in years so someone
else can work on them when we get to these areas again.
2024-07-01 15:32:30 +10:00
Osama Sayegh
4c1c25027d
DEV: Wiring for the admin about config page (#27492)
This commit continues work laid out by ffec8163b0 for the admin config page for the /about page. The last commit set up the user interface, and this one sets up all the wiring needed to make the input fields and save buttons actually work.

Internal topic: t/128544.
2024-07-01 05:40:37 +03:00
Jay Pfaffman
6a5ee0693d
FIX: rescue mmdb download failure (#27556)
* rescue mmdb download failure

If mmdb database fails to download a bootstrap fails. This is a trivial fix for that problem. A more elegant solution might check whether the dataabase was downloaded and provide a helpful error message.

* linting

---------

Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
2024-06-28 05:10:14 +08:00
Régis Hanol
a56321efb5 FIX: topic search order
When using the full page search and filtering down to a specific topic, the sort order was overwritten to by by "post_number".

This was confusing because we allow different type of sort order in the full search page.

This fixes it by only sorting by post_number when there's no "global" sort order defined.

Since the "new topic map" uses the search endpoint behind the scene, this also fixes the "most likes" popup.

Context - https://meta.discourse.org/t/searching-order-seems-to-be-broken-when-searching-in-topic/312303
2024-06-27 18:13:26 +02:00
Martin Brennan
cada172981
FIX: Do not capture OAuth user on 2FA page (#27617)
If the `enforce_second_factor_on_external_auth` setting
is disabled and a user logs in with an OAuth method,
they don't automatically get redirected to /preferences/second-factor
on login. However, they can get there manually, and once there
they cannot leave.

This commit fixes the issue and allows them to leave
and also does some refactors to indicate to the client
what login method is used as a followup to
0e1102b332
2024-06-27 10:27:49 +10:00
Loïc Guitaut
f58b844f45
Revert "DEV: Upgrade Rails to version 7.1" (#27625)
This reverts commit ce00f83173.
2024-06-26 18:55:05 +02:00
Régis Hanol
54a59be617 FEATURE: new 'should_add_email_attachments' plugin modifier
That can be used by plugins to control whether email attachments should be sent.

Internal ref - t/132149
2024-06-26 12:36:35 +02:00
Martin Brennan
a128ce5c4c
FIX: Missing multipart/mixed boundary on emails (#27599)
Followup 96a0781bc1

When sending emails where secure uploads is enabled
and secure_uploads_allow_embed_images_in_emails is
true, we attach the images to the email, and we
do some munging with the final email so the structure
of the MIME parts looks like this:

```
multipart/mixed
  multipart/alternative
    text/plain
    text/html
  image/jpeg
  image/jpeg
  image/png
```

However, we were not specifying the `boundary` of the
`multipart/mixed` main content-type of the email, so
sometimes the email would come through appearing to
have an empty body with the entire thing attached as
one attachment, and some mail parsers considered the
entire email as the "epilogue" and/or "preamble".

This commit fixes the issue by specifying the boundary
in the content-type header per https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html
2024-06-25 13:43:10 +10:00
Renato Atilio
55da8a7701
FEATURE: add Untranslated filter to admin text customization (#27555)
Adds a checkbox to filter untranslated text strings in the admin UI, behind a hidden and default `false` site setting `admin_allow_filter_untranslated_text`.
2024-06-24 06:24:06 -03:00
Loïc Guitaut
ce00f83173 DEV: Upgrade Rails to version 7.1
---------

Co-authored-by: Jarek Radosz <jradosz@gmail.com>
2024-06-24 11:16:14 +02:00
Sam
cb3456e18f
FEATURE: add JavaScript mode to ace (#27586)
This allows plugins to use Ace Editor in JavaScript mode. This will be used in Discourse AI when defining custom tools.
2024-06-24 13:38:24 +10:00
Penar Musaraj
f4108702c8
FIX: Regression in custom homepage modifier used in theme components (#27569) 2024-06-21 11:24:11 -04:00
Loïc Guitaut
160011793a Revert "DEV: Upgrade Rails to version 7.1 (#27539)"
This reverts commit ca4af53be8.
2024-06-21 11:20:40 +02:00
Kelv
60d5170587
DEV: add validation on content_security_policy_script_src site setting (#27564)
* DEV: add validation on content_security_policy_script_src site setting
2024-06-21 17:00:22 +08:00
Loïc Guitaut
ca4af53be8 DEV: Upgrade Rails to version 7.1 (#27539)
* DEV: Upgrade Rails to 7.1

* FIX: Remove references to `Rails.logger.chained`

`Rails.logger.chained` was provided by Logster before Rails 7.1
introduced their broadcast logger. Now all the loggers are added to
`Rails.logger.broadcasts`.

Some code in our initializers was still using `chained` instead of
`broadcasts`.

* DEV: Make parameters optional to all FakeLogger methods

* FIX: Set `override_level` on Logster loggers (#27519)

A followup to f595d599dd

* FIX: Don’t duplicate Rack response

---------

Co-authored-by: Jarek Radosz <jradosz@gmail.com>
2024-06-21 09:44:06 +02:00
Penar Musaraj
33de5abb6e
DEV: Extract theme resolution to a helper (#27426)
This ensures that the theme id is resolved as early as possible in the
request cycle. This is necessary for the custom homepage to skip
preloading the wrong data.
2024-06-20 11:33:46 -04:00
Loïc Guitaut
982c005979 Revert "DEV: Upgrade Rails to version 7.1 (#27539)"
This reverts commit 2301dddcff.
2024-06-20 11:43:35 +02:00
Loïc Guitaut
2301dddcff
DEV: Upgrade Rails to version 7.1 (#27539)
* DEV: Upgrade Rails to 7.1

* FIX: Remove references to `Rails.logger.chained`

`Rails.logger.chained` was provided by Logster before Rails 7.1
introduced their broadcast logger. Now all the loggers are added to
`Rails.logger.broadcasts`.

Some code in our initializers was still using `chained` instead of
`broadcasts`.

* DEV: Make parameters optional to all FakeLogger methods

* FIX: Set `override_level` on Logster loggers (#27519)

A followup to f595d599dd

* FIX: Don’t duplicate Rack response

---------

Co-authored-by: Jarek Radosz <jradosz@gmail.com>
2024-06-20 10:33:01 +02:00
Rafael dos Santos Silva
b2a9676f0b
FEATURE: Support Spotify Onebox (#27540) 2024-06-19 13:27:27 -03:00
Ted Johansson
96a0781bc1
FIX: Avoid duplicating e-mail body in summary e-mail (#27535)
We recently fixed a problem where secure upload images weren't re-attached when sending the activity summary e-mail.

This fix contained a bug that would lead to n copies of the e-mail body being included, n being the number of duplicates. This is because #fix_parts_after_attachments! was called once per attachment, and adding more parts to the multipart e-mail.

This PR fixes that by:

Adding a failing test case for the above.
Moving the looping over multiple posts into #fix_parts_after_attachments! itself.
2024-06-19 20:11:47 +08:00
Jarek Radosz
251b3a5c47
DEV: Remove discourse-moderator-attention (#27537)
…from the official plugins list. It's archived now.
2024-06-19 13:26:57 +02:00
Bianca Nenciu
2ce83b4d2e
FIX: Do not extract links for hotlinked images (#27538)
When a post is cooked the links are extracted and `TopicLink` instances
are created for each of them. These links are used in various places,
including the topic view, user summary page, etc.

In previous commit 48e5d1a, hotlinked images from Oneboxes have been
ignored from the texts, but hotlinked images turned into Lightboxes
were still extracted.
2024-06-19 14:25:29 +03:00
Martin Brennan
83361b2fc5
FEATURE: Introduce site settings which require confirmation (#27315)
Many site settings can be distructive or have huge side-effects
for a site that the admin may not be aware of when changing it.

This commit introduces a `requires_confirmation` attribute that
can be added to any site setting. When it is true, a confirmation
dialog will open if that setting is changed in the admin UI,
optionally with a custom message that is defined in client.en.yml.

If the admin does not confirm, we reset the setting to its previous
clean value and do not save the new value.
2024-06-19 16:01:24 +10:00
Krzysztof Kotlarek
cc4c199680
FEATURE: optional 2FA enforcement (#27506)
A new admin setting called `enforce_second_factor_on_external_auth`. It allows users to authenticate using external providers even when 2FA is forced with `enforce_second_factor` site setting.
2024-06-19 09:32:30 +10:00
Jarek Radosz
5cb84f8dcf
DEV: Revert rails 7.1 upgrade (#27522)
* Revert "FIX: Set `override_level` on Logster loggers (#27519)"

This reverts commit c1b0488c54.

* Revert "DEV: Make parameters optional to all FakeLogger methods"

This reverts commit 3318dad7b4.

* Revert "FIX: Remove references to `Rails.logger.chained`"

This reverts commit f595d599dd.

* Revert "DEV: Upgrade Rails to 7.1"

This reverts commit 081b00391e.
2024-06-18 23:48:30 +02:00
Loïc Guitaut
f595d599dd FIX: Remove references to Rails.logger.chained
`Rails.logger.chained` was provided by Logster before Rails 7.1
introduced their broadcast logger. Now all the loggers are added to
`Rails.logger.broadcasts`.

Some code in our initializers was still using `chained` instead of
`broadcasts`.
2024-06-18 17:46:40 +02:00
Loïc Guitaut
081b00391e DEV: Upgrade Rails to 7.1 2024-06-18 15:58:05 +02:00
Loïc Guitaut
2a22a3b51d FIX: Treat corrupt cache as cache miss
Currently when a cache entry is corrupt, we log the event without doing
anything else. It means the cache is still corrupt, and the proper value
isn’t computed again.

Normally, it’s very rare the cache becomes corrupt, but it can happen
when upgrading Rails for example and the cache format changes. This is
normally handled automatically by Rails but since we’re using a custom
cache class, we have to do it ourselves.

This patch takes the same approach the Rails team did, when a cache
entry is corrupt, we treat it as a miss, recomputing the proper value
and caching it in the new format.
2024-06-18 14:47:33 +02:00
Régis Hanol
53b3d2f0dc FIX: BBCode tag parser
Wasn't quite handling the cases where a closing bracket `]` was used in the value of one of the attributes.

```markdown
[chat quote=user channel="[broken]"]
```

Would not be correctly parsed because we would _greedily_ use the first `]` as the end of the tag even though it might be a valid character when inside proper quotes.

c39a4de139/app/assets/javascripts/discourse-markdown-it/src/features/bbcode-block.js (L62)

Re-wrote the `parseBBCodeTag` to properly handle the following cases

- A closing tag (aka `[/name]`) which are easy since they don't have any attributes
- An old `[quote=...]` format we used that doesn't uses quotes but still has various attributes of the form `key:value`
- All three valid BBCode opening tag formats we support
  - `[name]` without any attributes
  - `[name=foo]` with a default value
  - `[name foo=bar]` with some attributes

Ended up having to fix/rewrite the few bbcode rules that were using the `parseBBCodeTag` function, namely `d-wrap` and `discourse-local-dates`.

While working on this, I think I also found a way to get rid the of shims we had in place so that plugins could use the `parseBBCodeTag` function.

Reference - https://meta.discourse.org/t/having-a-right-bracket-in-a-channel-name-breaks-all-quotes-from-that-channel/308439
2024-06-18 10:47:18 +02:00
Kelv
2393234be5
DEV: remove legacy CSP implementation to make strict-dynamic only accepted behaviour (#27486)
* DEV: remove legacy CSP implementation that allowed for non-strict-dynamic behaviour
2024-06-18 16:40:53 +08:00
Alan Guo Xiang Tan
006169f782
DEV: Remove hostname from DiscourseLogstashLogger output (#27485)
This is a duplicate of the `host` field which means we are bloating the
logs unnecessarily.

Just remove without depreciation for now but we are open to properly
depreciating it if others depend on this field.
2024-06-18 07:24:42 +08:00
Daniel Waterworth
0a881a59d3
DEV: Don't enforce per ip rate limits for admin api requests (#27500) 2024-06-17 13:21:11 -05:00
Renato Atilio
49fdccbb1d
FIX: restrict a href protocols on form template description (#27472) 2024-06-14 11:39:43 -03:00
Osama Sayegh
4aea12fdcb
DEV: Allow fetching specific site settings and introduce a service for updating site settings (#27481)
This commit adds ability to fetch a subset of site settings from the `/admin/site_settings` endpoint so that it can be used in all places where the client app needs access to a subset of the site settings.

Additionally, this commit also introduces a new service class called `UpdateSiteSetting` that encapsulates all the logic that surrounds updating a site setting so that it can be used to update site setting(s) anywhere in the backend. This service comes in handy with, for example, the controller for the flags admin config area which may need to update some site settings related to flags.

Internal topic: t/130713.
2024-06-14 13:07:27 +03:00
Gabriel Grubba
93c2ae585c
FEATURE: Add tag_group option in /filter (#27427)
* FEATURE: Add tag_group option in `/filter`

* DEV: Update tag_group_filter in `/filter` to use SQL

* DEV: Add guardian to `/filter` by tag_group

* DEV: remove unused code

* DEV: Update tag_group filter implementation

* DEV: Add tests to tag_group filter
2024-06-13 14:03:49 -03:00
Jan Cernik
d7ece61252
UX: Rename FAQ to Guidelines (#27451) 2024-06-13 12:09:53 -03:00
Loïc Guitaut
c790c57415 DEV: Show parameters on a service contract failure
Now, when calling `StepsInspector#error` on a contract failure, the
output will contain the parameters provided to the contract.
2024-06-13 12:19:26 +02:00
Brendan Bell
0a2926ae03
DEV: Introduce users:exists rake task (#27163)
Checks if a user exists for given email address
2024-06-13 08:04:02 +01:00
Alan Guo Xiang Tan
ae1d8c50da
DEV: Add ability to log a warning when Sidekiq job runs for too long (#27446)
This commits introduces the `sidekiq_report_long_running_jobs_minutes`
global setting which allows a site administrator to log a warning in the
Rails log when a Sidekiq job has been running for too long.

The warning is logged with the backtrace of the thread that is
processing the Sidekiq job to make it easier to figure out what a
sidekiq job is stuck on.
2024-06-13 14:24:44 +08:00
Martin Brennan
e94ab11477
DEV: Allow for setting a message with SiteSetting.set_and_log (#27447)
When we turn on settings automatically for customers,
we sometimes use `.set_and_log` which will make a staff
action log for the site setting change. This is fine, but
there is no context for customers.

This change allows setting a message with `.set_and_log`, which
will be stored in the `details` column of the staff action log
created, which will show up on `/admin/logs/staff_action_logs`

---------

Co-authored-by: Kelv <kelv@discourse.org>
2024-06-13 14:59:49 +10:00
Ted Johansson
a5df029be3
FIX: Email::Sender expects type to be a string (#27463)
In #26642 we introduced a change that re-attaches securely uploaded images in the digest e-mail. However, this change assumed that the type argument to the Email::Sender constructor would be a symbol, but when it is coming from the UserEmail job it is a string. This PR fixes that.
2024-06-13 11:53:25 +08:00
Alan Guo Xiang Tan
1f4aa343d8
DEV: Move spec file to right folder (#27449)
Spec file is not being run because it was in the wrong folder.
2024-06-13 06:38:03 +08:00
Sam
dc8249c08a
FEATURE: align with /filter and allow multiple category search (#27440)
This introduces the syntax of

`category:a,b,c` which will search across multiple categories.

Previously there was no way to allow search across a wide selection of
categories.
2024-06-12 16:06:04 +10:00
Alan Guo Xiang Tan
be4f1e3350
PERF: Cache hostname in DiscourseLogstashLogger (#27442)
Logging events happen all the time. No need for us to keep running code
to fetch the hostname when it doesn't change between deploys.
2024-06-12 13:34:07 +08:00
Alan Guo Xiang Tan
781a5f60ea
DEV: Move RSpec test file into test folder (#27437)
Test was not being run because it was in the wrong folder
2024-06-12 06:58:44 +08:00
Arkshine
29460e1422 DEV: Provide a safe agent in check_crawler_limits() 2024-06-11 14:02:46 +02:00
Arkshine
30ae313dec DEV: Apply suggested changes 2024-06-11 14:02:46 +02:00
Arkshine
1fffb236b2 FIX: crawler requests exceptions for non UTF-8 user agents with invalid bytes 2024-06-11 14:02:46 +02:00
Alan Guo Xiang Tan
9fdcdcf58d
DEV: Log error encountered when reopening sidekiq logs (#27411)
We are seeing the following error in our logs when Sidekiq is sent a
`USR1` signal in production when logrotate happens:

```
log writing failed. stream closed in another thread
Error encountered while starting Sidekiq: can't be called from trap context\n/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/unicorn-6.1.0/lib/unicorn/util.rb:71:in `reopen'
```

I'm not quite sure where the error is triggered from so I'm improving
the way we log errors.
2024-06-11 12:29:48 +08:00
Martin Brennan
10b9a32abb
FIX: Message for bulk closing topics silently (#27400)
We were using `autoclose` as the topic status update
when silently closing topics using the bulk
actions (introduced in 0464ddcd9b).

However, this resulted in a message like this showing in
the topic as a small moderator post:

> This topic was automatically closed after X days.

This is not accurate, the topic was bulk closed by someone.
Instead, we can use `closed` as the status, and a more accurate

> Closed on DATE

message is used. `TopicStatusUpdater` needed an additional
option to keep the same "fake read" behaviour as autoclose
so we can keep the same functionality for silently closing
topics in bulk actions.
2024-06-11 09:36:54 +10:00
David Taylor
565c753dd2
DEV: @babel/plugin-proposal-decorators -> decorator-transforms (#27260)
decorator-transforms (https://github.com/ef4/decorator-transforms) is a modern replacement for babel's plugin-proposal-decorators. It provides a decorator implementation using modern browser features, without needing to enable babel's full suite of class feature transformations. This improves the developer experience and performance.

In local testing with Google's 'tachometer' tool, this reduces Discourse's 'init-to-render' time by around 3-4% (230ms -> 222ms).

It reduces our initial gzip'd JS payloads by 3.2% (2.43MB -> 2.35MB), or 7.5% (14.5MB -> 13.4MB) uncompressed.

This was previously reverted in 97847f6. This version includes a babel transformation which works around the bug in Safari <= 15.

For Cloudflare compatibility issues, check https://meta.discourse.org/t/311390
2024-06-10 15:51:48 +01:00
Gerhard Schlager
7bdf47b864 FIX: HtmlToMarkdown should keep HTML entities for <, > and & within HTML elements
Not all HTML elements are converted into Markdown. Some are kept as HTML.
Without this fix XML/HTML entities that are formatted as text instead of code are swallowed by Discourse.
This also fixes quotes in the `title` attribute of the `<abbr>` tag.
2024-06-10 16:03:30 +02:00
Gerhard Schlager
3c9d61d302 FIX: HtmlToMarkdown didn't keep text from within <center> tag
It should ignore the `<center>` tag, but keep the text from within the element.
2024-06-10 16:03:30 +02:00
Gerhard Schlager
b01905c724 FIX: HtmlToMarkdown didn't support tfoot in tables 2024-06-10 16:03:30 +02:00
Gerhard Schlager
52e81582b4 FEATURE: Use basic HTML table if it can't be converted to Markdown
Previously `HtmlToMarkdown` always converted HTML tables into Markdown tables. That lead to some badly formatted Markdown tables, e.g. when the table contained `rowspan` or `colspan`. This solves the issue by using very basic HTML tables in those cases.
2024-06-10 16:03:30 +02:00
Gerhard Schlager
b8f2cbf41c DEV: Add additional_allowed_tags to HtmlToMarkdown
Import script often use subclasses of `HtmlToMarkdown` and might need to allow additional tags that can be used within the custom class.
2024-06-10 16:03:30 +02:00
Jarek Radosz
42a529f9ae
PERF: Avoid excessive object creations in watched words (#27354)
Inline the helper functions, avoid creating and then immediately destructuring arrays, use complete strings instead of string interpolation, Map instead of a pojo.
2024-06-10 14:44:31 +02:00
JimmyJammyDodger
712f726764
DEV: Remove non-official plugins from metadata (#27050)
Removes BCC, Shared Edits, and Linkedin-auth from the official plugin list. 

Also corrects some alphabetising
2024-06-10 10:40:36 +01:00
Alan Guo Xiang Tan
8cf4ed5f88
DEV: Introduce hidden s3_inventory_bucket site setting (#27304)
This commit introduces a hidden `s3_inventory_bucket` site setting which
replaces the `enable_s3_inventory` and `s3_configure_inventory_policy`
site setting.

The reason `enable_s3_inventory` and `s3_configure_inventory_policy`
site settings are removed is because this feature has technically been
broken since it was introduced. When the `enable_s3_inventory` feature
is turned on, the app will because configure a daily inventory policy for the
`s3_upload_bucket` bucket and store the inventories under a prefix in
the bucket. The problem here is that once the inventories are created,
there is nothing cleaning up all these inventories so whoever that has
enabled this feature would have been paying the cost of storing a whole
bunch of inventory files which are never used. Given that we have not
received any complains about inventory files inflating S3 storage costs,
we think that it is very likely that this feature is no longer being
used and we are looking to drop support for this feature in the not too
distance future.

For now, we will still support a hidden `s3_inventory_bucket` site
setting which site administrators can configure via the
`DISCOURSE_S3_INVENTORY_BUCKET` env.
2024-06-10 13:16:00 +08:00
Gerhard Schlager
1fbc1cd326
DEV: Remove duplicate definition of RTL locales and add Uyghur to the list (#27387) 2024-06-08 21:24:39 +02:00
Jarek Radosz
b618d08d36
DEV: Search for -gnu gem variants (#27347)
> Platform names with a *-linux suffix are aliases for *-linux-gnu

Should fix the ffi upgrade issue
2024-06-05 14:58:02 +02:00
Krzysztof Kotlarek
aa88b07640
FEATURE: the ability to change the order of flags (#27269)
Continued work on moderate flags UI.
In this PR admins are allowed to change the order of flags. The notify user flag is always on top but all other flags can be moved.
2024-06-05 13:27:06 +10:00
Jarek Radosz
0b7563b804
DEV: Fix the order of operations in themes-frontend (#27317)
Previously "themes frontend" CI job would:

1. pull compatible versions of themes that happened to be in the base image
2. clone all official themes (overriding the compatible versions from 1.)
3. run tests
2024-06-04 15:16:44 +02:00
Mark VanLandingham
d42a1c8885
DEV: Pass recipient email address to message_builder modifiers (#27308) 2024-06-04 08:00:30 -05:00
Alan Guo Xiang Tan
e97ef7e9af
FEATURE: Allow site admin to mark a user's password as expired (#27314)
This commit adds the ability for site administrators to mark users'
passwords as expired. Note that this commit does not add any client side
interface to mark a user's password as expired.

The following changes are introduced in this commit:

1. Adds a `user_passwords` table and `UserPassword` model. While the
   `user_passwords` table is currently used to only store expired
   passwords, it will be used in the future to store a user's current
   password as well.

2. Adds a `UserPasswordExpirer.expire_user_password` method which can
   be used from the Rails console to mark a user's password as expired.

3. Updates `SessionsController#create` to check that the user's current
   password has not been marked as expired after confirming the
   password. If the password is determined to be expired based on the
   existence of a `UserPassword` record with the `password_expired_at`
   column set, we will not log the user in and will display a password
   expired notice. A forgot password email is automatically send out to
   the user as well.
2024-06-04 15:42:53 +08:00
Alan Guo Xiang Tan
aec892339e
DEV: Add rspec tests for SignalTrapLogger (#27302)
Follow-up to 23c38cbf11
2024-06-03 13:40:21 +08:00
Alan Guo Xiang Tan
23c38cbf11
DEV: Log Unicorn worker timeout backtraces to Rails.logger (#27257)
This commit introduces the following changes:

1. Introduce the `SignalTrapLogger` singleton which starts a single
   thread that polls a queue to log messages with the specified logger.
   This thread is necessary becasue most loggers cannot be used inside
   the `Signal.trap` context as they rely on mutexes which are not
   allowed within the context.

2. Moves the monkey patch in `freedom_patches/unicorn_http_server_patch.rb` to
   `config/unicorn.config.rb` which is already monkey patching
   `Unicorn::HttpServer`.

3. `Unicorn::HttpServer` will now automatically send a `USR2` signal to
   a unicorn worker 2 seconds before the worker is timed out by the
   Unicorn master.

4. When a Unicorn worker receives a `USR2` signal, it will now log only
   the main thread's backtraces to `Rails.logger`. Previously, it was
   `put`ing the backtraces to `STDOUT` which most people wouldn't read.
   Logging it via `Rails.logger` will make the backtraces easily
   accessible via `/logs`.
2024-06-03 12:51:12 +08:00
Alan Guo Xiang Tan
dc55b645b2
DEV: Allow site administrators to mark S3 uploads with a missing status (#27222)
This commit introduces the following changes which allows a site
administrator to mark `Upload` records with the `s3_file_missing`
verification status which will result in the `Upload` record being ignored when
`Discourse.store.list_missing_uploads` is ran on a site where S3 uploads
are enabled and `SiteSetting.enable_s3_inventory` is set to `true`.

1. Introduce `s3_file_missing` to `Upload.verification_statuses`
2. Introduce `Upload.mark_invalid_s3_uploads_as_missing` which updates
   `Upload#verification_status` of all `Upload` records from `invalid_etag` to `s3_file_missing`.
3. Introduce `rake uploads:mark_invalid_s3_uploads_as_missing` Rake task
   which allows a site administrator to change `Upload` records with
`invalid_etag` verification status to the `s3_file_missing`
verificaton_status.
4. Update `S3Inventory` to ignore `Upload` records with the
   `s3_file_missing` verification status.
2024-05-30 08:37:38 +08:00
Blake Erickson
f292e645b9
FEATURE: Show video thumbnail in composer (#27233)
When uploading a video, the composer will now show a thumbnail image in
the composer preview instead of just the video placeholder image.

If `enable_diffhtml_preview` is enabled the video will be rendered in
the composer preview and is playable.
2024-05-29 08:24:29 -06:00
Alan Guo Xiang Tan
f76d143919
DEV: Log backtrace along with the error (#27242)
Just logging the error message is useless if we don't know where the
error message is raised from.
2024-05-29 16:13:23 +08:00
Krzysztof Kotlarek
963b9fd157
FEATURE: admin can disable flags (#27171)
UI for admins to disable system flags.
2024-05-29 14:39:58 +10:00
Alan Guo Xiang Tan
e9c8e182d3
DEV: Use Unicorn logger to log Sidekiq signal handling events (#27239)
This commit updates all Sidekiq signal handling event logs to go through
Unicorn's logger instead of logging to STDOUT. Going through a proper logger
means the log messages are logged in the format which the logger has configured.
This means we get proper timestamp for the log messages.
2024-05-29 11:15:20 +08:00
Martin Brennan
9c85ea5945
DEV: Remove old TODOs for message-id formats (#27196)
Introduced back in 2022 in
e3d495850d,
our new more specific message-id format for inbound and
outbound emails has now been in use for a very long time,
we can remove the support for the old formats:

`topic/:topic_id/:post_id.:random@:host`
`topic/:topic_id@:host`
`topic/:topic_id.:random@:host`
2024-05-28 13:57:09 +10:00
Krzysztof Kotlarek
39902c148f
FIX: move ServiceRunner from Chat to Core (#27219)
In this PR service objects were moved to Core https://github.com/discourse/discourse/pull/26506

However, ServiceRunner should be moved as well. Mostly for CI to run effortlessly without loading plugins.
2024-05-28 13:55:46 +10:00
Loïc Guitaut
2a28cda15c DEV: Update to lastest rubocop-discourse 2024-05-27 18:06:14 +02:00
Osama Sayegh
361992bb74
FIX: Apply crawler rate limits to cached requests (#27174)
This commit moves the logic for crawler rate limits out of the application controller and into the request tracker middleware. The reason for this move is to apply rate limits to all crawler requests instead of just the requests that make it to the application controller. Some requests are served early from the middleware stack without reaching the Rails app for performance reasons (e.g. `AnonymousCache`) which results in crawlers getting 200 responses even though they've reached their limits and should be getting 429 responses.

Internal topic: t/128810.
2024-05-27 16:26:35 +03:00
Juan David Martínez Cubillos
7992d7a65a
DEV: Refactor to add Custom emoji sanitization (#24368)
Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
2024-05-27 15:24:55 +02:00
Discourse Translator Bot
9d8044a2ee
FEATURE: Add Uyghur language (#27183)
Co-authored-by: Gerhard Schlager <gerhard.schlager@discourse.org>
2024-05-27 09:58:18 +02:00
Gerhard Schlager
9aede9c8d8
FIX: "Replace Text" didn't list "Welcome Topic" when "General" category is missing (#27182)
Replacing the text of seeded topics doesn't require categories to exist, so this change removes that requirement for updates.
2024-05-27 09:57:30 +02:00
Alan Guo Xiang Tan
6cafe59c76
DEV: Add DISCOURSE_DUMP_BACKTRACES_ON_UNICORN_WORKER_TIMEOUT env (#27199)
This commit adds a `DISCOURSE_DUMP_BACKTRACES_ON_UNICORN_WORKER_TIMEOUT`
environment that will allow us to dump all backtraces for all threads of
a Unicorn worker 2 seconds before it times out. In development,
backtraces are dumped to `STDOUT` and in production we will dump it to
`unicorn.stdout.log`.

We want to dump all the backtraces to make it easier to identify the
cause of a Unicorn worker timing out.
2024-05-27 12:20:38 +08:00
Ted Johansson
69205cb1e5
DEV: Catch missing translations during test runs (#26258)
This configuration makes it so that a missing translation will raise an error during test execution. Better discover there than after deploy.
2024-05-24 22:15:53 +08:00
Alan Guo Xiang Tan
df16ab0758
FIX: S3Inventory to ignore files older than last backup restore date (#27166)
This commit updates `S3Inventory#files` to ignore S3 inventory files
which have a `last_modified` timestamp which are not at least 2 days
older than `BackupMetadata.last_restore_date` timestamp.

This check was previously only in `Jobs::EnsureS3UploadsExistence` but
`S3Inventory` can also be used via Rake tasks so this protection needs
to be in `S3Inventory` and not in the scheduled job.
2024-05-24 10:54:06 +08:00
Penar Musaraj
f9192835a7
FIX: Do not reset sidebar defaults in wizard (#27156)
We removed the option to enable/disable the sidebar in the wizard in
https://github.com/discourse/discourse/pull/26926, but these lines of
code were resetting it quietly to `header_dropdown`.
2024-05-23 16:45:47 -04:00
Jan Cernik
1ed1a1f96a
FIX: Add onebox support for x.com (#27140) 2024-05-23 10:25:42 -03:00
Krzysztof Kotlarek
cfbbfd177c
DEV: move post flags into database (#27125)
This is preparation for a feature that will allow admins to define their custom flags. Current behaviour should stay untouched.
2024-05-23 12:19:07 +10:00
marstall
d75339af76
DEV: let reply_by_email, visit_link_to_respond email strings be modified by plugins (#27133)
* DEV: allow reply_by_email, visit_link_to_respond strings to be modified by plugins

* DEV: separate visit_link_to_respond and reply_by_email modifiers out
2024-05-22 15:33:06 -04:00
Amanda Alves Branquinho
b0d95c8c78
FEATURE: Add bulk action to bookmark (#26856)
This PR aims to add bulk actions to the user's bookmarks.

After this feature, all users should be able to select multiple bookmarks and perform the actions of "deleting" or "clear reminders"
2024-05-22 12:50:21 -03:00
Régis Hanol
8f7a3e5b29 FIX: subfolder absolute links in summaries
This fixes the `PrettyText.make_all_links_absolute` to better handle subfolder.

In subfolder, when given the cooked version of a post, links to mentions includes the `Discourse.base_path` prefix. Adding the `Discourse.base_url` was doubling the `Discourse.base_path`.

The issue was hidden behind the specs which was stubbing `Discourse.base_url` instead of relying on `Discourse.base_path`.

This fixes both the "algorithm" used in `PrettyText.make_all_links_absolute` to better handle this case and correct the specs to properly handle subfolder cases.

There are lots of changes in the specs due to a refactoring to use squiggly heredoc strings for easier reading and less escaping.
2024-05-22 15:38:18 +02:00
Krzysztof Kotlarek
40d65dddf8
Revert "DEV: move post flags into database (#26951)" (#27102)
This reverts commit 7aff9806eb.
2024-05-21 16:21:07 +10:00
Krzysztof Kotlarek
7aff9806eb
DEV: move post flags into database (#26951)
This is preparation for a feature that will allow admins to define their custom flags. Current behaviour should stay untouched.
2024-05-21 13:15:32 +10:00
Alan Guo Xiang Tan
34c527d694
DEV: Pull compatible themes in tests workflow (#27093)
This commit adds a step in our tests workflow on Github actions to update the themes to
use the compatible version when not running aginast the `main` branch.
This is to ensure that we are not running
the tests for themes against an incompatible version of Discourse.
2024-05-21 10:38:41 +08:00
Ted Johansson
32aaf2e8d3
DEV: Remove deprecated AuthProvider#enabled_setting= (#27081)
AuthProvider#enabled_setting=, used primarily by plugins, has been deprecated since version 2.9, in favour of Authenticator#enabled?. This PR confirms we are seeing no more usage and removes the method.
2024-05-20 18:10:15 +08:00
Régis Hanol
b908abe35a
FIX: keep topic.word_count in sync (#27065)
Whenever one creates, updates, or deletes a post, we should keep the `topic.word_count` counter in sync.

Context - https://meta.discourse.org/t/-/308062
2024-05-17 17:05:49 +02:00
Jan Cernik
af6759f5e2
DEV: Use the correct open command in the version bump rake task (#27046) 2024-05-16 10:18:33 -03:00
Martin Brennan
d964709644
DEV: Add more _map extensions for list site settings (#27045)
Following on from eea74e0e32,
this commit adds the automatic _map splitting shorthand
for emoji_list, tag_list site settings.
2024-05-16 13:43:10 +10:00
Alan Guo Xiang Tan
e31cf66f11
FIX: FinalDestination#get forwarding Authorization header on redirects (#27043)
This commits updates `FinalDestination#get` to not forward
`Authorization` header on redirects since most HTTP clients I tested like
curl and wget does not it.

This also fixes a recent problem in `DiscourseIpInfo.mmdb_download`
where we will fail to download the databases when both `GlobalSetting.maxmind_account_id` and
`GlobalSetting.maxmind_license_key` has been set. The failure is due to
the bug above where the redirected URL given by MaxMind does not accept
an `Authorization` header.
2024-05-16 08:37:34 +08:00
Jan Cernik
fb63ddd7d2
Bump version to v3.3.0.beta3-dev 2024-05-15 12:24:11 -03:00
Jan Cernik
c723c126f2
Bump version to v3.3.0.beta2 2024-05-15 12:24:11 -03:00
Alan Guo Xiang Tan
2134ca9031
PERF: Optimise query for excluding topics in certain categories in TopicsFilter (#27027)
This commit optimises the database query generated by
`TopicsFilter#filter_categories` when the `-category:*` filter is used.
Previously, the method will add the `topics.category_id NOT IN
(<category ids to be excluded>)` filter to the resulting query. However,
we noticed that the performance of the query degrades as the number of
rows in the `topics` table grow and when the number of category ids to be
excluded is large.

Sample of query we ran on a large database in production to demonstrate
the improvement:

Before:

```
SELECT topics.id FROM topics WHERE topics.category_id NOT IN (83, 136, 149, 143, 153, 165, 161, 123, 155, 163, 144, 134, 69, 135, 158, 141, 151, 160, 131, 133, 89, 104, 150, 147, 132, 145, 108, 146, 122, 100, 128, 154, 95, 102, 140, 139, 88, 91, 87) ORDER BY topics.id DESC LIMIT 5;

                                                                                                       QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=27795.34..27795.34 rows=1 width=4) (actual time=29.317..30.165 rows=5 loops=1)
   ->  Sort  (cost=27795.34..27795.34 rows=1 width=4) (actual time=29.316..30.163 rows=5 loops=1)
         Sort Key: id DESC
         Sort Method: top-N heapsort  Memory: 25kB
         ->  Gather  (cost=1000.10..27795.33 rows=1 width=4) (actual time=0.187..26.132 rows=73478 loops=1)
               Workers Planned: 2
               Workers Launched: 2
               ->  Parallel Seq Scan on topics  (cost=0.10..26795.23 rows=1 width=4) (actual time=0.013..22.252 rows=24493 loops=3)
                     Filter: (category_id <> ALL ('{83,136,149,143,153,165,161,123,155,163,144,134,69,135,158,141,151,160,131,133,89,104,150,147,132,145,108,146,122,100,128,154,95,102,140,139,88,91,87}'::integer[]))
                     Rows Removed by Filter: 77276
 Planning Time: 0.140 ms
 Execution Time: 30.181 ms
```

After:

```
SELECT topics.id FROM topics WHERE NOT EXISTS (
  SELECT 1
  FROM unnest(array[83, 136, 149, 143, 153, 165, 161, 123, 155, 163, 144, 134, 69, 135, 158, 141, 151, 160, 131, 133, 89, 104, 150, 147, 132, 145, 108, 146, 122, 100, 128, 154, 95, 102, 140, 139, 88, 91, 87]) AS excluded_categories(category_id)
  WHERE topics.category_id IS NULL OR excluded_categories.category_id = topics.category_id
) ORDER BY topics.id DESC LIMIT 5 ;

                                                                        QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.42..13.52 rows=5 width=4) (actual time=0.028..0.110 rows=5 loops=1)
   ->  Nested Loop Anti Join  (cost=0.42..179929.62 rows=68715 width=4) (actual time=0.027..0.109 rows=5 loops=1)
         Join Filter: ((topics.category_id IS NULL) OR (excluded_categories.category_id = topics.category_id))
         Rows Removed by Join Filter: 239
         ->  Index Scan Backward using forum_threads_pkey on topics  (cost=0.42..108925.71 rows=305301 width=8) (actual time=0.012..0.062 rows=44 loops=1)
         ->  Function Scan on unnest excluded_categories  (cost=0.00..0.39 rows=39 width=4) (actual time=0.000..0.001 rows=6 loops=44)
 Planning Time: 0.126 ms
 Execution Time: 0.124 ms
(8 rows)
```
2024-05-15 11:03:07 +08:00
dsims
e6e3eaf472
FIX: avoid error from missing meta tags (#26927) 2024-05-14 11:41:53 -04:00
marstall
6df2f94bbc
DEV add modifiers to message_builder so plugins can customize subject/body/html (#26867) 2024-05-13 14:59:15 -04:00
Bianca Nenciu
ebc1763aa5
FIX: Change request method for categories/search (#26976)
This commit changes request method for "categories/search" from GET to
POST to make sure that long filters can be passed to the server. For
example, category selectors with many categories are setting the full
list of selected category IDs to ensure these are filtered out from the
list of choices. This can result in a long URL that exceeds the maximum
length.
2024-05-13 14:37:17 +03:00
David Taylor
97847f6cd8
Revert "DEV: @babel/plugin-proposal-decorators -> decorator-transforms (#25290)" (#26971)
This reverts commit 0f4520867b.

This has led to two problems:

1. An incompatibility with Cloudflare's "auto minify" feature. They've deprecated this feature because of incompatibility with modern JS syntax. But unfortunately it will remain enabled on existing properties until 2024-08-05.

2. Discourse fails to boot in Safari 15. This is strange, because Safari does support all the required features in our production JS bundles. Even more strangely, things start working as soon as you open the developer tools. That suggests the cause could be a Safari bug rather than a simple incompatibility.

Reverting while we work out a path forward on both those issues.
2024-05-10 12:48:16 +01:00
Alan Guo Xiang Tan
7079698cdf
FIX: Use MaxMind supplied permalinks to download MaxMind databases (#26847)
This commit switches `DiscourseIpInfo.mmdb_download` to use the
permalinks supplied by MaxMind to download the MaxMind databases as
specified in
https://dev.maxmind.com/geoip/updating-databases#directly-downloading-databases
which states:

```
To directly download databases, follow these steps:

1. In the "Download Links" column, click "Get Permalink(s)" for the desired database.
2. Copy the permalink(s) provided in the modal window.
3. Provide your account ID and your license key using Basic Authentication to authenticate.
```

Previously we are downloading from `https://download.maxmind.com/app/geoip_download` but this is not
documented anyway on MaxMind's docs so this URL can in theory break
in the future without warning. Therefore, we are taking a proactive
approach to download the databases from MaxMind the recommended way
instead of relying on a hidden URL. This old way of downloading the
databases with only a license key will be deprecated in 3.3 and be
removed in 3.4.
2024-05-09 15:11:56 +08:00
Alan Guo Xiang Tan
c8da2a33e8
FIX: Attempt to onebox even if response body exceeds max_download_kb (#26929)
In 95a82d608d, we lowered the default for
`Onebox.options.max_download_kb` from 10mb to 2mb for security hardening
purposes. However, this resulted in multiple bug reports where seemingly
nomral URLs stopped being oneboxed. It turns out that lowering
`Onebox.options.max_download_kb` resulted in `Onebox::Helpers::DownloadTooLarge` being raised
more often for more URLs  in `Onebox::Helpers.fetch_response` which
`Onebox::Helpers.fetch_html_doc` relies on. When
`Onebox::Helpers::DownloadTooLarge` is raised in
`Onebox::Helpers.fetch_response`, we throw away whatever response body
which we have already downloaded at that point. This is not ideal
because Nokogiri can parse incomplete HTML documents and there is a
really high chance that the incomplete HTML document still contains the
information which we need for oneboxing.

Therefore, this commit updates `Onebox::Helpers.fetch_html_doc` to not
throw away the response body when the size of the response body exceeds
`Onebox.options.max_download_size`. Instead, we just take whatever
response which we have and get Nokogiri to parse it.
2024-05-09 07:00:34 +08:00
David Taylor
ece0150cb7
FIX: Ensure RequestTracker handles bubbled exceptions correctly (#26940)
This can happen for various reasons including rate limiting and middleware bugs. This should resolve the warning we're seeing in the logs

```
RequestTracker.get_data failed : NoMethodError : undefined method `[]' for nil:NilClass
```
2024-05-08 16:08:39 +01:00
David Taylor
0f4520867b
DEV: @babel/plugin-proposal-decorators -> decorator-transforms (#25290)
decorator-transforms (https://github.com/ef4/decorator-transforms) is a modern replacement for babel's plugin-proposal-decorators. It provides a decorator implementation using modern browser features, without needing to enable babel's full suite of class feature transformations. This improves the developer experience and performance.

In local testing with Google's 'tachometer' tool, this reduces Discourse's 'init-to-render' time by around 3-4% (230ms -> 222ms).

It reduces our initial gzip'd JS payloads by 3.2% (2.43MB -> 2.35MB), or 7.5% (14.5MB -> 13.4MB) uncompressed.
2024-05-08 10:40:51 +01:00
Martin Brennan
ce2388e40b
FEATURE: Remove "Enable Sidebar" step from setup wizard (#26926)
This keeps coming up in user testing as something
we want to get rid of. The `navigation_menu` setting
has been set to sidebar by default for some time now,
and we are rolling out admin sidebar widely. It just
doesn't make sense to let people turn this off in
the first step of the wizard -- we _want_ people to
use the sidebar.
2024-05-08 10:00:40 +10:00
Régis Hanol
12cba2ce24 PERF: bail out of expensive post validations
Whenever a post already failed "lightweight" validations, we skip all the expensive validations (that cooks the post or run SQL queries) so that we reply as soon as possible.

Also skip validating polls when there's no "[/poll]" in the raw.

Internal ref - t/115890
2024-05-07 18:56:16 +02:00
misaka4e21
ba357dd6cc FIX: ignore SVGs when regenerating missing optimized images.
When running `rake uploads:regenerate_missing_optimized`,
a `Discourse::InvalidAccess` will be raised if an SVG
file is being processed as `OptimizedImage.prepend_decoder!`
doesn't support the svg extension. This commit simply copies
the original SVG file as the thumbnail, just like currently
`OptimizedImage.create_for` does.
2024-05-07 14:39:04 +02:00
Alan Guo Xiang Tan
0b947b6aab
DEV: Improve code comment about when ignored columns can be removed (#26894)
Ignored columns can only be dropped when its associated post-deploy
migration has been promoted to a regular migration. This is so because
Discourse doesn't rely on a schema file system to setup a brand new
database and thus the column information will be loaded by the
application first before the post-deploy migration runs.
2024-05-07 11:06:31 +08:00
Gerhard Schlager
1872047053
DEV: Uploads import script can download files (#26816)
Uploads import script can download files
2024-05-04 22:48:16 +02:00
David Taylor
f230767722
FIX: Serialization of staff_writes_only (#26866) 2024-05-03 14:36:13 -04:00
Daniel Waterworth
a9ca35b671
DEV: Use safer SQL functions for string queries when looking for tags (#26838) 2024-05-02 10:13:45 -05:00
David Taylor
9db5eafb15
PERF: Improve production JS build in low-memory environments (#26849)
- Use 'cheap-source-map' webpack config on low-memory machines

  This results in worse quality sourcemaps in browser dev tools, but it significantly reduces memory use in our webpack build. In approximate local testing it drops from 1100mb to 590mb. This should make the rebuild process on low-memory machines much faster and less likely to trigger OOM errors.

  In development, and on higher-memory machines, the higher-quality 'source-map' option is maintained.

- Disable Webpack's built-in `minimize` feature. Embroider already applies Terser after the webpack build is complete. There is no need to double-minimize the output.

- Update ember-cli-progress-ci to print to stderr instead of stdout. For some reason, pups (used by discourse_docker) buffers the stdout of commands and only prints when they are finished. stderr does not have this same limitation, so switching will mean sysadmins can see the progress of the ember build in real-time.

Given the number of variables it's hard to promise exact numbers. But, in my tests on a DO droplet with 1GB RAM (+2GB swap), this reduced the `ember build` portion of a `./launcher rebuild app` from ~50 minutes to ~15 minutes.
2024-05-02 11:43:59 +01:00
Daniel Waterworth
9f9c7f0a23
FIX: Handle tags with underscores correctly (#26839) 2024-05-01 20:01:39 -05:00
Alan Guo Xiang Tan
a6624af66e
DEV: Add isValidUrl helper function to theme migrations (#26817)
This commit adds a `isValidUrl` helper function to the context in
which theme migrations are ran in. This helper function is to make it
easier for theme developers to check if a string is a valid URL or path
when writing theme migrations. This can be helpful in cases when
migrating a string based setting to `type: objects` which contain `type:
string` properties with URL validations enabled.

This commit also introduces the `UrlHelper.is_valid_url?` method
which actually checks that the URL string is of the valid format instead of
only checking if the URL string is parseable which is what `UrlHelper.relaxed_parse` does
and is not sufficient for our needs.
2024-04-30 16:45:07 +08:00
Régis Hanol
bfc0f3f4cd FIX: prevent duplicate attachments in incoming emails - take 2
This is a follow up of 5fcb7c262d

It was missing the case where secure uploads is enabled, which creates a copy of the upload no matter what.

So this checks for the original_sha1 of the uploads as well when checking for duplicates.
2024-04-30 08:15:07 +02:00
Jan Cernik
9fb888923d
FIX: Do not show hidden posts in search results (#26800) 2024-04-29 12:32:02 -03:00
Régis Hanol
f7a1272fa4 DEV: cleanup custom filters to prevent leaks
Ensures we clean up any custom filters added in the specs to prevent any leaks when running the specs.

Follow up to https://github.com/discourse/discourse/pull/26770#discussion_r1582464760
2024-04-29 16:11:12 +02:00
David Taylor
620f76cec1
DEV: Log original exception/backtrace for RequestTracker errors (#26802) 2024-04-29 09:05:32 +01:00
Martin Brennan
edec941a87
FIX: Better tracking of topic visibility changes (#26709)
This commit introduces a few changes as a result of
customer issues with finding why a topic was relisted.
In one case, if a user edited the OP of a topic that was
unlisted and hidden because of too many flags, the topic
would get relisted by directly changing topic.visible,
instead of going via TopicStatusUpdater.

To improve tracking we:

* Introduce a visibility_reason_id to topic which functions
  in a similar way to hidden_reason_id on post, this column is
  set from the various places we change topic visibility
* Fix Post#unhide! which was directly modifying topic.visible,
  instead we use TopicStatusUpdater which sets visibility_reason_id
  and also makes a small action post
* Show the reason topic visibility changed when hovering the
  unlisted icon in topic status on topic titles
2024-04-29 10:34:46 +10:00
Régis Hanol
803c275bd7 DEV: add support for adding custom status filter
Those can be used in the /filter routes.
2024-04-26 14:04:03 +02:00
Osama Sayegh
2215fa0c8e
FIX: Pass values of objects typed settings to theme migrations (#26751)
This commit fixes a bug in theme settings migrations where values of `objects` typed theme settings aren't passed to migrations even when there are overriding values for those settings. What causes this bug is that, when creating the hash that contains all the overridden settings and will be passed to migrations, the values of `objects` typed settings are incorrectly retrieved from the `value` column (which is always nil for `objects` type) instead of `json_value`. `objects` settings are different from all other types in that they store their values in the `json_value` column and they need to be special-cased when retrieving their values.
2024-04-25 16:39:22 +03:00
David Taylor
2f2da72747
FEATURE: Add experimental tracking of 'real browser' pageviews (#26647)
Our 'page_view_crawler' / 'page_view_anon' metrics are based purely on the User Agent sent by clients. This means that 'badly behaved' bots which are imitating real user agents are counted towards 'anon' page views.

This commit introduces a new method of tracking visitors. When an initial HTML request is made, we assume it is a 'non-browser' request (i.e. a bot). Then, once the JS application has booted, we notify the server to count it as a 'browser' request. This reliance on a JavaScript-capable browser matches up more closely to dedicated analytics systems like Google Analytics.

Existing data collection and graphs are unchanged. Data collected via the new technique is available in a new 'experimental' report.
2024-04-25 11:00:01 +01:00
Alan Guo Xiang Tan
35bc27a36d
FIX: themes:update rake task not rolling back transaction on error (#26750)
This commit fixes a bug in the `themes:update` rake task which resulted
in the ActiveRecord transaction not being rolled back when an error was
encountered. The transaction was first introduced in
7f0682f4f2 which changed a `begin..rescue`
block to `transaction do..rescue`. The problem with that change
prevented the transaction from ever rolling back as the code block
looks something like this:

```
transaction do
  begin
    update_theme
  rescue => e
    # surpress error
  end
end
```

From the transaction's point of view now, it will never rollback even if
an error was encountered when updating the remote theme because it will
never see the error.

Instead we should have done something like this if we wanted to surpress
the errors encountered while still ensuring that the transaction is
rolled back.

```
begin
  transaction do
    update_theme
  end
rescue => e
  # surpress error
end
```
2024-04-25 15:19:23 +08:00
Alan Guo Xiang Tan
9b829216b2
DEV: Add site's DB name in themes:update rake task when printing error (#26749)
This is essential for us to determine which site is encountering an
error while updating remote themes. We are also including the theme's id
because themes can have the same name.
2024-04-25 12:41:14 +08:00
David Taylor
dcd994a9f1
DEV: Drop workbox dependency (#26735)
This service-worker caching functionality was disabled by default in 1c58395bca, and the setting to re-enable was marked as experimental. Now we are dropping all the related logic.
2024-04-24 10:19:12 +01:00
David Taylor
bca855f239
FIX: Improve handling of 'PublicExceptions' when bootstrap_error_pages enabled (#26700)
- Run the CSP-nonce-related middlewares on the generated response

- Fix the readonly mode checking to avoid empty strings being passed (the `check_readonly_mode` before_action will not execute in the case of these re-dispatched exceptions)

- Move the BlockRequestsMiddleware cookie-setting to the middleware, so that it is included even for unusual HTML responses like these exceptions
2024-04-24 09:40:13 +01:00
Penar Musaraj
98d400f7b5
DEV: Refactor discover setting reporting (#26706) 2024-04-23 09:52:01 -04:00
Michael Brown
a5ef7b1999 FIX: in EmailSettingsValidator, unset smtp authentication when there's no user and password
net-smtp 0.5.0 bails when authentication is set without username/password

followup-to: 7b8d60dc, 897be759
2024-04-19 14:02:22 -04:00
Ted Johansson
9e31135eca
FEATURE: Allow users to sign in using LinkedIn OpenID Connect (#26281)
LinkedIn has grandfathered its old OAuth2 provider. This can only be used by existing apps. New apps have to use the new OIDC provider.

This PR adds a linkedin_oidc provider to core. This will exist alongside the discourse-linkedin-auth plugin, which will be kept for those still using the deprecated provider.
2024-04-19 18:47:30 +08:00
Jarek Radosz
285acf0b86
DEV: Add missing svg icons to the svg_sprite list (#26674) 2024-04-18 13:01:48 +02:00
Ted Johansson
f3cad5f3a2
FIX: Correctly re-attach allowed images in activity summary e-mail (#26642)
For e-mails, secure uploads redacts all secure images, and later uses the access control post to re-attached allowed ones. We pass the ID of this post through the X-Discourse-Post-Id header. As the name suggests, this assumes there's only ever one access control post. This is not true for activity summary e-mails, as they summarize across posts.

This adds a new header, X-Discourse-Post-Ids, which is used the same way as the old header, but also works for the case where an e-mail is associated with multiple posts.
2024-04-18 10:27:46 +08:00
Krzysztof Kotlarek
98fc614162
FEATURE: mandatory fields for group site setting (#26612)
Automatically add `moderators` and `admins` auto groups to specific site settings.

In the new group-based permissions systems, we just want to check the user’s groups since it more accurately reflects reality

Affected settings:
- tag_topic_allowed_groups
- create_tag_allowed_groups
- send_email_messages_allowed_groups
- personal_message_enabled_groups
- here_mention_allowed_groups
- approve_unless_allowed_groups
- approve_new_topics_unless_allowed_groups
- skip_review_media_groups
- email_in_allowed_groups
- create_topic_allowed_groups
- edit_wiki_post_allowed_groups
- edit_post_allowed_groups
- self_wiki_allowed_groups
- flag_post_allowed_groups
- post_links_allowed_groups
- embedded_media_post_allowed_groups
- profile_background_allowed_groups
- user_card_background_allowed_groups
- invite_allowed_groups
- ignore_allowed_groups
- user_api_key_allowed_groups
2024-04-18 08:53:52 +10:00
Martin Brennan
380e5ca6cb
DEV: Move more service code to core (#26613)
This is to enable :array type attributes for Contract
attributes in services, this is a followup to the move
of services from chat to core here:

cab178a405

Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
2024-04-12 13:14:19 +02:00
Gerhard Schlager
4d045bfc61
DEV: Import script should insert more data into user_stats table (#26551)
This SQL tries to insert as much data as possible into the `user_stats` table by either calculating or by approximating stats based on existing. It also fixes an error in the calculation of `reply_count`which mistakenly contained all posts, not just replies.

This change also disables some steps in the `import:ensure_consistency` rake task by setting the `SKIP_USER_STATS` env variable. Otherwise, the rake task will overwrite the calculated data in the `user_stats` table with inaccurate data. I'm not changing or removing the logic from the rake task yet because other bulk import scripts seem to depend on it.
2024-04-11 14:05:21 +02:00
Martin Brennan
b7a2d29b7b
DEV: Introduce post_should_secure_uploads? plugin modifier (#26508)
This modifier allows plugins to alter the outcome of
`should_secure_uploads?` on a Post record, for cases when
plugins need post-attached uploads to always be secure (or
not secure) in specific scenarios.
2024-04-10 12:02:44 +10:00
Alan Guo Xiang Tan
e2ced85757
DEV: Allow enum typed theme objects property to be optional (#26571)
This commit changes enum typed theme objects property to be optional.
Previously, an enum typed property is always required but we have found
that this might not be ideal so we want to change it.
2024-04-09 11:26:24 +08:00
Martin Brennan
0d0dbd391a
DEV: Rename with_secure_uploads? to should_secure_uploads? on Post (#26549)
This method name is a bit confusing; with_secure_uploads implies
it may return a block or something with the uploads of the post,
and has_secure_uploads implies that it's checking whether the post
is linked to any secure uploads.

should_secure_uploads? communicates the true intent of this method --
which is to say whether uploads attached to this post should be
secure or not.
2024-04-09 13:23:11 +10:00
Alan Guo Xiang Tan
6edadeab27
FIX: DiscourseIpInfo.mmdb_download incorrectly joining URLs (#26545)
This commit changes `DiscourseIpInfo.mmdb_download` to use `File.join`
instead of `URI.join` when `GlobalSetting.maxmind_mirror_url` has been
configured. This is necessary because `URI.join` does not work the way I
expect it to work when I implemented it previously.

`URI.join("http://www.example.com/mirror", "test.tar.gz") results in
`http://www.example.com/test.tar.gz` instead of our expected
`http://www.exmaple.com/mirror/test.tar.gz`. For our simple use case
here, `File.join` is sufficient.
2024-04-08 10:27:40 +08:00
Blake Erickson
8da49b5664
FEATURE: Add message to bulk close topics (#26535)
This will allow you to type a single message when bulk closing topics
that will be applied to every topic, that way they all have the same
message.
2024-04-05 14:56:52 -06:00
Rafael dos Santos Silva
dd83a07550
FEATURE: Hide summarization in PMs (#26532) 2024-04-05 12:12:59 -03:00
Jan Cernik
cab178a405
DEV: Move chat service objects into core (#26506) 2024-04-04 10:57:41 -03:00
Gerhard Schlager
82c62fe44f
DEV: Correctly pluralize error messages (#26469) 2024-04-04 15:02:09 +02:00
David Taylor
0ad4034c58
DEV: Add a note about the version bump rake tasks (#26512) 2024-04-04 11:59:29 +01:00
Gerhard Schlager
689d2754f6
DEV: Bump the version to 3.3.0.beta2-dev (#26499)
This was accidentally omitted during the beta1 release process
2024-04-03 22:08:04 +01:00
Vinoth Kannan
9dc6325821
DEV: add logo URL and locale details to the Discover stats. (#26320)
We will be collecting the logo URL and the site's default locale values along with existing basic details to display the site on the Discourse Discover listing page. It will be included only if the site is opted-in by enabling the "`include_in_discourse_discover`" site setting.

Also, we no longer going to use `about.json` and `site/statistics.json` endpoints retrieve these data. We will be using only the `site/basic-info.json` endpoint.
2024-04-04 00:22:28 +05:30
Osama Sayegh
3d4faf3272
FEATURE: Merge discourse-automation (#26432)
Automation (previously known as discourse-automation) is now a core plugin.
2024-04-03 18:20:43 +03:00
David Taylor
929b4f89d6
DEV: Send proper 'stop' notification in turbo_rspec (#26488)
Doesn't actually seem to be used by any of our formatters, but let's send the proper data anyway for future-proofing. Followup to ff6cb1bc05 and 8098876bfa
2024-04-03 14:00:47 +01:00
David Taylor
ff6cb1bc05
DEV: Fix turbo_rspec formatters by sending real 'Start' notification (#26487) 2024-04-03 13:27:39 +01:00
David Taylor
8098876bfa
DEV: Restore RSpec 'documentation' output, but collapse in CI (#26485)
This reverts commit ef895f1c32 and 57df0d526e, but adds GitHub actions line grouping to the verbose output to reduce scrolling.
2024-04-03 10:54:55 +01:00
David Taylor
1e42e86601
DEV: Restore qunit 'tap' reporter, but collapse verbose output in CI (#26484)
This reverts commit 5adfb299ac, but uses GitHub actions grouping to achieve the same result: less scrolling to see failures.
2024-04-03 10:22:20 +01:00
Krzysztof Kotlarek
ba04fc6a01
FEATURE: ignore manually deactivated users when purging (#26478)
When a user is manually deactivated, they should not be deleted by our background job that purges inactive users.

In addition, site settings keywords should accept an array of keywords.
2024-04-03 14:06:31 +11:00
Penar Musaraj
1eb70973a2
DEV: allow themes to render their own custom homepage (#26291)
This PR adds a theme modifier and route so that custom themes can opt to show their own homepage. See PR description for example usage.
2024-04-02 11:05:08 -04:00
Vinoth Kannan
cd43985de0
UX: rename the word "Upgrade" to "Update" in the UI. (#26396)
In this PR, all references in the UI to the word "`upgrade`" are changed to "`update`". This is to differentiate the update process in self-hosted sites from the plan "upgrade" process in hosted sites.

Follow-up to the PR: https://github.com/discourse/docker_manager/pull/208
2024-04-02 20:04:37 +05:30
Alan Guo Xiang Tan
9182501366
DEV: Introduce maxmind_mirror_url GlobalSetting (#26458)
Why this change?

This allows downloading the MaxMind databases from a mirror in cases
where downloading directly from MaxMind's API endpoint is problematic
due to API limits.
2024-04-02 14:53:53 +08:00
Alan Guo Xiang Tan
972db687fe
FIX: maxmind:refresh does not respect refresh_maxmind_db_during_precompile_days (#26457)
Why this change?

We currently support `GlobalSetting.refresh_maxmind_db_during_precompile_days` which
should cache the maxmind databases on disk for the configured number of
days before it downloads the databases from maxmind again via the API.
This was previously added to help us avoid hitting the API rate limit from maxmind.
However, there was a bug in the `copy_maxmind` when we copied the latest
downloaded database to the cache directory. In particular, `FileUtils.cp` was called with
`preserve: true` which would preserve the modified time of the file
being copied. This is problematic because download the database from
maxmind on 2 April 2024 can give us a file with an mtime of 29 March
2024. If `GlobalSetting.refresh_maxmind_db_during_precompile_days` is
set to `2` for example, the cache will never be used since we will
think that the file has been downloaded for more than 2 days in our
checks.

What is the fix here?

While we want to preserve the owner and group of the file, we do not
want to preserve the modified time and hence we will call
`FileUtils.touch` when copying the file.
2024-04-02 14:25:05 +08:00
Alan Guo Xiang Tan
a84757fd91
FIX: Error not being raised for required typed categories property (#26443)
Why this change?

For a schema like this:

```
schema = {
  name: "section",
  properties: {
    category_property: {
      type: "categories",
      required: true,
    },
  },
}
```

When the value of the property is set to an empty array, we are not
raising an error which we should because the property is marked as
required.
2024-04-01 10:11:40 +08:00
carehabit
11877f3b9c
DEV: remove repetitive words (#26439) 2024-04-01 06:23:21 +08:00
Alan Guo Xiang Tan
a670d6d4af
DEV: Change group type to groups type for theme object schema (#26417)
Why this change?

This is a follow-up to 86b2e3a.

Basically, we want to allow people to select more than 1 group as well.

What does this change do?

1. Change `type: group` to `type: groups` and support `min` and `max`
   validations for `type: groups`.

2. Fix the `<SchemaThemeSetting::Types::Groups>` component to support the
   `min` and `max` validations and switch it to use the `<GroupChooser>` component
   instead of the `<ComboBoxComponent>` component which previously only supported
   selecting a single group.
2024-03-28 22:05:48 +08:00
Ted Johansson
0c875cb4d5
DEV: Make problem check registration more explicit (#26413)
Previously the problem check registry simply looked at the subclasses of ProblemCheck. This was causing some confusion in environments where eager loading is not enabled, as the registry would appear empty as a result of the classes never being referenced (and thus never loaded.)

This PR changes the approach to a more explicit one. I followed other implementations (bookmarkable and hashtag autocomplete.) As a bonus, this now has a neat plugin entry point as well.
2024-03-28 14:00:47 +08:00
Angus McLeod
7dc552c9cc
DEV: Add import_embed_unlisted site setting (#26222) 2024-03-27 08:57:43 -04:00
Osama Sayegh
0b34b4cbe9
DEV: Better error message when user is not allowed to user tags (#26380)
When a topic fails to be created due to the user not having permission to add tags to the topic, the error message that you get is `There was an error tagging the topic` which is very generic and doesn't explain where/what the problem is. This commit adds a clearer error message for this scenario.
2024-03-27 12:57:10 +03:00
Osama Sayegh
72c4709a5a
FIX: Skip tags-related validations when the skip_validations option is present (#26379)
The `TopicCreator` class has a `skip_validations` option that can force-create a topic without performing permission checks or validation rules. However, at the moment it doesn't skip validations that are related to tags, so topics that are created by the system or by some scrip can still fail if they use tags. This commit makes the `TopicCreator` class skip all tags-related checks if the `skip_validations` is specified.

Internal topic: t/124280.
2024-03-27 12:56:21 +03:00
Martin Brennan
8e08a3b31f
DEV: Use caller for plugin_file_from_fixtures (#26387)
Followup 0bbca318f2,
rather than making developers provide the plugin path
name (which may not always be the same depending on
dir names and git cloning etc) we can infer the plugin
dir from the caller in plugin_file_from_fixtures
2024-03-27 14:12:51 +11:00
Alan Guo Xiang Tan
476d91d233
DEV: Change category type to categories type for theme object schema (#26339)
Why this change?

This is a follow-up to 86b2e3aa3e.

Basically, we want to allow people to select more than 1 category as well.

What does this change do?

1. Change `type: category` to `type: categories` and support `min` and `max`
   validations for `type: categories`.

2. Fix the `<SchemaThemeSetting::Types::Categories>` component to support the
   `min` and `max` validations and switch it to use the `<CategorySelector>` component
   instead of the `<CategoryChooser>` component which only supports selecting one category.
2024-03-27 10:54:30 +08:00
Alan Guo Xiang Tan
957b945d25
DEV: Improve output of maxminddb:refresh (#26386)
Why this change?

We want to know when nothing was downloaded because stuff has been
cached on disk
2024-03-27 09:50:49 +08:00
Michael Brown
680f1ff19c FIX: Add content-type header to rate limiter error
It's best to always set a content-type header and one was missing here.
2024-03-26 12:39:42 -04:00
David Taylor
a8d20f92fb
FEATURE: Add page number to page titles for crawlers (#26367)
At the moment, all topic `?page=` views are served with exactly identical page titles. If you search for something which is mentioned many times in the same Discourse topic, this makes for some very hard-to-understand search results! All the result titles are exactly the same, with no indication of why there are multiple results showing.

This commit adds a `- Page #` suffix to the titles in this situation. This lines up with our existing strategy for topic-list pagination.
2024-03-26 15:19:00 +00:00
David Taylor
3329484e2d
FEATURE: Simplify crawler content for non-canonical post URLs (#26324)
When crawlers visit a post-specific URL like `/t/-/{topic-id}/{post-number}`, we use the canonical to direct them to the appropriate crawler-optimised paginated view (e.g. `?page=3`).

However, analysis of google results shows that the post-specific URLs are still being included in the index. Google doesn't tell us exactly why this is happening. However, as a general rule, 'A large portion of the duplicate page's content should be present on the canonical version'.

In our previous implementation, this wasn't 100% true all the time. That's because a request for a post-specific URL would include posts 'surrounding' that post, and won't exactly conform to the page boundaries which are used in the canonical version of the page. Essentially: in some cases, the content of the post-specific pages would include many posts which were not present on the canonical paginated version.

This commit aims to resolve that problem by simplifying the implementation. Instead of rendering posts surrounding the target post_number, we will only render the target post, and include a link to 'show post in topic'. With this new implementation, 100% of the post-specific page content will be present on the canonical paginated version, which will hopefully mean google reduces their  indexing of the non-canonical post-specific pages.
2024-03-26 15:18:46 +00:00
Loïc Guitaut
9c9526f0a8 DEV: Use Nimbus font instead of Helvetica
To generate letter avatars, we’re currently using the ImageMagick suite
and we’re using the Helvetica font family. However, that font isn’t
shipped anymore in the latest stable version of Debian (Bookworm).
Instead it seems to have been replaced by the Nimbus font. The rendering
is extremely similar (not to say it’s the same thing) so it shouldn’t be
noticeable.

That change is necessary for us to upgrade our docker images to Debian
Bookworm.
2024-03-26 14:42:40 +01:00
Martin Brennan
5adfb299ac
DEV: Change QUnit reporters to dot (#26360)
This is so the CI output on GitHub actions isn't showing
tons and tons of unnecessary log data every time you want
to see the important thing, which is the actual test failure.
2024-03-26 16:54:22 +10:00
Jan Cernik
532639f636
UX: Add threads fab icon (#26349) 2024-03-26 08:30:27 +08:00
Jay Pfaffman
63f6c1e81a
FIX: Allow redirect when downloading maxmind database (#26333)
Per https://dev.maxmind.com/geoip/release-notes/2024#presigned-urls-for-database-downloads 

MaxMind users who download databases should make sure that their HTTP client follows redirects and there are no proxy or firewall settings that would block requests to the host we are redirecting to.
2024-03-26 09:39:09 +11:00
David Taylor
8f44b805f6
DEV: Use -prod flag when building production assets (#26344)
We were previously using the `EMBER_ENV=production` environment variable, which appears to produce the same output. But, some parts of ember-cli don't seem to support it, which leads to a confusing 'Environment: development' being printed on the console.

This commit adds `-prod` by default, which is the more common way to invoke ember-cli for production builds.
2024-03-25 16:56:28 +00:00
Alan Guo Xiang Tan
86b2e3aa3e
DEV: Change tag type to tags type for theme object schema (#26315)
Why this change?

While working on the tag selector for the theme object editor, I
realised that there is an extremely high possibility that users might want to select
more than one tag. By supporting the ability to select more than one
tag, it also means that we get support for a single tag for free as
well.

What does this change do?

1. Change `type: tag` to `type: tags` and support `min` and `max`
   validations for `type: tags`.

2. Fix the `<SchemaThemeSetting::Types::Tags>` component to support the
   `min` and `max` validations
2024-03-22 15:32:00 +08:00
Alan Guo Xiang Tan
dfc406fdc2
FIX: Validate tags using Tag#name instead of Tag#id in ThemeSettingsObjectValidator (#26314)
Why this change?

Fortunately or unfortunately in Discourse core, we mainly use `Tag#name`
to look up tags and not its id. This assumption is built into the
frontend as well so we need to use the tag's name instead of the id
here.
2024-03-22 11:05:16 +08:00
Martin Brennan
61bd7d5d11
FIX: Anon users could not edit their own posts (#26283)
Followup 3094f32ff5,
this fixes an issue with the logic in this commit where
we were returning false if any of the conditionals here
were false, regardless of the type of `obj`, where we should
have only done this if `obj` was a `PostAction`, which lead
us to return false in cases where we were checking if the
user could edit their own post as anon.
2024-03-22 08:12:12 +10:00
Bianca Nenciu
a6e06915c4
FIX: Serialize parent categories first (#26294)
When categories are loaded by the frontend, the parent category is
looked up by ID and the `parentCategory` is set with the result. If the
categories returned are not in order, the parent category may miss.
2024-03-21 19:51:41 +02:00
Alan Guo Xiang Tan
0aa92500aa
DEV: ENV variable for brotli quality in assets:precompile rake task (#26286)
Why this change?

This ENV allows the brotli compression quality to be configurable such
that one can opt for a higher/lower level of compression based on their
preferences.
2024-03-21 19:07:35 +08:00
Martin Brennan
70f7c0ee6f
FEATURE: More flexible admin plugin config nav definition (#26254)
This commit changes the API for registering the plugin config
page nav configuration from a server-side to a JS one;
there is no need for it to be server-side.

It also makes some changes to allow for 2 different ways of displaying
navigation for plugin pages, depending on complexity:

* TOP - This is the best mode for simple plugins without a lot of different
  custom configuration pages, and it reuses the grey horizontal nav bar
  already used for admins.
* SIDEBAR - This is better for more complex plugins; likely this won't
  be used in the near future, but it's readily available if needed

There is a new AdminPluginConfigNavManager service too to manage which
plugin the admin is actively viewing, otherwise we would have trouble
hiding the main plugin nav for admins when viewing a single plugin.
2024-03-21 13:42:06 +10:00
Ella E
ef2a9da8e7
UX: Add chevron icons for expanding and collapsing all toggles in the admin sidebar (#26246) 2024-03-21 10:25:14 +11:00
Jarek Radosz
6d137a41c2
DEV: Remove the last transpile_js mentions (#26263) 2024-03-20 15:32:37 +01:00
David Taylor
aba0df7cf7
FEATURE: Increase default DiscourseConnect session length (#26261)
For some identity providers, 10 minutes isn't much time for a user to complete authentication/registration on the identity provider. Increasing the default to 30 minutes should help in those situations. The nonce is still tied to a single browser session, so there is no material impact on security.
2024-03-20 14:02:12 +00:00
Sam
34a14112a7
FIX: remove "fake" mentions from extract_mentions (#26253)
```
<a class="mention" href="/u/test1">bsam</a>
```

Is not a mention of the user sam. We expect an @ in front always.
2024-03-20 12:20:15 +11:00
Martin Brennan
78bafb331a
FEATURE: Allow site settings to be edited throughout admin UI (#26154)
This commit makes it so the site settings filter controls and
the list of settings input editors themselves can be used elsewhere
in the admin UI outside of /admin/site_settings

This allows us to provide more targeted groups of settings in different
UI areas where it makes sense to provide them, such as on plugin pages.
You could open a single page for a plugin where you can see information
about that plugin, change settings, and configure it with custom UIs
in the one place.

In future we will do this in "config areas" for other parts of the
admin UI.
2024-03-18 08:50:39 +10:00
Nat
ad7e3e04f3
Bump version to v3.3.0.beta1 2024-03-15 15:01:58 +08:00
Bianca Nenciu
b425fbc2a2
SECURITY: Generate more category CSS on client
This commit moves the generation of category background CSS from the
server side to the client side. This simplifies the server side code
because it does not need to check which categories are visible to the
current user.
2024-03-15 14:24:11 +08:00
David Taylor
2546817d07
FIX: correctly strip unneeded csp directives under strict-dynamic (#26180) 2024-03-14 18:50:09 +00:00
Rafael dos Santos Silva
679a773411
FIX: Handle nil post_search_data for search result (#26179)
Since we are introducing new ways to search in Discourse, like the AI
semantic search using embeddings, posts can be part of a search result
list without having any search data.

Since the code path already handles this, we only need to add a safety
check when accessing the post_search_data.
2024-03-14 14:40:46 -03:00
Ted Johansson
ea5c3a3bdc
DEV: Move non scheduled problem checks to classes (#26122)
In AdminDashboardData we have a bunch of problem checks implemented as methods on that class. This PR absolves it of the responsibility by promoting each of those checks to a first class ProblemCheck. This way each of them can have their own priority and arbitrary functionality can be isolated in its own class.

Think "extract class" refactoring over and over. Since they were all moved we can also get rid of the @@problem_syms class variable which was basically the old version of the registry now replaced by ProblemCheck.realtime.

In addition AdminDashboardData::Problem value object has been entirely replaced with the new ProblemCheck::Problem (with compatible API).

Lastly, I added some RSpec matchers to simplify testing of problem checks and provide helpful error messages when assertions fail.
2024-03-14 10:55:01 +08:00
David Taylor
551c6022dc
DEV: Introduce rake task to validate discourse-compatibility file (#26158) 2024-03-13 13:57:41 +00:00
Alan Guo Xiang Tan
1f71db426e
DEV: Remove cache PG connection type map freedom patch (#26153)
Why this change?

Previously, we identified that ActiveRecord's PostgreSQL adapter
executes 3 db queries each time a new connection is created. The 3 db
queries was identified when we looked at the `pg_stats_statement` table
on one of our multisite production cluster. At that time, the hypothesis
is that because we were agressively reaping and creating connections,
the db queries executed each time a connection is created is wasting
resources on our database servers. However, we didn't see any the needle
move much on our servers after deploying the patch so we have decided to
drop this patch as it makes it harder for us to upgrade ActiveRecord in
the future.
2024-03-13 13:28:06 +08:00
Martin Brennan
4e7a75a7ec
DEV: Single admin plugin page for consistent admin plugin UX (#26024)
This commit adds new plugin show routes (`/admin/plugins/:plugin_id`) as we move
towards every plugin having a consistent UI/landing page.

As part of this, we are introducing a consistent way for plugins
to show an inner sidebar in their config page, via a new plugin
API `register_admin_config_nav_routes`

This accepts an array of links with a label/text, and an
ember route. Once this commit is merged we can start the process
of conforming other plugins to follow this pattern, as well
as supporting a single-page version of this for simpler plugins
that don't require an inner sidebar.

Part of /t/122841 internally
2024-03-13 13:15:12 +10:00
Krzysztof Kotlarek
de00c9a3d3
EXPERIMENTAL: restructure admin sidebar menu (#26047)
New admin sidebar menu
2024-03-13 10:18:08 +11:00
David Taylor
127214c613
UX: Improve error handling for DiscourseConnect (#26140)
Previously, if the sso= payload was invalid Base64, but signed correctly, there would be no useful log or error. This commit improves things by:

- moving the base64 check before the signature checking so that it's properly surfaced
- split the ParseError exception into PayloadParseError and SignatureError
- add user-facing errors for both of those
- add/improve spec for both
2024-03-12 16:16:04 +00:00
Ted Johansson
2211ffa851
DEV: Move problem checks to app directory (#26120)
There are a couple of reasons for this.

The first one is practical, and related to eager loading. Since /lib is not eager loaded, when the application boots, ProblemCheck["identifier"] will be nil because the child classes aren't loaded.

The second one is more conceptual. There turns out to be a lot of inter-dependencies between the part of the problem check system that live in /app and the parts that live in /lib, which probably suggests it should all go in /app.
2024-03-11 13:36:22 +08:00
Alan Guo Xiang Tan
8d4f405da4
DEV: Allow typed objects theme settings to be saved via settings editor (#26100)
Why this change?

On the `/admin/customize/themes/<:id>` route, we allow admins to edit
all settings via a settings editor. Prior to this change, trying to edit
and save a typed objects theme settings will result in an error on the
server.
2024-03-11 08:42:12 +08:00
Alan Guo Xiang Tan
7d8dd0d8e3
UX: Show loading spinner while loading dependencies for ace-editor (#26099)
Why this change?

On a slow network, using the `AceEditor` component will result in a blob
of text being shown first before being swapped out with the `ace.js`
editor after it has completed loading.

There is also a problem when setting the theme for the editor which
would result in a "flash" as reported in
https://github.com/ajaxorg/ace/issues/3286. To avoid this, we need to
load the theme js file before displaying the editor.

What does this change do?

1. Adds a loading spinner and set the `div.ace` with a `.hidden` class.
2. Once all the relevant scripts and initialization is done, we will
   then remove the loading spinner and remove `div.ace`.
2024-03-11 06:56:17 +08:00
Osama Sayegh
f8964f8f8f
FIX: Allow nil for properties values when they're not required (#26112)
Properties of schema theme settings that are not marked `required: true` should accept nil as a value.
2024-03-09 14:25:30 +03:00
David Taylor
4b813427e7
DEV: Repair RSpec full_cause_backtrace under bin/turbo_rspec (#26087)
This commit fixes two issues:

1. The wrong exception was being printed as the 'cause' in turbo_rspec output. This was happening because RSpec [expects exceptions to be subclasses of `Exception`](d6e320dc11/lib/rspec/core/formatters/exception_presenter.rb (L102)). This commit resolves the issue by replacing the `FakeException` `Struct` with a subclass of `Exception`.

2. The `full_cause_backtrace` option we set in `rails_helper.rb` does not carry through to the RSpec formatters running in the turbo_rspec reporter process. To fix that, this commit duplicates the necessary config in `lib/turbo_tests.rb`.

Example before - note that the cause is a duplicate of the original exception, and only has three lines of backtrace:

```
     Failure/Error: raise capybara_timeout_error

     CapybaraTimeoutExtension::CapybaraTimedOut:
       This spec passed, but capybara waited for the full wait duration (4s) at least once. This will slow down the test suite. Beware of negating the result of selenium's RSpec matchers.

     [Screenshot Image]: /Users/david/discourse/discourse/tmp/capybara/failures_r_spec_example_groups_glimmer_header_when_cmd_f_keyboard_shortcut_pressed_when_within_a_topic_with_less_than20_posts_does_not_open_search_484.png

     ~~~~~~~ JS LOGS ~~~~~~~
     ~~~~~ END JS LOGS ~~~~~

     # ./spec/rails_helper.rb:372:in `block (3 levels) in <top (required)>'
     # ./spec/rails_helper.rb:472:in `block (2 levels) in <top (required)>'
     # /Users/david/.rvm/gems/ruby-3.2.1/gems/webmock-3.23.0/lib/webmock/rspec.rb:39:in `block (2 levels) in <top (required)>'
     # ------------------
     # --- Caused by: ---
     # CapybaraTimeoutExtension::CapybaraTimedOut:
     #   This spec passed, but capybara waited for the full wait duration (4s) at least once. This will slow down the test suite. Beware of negating the result of selenium's RSpec matchers.
     #   ./spec/rails_helper.rb:372:in `block (3 levels) in <top (required)>'
     #   ./spec/rails_helper.rb:472:in `block (2 levels) in <top (required)>'
     #   /Users/david/.rvm/gems/ruby-3.2.1/gems/webmock-3.23.0/lib/webmock/rspec.rb:39:in `block (2 levels) in <top (required)>'
```

After - note correct causing exception, and the full backtrace 🎉

```
     Failure/Error: raise capybara_timeout_error

     CapybaraTimeoutExtension::CapybaraTimedOut:
       This spec passed, but capybara waited for the full wait duration (4s) at least once. This will slow down the test suite. Beware of negating the result of selenium's RSpec matchers.

     [Screenshot Image]: /Users/david/discourse/discourse/tmp/capybara/failures_r_spec_example_groups_glimmer_header_when_cmd_f_keyboard_shortcut_pressed_when_within_a_topic_with_less_than20_posts_does_not_open_search_61.png

     ~~~~~~~ JS LOGS ~~~~~~~
     ~~~~~ END JS LOGS ~~~~~

     # ./spec/rails_helper.rb:372:in `block (3 levels) in <top (required)>'
     # ./spec/rails_helper.rb:472:in `block (2 levels) in <top (required)>'
     # /Users/david/.rvm/gems/ruby-3.2.1/gems/webmock-3.23.0/lib/webmock/rspec.rb:39:in `block (2 levels) in <top (required)>'
     # ------------------
     # --- Caused by: ---
     # Capybara::ExpectationNotMet:
     #   expected to find css ".search-menu .search-menu-panel" but there were no matches
     #   /Users/david/.rvm/gems/ruby-3.2.1/gems/capybara-3.40.0/lib/capybara/node/matchers.rb:112:in `block in assert_selector'
     #   /Users/david/.rvm/gems/ruby-3.2.1/gems/capybara-3.40.0/lib/capybara/node/matchers.rb:869:in `block in _verify_selector_result'
     #   /Users/david/.rvm/gems/ruby-3.2.1/gems/capybara-3.40.0/lib/capybara/node/base.rb:84:in `synchronize'
     #   ./spec/rails_helper.rb:345:in `synchronize'
     #   /Users/david/.rvm/gems/ruby-3.2.1/gems/capybara-3.40.0/lib/capybara/node/matchers.rb:868:in `_verify_selector_result'
     #   /Users/david/.rvm/gems/ruby-3.2.1/gems/capybara-3.40.0/lib/capybara/node/matchers.rb:110:in `assert_selector'
     #   /Users/david/.rvm/gems/ruby-3.2.1/gems/capybara-3.40.0/lib/capybara/node/matchers.rb:39:in `block in has_selector?'
     #   /Users/david/.rvm/gems/ruby-3.2.1/gems/capybara-3.40.0/lib/capybara/node/matchers.rb:902:in `make_predicate'
     #   /Users/david/.rvm/gems/ruby-3.2.1/gems/capybara-3.40.0/lib/capybara/node/matchers.rb:39:in `has_selector?'
     #   /Users/david/.rvm/gems/ruby-3.2.1/gems/capybara-3.40.0/lib/capybara/session.rb:774:in `has_selector?'
     #   ./spec/system/page_objects/pages/search.rb:46:in `has_search_menu_visible?'
     #   ./spec/system/header_spec.rb:206:in `block (4 levels) in <main>'
     #   ./spec/rails_helper.rb:472:in `block (2 levels) in <top (required)>'
     #   /Users/david/.rvm/gems/ruby-3.2.1/gems/webmock-3.23.0/lib/webmock/rspec.rb:39:in `block (2 levels) in <top (required)>'
```
2024-03-07 18:35:46 +00:00
David Taylor
92d357f91a
FEATURE: Enable strict-dynamic Content-Security-Policy by default (#26051)
Ref https://meta.discourse.org/t/298172 and https://meta.discourse.org/t/295603
2024-03-07 15:20:31 +00:00
Alan Guo Xiang Tan
b1495884eb
PERF: Avoid saving ThemeSetting twice when creating new db override (#26076)
Why this change?

When creating a new theme setting that does not have a corresponding row
in the `theme_settings` table, we end up writing to the database twice
because `ActiveRecord::Base#save!` is called once before the `value`
or `json_value` column is updated again with another database query with
another call to `ActiveRecord::Base#save!`.

What does this change do?

Adds the column to be updated to argument for the `ActiveRecord::Base#create!`
method call so that we only have one write query to the database.
2024-03-07 16:38:11 +08:00
Blake Erickson
0464ddcd9b
FEATURE: Bulk Silent Close Topics (#26043)
Using the new bulk select dropdown you can now choose to bulk close
topics silently.
2024-03-06 14:08:49 -07:00
David Taylor
dfc6bb4029
DEV: Remove direct minitest dependency to appease ruby-lsp (#26056)
Having minitest as a direct dependency causes ruby-lsp to use it as our test runner (per https://github.com/Shopify/ruby-lsp/blob/d1da8858a1/lib/ruby_lsp/requests/support/dependency_detector.rb#L40-L55). This makes VSCode's test explorer incorrectly display Minitest 'run' buttons above all our tests.

We were only using it in `emoji.rake`... and that wasn't even working with the latest version of Minitest. This commit refactors `emoji.rake` to work without minitest, and removes the dependency.
2024-03-06 15:41:14 +00:00
David Taylor
d1a2596889
DEV: Allow CSP nonce_placeholder to be generated outside Rails (#26052)
Sometimes we add scripts outside of Rails. This commit provides a way to generate a nonce placeholder even if you don't have access to an ApplicationController instance.
2024-03-06 13:01:32 +00:00
Joffrey JAFFEUX
00a5f67daf
FIX: brings back discourse-sidebar icon (#26050)
This has been removed from 76953cc356 by mistake
2024-03-06 12:16:28 +01:00
Joffrey JAFFEUX
76953cc356
FEATURE: allows to force a thread (#25987)
Forcing a thread will work even in channel which don't have `threading_enabled` or in direct message channels.

For now this feature is only available through the `ChatSDK`:

```ruby
ChatSDK::Message.create(in_reply_to_id: 1, guardian: guardian, raw: "foo bar baz", channel_id: 2, force_thread: true)
```
2024-03-06 12:03:42 +01:00
Ella E
f935b2ca6d
UX: Add sidebar icon (#26046) 2024-03-06 18:24:37 +10:00
Alan Guo Xiang Tan
94b09f3331
DEV: Open theme settings objects editor from admin customize theme page (#26006)
Why this change?

The `/admin/customize/themes/:id/schema/name` route is a work in
progress but we want to be able to start navigating to it from the
`/admin/customize/themes/:id` route.

What does this change do?

1. Move `adminCustomizeThemes.schema` to a child route of
   `adminCustomizeThemes.show`. This is because we need the model
   from the parent route and if it isn't a child route we end up
   having to load the theme model again from the server.

1. Add the `objects_schema` attribute to `ThemeSettingsSerializer`

1. Refactor `SiteSettingComponent` to be able to render a button
   so that we don't have to hardcode the button rendering into the
   `SiteSettings::String` component
2024-03-06 08:24:29 +08:00
Blake Erickson
40b707a690
FEATURE: Add onebox for loom (#26016)
Loom share links will now onebox and use the embedded loom player.
2024-03-04 15:12:08 -07:00
Alan Guo Xiang Tan
955339668b
FIX: ThemeSettingsObjectValidator not allowing URL paths for string (#26005)
Why this change?

Prior this change, we were using `URI.regexp` which was too strict as it
doesn't allow a URL path.

What does this change do?

Just parse the string using `URI.parse` and if it doesn't raise an error
we consider the string to be a valid URL
2024-03-04 13:22:14 +08:00
Martin Brennan
eca10e56b8
FEATURE: Allow specific groups to view raw email (#26003)
When a post is created by an incoming email, we show
an envelope icon on it which then opens a modal with the
raw email contents. Previously this was staff (admin+mod)
only, but now this commit adds the `view_raw_email_allowed_groups`
site setting, so any group can be added to give users permission
to see this.
2024-03-04 13:48:16 +10:00
Krzysztof Kotlarek
41f78b31a9
FIX: down downgrade trust level if all requirements are met. (#25953)
Currently, the trust level method  is calculating trust level based on maximum value from:
- locked trust level
- group automatic trust level
- previously granted trust level by admin

https://github.com/discourse/discourse/blob/main/lib/trust_level.rb#L33

Let's say the user belongs to groups with automatic trust level 1 and in the meantime meets all criteria to get trust level 2.

Each time, a user is removed from a group with automatic trust_level 1, they will be downgraded to trust_level 1 and promoted to trust_level 2

120a2f70a9/lib/promotion.rb (L142)

This will cause duplicated promotion messages.

Therefore, we have to check if the user meets the criteria, before downgrading.
2024-03-04 09:30:30 +11:00
Régis Hanol
5fcb7c262d
FIX: prevents duplicate attachments in incoming emails (#25986)
## What?

Depending on the email software used, when you reply to an email that has some attachments, they will be sent along, since they're part of the embedded (replied to) email.

When Discourse processes the reply as an incoming email, it will automatically add all the (valid) attachments at the end of the post. Including those that were sent as part of the "embedded reply".

This generates posts in Discourse with duplicate attachments 🙁

## How?

When processing attachments of an incoming email, before we add it to the bottom of the post, we check it against all the previous uploads in the same topic. If there already is an `Upload` record, it means that it's a duplicate and it is _therefore_ skipped.

All the inline attachments are left untouched since they're more likely new attachments added by the sender.
2024-03-01 18:38:49 +01:00
Gerhard Schlager
6847ed5be6
FIX: import:rebake_uncooked_* jobs couldn't be run in parallel (#25969)
If those jobs were started multiple times each process would have rebaked the same posts.
2024-02-29 19:54:19 +01:00
Daniel Waterworth
f0baa0ddfe
DEV: Prefer defer_get_set over get/set on DistributedCache (#25967) 2024-02-29 11:10:24 -06:00
Martin Brennan
5f119c57e8
DEV: Suppress verbose command failure output in plugin:turbo_spec (#25952)
Before this change, if the "Plugins backend" task on GitHub CI
failed, we would get a huge amount of extra output at the end
just to show the command that rake ran which failed (the bin/turbo_rspec
command). This is useless and just makes it hard to see the failing
specs. If you need the full command, it's already output at the
top of the "Plugins backend" task in the GitHub CI.
2024-02-29 14:35:31 +10:00
Martin Brennan
df4197c8b8
FIX: Show deleted bookmark reminders in user bookmarks menu (#25905)
When we send a bookmark reminder, there is an option to delete
the underlying bookmark. The Notification record stays around.
However, if you want to filter your notifications user menu
to only bookmark-based notifications, we were not showing unread
bookmark notifications for deleted bookmarks.

This commit fixes the issue _going forward_ by adding the
bookmarkable_id and bookmarkable_type to the Notification data,
so we can look up the underlying Post/Topic/Chat::Message
for a deleted bookmark and check user access in this way. Then,
it doesn't matter if the bookmark was deleted.
2024-02-29 09:03:49 +10:00
Alan Guo Xiang Tan
1a44c359b9
DEV: Fix reloading type map not clearing cache (#25924)
Why this change?

This is a follow up to 408d2f8e69. When
`ActiveRecord::ConnectionAdapaters::PostgreSQLAdatper#reload_type_map`
is called, we need to clear the type map cache otherwise migrations
adding an array column will end up throwing errors.
2024-02-28 14:56:15 +08:00
Alan Guo Xiang Tan
408d2f8e69
DEV: Adds a freedom patch to cache connection type map (#25923)
Why this change?

This patch has been added to address the problems identified in https://github.com/rails/rails/issues/35311. For every,
new connection created using the PostgreSQL adapter, 3 queries are executed to fetch type map information from the `pg_type`
system catalog, adding about 1ms overhead to every connection creation.

On multisite clusters where connections are reaped more aggressively, the 3 queries executed
accounts for a significant portion of CPU usage on the PostgreSQL cluster. This patch works around the problem by
caching the type map in a class level attribute to reuse across connections.
2024-02-28 14:29:13 +08:00
Alan Guo Xiang Tan
54a1fea74e
DEV: Refactor ThemeSettingsObjectValidator#validate (#25904)
What does this change do?

1. Reduce an additional loop through all the properties
2. Extract the validation of child objects into a dedicate method
2024-02-28 10:44:46 +08:00
Alan Guo Xiang Tan
afb0adf48d
DEV: Validate objects when updating typed objects theme settings (#25902)
Why this change?

This change ensures that we validate the value of the new objects
when updating typed objects theme settings.
2024-02-28 10:33:22 +08:00
Joffrey JAFFEUX
64e8ad170e
FIX: do not show send pm prompt when user cant pm (#25912)
Prior to this fix even when the user was not part of a group allowing sending pm we would show the prompt: "You've replied to ... X times, did you know you could send them a personal message instead?"
2024-02-27 11:54:05 +01:00
Jarek Radosz
3736d66f17
DEV: Extensively use exception: true in system() (#25911)
Specifically fixes a bug in smoke-test where it would just move on after failing to install latest js dependencies with yarn.
2024-02-27 11:33:28 +01:00
Jarek Radosz
b337ae5ae9
DEV: Update chrome-launcher from 0.15.2 to 1.1.0 (#25909) 2024-02-27 11:25:11 +01:00
Alan Guo Xiang Tan
18ca3d373d
FIX: ThemeSettingsValidator.validate_value returning wrong error (#25901)
Why this change?

Before this change, the error messages returned when validating theme
settings of typed objects was an array of array instead of just an
array.
2024-02-27 15:46:12 +08:00
Alan Guo Xiang Tan
52a4912475
DEV: Support topic, post, group, upload and tag type for theme objects setting (#25907)
Why this change?

Previously in cac60a2c6b, I added support
for `type: "category"` for a property in the theme objects schema. This
commit extend the work previously to add support for types `topic`,
`post`, `group`, `upload` and `tag`.
2024-02-27 14:27:10 +08:00
Alan Guo Xiang Tan
412b36cc93
FIX: Error when integer values are set as default of string type settings (#25898)
Why this change?

```
some_setting:
  default: 0
  type: string
```

A theme setting like the above will cause an error to be thrown on the
server when importing the theme because the default would be parsed as
an integer which caused an error to be thrown when we are validating the
value of the setting.

What does this change do?

Convert the value to a string when working with string typed theme
settings.
2024-02-27 10:18:38 +08:00
Alan Guo Xiang Tan
7bcfe60a76
DEV: Validate default value for type: objects theme settings (#25833)
Why this change?

This change adds validation for the default value for `type: objects` theme
settings when a setting theme field is uploaded. This helps the theme
author to ensure that the objects which they specifc in the default
value adhere to the schema which they have declared.

When an error is encountered in one of the objects, the error
message will look something like:

`"The property at JSON Pointer '/0/title' must be at least 5 characters
long."`

We use a JSON Pointer to reference the property in the object which is
something most json-schema validator uses as well.

What does this change do?

1. This commit once again changes the shape of hash returned by
   `ThemeSettingsObjectValidator.validate`. Instead of using the
   property name as the key previously, we have decided to avoid
   multiple levels of nesting and instead use a JSON Pointer as the key
   which helps to simplify the implementation.

2 Introduces `ThemeSettingsObjectValidator.validate_objects` which
  returns an array of validation error messages for all the objects
  passed to the method.
2024-02-27 09:16:37 +08:00
David Taylor
588a79c80c
DEV: Merge root JS packages (#25857)
Before this commit, we had a yarn package set up in the root directory and also in `app/assets/javascripts`. That meant two `yarn install` calls and two `node_modules` directories. This commit merges them both into the root location, and updates references to node_modules.

A previous attempt can be found at https://github.com/discourse/discourse/pull/21172. This commit re-uses that script to merge the `yarn.lock` files.

Co-authored-by: Jarek Radosz <jradosz@gmail.com>
2024-02-26 13:45:58 +00:00
Andrei Prigorshnev
b3a1199493
FEATURE: Hide user status when user is hiding public profile and presence (#24300)
Users can hide their public profile and presence information by checking 
“Hide my public profile and presence features” on the 
`u/{username}/preferences/interface` page. In that case, we also don't 
want to return user status from the server.

This work has been started in https://github.com/discourse/discourse/pull/23946. 
The current PR fixes all the remaining places in Core.

Note that the actual fix is quite simple – a5802f484d. 
But we had a fair amount of duplication in the code responsible for 
the user status serialization, so I had to dry that up first. The refactoring 
as well as adding some additional tests is the main part of this PR.
2024-02-26 17:40:48 +04:00
David Taylor
542cb22fd4 DEV: Drop Ember 3 feature flag 2024-02-26 12:22:05 +00:00
Ted Johansson
ed2496c59d
FEATURE: Add scheduled Twitter login problem check - Part 1 (#25830)
This PR adds a new scheduled problem check that simply tries to connect to Twitter OAuth endpoint to check that it's working. It is using the default retry strategy of 2 retries 30 seconds apart.
2024-02-26 12:08:12 +08:00
Vinoth Kannan
1bd9ca11e7
DEV: include more data in Discourse Discover enrollment payload. (#25846)
Adding forum's URL, title, and locale to the payload of enrollment can be helpful while managing it in the Discourse Discover.
2024-02-25 10:42:50 +05:30
Alan Guo Xiang Tan
ad0824b7e3
DEV: Fix connections timeout in system test (#25835)
Why this change?

This regressed in 6e9fbb5bab because we
had a `request.xhr?` check before we decide to block requests. However,
there could not none-xhr requests which we need to block as well at the
end of each system test when `@@block_requests` is true.

This also reverts commit 6437f27f90.
2024-02-23 16:03:46 +08:00
Vinoth Kannan
b3238bfc34
FEATURE: call hub API to update Discourse discover enrollment. (#25634)
Now forums can enroll their sites to be showcased in the Discourse [Discover](https://discourse.org/discover) directory. Once they enable the site setting `include_in_discourse_discover` to enroll their forum the `CallDiscourseHub` job will ping the `api.discourse.org/api/discover/enroll` endpoint. Then the Discourse Hub will fetch the basic details from the forum and add it to the review queue. If the site is approved then the forum details will be displayed in the `/discover` page.
2024-02-23 11:42:28 +05:30
Sam
207cb2052f
FIX: muted tags breaking hot page when filtered to tags (#25824)
Also, remove experimental setting and simply use top_menu for feature detection

This means that when people eventually enable the hot top menu, there will
be topics in it


Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2024-02-23 17:11:39 +11:00
Alan Guo Xiang Tan
6437f27f90
DEV: Debug AR connection pool queue on CI (#25828)
Why this change?

On CI, we have been seeing flaky system tests because ActiveRecord is
unable to checkout a connection. This patch is meant to help us debug
which thread is not returning the connection to the queue.

Example of timeout issue: https://github.com/discourse/discourse/actions/runs/8012541636/job/21888013082
2024-02-23 13:37:37 +08:00
Ted Johansson
a72dc2f420
DEV: Introduce a problem checks API (#25783)
Previously, problem checks were all added as either class methods or blocks in AdminDashboardData. Another set of class methods were used to add and run problem checks.

As of this PR, problem checks are promoted to first-class citizens. Each problem check receives their own class. This class of course contains the implementation for running the check, but also configuration items like retry strategies (for scheduled checks.)

In addition, the parent class ProblemCheck also serves as a registry for checks. For example we can get a list of all existing check classes through ProblemCheck.checks, or just the ones running on a schedule through ProblemCheck.scheduled.

After this refactor, the task of adding a new check is significantly simplified. You add a class that inherits ProblemCheck, you implement it, add a test, and you're good to go.
2024-02-23 11:20:32 +08:00
Daniel Waterworth
087712c1a6
DEV: Reduce duplication with DistributedMutex#clear_regex (#25795) 2024-02-21 14:19:04 -06:00
Daniel Waterworth
13291dc5ef
FIX: Cache keys should be strings (#25791)
* FIX: Cache keys should be strings

Otherwise, there are subtle bugs that don't show up with a single
process.
2024-02-21 10:55:48 -06:00
Bianca Nenciu
9199c52e5e
FIX: Load categories with search topic results (#25700)
Add categories to the serialized search results together with the topics
when lazy load categories is enabled. This is necessary in order for the
results to be rendered correctly and display the category information.
2024-02-21 17:29:47 +02:00
Alan Guo Xiang Tan
3e54351355
DEV: Change shape of errors in ThemeSettingsObjectValidator (#25784)
Why this change?

The current shape of errors returns the error messages after it has been
translated but there are cases where we want to customize the error
messages and the current way return only translated error messages is
making customization of error messages difficult. If we
wish to have the error messages in complete sentences like
"`some_property` property must be present in #link 1", this is not
possible at the moment with the current shape of the errors we return.

What does this change do?

This change introduces the `ThemeSettingsObjectValidator::ThemeSettingsObjectErrors`
and `ThemeSettingsObjectValidator::ThemeSettingsObjectError` classes to
hold the relevant error key and i18n translation options.
2024-02-21 15:27:42 +08:00
Daniel Waterworth
0529d20db4
DEV: Use DistributedCache#defer_get_set instead of getting and setting (#25778)
We use defer_get_set everywhere else, so for consistency and reducing
complexity, this is better.
2024-02-20 18:29:01 -06:00
Alan Guo Xiang Tan
cac60a2c6b
DEV: Support category type in theme setting object schema (#25760)
Why this change?

This change supports a property of `type: category` in the schema that
is declared for a theme setting object. Example:

```
sections:
  type: objects
  schema:
    name: section
    properties:
      category_property:
        type: category
```

The value of a property declared as `type: category` will have to be a
valid id of a row in the `categories` table.

What does this change do?

Adds a property value validation step for `type: category`. Care has
been taken to ensure that we do not spam the database with a ton of
requests if there are alot of category typed properties. This is done by
walking through the entire object and collecting all the values for
properties typed category. After which, a single database query is
executed to validate which values are valid.
2024-02-21 08:11:15 +08:00
Alan Guo Xiang Tan
6ca2396b12
DEV: Centralise logic for validating a theme setting value (#25764)
Why this change?

The logic for validating a theme setting's value and default value was
not consistent as each part of the code would implement its own logic.
This is not ideal as the default value may be validated differently than
when we are setting a new value. Therefore, this commit seeks to
refactor all the validation logic for a theme setting's value into a
single service class.

What does this change do?

Introduce the `ThemeSettingsValidator` service class which holds all the
necessary helper methods required to validate a theme setting's value
2024-02-21 08:08:26 +08:00
Martin Brennan
0b3180c86f
DEV: Add SecureUploadEndpointHelpers for controllers (#25758)
This commit moves some code out of UploadController#show_secure
so it can be reused in other controllers if a secure upload
needs to have permission checks run.
2024-02-20 11:19:22 +10:00
Alan Guo Xiang Tan
bf3c4b634a
DEV: Support validations options for string and numeral types (#25719)
Why this change?

This commit updates `ThemeSettingsObjectValidator` to validate a
property's value against the validations listed in the schema.

For string types, `min_length`, `max_length` and `url` are supported.
For integer and float types, `min` and `max` are supported.
2024-02-20 09:17:27 +08:00
Alan Guo Xiang Tan
a64f558f32
DEV: Add property value validation to ThemeSettingsObjectValidator (#25718)
Why this change?

This change adds property value validation to `ThemeSettingsObjectValidator`
for the following types: "string", "integer", "float", "boolean", "enum". Note
that this class is not being used anywhere yet and is still in
development.
2024-02-19 13:19:35 +08:00
Martin Brennan
a57280cb17
DEV: Change min_trust_level_to_allow_profile_background to trust level setting (#25721)
New setting name is profile_background_allowed_groups

c.f. https://meta.discourse.org/t/changes-coming-to-settings-for-giving-access-to-features-from-trust-levels-to-groups/283408
2024-02-19 10:47:47 +10:00
David Taylor
330cb837da
FIX: Remove strict-dynamic-specific logic from CSP extensions (#25725)
This data is cached, so we don't want to include any site-specific-logic in there. Let's just keep the old URL-collecting behaviour, and let it be stripped out by `CSP::Builder` at runtime.
2024-02-16 13:24:50 +00:00
David Taylor
1672a24490
DEV: Memoize CSP nonce placeholder on response (#25724)
That way, the same value is used even if the helper is called in the context of different controllers

Followup to c8a1b49ddd
2024-02-16 12:15:55 +00:00
David Taylor
b1f74ab59e
FEATURE: Add experimental option for strict-dynamic CSP (#25664)
The strict-dynamic CSP directive is supported in all our target browsers, and makes for a much simpler configuration. Instead of allowlisting paths, we use a per-request nonce to authorize `<script>` tags, and then those scripts are allowed to load additional scripts (or add additional inline scripts) without restriction.

This becomes especially useful when admins want to add external scripts like Google Tag Manager, or advertising scripts, which then go on to load a ton of other scripts.

All script tags introduced via themes will automatically have the nonce attribute applied, so it should be zero-effort for theme developers. Plugins *may* need some changes if they are inserting their own script tags.

This commit introduces a strict-dynamic-based CSP behind an experimental `content_security_policy_strict_dynamic` site setting.
2024-02-16 11:16:54 +00:00
Martin Brennan
3094f32ff5
FIX: is_my_own? check for users who are anonymously doing actions (#25716)
Followup to 978d52841a

It's complicated...we have multiple "anonymous" user concepts
in core, and even two classes called the exact same thing --
AnonymousUser.

The first case is Guardian::AnonymousUser, which is used for
people who are browsing the forum without being authenticated.

The second case is the model AnonymousUser, which is used when
a user is liking or posting anonymously via allow_anonymous_likes
or allow_anonymous_posting site settings.

We will untangle this naming nightmare later on...but for the
time being, only authenticated users who are pretending to be
anonymous should be able to like posts if allow_anonymous_likes
is on.
2024-02-16 14:28:12 +10:00
Alan Guo Xiang Tan
ad900ef9dd
Revert "DEV: Debug AR connection pool queue on CI (#25687)" (#25714)
This reverts commit 796af077c5.

We have not seen checkout timeout errors since c30aeafd9d
2024-02-16 10:11:46 +08:00
Alan Guo Xiang Tan
64b4e0d08d
DEV: First pass of ThemeSettingsObjectValidator (#25624)
Why this change?

This is a first pass at adding an objects validator which main's job is
to validate an object against a defined schema which we will support. In
this pass, we are simply validating that properties that has been marked
as required are present in the object.
2024-02-16 09:35:16 +08:00
Osama Sayegh
cfdb461e9a
PERF: Pass the -ping option to the identify ImageMagick command to speed it up (#25697)
The `-ping` option significantly speeds up the ImageMagick `identify` command per our testing and the [documentation](https://imagemagick.org/script/command-line-options.php#ping):

> -ping
Efficiently determine these image characteristics: image number, the file name, the width and height of the image, whether the image is colormapped or not, the number of colors in the image, the number of bytes in the image, the format of the image (JPEG, PNM, etc.). Use +ping to ensure accurate image properties.

We already pass the `-ping` option in other places where the `identify` command is used, so it makes sense to use the option everywhere.

Internal topic: t/121431.
2024-02-15 18:55:39 +03:00
Sam
4346abe260
FEATURE: apply pinning to hot topic lists (#25690)
pinned topics should be pinned even on hot lists so it can be used as a
home page
2024-02-15 18:27:54 +11:00
Alan Guo Xiang Tan
796af077c5
DEV: Debug AR connection pool queue on CI (#25687)
Why this change?

On CI, we have been seeing flaky system tests because ActiveRecord is
unable to checkout a connection. This patch is meant to help us debug
which thread is not returning the connection to the queue.
2024-02-15 14:00:30 +08:00
Blake Erickson
bb261094cf
FEATURE: Auto generate and display video preview image (#25633)
This change will allow auto generated video thumbnails to be used
instead of the black video thumbnail that overlays videos.

Follow up to: 2443446e62
2024-02-14 13:43:53 -07:00
Kris
faf0807b37
FIX: email category badges shouldn't use category text color (#25655) 2024-02-13 10:18:36 -05:00
David Battersby
d7dd871d9f
FIX: quoted private topic url respects subfolder install (#25643)
Fixes an issue where private topics that are quoted have an incorrectly formatted url when using a subfolder install.

This update returns a relative url that includes the base_path rather than a combination of base_url + base_path.
2024-02-13 13:20:24 +08:00
Penar Musaraj
021a02c3d8
FIX: Webauthn origin was incorrect for subfolder setups (#25651) 2024-02-12 16:27:24 -05:00
Martin Brennan
cf4d92f686
FIX: Change max_image_megapixels logic (#25625)
This commit changes `max_image_megapixels` to be used
as is without multiplying by 2 to give extra leway.
We found in reality this was just causing confusion
for admins, especially with the already permissive
40MP default.
2024-02-12 09:56:43 +10:00
Kris
c49eb373de
FIX: render category badge styles inline for email (#25487) 2024-02-09 15:29:11 -05:00
David Taylor
ee08a8c52b
Revert "FIX: Omit CSP nonce and hash values when unsafe-inline enabled (#25590)" (#25609)
This reverts commit 767b49232e.

If anything else (e.g. GTM integration) introduces a nonce/hash, then this change stops the splash screen JS to fail and makes sites unusable.
2024-02-08 11:44:09 +00:00
Alan Guo Xiang Tan
fb0e656cb7
DEV: Refactor subclasses in ThemeSettingsManager to individual files (#25605)
Why this change?

One Ruby class per file improves readability
2024-02-08 12:59:52 +08:00
Martin Brennan
7ce76143ac
FIX: Always trust admin and moderators with post edits (#25602)
Removes duplication from LimitedEdit to see who can edit
posts, and also removes the old trust level setting check
since it's no longer necessary.

Also make it so staff can always edit since can_edit_post?
already has a staff escape hatch.
2024-02-08 13:10:26 +10:00
Alan Guo Xiang Tan
9f884cdaab
DEV: Introduce experimental type: objects theme setting (#25538)
Why this change?

This commit introduces an experimental `type: objects` theme setting
which will allow theme developers to store a collection of objects as
JSON in the database. Currently, the feature is still in development and
this commit is simply setting up the ground work for us to introduce the
feature in smaller pieces.

What does this change do?

1. Adds a `json_value` column as `jsonb` data type to the `theme_settings` table.
2. Adds a `experimental_objects_type_for_theme_settings` site setting to
   determine whether `ThemeSetting` records of with the `objects` data
   type can be created.
3. Updates `ThemeSettingsManager` to support read/write access from the
   `ThemeSettings#json_value` column.
2024-02-08 10:20:59 +08:00
Martin Brennan
4ce1c2c030
FIX: Always allow staff (admins & mods) to post links (#25601)
Followup fb087b7ff6

post_links_allowed_groups is an odd check tied to
unrestricted_link_posting? in PostGuardian, in that
it doesn't have an escape hatch for staff like most
of the rest of these group based settings.

It doesn't make sense to exclude admins or mods from
posting links, so just always allow them to avoid confusion.
2024-02-08 11:19:28 +10:00
Daniel Waterworth
30922855f2
PERF: Don't allow a single user to monopolize the defer queue (#25593) 2024-02-07 13:47:50 -06:00
David Taylor
767b49232e
FIX: Omit CSP nonce and hash values when unsafe-inline enabled (#25590)
Browsers will ignore unsafe-inline if nonces or hashes are included in the CSP. When unsafe-inline is enabled, nonces and hashes are not required, so we can skip them.

Our strong recommendation remains that unsafe-inline should not be used in production.
2024-02-07 12:35:35 +00:00
David Taylor
dea753a204
FIX: Restore support for .js.es6 files in PrettyText (#25588)
Regressed in 1757a688c4

https://meta.discourse.org/t/294155
2024-02-07 09:34:31 +00:00
Ted Johansson
95a2d285d3
FEATURE: Add new 'illegal' flag reason (#25498)
To comply with Digital Services Act we need a way for users to flag a post as potentially illegal. This PR adds that functionality.
2024-02-07 10:12:22 +08:00
Arpit Jalan
badc390ebe
FEATURE: allow disabling user activity tab for non admin users (#25540)
* FEATURE: allow disabling user activity tab for non admin users

* add another test case
2024-02-05 14:30:36 +05:30
David Taylor
1df9186526
FIX: Plugin image assets in production (#25547)
Followup to 1757a688c4
2024-02-02 19:06:16 +00:00
Ted Johansson
2da7c74e60
DEV: Remove TagGuardian#can_create_tag? fallback (#25535)
We've changed access settings to be group membership based rather than based on the TL value directly. We kept both conditions here while we updated any plugins and themes. It should now be safe to remove.
2024-02-02 13:48:53 +08:00
David Taylor
1757a688c4
DEV: Remove sprockets from plugin 'extra js' pipeline (#25502)
JS assets added by plugins via `register_asset` will be outside the `assets/javascripts` directory, and are therefore exempt from being transpiled. That means that there isn't really any need to run them through DiscourseJsProcessor. Instead, we can just concatenate them together, and avoid the need for all the sprockets-wrangling.

This commit also takes the opportunity to clean up a number of plugin-asset-related codepaths which are no longer required (e.g. globs, handlebars)
2024-02-01 11:48:31 +00:00
Neil Lalonde
fb469e7e2f
FIX: use feedback_category placeholder in i18n string (#25514)
https://review.discourse.org/t/71308/4
2024-01-31 19:40:30 -05:00
Alan Guo Xiang Tan
e61608d080
FIX: Remap postgres text search proximity operator (#25497)
Why this change?

Since 1dba1aca27, we have been remapping
the `<->` proximity operator in a tsquery to `&`. However, there is
another variant of it which follows the `<N>` pattern. For example, the
following text "end-to-end" will eventually result in the following
tsquery `end-to-end:* <-> end:* <2> end:*` being generated by Postgres.
Before this fix, the tsquery is remapped to `end-to-end:* & end:* <2>
end:*` by us. This is requires the search data which we store to contain
`end` at exactly 2 position apart. Due to the way we limit the
number of duplicates in our search data, the search term may end up not
matching anything. In bd32912c5e, we made
it such that we do not allow any duplicates when indexing a topic's
title. Therefore, search for `end-to-end` against a topic title with
`end-to-end` will never match because our index will only contain one
`end` term.

What does this change do?

We will remap the `<N>` variant of the proximity operator.
2024-02-01 07:20:46 +08:00
Jordan Vidrine
f2ac9e4c12
UX: Refactor Do Not Disturb indicator (#25508) 2024-01-31 16:56:07 -06:00
Neil Lalonde
ba68ee4da7
FIX: missing translation of guidelines_topic.body (#25505)
Broken in https://github.com/discourse/discourse/pull/25253
2024-01-31 15:33:09 -05:00
Martin Brennan
575bc4af73
FIX: Remove newlines from img alt & title in HTML to markdown parser (#25473)
We were having a minor issue with emails with embedded images
that had newlines in the alt string; for example:

```
<p class="MsoNormal"><span style="font-size:11.0pt"><img width="898"
height="498" style="width:9.3541in;height:5.1875in" id="Picture_x0020_5"
src="cid:image003.png@01DA4EBA.0400B610" alt="A screenshot of a computer
program

Description automatically generated"></span><span
style="font-size:11.0pt"><o:p></o:p></span></p>
```

Once this was parsed and converted to markdown (or directly to HTML
in some cases), this caused an issue in the composer and the post
UI, where the markdown parser didn't know how to deal with this,
making the HTML show directly instead of showing an image.

The easiest way to deal with this is to just strip \n from image
alt and title attrs in the HTMLToMarkdown class.
2024-01-31 10:23:09 +10:00
Isaac Janzen
491e7a3429
DEV: Don't create backups during version bumps (#25484) 2024-01-30 11:41:15 -07:00
Isaac Janzen
a4022f69f3
Bump version to v3.3.0.beta1-dev 2024-01-30 10:37:25 -07:00
Isaac Janzen
b2b1e721b5
Bump version to v3.2.0.beta5 2024-01-30 10:37:25 -07:00
David Taylor
283fe48243
DEV: Update confirm-email flows to use central 2fa and ember rendering (#25404)
These routes were previously rendered using Rails, and had a fairly fragile 2fa implementation in vanilla-js. This commit refactors the routes to be handled in the Ember app, removes the custom vanilla-js bundles, and leans on our centralized 2fa implementation. It also introduces a set of system specs for the behavior.
2024-01-30 10:32:42 +00:00
David Taylor
2457553d0a
FIX: Avoid flash-of-unstyled-content in Safari with bug workaround (#25462)
Safari has a bug which means that scripts with the `defer` attribute are executed before stylesheets have finished loading. This is being tracked at https://bugs.webkit.org/show_bug.cgi?id=209261.

This commit works around the problem by introducing a no-op inline `<script>` to the end of our HTML document. This works because defer scripts are guaranteed to run after inline scripts, and inline scripts are guaranteed to run after any preceding stylesheets.

Technically we only need this for Safari. But given that the cost is so low, it makes sense to include it everywhere rather than incurring the complexity of gating it by user-agent.
2024-01-29 17:20:44 +00:00
David Taylor
1bfccdd4f2
DEV: Allow run_second_factor! to be used before login (#25420)
In a handful of situations, we need to verify a user's 2fa credentials before `current_user` is assigned. For example: login, email_login and change-email confirmation. This commit adds an explicit `target_user:` parameter to the centralized 2fa system so that it can be used for those situations.

For safety and clarity, this new parameter only works for anon. If some user is logged in, and target_user is set to a different user, an exception will be raised.
2024-01-29 12:28:47 +00:00
Ted Johansson
7e5d2a95ee
DEV: Convert min_trust_level_to_tag_topics to groups (#25273)
We're changing the implementation of trust levels to use groups. Part of this is to have site settings that reference trust levels use groups instead. It converts the min_trust_level_to_tag_topics site setting to tag_topic_allowed_groups.
2024-01-26 13:25:03 +08:00
Sérgio Saquetim
d34a7916c0
DEV: Move Plugin::Instance.register_search_group_query_callback out of protected (#25413) 2024-01-25 13:36:43 -03:00
Martin Brennan
74fd883a89
DEV: Improve site setting rename generator (#25354)
We need to be able to generate these migrations
for plugin settings as well. Also, we can use the
type supervisor to get the enum data in a nicer way.
2024-01-25 10:45:46 +10:00
Martin Brennan
0e50f88212
DEV: Move min_trust_to_post_embedded_media to group setting (#25238)
c.f. https://meta.discourse.org/t/we-are-changing-giving-access-to-features/283408
2024-01-25 09:50:59 +10:00
Leonardo Mosquera
508e2e601c
FIX: FinalDestination::HTTP: validate address argument (#25407)
This would only be empty due to a programming error elsewhere, but
checking this here is a failstop so that it doesn't go further.
2024-01-24 18:50:42 -03:00
Vinoth Kannan
ecb7fb0481
FEATURE: add option to sort topic query result via plugin. (#25349)
Previously, it was not possible to modify the sorting order of the `TopicQuery` result from a plugin. This feature adds support to specify custom sorting functionality in a plugin. We're using the `apply_modifier` method in the `DiscoursePluginRegistry` module to achieve it.

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2024-01-24 12:58:12 +05:30