Commit Graph

10340 Commits

Author SHA1 Message Date
Alan Guo Xiang Tan
e31cf66f11
FIX: FinalDestination#get forwarding Authorization header on redirects (#27043)
This commits updates `FinalDestination#get` to not forward
`Authorization` header on redirects since most HTTP clients I tested like
curl and wget does not it.

This also fixes a recent problem in `DiscourseIpInfo.mmdb_download`
where we will fail to download the databases when both `GlobalSetting.maxmind_account_id` and
`GlobalSetting.maxmind_license_key` has been set. The failure is due to
the bug above where the redirected URL given by MaxMind does not accept
an `Authorization` header.
2024-05-16 08:37:34 +08:00
Jan Cernik
fb63ddd7d2
Bump version to v3.3.0.beta3-dev 2024-05-15 12:24:11 -03:00
Jan Cernik
c723c126f2
Bump version to v3.3.0.beta2 2024-05-15 12:24:11 -03:00
Alan Guo Xiang Tan
2134ca9031
PERF: Optimise query for excluding topics in certain categories in TopicsFilter (#27027)
This commit optimises the database query generated by
`TopicsFilter#filter_categories` when the `-category:*` filter is used.
Previously, the method will add the `topics.category_id NOT IN
(<category ids to be excluded>)` filter to the resulting query. However,
we noticed that the performance of the query degrades as the number of
rows in the `topics` table grow and when the number of category ids to be
excluded is large.

Sample of query we ran on a large database in production to demonstrate
the improvement:

Before:

```
SELECT topics.id FROM topics WHERE topics.category_id NOT IN (83, 136, 149, 143, 153, 165, 161, 123, 155, 163, 144, 134, 69, 135, 158, 141, 151, 160, 131, 133, 89, 104, 150, 147, 132, 145, 108, 146, 122, 100, 128, 154, 95, 102, 140, 139, 88, 91, 87) ORDER BY topics.id DESC LIMIT 5;

                                                                                                       QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=27795.34..27795.34 rows=1 width=4) (actual time=29.317..30.165 rows=5 loops=1)
   ->  Sort  (cost=27795.34..27795.34 rows=1 width=4) (actual time=29.316..30.163 rows=5 loops=1)
         Sort Key: id DESC
         Sort Method: top-N heapsort  Memory: 25kB
         ->  Gather  (cost=1000.10..27795.33 rows=1 width=4) (actual time=0.187..26.132 rows=73478 loops=1)
               Workers Planned: 2
               Workers Launched: 2
               ->  Parallel Seq Scan on topics  (cost=0.10..26795.23 rows=1 width=4) (actual time=0.013..22.252 rows=24493 loops=3)
                     Filter: (category_id <> ALL ('{83,136,149,143,153,165,161,123,155,163,144,134,69,135,158,141,151,160,131,133,89,104,150,147,132,145,108,146,122,100,128,154,95,102,140,139,88,91,87}'::integer[]))
                     Rows Removed by Filter: 77276
 Planning Time: 0.140 ms
 Execution Time: 30.181 ms
```

After:

```
SELECT topics.id FROM topics WHERE NOT EXISTS (
  SELECT 1
  FROM unnest(array[83, 136, 149, 143, 153, 165, 161, 123, 155, 163, 144, 134, 69, 135, 158, 141, 151, 160, 131, 133, 89, 104, 150, 147, 132, 145, 108, 146, 122, 100, 128, 154, 95, 102, 140, 139, 88, 91, 87]) AS excluded_categories(category_id)
  WHERE topics.category_id IS NULL OR excluded_categories.category_id = topics.category_id
) ORDER BY topics.id DESC LIMIT 5 ;

                                                                        QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.42..13.52 rows=5 width=4) (actual time=0.028..0.110 rows=5 loops=1)
   ->  Nested Loop Anti Join  (cost=0.42..179929.62 rows=68715 width=4) (actual time=0.027..0.109 rows=5 loops=1)
         Join Filter: ((topics.category_id IS NULL) OR (excluded_categories.category_id = topics.category_id))
         Rows Removed by Join Filter: 239
         ->  Index Scan Backward using forum_threads_pkey on topics  (cost=0.42..108925.71 rows=305301 width=8) (actual time=0.012..0.062 rows=44 loops=1)
         ->  Function Scan on unnest excluded_categories  (cost=0.00..0.39 rows=39 width=4) (actual time=0.000..0.001 rows=6 loops=44)
 Planning Time: 0.126 ms
 Execution Time: 0.124 ms
(8 rows)
```
2024-05-15 11:03:07 +08:00
dsims
e6e3eaf472
FIX: avoid error from missing meta tags (#26927) 2024-05-14 11:41:53 -04:00
marstall
6df2f94bbc
DEV add modifiers to message_builder so plugins can customize subject/body/html (#26867) 2024-05-13 14:59:15 -04:00
Bianca Nenciu
ebc1763aa5
FIX: Change request method for categories/search (#26976)
This commit changes request method for "categories/search" from GET to
POST to make sure that long filters can be passed to the server. For
example, category selectors with many categories are setting the full
list of selected category IDs to ensure these are filtered out from the
list of choices. This can result in a long URL that exceeds the maximum
length.
2024-05-13 14:37:17 +03:00
David Taylor
97847f6cd8
Revert "DEV: @babel/plugin-proposal-decorators -> decorator-transforms (#25290)" (#26971)
This reverts commit 0f4520867b.

This has led to two problems:

1. An incompatibility with Cloudflare's "auto minify" feature. They've deprecated this feature because of incompatibility with modern JS syntax. But unfortunately it will remain enabled on existing properties until 2024-08-05.

2. Discourse fails to boot in Safari 15. This is strange, because Safari does support all the required features in our production JS bundles. Even more strangely, things start working as soon as you open the developer tools. That suggests the cause could be a Safari bug rather than a simple incompatibility.

Reverting while we work out a path forward on both those issues.
2024-05-10 12:48:16 +01:00
Alan Guo Xiang Tan
7079698cdf
FIX: Use MaxMind supplied permalinks to download MaxMind databases (#26847)
This commit switches `DiscourseIpInfo.mmdb_download` to use the
permalinks supplied by MaxMind to download the MaxMind databases as
specified in
https://dev.maxmind.com/geoip/updating-databases#directly-downloading-databases
which states:

```
To directly download databases, follow these steps:

1. In the "Download Links" column, click "Get Permalink(s)" for the desired database.
2. Copy the permalink(s) provided in the modal window.
3. Provide your account ID and your license key using Basic Authentication to authenticate.
```

Previously we are downloading from `https://download.maxmind.com/app/geoip_download` but this is not
documented anyway on MaxMind's docs so this URL can in theory break
in the future without warning. Therefore, we are taking a proactive
approach to download the databases from MaxMind the recommended way
instead of relying on a hidden URL. This old way of downloading the
databases with only a license key will be deprecated in 3.3 and be
removed in 3.4.
2024-05-09 15:11:56 +08:00
Alan Guo Xiang Tan
c8da2a33e8
FIX: Attempt to onebox even if response body exceeds max_download_kb (#26929)
In 95a82d608d, we lowered the default for
`Onebox.options.max_download_kb` from 10mb to 2mb for security hardening
purposes. However, this resulted in multiple bug reports where seemingly
nomral URLs stopped being oneboxed. It turns out that lowering
`Onebox.options.max_download_kb` resulted in `Onebox::Helpers::DownloadTooLarge` being raised
more often for more URLs  in `Onebox::Helpers.fetch_response` which
`Onebox::Helpers.fetch_html_doc` relies on. When
`Onebox::Helpers::DownloadTooLarge` is raised in
`Onebox::Helpers.fetch_response`, we throw away whatever response body
which we have already downloaded at that point. This is not ideal
because Nokogiri can parse incomplete HTML documents and there is a
really high chance that the incomplete HTML document still contains the
information which we need for oneboxing.

Therefore, this commit updates `Onebox::Helpers.fetch_html_doc` to not
throw away the response body when the size of the response body exceeds
`Onebox.options.max_download_size`. Instead, we just take whatever
response which we have and get Nokogiri to parse it.
2024-05-09 07:00:34 +08:00
David Taylor
ece0150cb7
FIX: Ensure RequestTracker handles bubbled exceptions correctly (#26940)
This can happen for various reasons including rate limiting and middleware bugs. This should resolve the warning we're seeing in the logs

```
RequestTracker.get_data failed : NoMethodError : undefined method `[]' for nil:NilClass
```
2024-05-08 16:08:39 +01:00
David Taylor
0f4520867b
DEV: @babel/plugin-proposal-decorators -> decorator-transforms (#25290)
decorator-transforms (https://github.com/ef4/decorator-transforms) is a modern replacement for babel's plugin-proposal-decorators. It provides a decorator implementation using modern browser features, without needing to enable babel's full suite of class feature transformations. This improves the developer experience and performance.

In local testing with Google's 'tachometer' tool, this reduces Discourse's 'init-to-render' time by around 3-4% (230ms -> 222ms).

It reduces our initial gzip'd JS payloads by 3.2% (2.43MB -> 2.35MB), or 7.5% (14.5MB -> 13.4MB) uncompressed.
2024-05-08 10:40:51 +01:00
Martin Brennan
ce2388e40b
FEATURE: Remove "Enable Sidebar" step from setup wizard (#26926)
This keeps coming up in user testing as something
we want to get rid of. The `navigation_menu` setting
has been set to sidebar by default for some time now,
and we are rolling out admin sidebar widely. It just
doesn't make sense to let people turn this off in
the first step of the wizard -- we _want_ people to
use the sidebar.
2024-05-08 10:00:40 +10:00
Régis Hanol
12cba2ce24 PERF: bail out of expensive post validations
Whenever a post already failed "lightweight" validations, we skip all the expensive validations (that cooks the post or run SQL queries) so that we reply as soon as possible.

Also skip validating polls when there's no "[/poll]" in the raw.

Internal ref - t/115890
2024-05-07 18:56:16 +02:00
misaka4e21
ba357dd6cc FIX: ignore SVGs when regenerating missing optimized images.
When running `rake uploads:regenerate_missing_optimized`,
a `Discourse::InvalidAccess` will be raised if an SVG
file is being processed as `OptimizedImage.prepend_decoder!`
doesn't support the svg extension. This commit simply copies
the original SVG file as the thumbnail, just like currently
`OptimizedImage.create_for` does.
2024-05-07 14:39:04 +02:00
Alan Guo Xiang Tan
0b947b6aab
DEV: Improve code comment about when ignored columns can be removed (#26894)
Ignored columns can only be dropped when its associated post-deploy
migration has been promoted to a regular migration. This is so because
Discourse doesn't rely on a schema file system to setup a brand new
database and thus the column information will be loaded by the
application first before the post-deploy migration runs.
2024-05-07 11:06:31 +08:00
Gerhard Schlager
1872047053
DEV: Uploads import script can download files (#26816)
Uploads import script can download files
2024-05-04 22:48:16 +02:00
David Taylor
f230767722
FIX: Serialization of staff_writes_only (#26866) 2024-05-03 14:36:13 -04:00
Daniel Waterworth
a9ca35b671
DEV: Use safer SQL functions for string queries when looking for tags (#26838) 2024-05-02 10:13:45 -05:00
David Taylor
9db5eafb15
PERF: Improve production JS build in low-memory environments (#26849)
- Use 'cheap-source-map' webpack config on low-memory machines

  This results in worse quality sourcemaps in browser dev tools, but it significantly reduces memory use in our webpack build. In approximate local testing it drops from 1100mb to 590mb. This should make the rebuild process on low-memory machines much faster and less likely to trigger OOM errors.

  In development, and on higher-memory machines, the higher-quality 'source-map' option is maintained.

- Disable Webpack's built-in `minimize` feature. Embroider already applies Terser after the webpack build is complete. There is no need to double-minimize the output.

- Update ember-cli-progress-ci to print to stderr instead of stdout. For some reason, pups (used by discourse_docker) buffers the stdout of commands and only prints when they are finished. stderr does not have this same limitation, so switching will mean sysadmins can see the progress of the ember build in real-time.

Given the number of variables it's hard to promise exact numbers. But, in my tests on a DO droplet with 1GB RAM (+2GB swap), this reduced the `ember build` portion of a `./launcher rebuild app` from ~50 minutes to ~15 minutes.
2024-05-02 11:43:59 +01:00
Daniel Waterworth
9f9c7f0a23
FIX: Handle tags with underscores correctly (#26839) 2024-05-01 20:01:39 -05:00
Alan Guo Xiang Tan
a6624af66e
DEV: Add isValidUrl helper function to theme migrations (#26817)
This commit adds a `isValidUrl` helper function to the context in
which theme migrations are ran in. This helper function is to make it
easier for theme developers to check if a string is a valid URL or path
when writing theme migrations. This can be helpful in cases when
migrating a string based setting to `type: objects` which contain `type:
string` properties with URL validations enabled.

This commit also introduces the `UrlHelper.is_valid_url?` method
which actually checks that the URL string is of the valid format instead of
only checking if the URL string is parseable which is what `UrlHelper.relaxed_parse` does
and is not sufficient for our needs.
2024-04-30 16:45:07 +08:00
Régis Hanol
bfc0f3f4cd FIX: prevent duplicate attachments in incoming emails - take 2
This is a follow up of 5fcb7c262d

It was missing the case where secure uploads is enabled, which creates a copy of the upload no matter what.

So this checks for the original_sha1 of the uploads as well when checking for duplicates.
2024-04-30 08:15:07 +02:00
Jan Cernik
9fb888923d
FIX: Do not show hidden posts in search results (#26800) 2024-04-29 12:32:02 -03:00
Régis Hanol
f7a1272fa4 DEV: cleanup custom filters to prevent leaks
Ensures we clean up any custom filters added in the specs to prevent any leaks when running the specs.

Follow up to https://github.com/discourse/discourse/pull/26770#discussion_r1582464760
2024-04-29 16:11:12 +02:00
David Taylor
620f76cec1
DEV: Log original exception/backtrace for RequestTracker errors (#26802) 2024-04-29 09:05:32 +01:00
Martin Brennan
edec941a87
FIX: Better tracking of topic visibility changes (#26709)
This commit introduces a few changes as a result of
customer issues with finding why a topic was relisted.
In one case, if a user edited the OP of a topic that was
unlisted and hidden because of too many flags, the topic
would get relisted by directly changing topic.visible,
instead of going via TopicStatusUpdater.

To improve tracking we:

* Introduce a visibility_reason_id to topic which functions
  in a similar way to hidden_reason_id on post, this column is
  set from the various places we change topic visibility
* Fix Post#unhide! which was directly modifying topic.visible,
  instead we use TopicStatusUpdater which sets visibility_reason_id
  and also makes a small action post
* Show the reason topic visibility changed when hovering the
  unlisted icon in topic status on topic titles
2024-04-29 10:34:46 +10:00
Régis Hanol
803c275bd7 DEV: add support for adding custom status filter
Those can be used in the /filter routes.
2024-04-26 14:04:03 +02:00
Osama Sayegh
2215fa0c8e
FIX: Pass values of objects typed settings to theme migrations (#26751)
This commit fixes a bug in theme settings migrations where values of `objects` typed theme settings aren't passed to migrations even when there are overriding values for those settings. What causes this bug is that, when creating the hash that contains all the overridden settings and will be passed to migrations, the values of `objects` typed settings are incorrectly retrieved from the `value` column (which is always nil for `objects` type) instead of `json_value`. `objects` settings are different from all other types in that they store their values in the `json_value` column and they need to be special-cased when retrieving their values.
2024-04-25 16:39:22 +03:00
David Taylor
2f2da72747
FEATURE: Add experimental tracking of 'real browser' pageviews (#26647)
Our 'page_view_crawler' / 'page_view_anon' metrics are based purely on the User Agent sent by clients. This means that 'badly behaved' bots which are imitating real user agents are counted towards 'anon' page views.

This commit introduces a new method of tracking visitors. When an initial HTML request is made, we assume it is a 'non-browser' request (i.e. a bot). Then, once the JS application has booted, we notify the server to count it as a 'browser' request. This reliance on a JavaScript-capable browser matches up more closely to dedicated analytics systems like Google Analytics.

Existing data collection and graphs are unchanged. Data collected via the new technique is available in a new 'experimental' report.
2024-04-25 11:00:01 +01:00
Alan Guo Xiang Tan
35bc27a36d
FIX: themes:update rake task not rolling back transaction on error (#26750)
This commit fixes a bug in the `themes:update` rake task which resulted
in the ActiveRecord transaction not being rolled back when an error was
encountered. The transaction was first introduced in
7f0682f4f2 which changed a `begin..rescue`
block to `transaction do..rescue`. The problem with that change
prevented the transaction from ever rolling back as the code block
looks something like this:

```
transaction do
  begin
    update_theme
  rescue => e
    # surpress error
  end
end
```

From the transaction's point of view now, it will never rollback even if
an error was encountered when updating the remote theme because it will
never see the error.

Instead we should have done something like this if we wanted to surpress
the errors encountered while still ensuring that the transaction is
rolled back.

```
begin
  transaction do
    update_theme
  end
rescue => e
  # surpress error
end
```
2024-04-25 15:19:23 +08:00
Alan Guo Xiang Tan
9b829216b2
DEV: Add site's DB name in themes:update rake task when printing error (#26749)
This is essential for us to determine which site is encountering an
error while updating remote themes. We are also including the theme's id
because themes can have the same name.
2024-04-25 12:41:14 +08:00
David Taylor
dcd994a9f1
DEV: Drop workbox dependency (#26735)
This service-worker caching functionality was disabled by default in 1c58395bca, and the setting to re-enable was marked as experimental. Now we are dropping all the related logic.
2024-04-24 10:19:12 +01:00
David Taylor
bca855f239
FIX: Improve handling of 'PublicExceptions' when bootstrap_error_pages enabled (#26700)
- Run the CSP-nonce-related middlewares on the generated response

- Fix the readonly mode checking to avoid empty strings being passed (the `check_readonly_mode` before_action will not execute in the case of these re-dispatched exceptions)

- Move the BlockRequestsMiddleware cookie-setting to the middleware, so that it is included even for unusual HTML responses like these exceptions
2024-04-24 09:40:13 +01:00
Penar Musaraj
98d400f7b5
DEV: Refactor discover setting reporting (#26706) 2024-04-23 09:52:01 -04:00
Michael Brown
a5ef7b1999 FIX: in EmailSettingsValidator, unset smtp authentication when there's no user and password
net-smtp 0.5.0 bails when authentication is set without username/password

followup-to: 7b8d60dc, 897be759
2024-04-19 14:02:22 -04:00
Ted Johansson
9e31135eca
FEATURE: Allow users to sign in using LinkedIn OpenID Connect (#26281)
LinkedIn has grandfathered its old OAuth2 provider. This can only be used by existing apps. New apps have to use the new OIDC provider.

This PR adds a linkedin_oidc provider to core. This will exist alongside the discourse-linkedin-auth plugin, which will be kept for those still using the deprecated provider.
2024-04-19 18:47:30 +08:00
Jarek Radosz
285acf0b86
DEV: Add missing svg icons to the svg_sprite list (#26674) 2024-04-18 13:01:48 +02:00
Ted Johansson
f3cad5f3a2
FIX: Correctly re-attach allowed images in activity summary e-mail (#26642)
For e-mails, secure uploads redacts all secure images, and later uses the access control post to re-attached allowed ones. We pass the ID of this post through the X-Discourse-Post-Id header. As the name suggests, this assumes there's only ever one access control post. This is not true for activity summary e-mails, as they summarize across posts.

This adds a new header, X-Discourse-Post-Ids, which is used the same way as the old header, but also works for the case where an e-mail is associated with multiple posts.
2024-04-18 10:27:46 +08:00
Krzysztof Kotlarek
98fc614162
FEATURE: mandatory fields for group site setting (#26612)
Automatically add `moderators` and `admins` auto groups to specific site settings.

In the new group-based permissions systems, we just want to check the user’s groups since it more accurately reflects reality

Affected settings:
- tag_topic_allowed_groups
- create_tag_allowed_groups
- send_email_messages_allowed_groups
- personal_message_enabled_groups
- here_mention_allowed_groups
- approve_unless_allowed_groups
- approve_new_topics_unless_allowed_groups
- skip_review_media_groups
- email_in_allowed_groups
- create_topic_allowed_groups
- edit_wiki_post_allowed_groups
- edit_post_allowed_groups
- self_wiki_allowed_groups
- flag_post_allowed_groups
- post_links_allowed_groups
- embedded_media_post_allowed_groups
- profile_background_allowed_groups
- user_card_background_allowed_groups
- invite_allowed_groups
- ignore_allowed_groups
- user_api_key_allowed_groups
2024-04-18 08:53:52 +10:00
Martin Brennan
380e5ca6cb
DEV: Move more service code to core (#26613)
This is to enable :array type attributes for Contract
attributes in services, this is a followup to the move
of services from chat to core here:

cab178a405

Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
2024-04-12 13:14:19 +02:00
Gerhard Schlager
4d045bfc61
DEV: Import script should insert more data into user_stats table (#26551)
This SQL tries to insert as much data as possible into the `user_stats` table by either calculating or by approximating stats based on existing. It also fixes an error in the calculation of `reply_count`which mistakenly contained all posts, not just replies.

This change also disables some steps in the `import:ensure_consistency` rake task by setting the `SKIP_USER_STATS` env variable. Otherwise, the rake task will overwrite the calculated data in the `user_stats` table with inaccurate data. I'm not changing or removing the logic from the rake task yet because other bulk import scripts seem to depend on it.
2024-04-11 14:05:21 +02:00
Martin Brennan
b7a2d29b7b
DEV: Introduce post_should_secure_uploads? plugin modifier (#26508)
This modifier allows plugins to alter the outcome of
`should_secure_uploads?` on a Post record, for cases when
plugins need post-attached uploads to always be secure (or
not secure) in specific scenarios.
2024-04-10 12:02:44 +10:00
Alan Guo Xiang Tan
e2ced85757
DEV: Allow enum typed theme objects property to be optional (#26571)
This commit changes enum typed theme objects property to be optional.
Previously, an enum typed property is always required but we have found
that this might not be ideal so we want to change it.
2024-04-09 11:26:24 +08:00
Martin Brennan
0d0dbd391a
DEV: Rename with_secure_uploads? to should_secure_uploads? on Post (#26549)
This method name is a bit confusing; with_secure_uploads implies
it may return a block or something with the uploads of the post,
and has_secure_uploads implies that it's checking whether the post
is linked to any secure uploads.

should_secure_uploads? communicates the true intent of this method --
which is to say whether uploads attached to this post should be
secure or not.
2024-04-09 13:23:11 +10:00
Alan Guo Xiang Tan
6edadeab27
FIX: DiscourseIpInfo.mmdb_download incorrectly joining URLs (#26545)
This commit changes `DiscourseIpInfo.mmdb_download` to use `File.join`
instead of `URI.join` when `GlobalSetting.maxmind_mirror_url` has been
configured. This is necessary because `URI.join` does not work the way I
expect it to work when I implemented it previously.

`URI.join("http://www.example.com/mirror", "test.tar.gz") results in
`http://www.example.com/test.tar.gz` instead of our expected
`http://www.exmaple.com/mirror/test.tar.gz`. For our simple use case
here, `File.join` is sufficient.
2024-04-08 10:27:40 +08:00
Blake Erickson
8da49b5664
FEATURE: Add message to bulk close topics (#26535)
This will allow you to type a single message when bulk closing topics
that will be applied to every topic, that way they all have the same
message.
2024-04-05 14:56:52 -06:00
Rafael dos Santos Silva
dd83a07550
FEATURE: Hide summarization in PMs (#26532) 2024-04-05 12:12:59 -03:00
Jan Cernik
cab178a405
DEV: Move chat service objects into core (#26506) 2024-04-04 10:57:41 -03:00
Gerhard Schlager
82c62fe44f
DEV: Correctly pluralize error messages (#26469) 2024-04-04 15:02:09 +02:00
David Taylor
0ad4034c58
DEV: Add a note about the version bump rake tasks (#26512) 2024-04-04 11:59:29 +01:00
Gerhard Schlager
689d2754f6
DEV: Bump the version to 3.3.0.beta2-dev (#26499)
This was accidentally omitted during the beta1 release process
2024-04-03 22:08:04 +01:00
Vinoth Kannan
9dc6325821
DEV: add logo URL and locale details to the Discover stats. (#26320)
We will be collecting the logo URL and the site's default locale values along with existing basic details to display the site on the Discourse Discover listing page. It will be included only if the site is opted-in by enabling the "`include_in_discourse_discover`" site setting.

Also, we no longer going to use `about.json` and `site/statistics.json` endpoints retrieve these data. We will be using only the `site/basic-info.json` endpoint.
2024-04-04 00:22:28 +05:30
Osama Sayegh
3d4faf3272
FEATURE: Merge discourse-automation (#26432)
Automation (previously known as discourse-automation) is now a core plugin.
2024-04-03 18:20:43 +03:00
David Taylor
929b4f89d6
DEV: Send proper 'stop' notification in turbo_rspec (#26488)
Doesn't actually seem to be used by any of our formatters, but let's send the proper data anyway for future-proofing. Followup to ff6cb1bc05 and 8098876bfa
2024-04-03 14:00:47 +01:00
David Taylor
ff6cb1bc05
DEV: Fix turbo_rspec formatters by sending real 'Start' notification (#26487) 2024-04-03 13:27:39 +01:00
David Taylor
8098876bfa
DEV: Restore RSpec 'documentation' output, but collapse in CI (#26485)
This reverts commit ef895f1c32 and 57df0d526e, but adds GitHub actions line grouping to the verbose output to reduce scrolling.
2024-04-03 10:54:55 +01:00
David Taylor
1e42e86601
DEV: Restore qunit 'tap' reporter, but collapse verbose output in CI (#26484)
This reverts commit 5adfb299ac, but uses GitHub actions grouping to achieve the same result: less scrolling to see failures.
2024-04-03 10:22:20 +01:00
Krzysztof Kotlarek
ba04fc6a01
FEATURE: ignore manually deactivated users when purging (#26478)
When a user is manually deactivated, they should not be deleted by our background job that purges inactive users.

In addition, site settings keywords should accept an array of keywords.
2024-04-03 14:06:31 +11:00
Penar Musaraj
1eb70973a2
DEV: allow themes to render their own custom homepage (#26291)
This PR adds a theme modifier and route so that custom themes can opt to show their own homepage. See PR description for example usage.
2024-04-02 11:05:08 -04:00
Vinoth Kannan
cd43985de0
UX: rename the word "Upgrade" to "Update" in the UI. (#26396)
In this PR, all references in the UI to the word "`upgrade`" are changed to "`update`". This is to differentiate the update process in self-hosted sites from the plan "upgrade" process in hosted sites.

Follow-up to the PR: https://github.com/discourse/docker_manager/pull/208
2024-04-02 20:04:37 +05:30
Alan Guo Xiang Tan
9182501366
DEV: Introduce maxmind_mirror_url GlobalSetting (#26458)
Why this change?

This allows downloading the MaxMind databases from a mirror in cases
where downloading directly from MaxMind's API endpoint is problematic
due to API limits.
2024-04-02 14:53:53 +08:00
Alan Guo Xiang Tan
972db687fe
FIX: maxmind:refresh does not respect refresh_maxmind_db_during_precompile_days (#26457)
Why this change?

We currently support `GlobalSetting.refresh_maxmind_db_during_precompile_days` which
should cache the maxmind databases on disk for the configured number of
days before it downloads the databases from maxmind again via the API.
This was previously added to help us avoid hitting the API rate limit from maxmind.
However, there was a bug in the `copy_maxmind` when we copied the latest
downloaded database to the cache directory. In particular, `FileUtils.cp` was called with
`preserve: true` which would preserve the modified time of the file
being copied. This is problematic because download the database from
maxmind on 2 April 2024 can give us a file with an mtime of 29 March
2024. If `GlobalSetting.refresh_maxmind_db_during_precompile_days` is
set to `2` for example, the cache will never be used since we will
think that the file has been downloaded for more than 2 days in our
checks.

What is the fix here?

While we want to preserve the owner and group of the file, we do not
want to preserve the modified time and hence we will call
`FileUtils.touch` when copying the file.
2024-04-02 14:25:05 +08:00
Alan Guo Xiang Tan
a84757fd91
FIX: Error not being raised for required typed categories property (#26443)
Why this change?

For a schema like this:

```
schema = {
  name: "section",
  properties: {
    category_property: {
      type: "categories",
      required: true,
    },
  },
}
```

When the value of the property is set to an empty array, we are not
raising an error which we should because the property is marked as
required.
2024-04-01 10:11:40 +08:00
carehabit
11877f3b9c
DEV: remove repetitive words (#26439) 2024-04-01 06:23:21 +08:00
Alan Guo Xiang Tan
a670d6d4af
DEV: Change group type to groups type for theme object schema (#26417)
Why this change?

This is a follow-up to 86b2e3a.

Basically, we want to allow people to select more than 1 group as well.

What does this change do?

1. Change `type: group` to `type: groups` and support `min` and `max`
   validations for `type: groups`.

2. Fix the `<SchemaThemeSetting::Types::Groups>` component to support the
   `min` and `max` validations and switch it to use the `<GroupChooser>` component
   instead of the `<ComboBoxComponent>` component which previously only supported
   selecting a single group.
2024-03-28 22:05:48 +08:00
Ted Johansson
0c875cb4d5
DEV: Make problem check registration more explicit (#26413)
Previously the problem check registry simply looked at the subclasses of ProblemCheck. This was causing some confusion in environments where eager loading is not enabled, as the registry would appear empty as a result of the classes never being referenced (and thus never loaded.)

This PR changes the approach to a more explicit one. I followed other implementations (bookmarkable and hashtag autocomplete.) As a bonus, this now has a neat plugin entry point as well.
2024-03-28 14:00:47 +08:00
Angus McLeod
7dc552c9cc
DEV: Add import_embed_unlisted site setting (#26222) 2024-03-27 08:57:43 -04:00
Osama Sayegh
0b34b4cbe9
DEV: Better error message when user is not allowed to user tags (#26380)
When a topic fails to be created due to the user not having permission to add tags to the topic, the error message that you get is `There was an error tagging the topic` which is very generic and doesn't explain where/what the problem is. This commit adds a clearer error message for this scenario.
2024-03-27 12:57:10 +03:00
Osama Sayegh
72c4709a5a
FIX: Skip tags-related validations when the skip_validations option is present (#26379)
The `TopicCreator` class has a `skip_validations` option that can force-create a topic without performing permission checks or validation rules. However, at the moment it doesn't skip validations that are related to tags, so topics that are created by the system or by some scrip can still fail if they use tags. This commit makes the `TopicCreator` class skip all tags-related checks if the `skip_validations` is specified.

Internal topic: t/124280.
2024-03-27 12:56:21 +03:00
Martin Brennan
8e08a3b31f
DEV: Use caller for plugin_file_from_fixtures (#26387)
Followup 0bbca318f2,
rather than making developers provide the plugin path
name (which may not always be the same depending on
dir names and git cloning etc) we can infer the plugin
dir from the caller in plugin_file_from_fixtures
2024-03-27 14:12:51 +11:00
Alan Guo Xiang Tan
476d91d233
DEV: Change category type to categories type for theme object schema (#26339)
Why this change?

This is a follow-up to 86b2e3aa3e.

Basically, we want to allow people to select more than 1 category as well.

What does this change do?

1. Change `type: category` to `type: categories` and support `min` and `max`
   validations for `type: categories`.

2. Fix the `<SchemaThemeSetting::Types::Categories>` component to support the
   `min` and `max` validations and switch it to use the `<CategorySelector>` component
   instead of the `<CategoryChooser>` component which only supports selecting one category.
2024-03-27 10:54:30 +08:00
Alan Guo Xiang Tan
957b945d25
DEV: Improve output of maxminddb:refresh (#26386)
Why this change?

We want to know when nothing was downloaded because stuff has been
cached on disk
2024-03-27 09:50:49 +08:00
Michael Brown
680f1ff19c FIX: Add content-type header to rate limiter error
It's best to always set a content-type header and one was missing here.
2024-03-26 12:39:42 -04:00
David Taylor
a8d20f92fb
FEATURE: Add page number to page titles for crawlers (#26367)
At the moment, all topic `?page=` views are served with exactly identical page titles. If you search for something which is mentioned many times in the same Discourse topic, this makes for some very hard-to-understand search results! All the result titles are exactly the same, with no indication of why there are multiple results showing.

This commit adds a `- Page #` suffix to the titles in this situation. This lines up with our existing strategy for topic-list pagination.
2024-03-26 15:19:00 +00:00
David Taylor
3329484e2d
FEATURE: Simplify crawler content for non-canonical post URLs (#26324)
When crawlers visit a post-specific URL like `/t/-/{topic-id}/{post-number}`, we use the canonical to direct them to the appropriate crawler-optimised paginated view (e.g. `?page=3`).

However, analysis of google results shows that the post-specific URLs are still being included in the index. Google doesn't tell us exactly why this is happening. However, as a general rule, 'A large portion of the duplicate page's content should be present on the canonical version'.

In our previous implementation, this wasn't 100% true all the time. That's because a request for a post-specific URL would include posts 'surrounding' that post, and won't exactly conform to the page boundaries which are used in the canonical version of the page. Essentially: in some cases, the content of the post-specific pages would include many posts which were not present on the canonical paginated version.

This commit aims to resolve that problem by simplifying the implementation. Instead of rendering posts surrounding the target post_number, we will only render the target post, and include a link to 'show post in topic'. With this new implementation, 100% of the post-specific page content will be present on the canonical paginated version, which will hopefully mean google reduces their  indexing of the non-canonical post-specific pages.
2024-03-26 15:18:46 +00:00
Loïc Guitaut
9c9526f0a8 DEV: Use Nimbus font instead of Helvetica
To generate letter avatars, we’re currently using the ImageMagick suite
and we’re using the Helvetica font family. However, that font isn’t
shipped anymore in the latest stable version of Debian (Bookworm).
Instead it seems to have been replaced by the Nimbus font. The rendering
is extremely similar (not to say it’s the same thing) so it shouldn’t be
noticeable.

That change is necessary for us to upgrade our docker images to Debian
Bookworm.
2024-03-26 14:42:40 +01:00
Martin Brennan
5adfb299ac
DEV: Change QUnit reporters to dot (#26360)
This is so the CI output on GitHub actions isn't showing
tons and tons of unnecessary log data every time you want
to see the important thing, which is the actual test failure.
2024-03-26 16:54:22 +10:00
Jan Cernik
532639f636
UX: Add threads fab icon (#26349) 2024-03-26 08:30:27 +08:00
Jay Pfaffman
63f6c1e81a
FIX: Allow redirect when downloading maxmind database (#26333)
Per https://dev.maxmind.com/geoip/release-notes/2024#presigned-urls-for-database-downloads 

MaxMind users who download databases should make sure that their HTTP client follows redirects and there are no proxy or firewall settings that would block requests to the host we are redirecting to.
2024-03-26 09:39:09 +11:00
David Taylor
8f44b805f6
DEV: Use -prod flag when building production assets (#26344)
We were previously using the `EMBER_ENV=production` environment variable, which appears to produce the same output. But, some parts of ember-cli don't seem to support it, which leads to a confusing 'Environment: development' being printed on the console.

This commit adds `-prod` by default, which is the more common way to invoke ember-cli for production builds.
2024-03-25 16:56:28 +00:00
Alan Guo Xiang Tan
86b2e3aa3e
DEV: Change tag type to tags type for theme object schema (#26315)
Why this change?

While working on the tag selector for the theme object editor, I
realised that there is an extremely high possibility that users might want to select
more than one tag. By supporting the ability to select more than one
tag, it also means that we get support for a single tag for free as
well.

What does this change do?

1. Change `type: tag` to `type: tags` and support `min` and `max`
   validations for `type: tags`.

2. Fix the `<SchemaThemeSetting::Types::Tags>` component to support the
   `min` and `max` validations
2024-03-22 15:32:00 +08:00
Alan Guo Xiang Tan
dfc406fdc2
FIX: Validate tags using Tag#name instead of Tag#id in ThemeSettingsObjectValidator (#26314)
Why this change?

Fortunately or unfortunately in Discourse core, we mainly use `Tag#name`
to look up tags and not its id. This assumption is built into the
frontend as well so we need to use the tag's name instead of the id
here.
2024-03-22 11:05:16 +08:00
Martin Brennan
61bd7d5d11
FIX: Anon users could not edit their own posts (#26283)
Followup 3094f32ff5,
this fixes an issue with the logic in this commit where
we were returning false if any of the conditionals here
were false, regardless of the type of `obj`, where we should
have only done this if `obj` was a `PostAction`, which lead
us to return false in cases where we were checking if the
user could edit their own post as anon.
2024-03-22 08:12:12 +10:00
Bianca Nenciu
a6e06915c4
FIX: Serialize parent categories first (#26294)
When categories are loaded by the frontend, the parent category is
looked up by ID and the `parentCategory` is set with the result. If the
categories returned are not in order, the parent category may miss.
2024-03-21 19:51:41 +02:00
Alan Guo Xiang Tan
0aa92500aa
DEV: ENV variable for brotli quality in assets:precompile rake task (#26286)
Why this change?

This ENV allows the brotli compression quality to be configurable such
that one can opt for a higher/lower level of compression based on their
preferences.
2024-03-21 19:07:35 +08:00
Martin Brennan
70f7c0ee6f
FEATURE: More flexible admin plugin config nav definition (#26254)
This commit changes the API for registering the plugin config
page nav configuration from a server-side to a JS one;
there is no need for it to be server-side.

It also makes some changes to allow for 2 different ways of displaying
navigation for plugin pages, depending on complexity:

* TOP - This is the best mode for simple plugins without a lot of different
  custom configuration pages, and it reuses the grey horizontal nav bar
  already used for admins.
* SIDEBAR - This is better for more complex plugins; likely this won't
  be used in the near future, but it's readily available if needed

There is a new AdminPluginConfigNavManager service too to manage which
plugin the admin is actively viewing, otherwise we would have trouble
hiding the main plugin nav for admins when viewing a single plugin.
2024-03-21 13:42:06 +10:00
Ella E
ef2a9da8e7
UX: Add chevron icons for expanding and collapsing all toggles in the admin sidebar (#26246) 2024-03-21 10:25:14 +11:00
Jarek Radosz
6d137a41c2
DEV: Remove the last transpile_js mentions (#26263) 2024-03-20 15:32:37 +01:00
David Taylor
aba0df7cf7
FEATURE: Increase default DiscourseConnect session length (#26261)
For some identity providers, 10 minutes isn't much time for a user to complete authentication/registration on the identity provider. Increasing the default to 30 minutes should help in those situations. The nonce is still tied to a single browser session, so there is no material impact on security.
2024-03-20 14:02:12 +00:00
Sam
34a14112a7
FIX: remove "fake" mentions from extract_mentions (#26253)
```
<a class="mention" href="/u/test1">bsam</a>
```

Is not a mention of the user sam. We expect an @ in front always.
2024-03-20 12:20:15 +11:00
Martin Brennan
78bafb331a
FEATURE: Allow site settings to be edited throughout admin UI (#26154)
This commit makes it so the site settings filter controls and
the list of settings input editors themselves can be used elsewhere
in the admin UI outside of /admin/site_settings

This allows us to provide more targeted groups of settings in different
UI areas where it makes sense to provide them, such as on plugin pages.
You could open a single page for a plugin where you can see information
about that plugin, change settings, and configure it with custom UIs
in the one place.

In future we will do this in "config areas" for other parts of the
admin UI.
2024-03-18 08:50:39 +10:00
Nat
ad7e3e04f3
Bump version to v3.3.0.beta1 2024-03-15 15:01:58 +08:00
Bianca Nenciu
b425fbc2a2
SECURITY: Generate more category CSS on client
This commit moves the generation of category background CSS from the
server side to the client side. This simplifies the server side code
because it does not need to check which categories are visible to the
current user.
2024-03-15 14:24:11 +08:00
David Taylor
2546817d07
FIX: correctly strip unneeded csp directives under strict-dynamic (#26180) 2024-03-14 18:50:09 +00:00
Rafael dos Santos Silva
679a773411
FIX: Handle nil post_search_data for search result (#26179)
Since we are introducing new ways to search in Discourse, like the AI
semantic search using embeddings, posts can be part of a search result
list without having any search data.

Since the code path already handles this, we only need to add a safety
check when accessing the post_search_data.
2024-03-14 14:40:46 -03:00
Ted Johansson
ea5c3a3bdc
DEV: Move non scheduled problem checks to classes (#26122)
In AdminDashboardData we have a bunch of problem checks implemented as methods on that class. This PR absolves it of the responsibility by promoting each of those checks to a first class ProblemCheck. This way each of them can have their own priority and arbitrary functionality can be isolated in its own class.

Think "extract class" refactoring over and over. Since they were all moved we can also get rid of the @@problem_syms class variable which was basically the old version of the registry now replaced by ProblemCheck.realtime.

In addition AdminDashboardData::Problem value object has been entirely replaced with the new ProblemCheck::Problem (with compatible API).

Lastly, I added some RSpec matchers to simplify testing of problem checks and provide helpful error messages when assertions fail.
2024-03-14 10:55:01 +08:00
David Taylor
551c6022dc
DEV: Introduce rake task to validate discourse-compatibility file (#26158) 2024-03-13 13:57:41 +00:00
Alan Guo Xiang Tan
1f71db426e
DEV: Remove cache PG connection type map freedom patch (#26153)
Why this change?

Previously, we identified that ActiveRecord's PostgreSQL adapter
executes 3 db queries each time a new connection is created. The 3 db
queries was identified when we looked at the `pg_stats_statement` table
on one of our multisite production cluster. At that time, the hypothesis
is that because we were agressively reaping and creating connections,
the db queries executed each time a connection is created is wasting
resources on our database servers. However, we didn't see any the needle
move much on our servers after deploying the patch so we have decided to
drop this patch as it makes it harder for us to upgrade ActiveRecord in
the future.
2024-03-13 13:28:06 +08:00
Martin Brennan
4e7a75a7ec
DEV: Single admin plugin page for consistent admin plugin UX (#26024)
This commit adds new plugin show routes (`/admin/plugins/:plugin_id`) as we move
towards every plugin having a consistent UI/landing page.

As part of this, we are introducing a consistent way for plugins
to show an inner sidebar in their config page, via a new plugin
API `register_admin_config_nav_routes`

This accepts an array of links with a label/text, and an
ember route. Once this commit is merged we can start the process
of conforming other plugins to follow this pattern, as well
as supporting a single-page version of this for simpler plugins
that don't require an inner sidebar.

Part of /t/122841 internally
2024-03-13 13:15:12 +10:00
Krzysztof Kotlarek
de00c9a3d3
EXPERIMENTAL: restructure admin sidebar menu (#26047)
New admin sidebar menu
2024-03-13 10:18:08 +11:00
David Taylor
127214c613
UX: Improve error handling for DiscourseConnect (#26140)
Previously, if the sso= payload was invalid Base64, but signed correctly, there would be no useful log or error. This commit improves things by:

- moving the base64 check before the signature checking so that it's properly surfaced
- split the ParseError exception into PayloadParseError and SignatureError
- add user-facing errors for both of those
- add/improve spec for both
2024-03-12 16:16:04 +00:00
Ted Johansson
2211ffa851
DEV: Move problem checks to app directory (#26120)
There are a couple of reasons for this.

The first one is practical, and related to eager loading. Since /lib is not eager loaded, when the application boots, ProblemCheck["identifier"] will be nil because the child classes aren't loaded.

The second one is more conceptual. There turns out to be a lot of inter-dependencies between the part of the problem check system that live in /app and the parts that live in /lib, which probably suggests it should all go in /app.
2024-03-11 13:36:22 +08:00
Alan Guo Xiang Tan
8d4f405da4
DEV: Allow typed objects theme settings to be saved via settings editor (#26100)
Why this change?

On the `/admin/customize/themes/<:id>` route, we allow admins to edit
all settings via a settings editor. Prior to this change, trying to edit
and save a typed objects theme settings will result in an error on the
server.
2024-03-11 08:42:12 +08:00
Alan Guo Xiang Tan
7d8dd0d8e3
UX: Show loading spinner while loading dependencies for ace-editor (#26099)
Why this change?

On a slow network, using the `AceEditor` component will result in a blob
of text being shown first before being swapped out with the `ace.js`
editor after it has completed loading.

There is also a problem when setting the theme for the editor which
would result in a "flash" as reported in
https://github.com/ajaxorg/ace/issues/3286. To avoid this, we need to
load the theme js file before displaying the editor.

What does this change do?

1. Adds a loading spinner and set the `div.ace` with a `.hidden` class.
2. Once all the relevant scripts and initialization is done, we will
   then remove the loading spinner and remove `div.ace`.
2024-03-11 06:56:17 +08:00
Osama Sayegh
f8964f8f8f
FIX: Allow nil for properties values when they're not required (#26112)
Properties of schema theme settings that are not marked `required: true` should accept nil as a value.
2024-03-09 14:25:30 +03:00
David Taylor
4b813427e7
DEV: Repair RSpec full_cause_backtrace under bin/turbo_rspec (#26087)
This commit fixes two issues:

1. The wrong exception was being printed as the 'cause' in turbo_rspec output. This was happening because RSpec [expects exceptions to be subclasses of `Exception`](d6e320dc11/lib/rspec/core/formatters/exception_presenter.rb (L102)). This commit resolves the issue by replacing the `FakeException` `Struct` with a subclass of `Exception`.

2. The `full_cause_backtrace` option we set in `rails_helper.rb` does not carry through to the RSpec formatters running in the turbo_rspec reporter process. To fix that, this commit duplicates the necessary config in `lib/turbo_tests.rb`.

Example before - note that the cause is a duplicate of the original exception, and only has three lines of backtrace:

```
     Failure/Error: raise capybara_timeout_error

     CapybaraTimeoutExtension::CapybaraTimedOut:
       This spec passed, but capybara waited for the full wait duration (4s) at least once. This will slow down the test suite. Beware of negating the result of selenium's RSpec matchers.

     [Screenshot Image]: /Users/david/discourse/discourse/tmp/capybara/failures_r_spec_example_groups_glimmer_header_when_cmd_f_keyboard_shortcut_pressed_when_within_a_topic_with_less_than20_posts_does_not_open_search_484.png

     ~~~~~~~ JS LOGS ~~~~~~~
     ~~~~~ END JS LOGS ~~~~~

     # ./spec/rails_helper.rb:372:in `block (3 levels) in <top (required)>'
     # ./spec/rails_helper.rb:472:in `block (2 levels) in <top (required)>'
     # /Users/david/.rvm/gems/ruby-3.2.1/gems/webmock-3.23.0/lib/webmock/rspec.rb:39:in `block (2 levels) in <top (required)>'
     # ------------------
     # --- Caused by: ---
     # CapybaraTimeoutExtension::CapybaraTimedOut:
     #   This spec passed, but capybara waited for the full wait duration (4s) at least once. This will slow down the test suite. Beware of negating the result of selenium's RSpec matchers.
     #   ./spec/rails_helper.rb:372:in `block (3 levels) in <top (required)>'
     #   ./spec/rails_helper.rb:472:in `block (2 levels) in <top (required)>'
     #   /Users/david/.rvm/gems/ruby-3.2.1/gems/webmock-3.23.0/lib/webmock/rspec.rb:39:in `block (2 levels) in <top (required)>'
```

After - note correct causing exception, and the full backtrace 🎉

```
     Failure/Error: raise capybara_timeout_error

     CapybaraTimeoutExtension::CapybaraTimedOut:
       This spec passed, but capybara waited for the full wait duration (4s) at least once. This will slow down the test suite. Beware of negating the result of selenium's RSpec matchers.

     [Screenshot Image]: /Users/david/discourse/discourse/tmp/capybara/failures_r_spec_example_groups_glimmer_header_when_cmd_f_keyboard_shortcut_pressed_when_within_a_topic_with_less_than20_posts_does_not_open_search_61.png

     ~~~~~~~ JS LOGS ~~~~~~~
     ~~~~~ END JS LOGS ~~~~~

     # ./spec/rails_helper.rb:372:in `block (3 levels) in <top (required)>'
     # ./spec/rails_helper.rb:472:in `block (2 levels) in <top (required)>'
     # /Users/david/.rvm/gems/ruby-3.2.1/gems/webmock-3.23.0/lib/webmock/rspec.rb:39:in `block (2 levels) in <top (required)>'
     # ------------------
     # --- Caused by: ---
     # Capybara::ExpectationNotMet:
     #   expected to find css ".search-menu .search-menu-panel" but there were no matches
     #   /Users/david/.rvm/gems/ruby-3.2.1/gems/capybara-3.40.0/lib/capybara/node/matchers.rb:112:in `block in assert_selector'
     #   /Users/david/.rvm/gems/ruby-3.2.1/gems/capybara-3.40.0/lib/capybara/node/matchers.rb:869:in `block in _verify_selector_result'
     #   /Users/david/.rvm/gems/ruby-3.2.1/gems/capybara-3.40.0/lib/capybara/node/base.rb:84:in `synchronize'
     #   ./spec/rails_helper.rb:345:in `synchronize'
     #   /Users/david/.rvm/gems/ruby-3.2.1/gems/capybara-3.40.0/lib/capybara/node/matchers.rb:868:in `_verify_selector_result'
     #   /Users/david/.rvm/gems/ruby-3.2.1/gems/capybara-3.40.0/lib/capybara/node/matchers.rb:110:in `assert_selector'
     #   /Users/david/.rvm/gems/ruby-3.2.1/gems/capybara-3.40.0/lib/capybara/node/matchers.rb:39:in `block in has_selector?'
     #   /Users/david/.rvm/gems/ruby-3.2.1/gems/capybara-3.40.0/lib/capybara/node/matchers.rb:902:in `make_predicate'
     #   /Users/david/.rvm/gems/ruby-3.2.1/gems/capybara-3.40.0/lib/capybara/node/matchers.rb:39:in `has_selector?'
     #   /Users/david/.rvm/gems/ruby-3.2.1/gems/capybara-3.40.0/lib/capybara/session.rb:774:in `has_selector?'
     #   ./spec/system/page_objects/pages/search.rb:46:in `has_search_menu_visible?'
     #   ./spec/system/header_spec.rb:206:in `block (4 levels) in <main>'
     #   ./spec/rails_helper.rb:472:in `block (2 levels) in <top (required)>'
     #   /Users/david/.rvm/gems/ruby-3.2.1/gems/webmock-3.23.0/lib/webmock/rspec.rb:39:in `block (2 levels) in <top (required)>'
```
2024-03-07 18:35:46 +00:00
David Taylor
92d357f91a
FEATURE: Enable strict-dynamic Content-Security-Policy by default (#26051)
Ref https://meta.discourse.org/t/298172 and https://meta.discourse.org/t/295603
2024-03-07 15:20:31 +00:00
Alan Guo Xiang Tan
b1495884eb
PERF: Avoid saving ThemeSetting twice when creating new db override (#26076)
Why this change?

When creating a new theme setting that does not have a corresponding row
in the `theme_settings` table, we end up writing to the database twice
because `ActiveRecord::Base#save!` is called once before the `value`
or `json_value` column is updated again with another database query with
another call to `ActiveRecord::Base#save!`.

What does this change do?

Adds the column to be updated to argument for the `ActiveRecord::Base#create!`
method call so that we only have one write query to the database.
2024-03-07 16:38:11 +08:00
Blake Erickson
0464ddcd9b
FEATURE: Bulk Silent Close Topics (#26043)
Using the new bulk select dropdown you can now choose to bulk close
topics silently.
2024-03-06 14:08:49 -07:00
David Taylor
dfc6bb4029
DEV: Remove direct minitest dependency to appease ruby-lsp (#26056)
Having minitest as a direct dependency causes ruby-lsp to use it as our test runner (per https://github.com/Shopify/ruby-lsp/blob/d1da8858a1/lib/ruby_lsp/requests/support/dependency_detector.rb#L40-L55). This makes VSCode's test explorer incorrectly display Minitest 'run' buttons above all our tests.

We were only using it in `emoji.rake`... and that wasn't even working with the latest version of Minitest. This commit refactors `emoji.rake` to work without minitest, and removes the dependency.
2024-03-06 15:41:14 +00:00
David Taylor
d1a2596889
DEV: Allow CSP nonce_placeholder to be generated outside Rails (#26052)
Sometimes we add scripts outside of Rails. This commit provides a way to generate a nonce placeholder even if you don't have access to an ApplicationController instance.
2024-03-06 13:01:32 +00:00
Joffrey JAFFEUX
00a5f67daf
FIX: brings back discourse-sidebar icon (#26050)
This has been removed from 76953cc356 by mistake
2024-03-06 12:16:28 +01:00
Joffrey JAFFEUX
76953cc356
FEATURE: allows to force a thread (#25987)
Forcing a thread will work even in channel which don't have `threading_enabled` or in direct message channels.

For now this feature is only available through the `ChatSDK`:

```ruby
ChatSDK::Message.create(in_reply_to_id: 1, guardian: guardian, raw: "foo bar baz", channel_id: 2, force_thread: true)
```
2024-03-06 12:03:42 +01:00
Ella E
f935b2ca6d
UX: Add sidebar icon (#26046) 2024-03-06 18:24:37 +10:00
Alan Guo Xiang Tan
94b09f3331
DEV: Open theme settings objects editor from admin customize theme page (#26006)
Why this change?

The `/admin/customize/themes/:id/schema/name` route is a work in
progress but we want to be able to start navigating to it from the
`/admin/customize/themes/:id` route.

What does this change do?

1. Move `adminCustomizeThemes.schema` to a child route of
   `adminCustomizeThemes.show`. This is because we need the model
   from the parent route and if it isn't a child route we end up
   having to load the theme model again from the server.

1. Add the `objects_schema` attribute to `ThemeSettingsSerializer`

1. Refactor `SiteSettingComponent` to be able to render a button
   so that we don't have to hardcode the button rendering into the
   `SiteSettings::String` component
2024-03-06 08:24:29 +08:00
Blake Erickson
40b707a690
FEATURE: Add onebox for loom (#26016)
Loom share links will now onebox and use the embedded loom player.
2024-03-04 15:12:08 -07:00
Alan Guo Xiang Tan
955339668b
FIX: ThemeSettingsObjectValidator not allowing URL paths for string (#26005)
Why this change?

Prior this change, we were using `URI.regexp` which was too strict as it
doesn't allow a URL path.

What does this change do?

Just parse the string using `URI.parse` and if it doesn't raise an error
we consider the string to be a valid URL
2024-03-04 13:22:14 +08:00
Martin Brennan
eca10e56b8
FEATURE: Allow specific groups to view raw email (#26003)
When a post is created by an incoming email, we show
an envelope icon on it which then opens a modal with the
raw email contents. Previously this was staff (admin+mod)
only, but now this commit adds the `view_raw_email_allowed_groups`
site setting, so any group can be added to give users permission
to see this.
2024-03-04 13:48:16 +10:00
Krzysztof Kotlarek
41f78b31a9
FIX: down downgrade trust level if all requirements are met. (#25953)
Currently, the trust level method  is calculating trust level based on maximum value from:
- locked trust level
- group automatic trust level
- previously granted trust level by admin

https://github.com/discourse/discourse/blob/main/lib/trust_level.rb#L33

Let's say the user belongs to groups with automatic trust level 1 and in the meantime meets all criteria to get trust level 2.

Each time, a user is removed from a group with automatic trust_level 1, they will be downgraded to trust_level 1 and promoted to trust_level 2

120a2f70a9/lib/promotion.rb (L142)

This will cause duplicated promotion messages.

Therefore, we have to check if the user meets the criteria, before downgrading.
2024-03-04 09:30:30 +11:00
Régis Hanol
5fcb7c262d
FIX: prevents duplicate attachments in incoming emails (#25986)
## What?

Depending on the email software used, when you reply to an email that has some attachments, they will be sent along, since they're part of the embedded (replied to) email.

When Discourse processes the reply as an incoming email, it will automatically add all the (valid) attachments at the end of the post. Including those that were sent as part of the "embedded reply".

This generates posts in Discourse with duplicate attachments 🙁

## How?

When processing attachments of an incoming email, before we add it to the bottom of the post, we check it against all the previous uploads in the same topic. If there already is an `Upload` record, it means that it's a duplicate and it is _therefore_ skipped.

All the inline attachments are left untouched since they're more likely new attachments added by the sender.
2024-03-01 18:38:49 +01:00
Gerhard Schlager
6847ed5be6
FIX: import:rebake_uncooked_* jobs couldn't be run in parallel (#25969)
If those jobs were started multiple times each process would have rebaked the same posts.
2024-02-29 19:54:19 +01:00
Daniel Waterworth
f0baa0ddfe
DEV: Prefer defer_get_set over get/set on DistributedCache (#25967) 2024-02-29 11:10:24 -06:00
Martin Brennan
5f119c57e8
DEV: Suppress verbose command failure output in plugin:turbo_spec (#25952)
Before this change, if the "Plugins backend" task on GitHub CI
failed, we would get a huge amount of extra output at the end
just to show the command that rake ran which failed (the bin/turbo_rspec
command). This is useless and just makes it hard to see the failing
specs. If you need the full command, it's already output at the
top of the "Plugins backend" task in the GitHub CI.
2024-02-29 14:35:31 +10:00
Martin Brennan
df4197c8b8
FIX: Show deleted bookmark reminders in user bookmarks menu (#25905)
When we send a bookmark reminder, there is an option to delete
the underlying bookmark. The Notification record stays around.
However, if you want to filter your notifications user menu
to only bookmark-based notifications, we were not showing unread
bookmark notifications for deleted bookmarks.

This commit fixes the issue _going forward_ by adding the
bookmarkable_id and bookmarkable_type to the Notification data,
so we can look up the underlying Post/Topic/Chat::Message
for a deleted bookmark and check user access in this way. Then,
it doesn't matter if the bookmark was deleted.
2024-02-29 09:03:49 +10:00
Alan Guo Xiang Tan
1a44c359b9
DEV: Fix reloading type map not clearing cache (#25924)
Why this change?

This is a follow up to 408d2f8e69. When
`ActiveRecord::ConnectionAdapaters::PostgreSQLAdatper#reload_type_map`
is called, we need to clear the type map cache otherwise migrations
adding an array column will end up throwing errors.
2024-02-28 14:56:15 +08:00
Alan Guo Xiang Tan
408d2f8e69
DEV: Adds a freedom patch to cache connection type map (#25923)
Why this change?

This patch has been added to address the problems identified in https://github.com/rails/rails/issues/35311. For every,
new connection created using the PostgreSQL adapter, 3 queries are executed to fetch type map information from the `pg_type`
system catalog, adding about 1ms overhead to every connection creation.

On multisite clusters where connections are reaped more aggressively, the 3 queries executed
accounts for a significant portion of CPU usage on the PostgreSQL cluster. This patch works around the problem by
caching the type map in a class level attribute to reuse across connections.
2024-02-28 14:29:13 +08:00
Alan Guo Xiang Tan
54a1fea74e
DEV: Refactor ThemeSettingsObjectValidator#validate (#25904)
What does this change do?

1. Reduce an additional loop through all the properties
2. Extract the validation of child objects into a dedicate method
2024-02-28 10:44:46 +08:00
Alan Guo Xiang Tan
afb0adf48d
DEV: Validate objects when updating typed objects theme settings (#25902)
Why this change?

This change ensures that we validate the value of the new objects
when updating typed objects theme settings.
2024-02-28 10:33:22 +08:00
Joffrey JAFFEUX
64e8ad170e
FIX: do not show send pm prompt when user cant pm (#25912)
Prior to this fix even when the user was not part of a group allowing sending pm we would show the prompt: "You've replied to ... X times, did you know you could send them a personal message instead?"
2024-02-27 11:54:05 +01:00
Jarek Radosz
3736d66f17
DEV: Extensively use exception: true in system() (#25911)
Specifically fixes a bug in smoke-test where it would just move on after failing to install latest js dependencies with yarn.
2024-02-27 11:33:28 +01:00
Jarek Radosz
b337ae5ae9
DEV: Update chrome-launcher from 0.15.2 to 1.1.0 (#25909) 2024-02-27 11:25:11 +01:00
Alan Guo Xiang Tan
18ca3d373d
FIX: ThemeSettingsValidator.validate_value returning wrong error (#25901)
Why this change?

Before this change, the error messages returned when validating theme
settings of typed objects was an array of array instead of just an
array.
2024-02-27 15:46:12 +08:00
Alan Guo Xiang Tan
52a4912475
DEV: Support topic, post, group, upload and tag type for theme objects setting (#25907)
Why this change?

Previously in cac60a2c6b, I added support
for `type: "category"` for a property in the theme objects schema. This
commit extend the work previously to add support for types `topic`,
`post`, `group`, `upload` and `tag`.
2024-02-27 14:27:10 +08:00
Alan Guo Xiang Tan
412b36cc93
FIX: Error when integer values are set as default of string type settings (#25898)
Why this change?

```
some_setting:
  default: 0
  type: string
```

A theme setting like the above will cause an error to be thrown on the
server when importing the theme because the default would be parsed as
an integer which caused an error to be thrown when we are validating the
value of the setting.

What does this change do?

Convert the value to a string when working with string typed theme
settings.
2024-02-27 10:18:38 +08:00
Alan Guo Xiang Tan
7bcfe60a76
DEV: Validate default value for type: objects theme settings (#25833)
Why this change?

This change adds validation for the default value for `type: objects` theme
settings when a setting theme field is uploaded. This helps the theme
author to ensure that the objects which they specifc in the default
value adhere to the schema which they have declared.

When an error is encountered in one of the objects, the error
message will look something like:

`"The property at JSON Pointer '/0/title' must be at least 5 characters
long."`

We use a JSON Pointer to reference the property in the object which is
something most json-schema validator uses as well.

What does this change do?

1. This commit once again changes the shape of hash returned by
   `ThemeSettingsObjectValidator.validate`. Instead of using the
   property name as the key previously, we have decided to avoid
   multiple levels of nesting and instead use a JSON Pointer as the key
   which helps to simplify the implementation.

2 Introduces `ThemeSettingsObjectValidator.validate_objects` which
  returns an array of validation error messages for all the objects
  passed to the method.
2024-02-27 09:16:37 +08:00
David Taylor
588a79c80c
DEV: Merge root JS packages (#25857)
Before this commit, we had a yarn package set up in the root directory and also in `app/assets/javascripts`. That meant two `yarn install` calls and two `node_modules` directories. This commit merges them both into the root location, and updates references to node_modules.

A previous attempt can be found at https://github.com/discourse/discourse/pull/21172. This commit re-uses that script to merge the `yarn.lock` files.

Co-authored-by: Jarek Radosz <jradosz@gmail.com>
2024-02-26 13:45:58 +00:00
Andrei Prigorshnev
b3a1199493
FEATURE: Hide user status when user is hiding public profile and presence (#24300)
Users can hide their public profile and presence information by checking 
“Hide my public profile and presence features” on the 
`u/{username}/preferences/interface` page. In that case, we also don't 
want to return user status from the server.

This work has been started in https://github.com/discourse/discourse/pull/23946. 
The current PR fixes all the remaining places in Core.

Note that the actual fix is quite simple – a5802f484d. 
But we had a fair amount of duplication in the code responsible for 
the user status serialization, so I had to dry that up first. The refactoring 
as well as adding some additional tests is the main part of this PR.
2024-02-26 17:40:48 +04:00
David Taylor
542cb22fd4 DEV: Drop Ember 3 feature flag 2024-02-26 12:22:05 +00:00
Ted Johansson
ed2496c59d
FEATURE: Add scheduled Twitter login problem check - Part 1 (#25830)
This PR adds a new scheduled problem check that simply tries to connect to Twitter OAuth endpoint to check that it's working. It is using the default retry strategy of 2 retries 30 seconds apart.
2024-02-26 12:08:12 +08:00
Vinoth Kannan
1bd9ca11e7
DEV: include more data in Discourse Discover enrollment payload. (#25846)
Adding forum's URL, title, and locale to the payload of enrollment can be helpful while managing it in the Discourse Discover.
2024-02-25 10:42:50 +05:30
Alan Guo Xiang Tan
ad0824b7e3
DEV: Fix connections timeout in system test (#25835)
Why this change?

This regressed in 6e9fbb5bab because we
had a `request.xhr?` check before we decide to block requests. However,
there could not none-xhr requests which we need to block as well at the
end of each system test when `@@block_requests` is true.

This also reverts commit 6437f27f90.
2024-02-23 16:03:46 +08:00
Vinoth Kannan
b3238bfc34
FEATURE: call hub API to update Discourse discover enrollment. (#25634)
Now forums can enroll their sites to be showcased in the Discourse [Discover](https://discourse.org/discover) directory. Once they enable the site setting `include_in_discourse_discover` to enroll their forum the `CallDiscourseHub` job will ping the `api.discourse.org/api/discover/enroll` endpoint. Then the Discourse Hub will fetch the basic details from the forum and add it to the review queue. If the site is approved then the forum details will be displayed in the `/discover` page.
2024-02-23 11:42:28 +05:30
Sam
207cb2052f
FIX: muted tags breaking hot page when filtered to tags (#25824)
Also, remove experimental setting and simply use top_menu for feature detection

This means that when people eventually enable the hot top menu, there will
be topics in it


Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2024-02-23 17:11:39 +11:00
Alan Guo Xiang Tan
6437f27f90
DEV: Debug AR connection pool queue on CI (#25828)
Why this change?

On CI, we have been seeing flaky system tests because ActiveRecord is
unable to checkout a connection. This patch is meant to help us debug
which thread is not returning the connection to the queue.

Example of timeout issue: https://github.com/discourse/discourse/actions/runs/8012541636/job/21888013082
2024-02-23 13:37:37 +08:00
Ted Johansson
a72dc2f420
DEV: Introduce a problem checks API (#25783)
Previously, problem checks were all added as either class methods or blocks in AdminDashboardData. Another set of class methods were used to add and run problem checks.

As of this PR, problem checks are promoted to first-class citizens. Each problem check receives their own class. This class of course contains the implementation for running the check, but also configuration items like retry strategies (for scheduled checks.)

In addition, the parent class ProblemCheck also serves as a registry for checks. For example we can get a list of all existing check classes through ProblemCheck.checks, or just the ones running on a schedule through ProblemCheck.scheduled.

After this refactor, the task of adding a new check is significantly simplified. You add a class that inherits ProblemCheck, you implement it, add a test, and you're good to go.
2024-02-23 11:20:32 +08:00
Daniel Waterworth
087712c1a6
DEV: Reduce duplication with DistributedMutex#clear_regex (#25795) 2024-02-21 14:19:04 -06:00
Daniel Waterworth
13291dc5ef
FIX: Cache keys should be strings (#25791)
* FIX: Cache keys should be strings

Otherwise, there are subtle bugs that don't show up with a single
process.
2024-02-21 10:55:48 -06:00
Bianca Nenciu
9199c52e5e
FIX: Load categories with search topic results (#25700)
Add categories to the serialized search results together with the topics
when lazy load categories is enabled. This is necessary in order for the
results to be rendered correctly and display the category information.
2024-02-21 17:29:47 +02:00
Alan Guo Xiang Tan
3e54351355
DEV: Change shape of errors in ThemeSettingsObjectValidator (#25784)
Why this change?

The current shape of errors returns the error messages after it has been
translated but there are cases where we want to customize the error
messages and the current way return only translated error messages is
making customization of error messages difficult. If we
wish to have the error messages in complete sentences like
"`some_property` property must be present in #link 1", this is not
possible at the moment with the current shape of the errors we return.

What does this change do?

This change introduces the `ThemeSettingsObjectValidator::ThemeSettingsObjectErrors`
and `ThemeSettingsObjectValidator::ThemeSettingsObjectError` classes to
hold the relevant error key and i18n translation options.
2024-02-21 15:27:42 +08:00
Daniel Waterworth
0529d20db4
DEV: Use DistributedCache#defer_get_set instead of getting and setting (#25778)
We use defer_get_set everywhere else, so for consistency and reducing
complexity, this is better.
2024-02-20 18:29:01 -06:00
Alan Guo Xiang Tan
cac60a2c6b
DEV: Support category type in theme setting object schema (#25760)
Why this change?

This change supports a property of `type: category` in the schema that
is declared for a theme setting object. Example:

```
sections:
  type: objects
  schema:
    name: section
    properties:
      category_property:
        type: category
```

The value of a property declared as `type: category` will have to be a
valid id of a row in the `categories` table.

What does this change do?

Adds a property value validation step for `type: category`. Care has
been taken to ensure that we do not spam the database with a ton of
requests if there are alot of category typed properties. This is done by
walking through the entire object and collecting all the values for
properties typed category. After which, a single database query is
executed to validate which values are valid.
2024-02-21 08:11:15 +08:00
Alan Guo Xiang Tan
6ca2396b12
DEV: Centralise logic for validating a theme setting value (#25764)
Why this change?

The logic for validating a theme setting's value and default value was
not consistent as each part of the code would implement its own logic.
This is not ideal as the default value may be validated differently than
when we are setting a new value. Therefore, this commit seeks to
refactor all the validation logic for a theme setting's value into a
single service class.

What does this change do?

Introduce the `ThemeSettingsValidator` service class which holds all the
necessary helper methods required to validate a theme setting's value
2024-02-21 08:08:26 +08:00
Martin Brennan
0b3180c86f
DEV: Add SecureUploadEndpointHelpers for controllers (#25758)
This commit moves some code out of UploadController#show_secure
so it can be reused in other controllers if a secure upload
needs to have permission checks run.
2024-02-20 11:19:22 +10:00
Alan Guo Xiang Tan
bf3c4b634a
DEV: Support validations options for string and numeral types (#25719)
Why this change?

This commit updates `ThemeSettingsObjectValidator` to validate a
property's value against the validations listed in the schema.

For string types, `min_length`, `max_length` and `url` are supported.
For integer and float types, `min` and `max` are supported.
2024-02-20 09:17:27 +08:00
Alan Guo Xiang Tan
a64f558f32
DEV: Add property value validation to ThemeSettingsObjectValidator (#25718)
Why this change?

This change adds property value validation to `ThemeSettingsObjectValidator`
for the following types: "string", "integer", "float", "boolean", "enum". Note
that this class is not being used anywhere yet and is still in
development.
2024-02-19 13:19:35 +08:00
Martin Brennan
a57280cb17
DEV: Change min_trust_level_to_allow_profile_background to trust level setting (#25721)
New setting name is profile_background_allowed_groups

c.f. https://meta.discourse.org/t/changes-coming-to-settings-for-giving-access-to-features-from-trust-levels-to-groups/283408
2024-02-19 10:47:47 +10:00
David Taylor
330cb837da
FIX: Remove strict-dynamic-specific logic from CSP extensions (#25725)
This data is cached, so we don't want to include any site-specific-logic in there. Let's just keep the old URL-collecting behaviour, and let it be stripped out by `CSP::Builder` at runtime.
2024-02-16 13:24:50 +00:00
David Taylor
1672a24490
DEV: Memoize CSP nonce placeholder on response (#25724)
That way, the same value is used even if the helper is called in the context of different controllers

Followup to c8a1b49ddd
2024-02-16 12:15:55 +00:00
David Taylor
b1f74ab59e
FEATURE: Add experimental option for strict-dynamic CSP (#25664)
The strict-dynamic CSP directive is supported in all our target browsers, and makes for a much simpler configuration. Instead of allowlisting paths, we use a per-request nonce to authorize `<script>` tags, and then those scripts are allowed to load additional scripts (or add additional inline scripts) without restriction.

This becomes especially useful when admins want to add external scripts like Google Tag Manager, or advertising scripts, which then go on to load a ton of other scripts.

All script tags introduced via themes will automatically have the nonce attribute applied, so it should be zero-effort for theme developers. Plugins *may* need some changes if they are inserting their own script tags.

This commit introduces a strict-dynamic-based CSP behind an experimental `content_security_policy_strict_dynamic` site setting.
2024-02-16 11:16:54 +00:00
Martin Brennan
3094f32ff5
FIX: is_my_own? check for users who are anonymously doing actions (#25716)
Followup to 978d52841a

It's complicated...we have multiple "anonymous" user concepts
in core, and even two classes called the exact same thing --
AnonymousUser.

The first case is Guardian::AnonymousUser, which is used for
people who are browsing the forum without being authenticated.

The second case is the model AnonymousUser, which is used when
a user is liking or posting anonymously via allow_anonymous_likes
or allow_anonymous_posting site settings.

We will untangle this naming nightmare later on...but for the
time being, only authenticated users who are pretending to be
anonymous should be able to like posts if allow_anonymous_likes
is on.
2024-02-16 14:28:12 +10:00
Alan Guo Xiang Tan
ad900ef9dd
Revert "DEV: Debug AR connection pool queue on CI (#25687)" (#25714)
This reverts commit 796af077c5.

We have not seen checkout timeout errors since c30aeafd9d
2024-02-16 10:11:46 +08:00
Alan Guo Xiang Tan
64b4e0d08d
DEV: First pass of ThemeSettingsObjectValidator (#25624)
Why this change?

This is a first pass at adding an objects validator which main's job is
to validate an object against a defined schema which we will support. In
this pass, we are simply validating that properties that has been marked
as required are present in the object.
2024-02-16 09:35:16 +08:00
Osama Sayegh
cfdb461e9a
PERF: Pass the -ping option to the identify ImageMagick command to speed it up (#25697)
The `-ping` option significantly speeds up the ImageMagick `identify` command per our testing and the [documentation](https://imagemagick.org/script/command-line-options.php#ping):

> -ping
Efficiently determine these image characteristics: image number, the file name, the width and height of the image, whether the image is colormapped or not, the number of colors in the image, the number of bytes in the image, the format of the image (JPEG, PNM, etc.). Use +ping to ensure accurate image properties.

We already pass the `-ping` option in other places where the `identify` command is used, so it makes sense to use the option everywhere.

Internal topic: t/121431.
2024-02-15 18:55:39 +03:00
Sam
4346abe260
FEATURE: apply pinning to hot topic lists (#25690)
pinned topics should be pinned even on hot lists so it can be used as a
home page
2024-02-15 18:27:54 +11:00
Alan Guo Xiang Tan
796af077c5
DEV: Debug AR connection pool queue on CI (#25687)
Why this change?

On CI, we have been seeing flaky system tests because ActiveRecord is
unable to checkout a connection. This patch is meant to help us debug
which thread is not returning the connection to the queue.
2024-02-15 14:00:30 +08:00
Blake Erickson
bb261094cf
FEATURE: Auto generate and display video preview image (#25633)
This change will allow auto generated video thumbnails to be used
instead of the black video thumbnail that overlays videos.

Follow up to: 2443446e62
2024-02-14 13:43:53 -07:00
Kris
faf0807b37
FIX: email category badges shouldn't use category text color (#25655) 2024-02-13 10:18:36 -05:00
David Battersby
d7dd871d9f
FIX: quoted private topic url respects subfolder install (#25643)
Fixes an issue where private topics that are quoted have an incorrectly formatted url when using a subfolder install.

This update returns a relative url that includes the base_path rather than a combination of base_url + base_path.
2024-02-13 13:20:24 +08:00
Penar Musaraj
021a02c3d8
FIX: Webauthn origin was incorrect for subfolder setups (#25651) 2024-02-12 16:27:24 -05:00
Martin Brennan
cf4d92f686
FIX: Change max_image_megapixels logic (#25625)
This commit changes `max_image_megapixels` to be used
as is without multiplying by 2 to give extra leway.
We found in reality this was just causing confusion
for admins, especially with the already permissive
40MP default.
2024-02-12 09:56:43 +10:00
Kris
c49eb373de
FIX: render category badge styles inline for email (#25487) 2024-02-09 15:29:11 -05:00
David Taylor
ee08a8c52b
Revert "FIX: Omit CSP nonce and hash values when unsafe-inline enabled (#25590)" (#25609)
This reverts commit 767b49232e.

If anything else (e.g. GTM integration) introduces a nonce/hash, then this change stops the splash screen JS to fail and makes sites unusable.
2024-02-08 11:44:09 +00:00
Alan Guo Xiang Tan
fb0e656cb7
DEV: Refactor subclasses in ThemeSettingsManager to individual files (#25605)
Why this change?

One Ruby class per file improves readability
2024-02-08 12:59:52 +08:00
Martin Brennan
7ce76143ac
FIX: Always trust admin and moderators with post edits (#25602)
Removes duplication from LimitedEdit to see who can edit
posts, and also removes the old trust level setting check
since it's no longer necessary.

Also make it so staff can always edit since can_edit_post?
already has a staff escape hatch.
2024-02-08 13:10:26 +10:00
Alan Guo Xiang Tan
9f884cdaab
DEV: Introduce experimental type: objects theme setting (#25538)
Why this change?

This commit introduces an experimental `type: objects` theme setting
which will allow theme developers to store a collection of objects as
JSON in the database. Currently, the feature is still in development and
this commit is simply setting up the ground work for us to introduce the
feature in smaller pieces.

What does this change do?

1. Adds a `json_value` column as `jsonb` data type to the `theme_settings` table.
2. Adds a `experimental_objects_type_for_theme_settings` site setting to
   determine whether `ThemeSetting` records of with the `objects` data
   type can be created.
3. Updates `ThemeSettingsManager` to support read/write access from the
   `ThemeSettings#json_value` column.
2024-02-08 10:20:59 +08:00
Martin Brennan
4ce1c2c030
FIX: Always allow staff (admins & mods) to post links (#25601)
Followup fb087b7ff6

post_links_allowed_groups is an odd check tied to
unrestricted_link_posting? in PostGuardian, in that
it doesn't have an escape hatch for staff like most
of the rest of these group based settings.

It doesn't make sense to exclude admins or mods from
posting links, so just always allow them to avoid confusion.
2024-02-08 11:19:28 +10:00
Daniel Waterworth
30922855f2
PERF: Don't allow a single user to monopolize the defer queue (#25593) 2024-02-07 13:47:50 -06:00
David Taylor
767b49232e
FIX: Omit CSP nonce and hash values when unsafe-inline enabled (#25590)
Browsers will ignore unsafe-inline if nonces or hashes are included in the CSP. When unsafe-inline is enabled, nonces and hashes are not required, so we can skip them.

Our strong recommendation remains that unsafe-inline should not be used in production.
2024-02-07 12:35:35 +00:00
David Taylor
dea753a204
FIX: Restore support for .js.es6 files in PrettyText (#25588)
Regressed in 1757a688c4

https://meta.discourse.org/t/294155
2024-02-07 09:34:31 +00:00
Ted Johansson
95a2d285d3
FEATURE: Add new 'illegal' flag reason (#25498)
To comply with Digital Services Act we need a way for users to flag a post as potentially illegal. This PR adds that functionality.
2024-02-07 10:12:22 +08:00
Arpit Jalan
badc390ebe
FEATURE: allow disabling user activity tab for non admin users (#25540)
* FEATURE: allow disabling user activity tab for non admin users

* add another test case
2024-02-05 14:30:36 +05:30
David Taylor
1df9186526
FIX: Plugin image assets in production (#25547)
Followup to 1757a688c4
2024-02-02 19:06:16 +00:00
Ted Johansson
2da7c74e60
DEV: Remove TagGuardian#can_create_tag? fallback (#25535)
We've changed access settings to be group membership based rather than based on the TL value directly. We kept both conditions here while we updated any plugins and themes. It should now be safe to remove.
2024-02-02 13:48:53 +08:00
David Taylor
1757a688c4
DEV: Remove sprockets from plugin 'extra js' pipeline (#25502)
JS assets added by plugins via `register_asset` will be outside the `assets/javascripts` directory, and are therefore exempt from being transpiled. That means that there isn't really any need to run them through DiscourseJsProcessor. Instead, we can just concatenate them together, and avoid the need for all the sprockets-wrangling.

This commit also takes the opportunity to clean up a number of plugin-asset-related codepaths which are no longer required (e.g. globs, handlebars)
2024-02-01 11:48:31 +00:00
Neil Lalonde
fb469e7e2f
FIX: use feedback_category placeholder in i18n string (#25514)
https://review.discourse.org/t/71308/4
2024-01-31 19:40:30 -05:00
Alan Guo Xiang Tan
e61608d080
FIX: Remap postgres text search proximity operator (#25497)
Why this change?

Since 1dba1aca27, we have been remapping
the `<->` proximity operator in a tsquery to `&`. However, there is
another variant of it which follows the `<N>` pattern. For example, the
following text "end-to-end" will eventually result in the following
tsquery `end-to-end:* <-> end:* <2> end:*` being generated by Postgres.
Before this fix, the tsquery is remapped to `end-to-end:* & end:* <2>
end:*` by us. This is requires the search data which we store to contain
`end` at exactly 2 position apart. Due to the way we limit the
number of duplicates in our search data, the search term may end up not
matching anything. In bd32912c5e, we made
it such that we do not allow any duplicates when indexing a topic's
title. Therefore, search for `end-to-end` against a topic title with
`end-to-end` will never match because our index will only contain one
`end` term.

What does this change do?

We will remap the `<N>` variant of the proximity operator.
2024-02-01 07:20:46 +08:00
Jordan Vidrine
f2ac9e4c12
UX: Refactor Do Not Disturb indicator (#25508) 2024-01-31 16:56:07 -06:00
Neil Lalonde
ba68ee4da7
FIX: missing translation of guidelines_topic.body (#25505)
Broken in https://github.com/discourse/discourse/pull/25253
2024-01-31 15:33:09 -05:00
Martin Brennan
575bc4af73
FIX: Remove newlines from img alt & title in HTML to markdown parser (#25473)
We were having a minor issue with emails with embedded images
that had newlines in the alt string; for example:

```
<p class="MsoNormal"><span style="font-size:11.0pt"><img width="898"
height="498" style="width:9.3541in;height:5.1875in" id="Picture_x0020_5"
src="cid:image003.png@01DA4EBA.0400B610" alt="A screenshot of a computer
program

Description automatically generated"></span><span
style="font-size:11.0pt"><o:p></o:p></span></p>
```

Once this was parsed and converted to markdown (or directly to HTML
in some cases), this caused an issue in the composer and the post
UI, where the markdown parser didn't know how to deal with this,
making the HTML show directly instead of showing an image.

The easiest way to deal with this is to just strip \n from image
alt and title attrs in the HTMLToMarkdown class.
2024-01-31 10:23:09 +10:00
Isaac Janzen
491e7a3429
DEV: Don't create backups during version bumps (#25484) 2024-01-30 11:41:15 -07:00
Isaac Janzen
a4022f69f3
Bump version to v3.3.0.beta1-dev 2024-01-30 10:37:25 -07:00
Isaac Janzen
b2b1e721b5
Bump version to v3.2.0.beta5 2024-01-30 10:37:25 -07:00
David Taylor
283fe48243
DEV: Update confirm-email flows to use central 2fa and ember rendering (#25404)
These routes were previously rendered using Rails, and had a fairly fragile 2fa implementation in vanilla-js. This commit refactors the routes to be handled in the Ember app, removes the custom vanilla-js bundles, and leans on our centralized 2fa implementation. It also introduces a set of system specs for the behavior.
2024-01-30 10:32:42 +00:00
David Taylor
2457553d0a
FIX: Avoid flash-of-unstyled-content in Safari with bug workaround (#25462)
Safari has a bug which means that scripts with the `defer` attribute are executed before stylesheets have finished loading. This is being tracked at https://bugs.webkit.org/show_bug.cgi?id=209261.

This commit works around the problem by introducing a no-op inline `<script>` to the end of our HTML document. This works because defer scripts are guaranteed to run after inline scripts, and inline scripts are guaranteed to run after any preceding stylesheets.

Technically we only need this for Safari. But given that the cost is so low, it makes sense to include it everywhere rather than incurring the complexity of gating it by user-agent.
2024-01-29 17:20:44 +00:00
David Taylor
1bfccdd4f2
DEV: Allow run_second_factor! to be used before login (#25420)
In a handful of situations, we need to verify a user's 2fa credentials before `current_user` is assigned. For example: login, email_login and change-email confirmation. This commit adds an explicit `target_user:` parameter to the centralized 2fa system so that it can be used for those situations.

For safety and clarity, this new parameter only works for anon. If some user is logged in, and target_user is set to a different user, an exception will be raised.
2024-01-29 12:28:47 +00:00
Ted Johansson
7e5d2a95ee
DEV: Convert min_trust_level_to_tag_topics to groups (#25273)
We're changing the implementation of trust levels to use groups. Part of this is to have site settings that reference trust levels use groups instead. It converts the min_trust_level_to_tag_topics site setting to tag_topic_allowed_groups.
2024-01-26 13:25:03 +08:00
Sérgio Saquetim
d34a7916c0
DEV: Move Plugin::Instance.register_search_group_query_callback out of protected (#25413) 2024-01-25 13:36:43 -03:00
Martin Brennan
74fd883a89
DEV: Improve site setting rename generator (#25354)
We need to be able to generate these migrations
for plugin settings as well. Also, we can use the
type supervisor to get the enum data in a nicer way.
2024-01-25 10:45:46 +10:00
Martin Brennan
0e50f88212
DEV: Move min_trust_to_post_embedded_media to group setting (#25238)
c.f. https://meta.discourse.org/t/we-are-changing-giving-access-to-features/283408
2024-01-25 09:50:59 +10:00
Leonardo Mosquera
508e2e601c
FIX: FinalDestination::HTTP: validate address argument (#25407)
This would only be empty due to a programming error elsewhere, but
checking this here is a failstop so that it doesn't go further.
2024-01-24 18:50:42 -03:00
Vinoth Kannan
ecb7fb0481
FEATURE: add option to sort topic query result via plugin. (#25349)
Previously, it was not possible to modify the sorting order of the `TopicQuery` result from a plugin. This feature adds support to specify custom sorting functionality in a plugin. We're using the `apply_modifier` method in the `DiscoursePluginRegistry` module to achieve it.

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2024-01-24 12:58:12 +05:30
Roman Rizzi
a709b7e861
FIX: Allow sanitized-HTML in GH issues and categories oneboxes. (#25374)
Follow-up to d78357917c

Related meta topic: https://meta.discourse.org/t/html-is-not-render-on-category-onebox-description/289424:
2024-01-22 15:25:29 -03:00
Jarek Radosz
5278734fe2
FIX: Ignore invalid images when shrinking uploads (#25346) 2024-01-22 12:10:29 +01:00
Jan Cernik
d401502834
FIX: Localize text in github oneboxes (#25327) 2024-01-19 11:26:06 -03:00
Gerhard Schlager
a417760337
FIX: Rake task executed wrong method (#25323)
Rake files share methods with all other rake files and there is already a `rebake_posts` method in another rake file.
2024-01-19 12:55:24 +01:00
Ted Johansson
d17ae1563d
DEV: Convert min_trust_level_for_user_api_key to groups (#25299)
We're changing the implementation of trust levels to use groups. Part of this is to have site settings that reference trust levels use groups instead. It converts the min_trust_level_for_user_api_key  site setting to user_api_key_allowed_groups.

This isn't used by any of our plugins or themes, so very little fallout.
2024-01-19 11:25:24 +08:00
Ted Johansson
46f1c209be
FIX: Account for moderators in group to TL mapping (#25326)
If configuring only moderators in a group based access setting, the mapping to the old setting wouldn't work correctly, because the case was unaccounted for.

This PR accounts for moderators group when doing the mapping.
2024-01-19 11:10:28 +08:00
Ruben Oussoren
0c0f486647
Corrected reply_to_user_id to reference original posters ID instead of itself. (#25307) 2024-01-18 09:02:45 -05:00
Ted Johansson
fb087b7ff6
DEV: Convert min_trust_to_post_links to groups (#25298)
We're changing the implementation of trust levels to use groups. Part of this is to have site settings that reference trust levels use groups instead. It converts the min_trust_to_post_links  site setting to post_links_allowed_groups.

This isn't used by any of our plugins or themes, so very little fallout.
2024-01-18 14:08:40 +08:00
Gerhard Schlager
1a8cf92be1
DEV: Update reply_to_user_id after bulk imports (#25289) 2024-01-17 20:47:01 +01:00
Bianca Nenciu
abad38c2e7
DEV: Make lazy_load_categories setting use groups (#25282)
This allows certain users to test the new feature and avoid disruptions
in other's workflows.
2024-01-17 20:26:51 +02:00
Sam
df8bb947b2
FEATURE: improvements to hot algorithm (#25295)
- Decrease gravity, we come in too hot prioritizing too many new topics
- Remove all muted topics / categories and tags from the hot list
- Punish topics with zero likes in algorithm
2024-01-17 16:12:03 +11:00
Sam
ebd3971533
FEATURE: experiment with hot sort order (#25274)
This introduces a new experimental hot sort ordering. 

It attempts to float top conversations by first prioritizing a  topics with lots of recent activity (likes and users responding) 

The schedule that updates hot topics is disabled unless the hidden site setting: `experimental_hot_topics` is enabled. 

You can control "decay" with `hot_topic_gravity` and `recency` with `hot_topics_recent_days` 

Data is stored in the new `topic_hot_scores` table and you can check it out on the `/hot` route once 
enabled. 
---------

Co-authored-by: Penar Musaraj <pmusaraj@gmail.com>
2024-01-17 13:01:04 +11:00
Gerhard Schlager
241bf48497 DEV: Allow rebakes to generate optimized images at the same time
Previously only Sidekiq was allowed to generate more than one optimized image at the same time per machine. This adds an easy mechanism to allow the same in rake tasks and other tools.
2024-01-16 14:33:16 +01:00
Alan Guo Xiang Tan
c33a8d658b
DEV: Avoid duplicating constant between client and server (#25179)
Why this change?

While the constant does not change very often, we should still avoid
duplicating the value of a constant used on the server side in the
client side to avoid the values going out of sync.
2024-01-16 09:50:55 +08:00
Alan Guo Xiang Tan
22614ca85b
DEV: Compile theme migrations javascript files when running theme qunit (#25219)
Why this change?

Currently, is it hard to iteratively write a theme settings migrations
because our theme migrations system does not rollback. Therefore, we
want to allow theme developers to be able to write QUnit tests for their
theme migrations files enabling them to iteratively write their theme
migrations.

What does this change do?

1. Update `Theme#baked_js_tests_with_digest` to include all `ThemeField`
records of `ThemeField#target` equal to `migrations`. Note that we do
not include the `settings` and `themePrefix` variables for migration files.

2. Don't minify JavaScript test files becasue it makes debugging in
   development hard.
2024-01-16 09:50:44 +08:00
David Taylor
a562214f56
FIX: Update global rate limiter keys/messages to clarify user vs ip (#25264) 2024-01-15 19:54:50 +00:00
David Taylor
59c2407e18
FEATURE: add username header to global-rate-limited responses (#25265)
This will make it easier to analyze rate limiting in reverse-proxy logs. To make this possible without a database lookup, we add the username to the encrypted `_t` cookie data.
2024-01-15 19:50:37 +00:00
Penar Musaraj
f2cf5434f3
Revert "DEV: Convert min_trust_level_to_tag_topics to groups (#25258)" (#25262)
This reverts commit c7e3d27624 due to
test failures. This is temporary.
2024-01-15 11:33:47 -05:00
Ted Johansson
c7e3d27624
DEV: Convert min_trust_level_to_tag_topics to groups (#25258)
We're changing the implementation of trust levels to use groups. Part of this is to have site settings that reference trust levels use groups instead. It converts the min_trust_level_to_tag_topics site setting to tag_topic_allowed_groups.
2024-01-15 20:59:08 +08:00
Mark VanLandingham
66fb2257cf
DEV: Add apply_modifier in Email::Renderer for html modifications (#25205) 2024-01-12 09:14:55 -06:00
Daniel Waterworth
30bea5c7c2
FIX: Don't use joins to filter (#24904)
Posts may have multiple uploads/upload references.
2024-01-11 16:11:29 +11:00
Martin Brennan
08d641d932
DEV: Convert review_media_unless_trust_level to group-based setting (#25201)
This commit moves the review_media_unless_trust_level setting
to skip_review_media_groups as part of our move from TL to group
based settings.

c.f. https://meta.discourse.org/t/changes-coming-to-settings-for-giving-access-to-features-from-trust-levels-to-groups/283408
2024-01-11 13:43:01 +10:00
Blake Erickson
6ebe61ecec
FIX: Logs api scope not working (#25215) 2024-01-10 19:30:10 -07:00
Bianca Nenciu
c916806fe8
FIX: Serialize categories when viewing a topic (#25206)
When navigating straight to a topic the category was not displayed at
all because the categories were not loaded. Similarly, the categories
for suggested topics were not loaded either.

This commit adds a list of categories to topic view model class and
serializer.
2024-01-10 20:30:59 +02:00
David Taylor
7a8cbf8422
DEV: Switch default Ember version to 5 (#25203)
https://meta.discourse.org/t/287211
2024-01-10 12:12:36 +00:00
Vinoth Kannan
992211350a
FEATURE: option to sort user and group private messages. (#25146)
The UI will be the same as the one we're using in the topic list in "latest", "top" etc.,
2024-01-10 13:33:30 +05:30
Ted Johansson
e4494b26fd
DEV: Convert min_trust_to_send_email_messages to groups (#24942)
We're changing the implementation of trust levels to use groups. Part of this is to have site settings that reference trust levels use groups instead. It converts the min_trust_to_send_email_messages site setting to send_email_messages_allowed_groups.
2024-01-09 09:47:06 +08:00
marstall
3837657449
FIX: refactor calling of timed backup deletion
refactor calling of timed backup deletion so it runs regardless of SiteSetting.automatic_backups_enabled value
2024-01-08 15:41:00 -05:00
Penar Musaraj
d795a59fab
Bump version to v3.2.0.beta5-dev 2024-01-08 12:41:52 -05:00
Penar Musaraj
f953d9b113
Bump version to v3.2.0.beta4 2024-01-08 12:41:51 -05:00
Isaac Janzen
e463bc9a67
Revert "DEV: Version bumps for main: v3.2.0.beta4, v3.2.0.beta5-dev (#25168)" (#25169)
This reverts commit ab0fea3f42.
2024-01-08 10:24:04 -07:00
Isaac Janzen
ab0fea3f42
DEV: Version bumps for main: v3.2.0.beta4, v3.2.0.beta5-dev (#25168)
* Bump version to v3.2.0.beta4
* Bump version to v3.2.0.beta5-dev
2024-01-08 10:07:33 -07:00
Martin Brennan
628873de24
FIX: Sort plugins by their setting category name (#25128)
Some plugins have names (e.g. discourse-x-yz) that
are totally different from what they are actually called,
and that causes issues when showing them in a sorted way
in the admin plugin list.

Now, we should use the setting category name from client.en.yml
if it exists, otherwise fall back to the name, for sorting.
This is what we do on the client to determine what text to
show for the plugin name as well.
2024-01-08 09:57:25 +10:00
Ted Johansson
a5f0935307
DEV: Convert min_trust_level_to_create_tag to groups (#24899)
We're changing the implementation of trust levels to use groups. Part of this is to have site settings that reference trust levels use groups instead. It converts the min_trust_level_to_create_tag  site setting to create_tag_allowed_groups.

This PR maintains backwards compatibility until we can update plugins and themes using this.
2024-01-05 10:19:43 +08:00
David Taylor
451581d50a
DEV: Remove unmaintained tidy-jsdoc dependency (#25110)
This was used by chat's HTML documentation experiment. That documentation experiment isn't being actively used/updated, but may be revisited in future. Therefore, this commit updates the jsdoc config to remove the custom theme, but keeps it functional (with the default jsdoc theme).
2024-01-03 10:30:54 +00:00
Martin Brennan
e8deed874b
FIX: Do not allow setting admin and staff for TrustLevelSetting (#25107)
This fixes an issue where any string for an enum site setting
(such as TrustLevelSetting) would be converted to an integer
if the default value for the enum was an integer. This is an
issue because things like "admin" and "staff" would get silently
converted to 0 which is "valid" because it's TrustLevel[0],
but it's unexpected behaviour. It's best to just let the site
setting validator catch this broken value.
2024-01-03 16:55:28 +10:00
Alan Guo Xiang Tan
5530cb574b
DEV: Fix test incorrectly removing stylesheet cache of other processes (#25103)
Why this change?

The `can survive cache miss` test in `spec/requests/stylesheets_controller_spec.rb`
was failing because the file was not found on disk for the cache to be
regenerated. This is because a test in
`spec/lib/stylesheet/manager_spec.rb` was removing the entire
`tmp/stylesheet-cache` directory which is incorrect because the folder
in the test environment further segretates the stylesheet caches based
on the process of the test.

What does this change do?

1. Introduce `Stylesheet::Manager.rm_cache_folder` method for the test
   environment to properly clean up the cache folder.

2. Make `Stylesheet::Manager::CACHE_PATH` a private constant since the
   cache path should be obtained from the `Stylesheet::Manager.cache_fullpath` method.
2024-01-03 13:15:35 +08:00
Kris
e8509ddf50
FIX: github onebox styles for commits (#25098) 2024-01-02 17:29:56 -05:00
David Taylor
7d0e0c3110
DEV: Collect css assets from webpack build (#25087)
fb95ab8e00 started depending on webpack-bundled CSS, but css files weren't being collected by Sprockets or uploaded to S3.
2024-01-02 12:45:34 +00:00
Alan Guo Xiang Tan
655c106101
DEV: Capture and log AR debug logs on GitHub actions for flaky tests (#25048)
Why this change?

We have been running into flaky tests which seems to be related to
AR transaction problems. However, we are not able to reproduce this
locally and do not have sufficient information on our builds now to
debug the problem.

What does this change do?

Noe the following changes only applies when `ENV["GITHUB_ACTIONS"]` is
present.

This change introduces an RSpec around hook when `capture_log: true` has
been set for a test. The responsibility of the hook is to capture the
ActiveRecord debug logs and print them out.
2023-12-27 14:40:00 +08:00
Ted Johansson
b890eb1bd2
DEV: Convert min_trust_to_allow_self_wiki to groups (#25009)
We're changing the implementation of trust levels to use groups. Part of this is to have site settings that reference trust levels use groups instead. It converts the min_trust_to_allow_self_wiki site setting to self_wiki_allowed_groups.

Nothing of note here. This is used in exactly one place, and there's no fallout.
2023-12-27 09:21:39 +08:00
Martin Brennan
89705be722
DEV: Add auto map from TL -> group site settings in DeprecatedSettings (#24959)
When setting an old TL based site setting in the console e.g.:

SiteSetting.min_trust_level_to_allow_ignore = TrustLevel[3]

We will silently convert this to the corresponding Group::AUTO_GROUP. And vice-versa, when we read the value on the old setting, we will automatically get the lowest trust level corresponding to the lowest auto group for the new setting in the database.
2023-12-26 14:39:18 +08:00
Alan Guo Xiang Tan
cfec408bc1
DEV: Remove flaky tests report when there are too many failures (#25031)
Why this change?

Currently we only rerun failing tests to check if they are flaky tests
when there are 10 or less failing tests. When there are more than 10
failing tests in the first run, we assume that the odds of those tests
being flaky are low and do not rerun the tests. However, there was a bug
where we do not clean up the potential flaky tests being logged when
there are too many test failures. This resulted in those test failures
being treated as flaky tests.

What does this change do?

Clean up the flaky tests report when we do not rerun the tests.
2023-12-26 09:23:17 +08:00
Krzysztof Kotlarek
d03f6727b1
FIX: TL3 can convert their post to a wiki (#25023)
A bug that allowed TL1 to convert other's posts to wiki.

The issue was introduced in this PR: https://github.com/discourse/discourse/pull/24999/files

The wiki can be created if a user is TL3 and it is their own post - default 3 for setting `SiteSetting.min_trust_to_allow_self_wiki`

In addition, a wiki can be created by staff and TL4 users for any post.
2023-12-23 21:31:46 +11:00
Krzysztof Kotlarek
025e40354c
FIX: correct typo minmin_trust_to_edit_wiki_post (#24999)
Typo introduced here https://github.com/discourse/discourse/pull/24766#pullrequestreview-1792187422

In addition, use setting instead of hard-coded @user.has_trust_level?(TrustLevel[4])
2023-12-22 08:39:42 +11:00
Martin Brennan
7fcef5f2f9
FIX: Show admin plugin route sub-links in sidebar (#24982)
This changes the Plugins link in the admin sidebar to
be a section instead, which then shows all enabled plugin
admin routes (which are custom routes some plugins e.g.
chat define).

This is done via adding some special preloaded data for
all controllers based on AdminController, and also specifically
on Admin::PluginsController, to have the routes loaded without
additional requests on page load.

We just use a cog for all the route icons for now...we don't
have anything better.
2023-12-21 11:37:20 +10:00
Blake Erickson
43a6c1b7cc
FIX: Have file size restriction type return integers (#24989) 2023-12-20 10:17:10 -07:00
Krzysztof Kotlarek
4c8bc34475
DEV: Custom generator for move setting from tl to groups (#24912)
Ability to automatically generate migration when site setting is changed from trust level to groups.

Example usage:

rails generate site_setting_move_to_groups_migration min_trust_to_create_topic create_topic_allowed_groups
2023-12-19 10:52:28 +11:00
Ted Johansson
0edf39409c
DEV: Convert min_trust_level_to_allow_ignore to groups (#24894)
We're changing the implementation of trust levels to use groups. Part of this is to have site settings that reference trust levels use groups instead. It converts the min_trust_level_to_allow_ignore  site setting to ignore_allowed_groups.

This PR maintains backwards compatibility until we can update plugins and themes using this.
2023-12-18 13:04:37 +08:00
Ted Johansson
6ab1a19e93
DEV: Convert min_trust_level_to_allow_invite to groups (#24893)
We're changing the implementation of trust levels to use groups. Part of this is to have site settings that reference trust levels use groups instead. It converts the min_trust_level_to_allow_invite  site setting to invite_allowed_groups.

Nothing much of note. This is used in one place and there's no fallout.
2023-12-18 12:07:36 +08:00
Krzysztof Kotlarek
1f72152e47
DEV: Remove usage of min_trust_to_create_topic SiteSetting (#24887)
Using min_trust_to_create_topic and create_topic_allowed_groups together was part of #24740

Now, when plugins specs are fixed, we can safely remove that part of logic.
2023-12-18 13:39:53 +11:00
Martin Brennan
6de00f89c2
FEATURE: Initial admin sidebar navigation (#24789)
This is v0 of admin sidebar navigation, which moves
all of the top-level admin nav from the top of the page
into a sidebar. This is hidden behind a enable_admin_sidebar_navigation
site setting, and is opt-in for now.

This sidebar is dynamically shown whenever the user enters an
admin route in the UI, and is hidden and replaced with either
the:

* Main forum sidebar
* Chat sidebar

Depending on where they navigate to. For now, custom sections
are not supported in the admin sidebar.

This commit removes the experimental admin sidebar generation rake
task but keeps the experimental sidebar UI for now for further
testing; it just uses the real nav as the default now.
2023-12-18 11:48:25 +10:00
Andrei Prigorshnev
6e2201135f
DEV: Introduce plugin API for getting stats (#24829)
Before, when needed to get stats in a plugin, we called Core classes directly. 
Introducing plugin API will decouple plugins from Core and give as more freedom 
in refactoring stats in Core. Without this API, I wasn't able to do all refactorings 
I wanted when working on d91456f.
2023-12-15 23:47:20 +04:00
Kelv
2477bcc32e
DEV: lint against Layout/EmptyLineBetweenDefs (#24914) 2023-12-15 23:46:04 +08:00
Blake Erickson
7aeb5d6012
FIX: Unable to move pm to public topic (#24903)
* FIX: guard against empty category_ids when creating small action post for changing of category

Co-authored-by: Kelvin Tan <kelv@discourse.org>
2023-12-14 12:31:38 -07:00
Alan Guo Xiang Tan
c437b9f5f2
DEV: Include exception details for each test in flaky tests report (#24892)
Why this change?

The exception message and name is useful when analyzing why a test
failed.
2023-12-14 11:11:11 +08:00
Ted Johansson
53d40672a7
DEV: Convert min_trust_level_to_allow_user_card_background to groups (#24891)
We're changing the implementation of trust levels to use groups. Part of this is to have site settings that reference trust levels use groups instead. It converts the min_trust_level_to_allow_user_card_background site setting to user_card_background_allowed_groups.

Nothing of note here. This is used in exactly one place, and there's no fallout.
2023-12-14 10:57:58 +08:00
Ted Johansson
f029d8142b
DEV: Validate that passed in groups exist in AtLeastOneGroupValidator (#24890)
This validator is used for site settings where one or more groups are to be input.

At the moment this validator just checks that the value isn't blank. This PR adds a validation for the existence of the groups passed in.
2023-12-14 10:00:53 +08:00
Ted Johansson
48116186af
DEV: Convert tl4_delete_posts_and_topics to groups (#24866)
We're changing the implementation of trust levels to use groups. Part of this is to have site settings that reference trust levels use groups instead. It converts the tl4_delete_posts_and_topics  site setting to delete_all_posts_and_topics_allowed_groups.

This one is a bit different from previous ones, as it's a boolean flag, and the default should be no group. Pay special attention to the migration during review.
2023-12-14 09:56:42 +08:00
Blake Erickson
00209f03e6
DEV: Add file_size_restriction site setting type (#24704)
This change will allow admins to configure file sizes in mb instead of
having to convert to kb.

Co-authored-by: Ella <ella.estigoy@gmail.com>
2023-12-13 16:22:48 -07:00
marstall
0513865c3c
FEATURE: Delete backups based on time window (#24296)
* FEATURE: core code, tests for feature to allow backups to removed based on a time window

* FEATURE: getting tests working for time-based backup

* FEATURE: getting tests running

* FEATURE: linting
2023-12-13 13:00:27 -05:00
David Taylor
6731eec42a
DEV: Summarize JS deprecations at end of system spec run (#24824) 2023-12-13 16:04:25 +00:00
Ted Johansson
294febf3c4
DEV: Convert min_trust_to_flag_posts setting to groups (#24864)
We're changing the implementation of trust levels to use groups. Part of this is to have site settings that reference trust levels use groups instead. It converts the min_trust_to_flag_posts site setting to flag_post_allowed_groups.

Note: In the original setting, "posts" is plural. I have changed this to "post" singular in the new setting to match others.
2023-12-13 17:18:42 +08:00
Ted Johansson
36057638ca
DEV: Convert min_trust_to_edit_post to groups (#24840)
We're changing the implementation of trust levels to use groups. Part of this is to have site settings that reference trust levels use groups instead. It converts the min_trust_to_edit_post site setting to edit_post_allowed_groups.

The old implementation will co-exist for a short period while I update any references in plugins and themes.
2023-12-13 13:25:13 +08:00
Krzysztof Kotlarek
702d0620d7
DEV: Convert min_trust_to_create_topic to groups (#24740)
This change converts the min_trust_to_create_topic site setting to
create_topic_allowed_groups.

See: https://meta.discourse.org/t/283408

- Hides the old setting
- Adds the new site setting
- Add a deprecation warning
- Updates to use the new setting
- Adds a migration to fill in the new setting if the old setting was
changed
- Adds an entry to the site_setting.keywords section
- Updates tests to account for the new change
- After a couple of months, we will remove the min_trust_to_create_topicsetting entirely.

Internal ref: /t/117248
2023-12-13 14:50:13 +11:00
Krzysztof Kotlarek
1017820012
DEV: Convert allow_uploaded_avatars to groups (#24810)
This change converts the allow_uploaded_avatars site setting to uploaded_avatars_allowed_groups.

See: https://meta.discourse.org/t/283408

Hides the old setting
Adds the new site setting
Adds a deprecation warning
Updates to use the new setting
Adds a migration to fill in the new setting if the old setting was changed
Adds an entry to the site_setting.keywords section
Updates tests to account for the new change
After a couple of months, we will remove the allow_uploaded_avatars setting entirely.

Internal ref: /t/117248
2023-12-13 10:53:19 +11:00
Alan Guo Xiang Tan
39da9106ba
DEV: Introduce automatic reruns to RSpec tests on Github actions (#24811)
What motivated this change?

Our builds on Github actions have been extremely flaky mostly due to system tests. This has led to a drop in confidence
in our test suite where our developers tend to assume that a failed job is due to a flaky system test. As a result, we
have had occurrences where changes that resulted in legitimate test failures are merged into the `main` branch because developers
assumed it was a flaky test.

What does this change do?

This change seeks to reduce the flakiness of our builds on Github Actions by automatically re-running RSpec tests once when
they fail. If a failed test passes subsequently in the re-run, we mark the test as flaky by logging it into a file on disk
which is then uploaded as an artifact of the Github workflow run. We understand that automatically re-runs will lead to 
lower accuracy of our tests but we accept this as an acceptable trade-off since a fragile build has a much greater impact
on our developers' time. Internally, the Discourse development team will be running a service to fetch the flaky tests 
which have been logged for internal monitoring.

How is the change implemented?

1. A `--retry-and-log-flaky-tests` CLI flag is added to the `bin/turbo_rspec` CLI which will then initialize `TurboTests::Runner` 
with the `retry_and_log_flaky_tests` kwarg set to `true`. 

2. When the `retry_and_log_flaky_tests` kwarg is set to `true` for `TurboTests::Runner`, we will register an additional 
formatter `Flaky::FailuresLoggerFormatter` to the `TurboTests::Reporter` in the `TurboTests::Runner#run` method. 
The `Flaky::FailuresLoggerFormatter` has a simple job of logging all failed examples to a file on disk when running all the 
tests. The details of the failed example which are logged can be found in `TurboTests::Flaky::FailedExample.to_h`.

3. Once all the tests have been run once, we check the result for any failed examples and if there are, we read the file on
disk to fetch the `location_rerun_location` of the failed examples which is then used to run the tests in a new RSpec process.
In the rerun, we configure a `TurboTests::Flaky::FlakyDetectorFormatter` with RSpec which removes all failed examples from the log file on disk since those examples are not flaky tests. Note that if there are too many failed examples on the first run, we will deem the failures to likely not be due to flaky tests and not re-run the test failures. As of writing, the threshold of failed examples is set to 10. If there are more than 10 failed examples, we will not re-run the failures.
2023-12-13 07:18:27 +08:00
Krzysztof Kotlarek
5055e431a8
DEV: Custom generator for rename site setting migration (#24841)
Ability to automatically generate migration when site setting name is changed.
Example usage: `rails generate site_setting_rename_migration site_description contact_email`
2023-12-13 09:58:45 +11:00
Rafael dos Santos Silva
de936f07e5
PERF: Update node_options during ember build for low end servers (#24850)
See https://meta.discourse.org/t/286643/14

Co-authored-by: David Taylor <david@taylorhq.com>
2023-12-12 17:05:20 +00:00
Angus McLeod
95c61b88dc
Apply embed unlisted setting consistently (#24294)
Applies the embed_unlisted site setting consistently across topic embeds, including those created via the WP Discourse plugin. Relatedly, adds a embed exception to can_create_unlisted_topic? check. Users creating embedded topics are not always staff.
2023-12-12 09:35:26 -05:00
Krzysztof Kotlarek
7dd150bc95
DEV: Convert min_trust_to_edit_wiki_post to groups (#24766)
This change converts the min_trust_to_edit_wiki_post site setting to edit_wiki_post_allowed_groups.

See: https://meta.discourse.org/t/283408

Hides the old setting
Adds the new site setting
Add a deprecation warning
Updates to use the new setting
Adds a migration to fill in the new setting if the old setting was changed
Adds an entry to the site_setting.keywords section
Updates tests to account for the new change
After a couple of months, we will remove the email_in_min_trust setting entirely.

Internal ref: /t/117248
2023-12-12 15:20:37 +11:00
Gerhard Schlager
dc8c6b8958 DEV: Lots of improvements to the generic_bulk import script
Notable changes:
* Imports a lot more tables from core and plugins
  * site settings
  * uploads with necessary upload references
  * groups and group members
  * user profiles
  * user options
  * user fields & values
  * muted users
  * user notes (plugin)
  * user followers (plugin)
  * user avatars
  * tag groups and tags
  * tag users (notification settings for tags / user)
  * category permissions
  * polls with options and votes
  * post votes (plugin)
  * solutions (plugin)
  * gamification scores (plugin)
  * events (plugin)
  * badges and badge groupings
  * user badges
  * optimized images
  * topic users (notification settings for topics)
  * post custom fields
  * permalinks and permalink normalizations

* It creates the `migration_mappings` table which is used to store the mapping for a handful of imported tables

* Detects duplicate group names and renames them

* Pre-cooking for attachments, images and mentions

* Outputs instructions when gems are missing

* Supports importing uploads from a DB generated by `uploads_importer.rb`

* Checks that all required plugins exists and enables them if needed

* A couple of optimizations and additions in `import.rake`
2023-12-11 16:23:07 +01:00
Penar Musaraj
28956a5415
DEV: Switch to sassc-embedded (#24639)
Sassc-embedded fixes a performance issue with a leaking DartSass process. And it also fixes an issue with source map file paths (without any extra flags).
2023-12-08 10:34:03 -05:00
David Taylor
e4c373194d
DEV: Refactor Wizard components (#24770)
This commit refactors the Wizard component code in preparation for moving it to the 'static' directory for Embroider route-splitting. It also includes a number of general improvements and simplifications.

Extracted from https://github.com/discourse/discourse/pull/23678

Co-authored-by: Godfrey Chan <godfreykfc@gmail.com>
2023-12-07 16:33:38 +00:00
Martin Brennan
7afb5fc481
DEV: Use Discourse::SYSTEM_USER_ID in fixtures/009_users (#24743)
I couldn't find where we created the system user and
this is why -- everywhere else in the app we reference
SYSTEM_USER_ID but here.
2023-12-07 09:04:45 +10:00
Jarek Radosz
694b5f108b
DEV: Fix various rubocop lints (#24749)
These (21 + 3 from previous PRs) are soon to be enabled in rubocop-discourse:

Capybara/VisibilityMatcher
Lint/DeprecatedOpenSSLConstant
Lint/DisjunctiveAssignmentInConstructor
Lint/EmptyConditionalBody
Lint/EmptyEnsure
Lint/LiteralInInterpolation
Lint/NonLocalExitFromIterator
Lint/ParenthesesAsGroupedExpression
Lint/RedundantCopDisableDirective
Lint/RedundantRequireStatement
Lint/RedundantSafeNavigation
Lint/RedundantStringCoercion
Lint/RedundantWithIndex
Lint/RedundantWithObject
Lint/SafeNavigationChain
Lint/SafeNavigationConsistency
Lint/SelfAssignment
Lint/UnreachableCode
Lint/UselessMethodDefinition
Lint/Void

Previous PRs:
Lint/ShadowedArgument
Lint/DuplicateMethods
Lint/BooleanSymbol
RSpec/SpecFilePathSuffix
2023-12-06 23:25:00 +01:00
David Taylor
48ec946702
UX: Restore category badge colours on 404 page (#24754) 2023-12-06 17:49:19 +00:00
Jarek Radosz
6a66dc1cfb
DEV: Fix Lint/BooleanSymbol (#24747) 2023-12-06 13:19:09 +01:00
Jarek Radosz
138bf486d3
DEV: Fix Lint/DuplicateMethods (#24746) 2023-12-06 13:18:34 +01:00
Jarek Radosz
4280c01153
DEV: Fix Lint/ShadowedArgument (#24733) 2023-12-06 13:16:10 +01:00
David Battersby
8b46dc8bb5
FEATURE: Add thumbnails for chat image uploads (#24328)
Introduces the concept of image thumbnails in chat, prior to this we uploaded and used full size chat images within channels and direct messages.

The following changes are covered:
- Post processing of image uploads to create the thumbnail within Chat::MessageProcessor
- Extract responsive image ratios into CookedProcessorMixin (used for creating upload variations)
- Add thumbnail to upload serializer from plugin.rb
- Convert chat upload template to glimmer component using .gjs format
- Use thumbnail image within chat upload component (stores full size img in orig-src data attribute)
- Old uploads which don't have thumbnails will fallback to full size images in channels/DMs
- Update Magnific lightbox to use full size image when clicked
- Update Glimmer lightbox to use full size image (enables zooming for chat images)
2023-12-06 14:59:18 +08:00
Martin Brennan
30d5e752d7
DEV: Revert guardian changes (#24742)
I took the wrong approach here, need to rethink.

* Revert "FIX: Use Guardian.basic_user instead of new (anon) (#24705)"

This reverts commit 9057272ee2.

* Revert "DEV: Remove unnecessary method_missing from GuardianUser (#24735)"

This reverts commit a5d4bf6dd2.

* Revert "DEV: Improve Guardian devex (#24706)"

This reverts commit 77b6a038ba.

* Revert "FIX: Introduce Guardian::BasicUser for oneboxing checks (#24681)"

This reverts commit de983796e1.
2023-12-06 16:37:32 +10:00
Martin Brennan
9057272ee2
FIX: Use Guardian.basic_user instead of new (anon) (#24705)
c.f. de983796e1

There will soon be additional login_required checks
for Guardian, and the intent of many checks by automated
systems is better fulfilled by using BasicUser, which
simulates a logged in TL0 forum user, rather than an
anon user.

In some cases the use of anon still makes sense (e.g.
anonymous_cache), and in that case the more explicit
`Guardian.anon_user` is used
2023-12-06 11:56:21 +10:00
Martin Brennan
a5d4bf6dd2
DEV: Remove unnecessary method_missing from GuardianUser (#24735)
Followup to 77b6a038ba, this
was a mistake and should have been removed before merge.
2023-12-06 11:24:19 +10:00
Martin Brennan
77b6a038ba
DEV: Improve Guardian devex (#24706)
It's quite confusing for blank? to be overridden
on AnonymousUser and BasicUser to represent
whether the fake user is authenticated or not;
we can achieve the same thing more clearly with
a wrapper GuardianUser class around these
user classes. Also fixes an issue where
`def user` would be returning nil.
2023-12-06 10:57:04 +10:00
Martin Brennan
de983796e1
FIX: Introduce Guardian::BasicUser for oneboxing checks (#24681)
Through internal discussion, it has become clear that
we need a conceptual Guardian user that bridges the
gap between anon users and a logged in forum user with
an absolute baseline level of access to public topics,
which can be used in cases where:

1. Automated systems are running which shouldn't see any
   private data
1. A baseline level of user access is needed

In this case we are fixing the latter; when oneboxing a local
topic, and we are linking to a topic in another category from
the current one, we need to operate off a baseline level of
access, since not all users have access to the same categories,
and we don't want e.g. editing a post with an internal link to
expose sensitive internal information.
2023-12-05 09:25:23 +10:00
David Taylor
3aeff56faf
DEV: Run assets:precompile ember build with CI=1 (#24696)
This will improve the output to print the current step (rather than the existing behavior which just says 'building...')
2023-12-04 15:49:50 +00:00
Jarek Radosz
7196613e2e
DEV: Fix various spec linting issues (#24672)
Duplicated specs, incorrect descriptions, incorrect assertions, incorrect filenames, old todo
2023-12-04 13:45:19 +01:00
David Taylor
c2887d3f8c
DEV: Remove unused Ember::Handlebars freedom patch (#24688)
We no longer compile Ember templates in ruby, so this is unused
2023-12-04 12:40:08 +00:00
Jarek Radosz
74011232e9
FIX: Request html when fetching inline onebox data (#24674)
We do expect to receive html
2023-12-04 11:36:42 +10:00
David Taylor
ecf7a4f0c6
FIX: Ensure app-cdn CORS is not overridden by cors_origin setting (#24661)
We add `Access-Control-Allow-Origin: *` to all asset requests which are requested via a configured CDN. This is particularly important now that we're using browser-native `import()` to load the highlightjs bundle. Unfortunately, user-configurable 'cors_origins' site setting was overriding the wldcard value on CDN assets and causing CORS errors.

This commit updates the logic to give the `*` value precedence, and adds a spec for the situation. It also invalidates the cache of hljs assets (because CDNs will have cached the bad Access-Control-Allow-Origin header).

The rack-cors middleware is also slightly tweaked so that it is always inserted. This makes things easier to test and more consistent.
2023-12-01 12:57:11 +00:00
Ted Johansson
54e813e964
FIX: Don't error out when trying to retrieve title and URL won't encode (#24660) 2023-12-01 15:03:06 +08:00
Martin Brennan
c58cd697d2
FIX: Further improvements for plugin list (#24622)
Followup e37fb3042d

* Automatically remove the prefix `Discourse ` from all the plugin titles to avoid repetition
* Remove the :discourse_dev: icon from the author. Consider a "By Discourse" with no labels as official
* We add a `label` metadata to plugin.rb
  * Only plugins made by us in `discourse` and `discourse-org` GitHub organizations will show these in the list
* Make the plugin author font size a little smaller
* Make the commit sha look like a link so it's more obvious it goes to the code

Also I added some validation and truncation for plugin metadata
parsing since currently you can put absolutely anything in there
and it will show on the plugin list.
2023-11-30 10:53:17 +10:00
Daniel Waterworth
eef93ac926
DEV: Allow setting max_length for field types using the plugin API (#24635) 2023-11-29 14:17:12 -06:00
David Taylor
265a8cd2b2
DEV: Fixup plugin initialization guard (#24628)
In development, I sometimes get `nil` `location.absolute_path` values. It looks like this is sometimes expected (https://bugs.ruby-lang.org/issues/10561) so we should fallback to `.path` and add a nil check.
2023-11-29 11:44:09 +00:00
Blake Erickson
21d614215b
DEV: Use staged user check instead (#24578)
This change refactors the check `user.groups.any?` and instead uses
`user.staged?` to check if the user is staged or not.

Also fixes several tests to ensure the users have their auto trust level
groups created.

Follow up to:

- 8a45f84277
- 447d9b2105
- c89edd9e86
2023-11-28 07:34:02 -07:00
David Taylor
5783f231f8
DEV: Introduce DISCOURSE_ASSET_URL_SALT (#24596)
This value is included when generating static asset URLs. Updating the value will allow site operators to invalidate all asset urls to recover from configuration issues which may have been cached by CDNs/browsers.
2023-11-28 11:28:40 +00:00
David Taylor
16b6e86932 DEV: Introduce feature-flag for Ember 5 upgrade
This commit introduces the scaffolding for us to easily switch between Ember 3.28 and Ember 5 on the `main` branch of Discourse. Unfortunately, there is no built-in system to apply this kind of flagging within yarn / ember-cli. There are projects like `ember-try` which are designed for running against multiple version of a dependency, but they do not allow us to 'lock' dependency/sub-dependency versions, and are therefore unsuitable for our use in production.

Instead, we will be maintaining two root `package.json` files, and two `yarn.lock` files. For ember-3, they remain as-is. For ember5, we use a yarn 'resolution' to override the version for ember-source across the entire yarn workspace.

To allow for easy switching with minimal diff against the repository, `package.json` and `yarn.lock` are symlinks which point to `package-ember3.json` and `yarn-ember3.lock` by default. To switch to Ember 5, we can run `script/switch ember version 5` to update the symlinks to point to `package-ember5.json` and `package-ember3.json` respectively. In production, and when using `bin/ember-cli` for development, the ember version can also be upgraded using the `EMBER_VERSION=5` environment variable.

When making changes to dependencies, these should be made against the default `ember3` versions, and then `script/regen_ember_5_lockfile` should be used to regenerate `yarn-ember5.lock` accordingly. A new 'Ember Version Lockfiles' GitHub workflow will automate this process on Dependabot PRs.

When running a local environment against Ember 5, the two symlink changes will show up as git diffs. To avoid us accidentally committing/pushing that change, another GitHub workflow is introduced which checks the default Ember version and raises an error if it is greater than v3.

Supporting two ember versions simultaneously obviously carries significant overhead, so our aim will be to get themes/plugins updated as quickly as possible, and then drop this flag.
2023-11-27 16:40:22 +00:00
Martin Brennan
09c446c1ae
FIX: Handle missing git repo details in plugin list (#24539)
Followup to e37fb3042d,
in some cases we cannot get git information for the
plugin folder (e.g. permission issues), so we need
to only try and get information about it if
commit_hash is present.
2023-11-24 12:08:10 +10:00
Sam
c2fd090d7d
DEV: revert missing license for maxmind changes (#24538)
Reverts
 
 - DEV: maxmind license checking failing tests #24534 
 - UX: Show if MaxMind key is missing on IP lookup #18993

These changes are leading to surprising results, our logs are now filling up with warnings on dev environments 

We need the change to be redone
2023-11-24 11:31:11 +11:00
Sam
85d2b5fa48
DEV: maxmind license checking failing tests (#24534)
This improves the implementation of #18993

1. Error message displayed to user is clearer
2. open_db will also be called, even if license key is blank, as it was previously
3. This in turn means no need to keep stubbing 'maxmind_license_key'
2023-11-24 09:38:46 +11:00
MichaIng
c58a41cb3e
UX: Show on IP lookup if MaxMind key is missing (#18993)
as discussed in https://meta.discourse.org/t/maxminddb-not-found-error/148512/7.
 
shows a warning to the admin if no license for maxmind is found
2023-11-24 08:02:05 +11:00
Alan Guo Xiang Tan
aaadce0652
Revert "FIX: plugin:install_all_gems Rake task not installing plugin gem (#24522)" (#24524)
This breaks the `plugin:install_all_gems` Rake task when used before
Redis is running. Need to go back to the drawing board.

This reverts commit 189aa5fa4e.
2023-11-23 13:01:54 +08:00
Blake Erickson
c89edd9e86
DEV: Convert email_in_min_trust to groups (#24515)
This change converts the `email_in_min_trust` site setting to
`email_in_allowed_groups`.

See: https://meta.discourse.org/t/283408

- Hides the old setting
- Adds the new site setting
- Add a deprecation warning
- Updates to use the new setting
- Adds a migration to fill in the new setting if the old setting was
  changed
- Adds an entry to the site_setting.keywords section
- Updates tests to account for the new change

After a couple of months we will remove the
`email_in_min_trust` setting entirely.

Internal ref: /t/115696
2023-11-22 18:03:28 -07:00
Alan Guo Xiang Tan
189aa5fa4e
FIX: plugin:install_all_gems Rake task not installing plugin gem (#24522)
Why this change?

This regressed in dec68d780c where the
commit assumes that plugin gems are always installed when the
`plugin:install_all_gems` Rake task is ran as it would run the our Rails
initializers which activates plugins and install the gems. However, this
assumption only holds true when the `LOAD_PLUGINS` is present and set to
`1`.

What does this change do?

This commit changes the `plugin:install_all_gems` to load the Rails
environment with `LOAD_PLUGINS` set to `1` such that the plugin gems
will be installed as part of our initialization process for the app.

The commit also removes the `plugin:install_gems` Rake task which is
currently a noop and does not seem to be used anywhere..
2023-11-23 08:29:51 +08:00
Gerhard Schlager
5b97f79569
DEV: Replace starts_with? with start_with? in HtmlToMarkdown (#24521)
This allows us to use that class without loading Rails, e.g. in imports (converters).
2023-11-23 00:57:24 +01:00
Gerhard Schlager
9f3399ca8f
FIX: Remove multiple consecutive dashes from tag names (#24520) 2023-11-23 00:57:12 +01:00
Alan Guo Xiang Tan
1e290eed7b
PERF: Switch plugins:update_all to git pull concurrently (#24513)
Why this change?

Similar to d0117ff6e3, `plugins:update_all` spends most of its time waiting
on the network. On my local machine, this takes up to 2 mins when I have
all the official plugins installed. On a 32 cores machine, the total
time is cut down to 4 seconds.

What does this change do?

1. Move the logic in the `plugin:update` Rake task into a method.
2. Updates the `plugin:update` and `plugin:update_all` to rely on the
   new method.
3. Wraps the method call to update a plugin in `plugin:update_all` in a
   `Concurrent::Promise`

This change also adds the `--quiet` option to the `git pull` option
since the `git pull` output is just noise for 99% of the time.
2023-11-23 07:08:32 +08:00
Martin Brennan
e395e5e002
FIX: Use plugin category name for plugin list (#24477)
Followup to e37fb3042d

Some plugins like discourse-ai and discourse-saml do not
nicely change from kebab-case to Title Case (e.g. Ai, Saml),
and anyway this method of getting the plugin name is not
translated either.

Better to use the plugin setting category if it exists,
since that is written by a human and is translated.
2023-11-23 08:40:55 +10:00
Blake Erickson
8a45f84277
DEV: Convert approve_new_topics_unless_trust_level to groups (#24504)
* DEV: Convert approve_new_topics_unless_trust_level to groups

This change converts the `approve_new_topics_unless_trust_level` site
setting to `approve_new_topics_unless_allowed_groups`.

See: https://meta.discourse.org/t/283408

- Hides the old setting
- Adds the new site setting
- Add a deprecation warning
- Updates to use the new setting
- Adds a migration to fill in the new setting if the old setting was
  changed
- Adds an entry to the site_setting.keywords section
- Updates tests to account for the new change

After a couple of months we will remove the
`approve_new_topics_unless_trust_level` setting entirely.

Internal ref: /t/115696

* add missing translation

* Add keyword entry

* Add migration
2023-11-22 10:44:59 -07:00
Alan Guo Xiang Tan
d0117ff6e3
PERF: Switch plugin:install_all_official to clone plugins concurrently (#24511)
Why this change?

`plugin:install_all_official` is quite slow at the moment taking roughly
1 minute and 51 seconds on my machine. Since most of the time is spent
waiting on the network, we can actually speed up the Rake task
significantly by executing the cloning concurrently. With a 8 cores
machine, cloning all plugins will only take 15 seconds.

What does this change do?

This change wraps the `git clone` operation in the
`plugin:install_all_official` Rake task in a `Concurrent::Promise` which
basically runs the `git clone` operation in a Thread. The `--quiet`
option has also been added to `git clone` since running stuff
concurrently messes up the output. That could be fixed but it has been
determined to be not worth it since the output from `git clone` is
meaningless to us.
2023-11-22 21:43:42 +08:00
Alan Guo Xiang Tan
48f3c638cc
DEV: Support customizing docker:test:setup with envs (#24508)
Why this change?

There are instances where we would like to customize what the
`docker:test:setup` Rake task does.

What does this change do?

Adds a bunch of env variables that could be set to customize what the
`docker:test:setup` Rake test does.
2023-11-22 15:12:02 +08:00
Martin Brennan
4e7929abb6
FIX: Wrong argument error being thrown in UrlHelper (#24506)
We were throwing ArgumentError in UrlHelper.normalised_encode,
but it was incorrect -- we were passing ArgumentError.new
2 arguments which is not supported. Fix this and have a hint
of which URL is causing the issue for debugging.
2023-11-22 15:19:40 +10:00
Alan Guo Xiang Tan
86da47f58d
FIX: docker:test Rake task did not run system tests in parallel (#24507)
Why this change?

We support a `USE_TURBO` environment variable which tells the
`docker:test` rake task to run rspec tests in parallel. However, this
currently does not apply to system tests.

What does this change do?

This commit runs system specs for both core and plugins using
`./bin/turbo_rspec` when the `USE_TURBO` environment is present. Note
that when running system specs, we will only spawn X number of test
processes where X is half the number of available CPU cores. This is
done because we have to leave CPU resources for the chrome processes
that will be created.
2023-11-22 11:51:43 +08:00
Blake Erickson
447d9b2105
DEV: Convert approve_unless_trust_level to groups (#24357)
This change converts the `approve_unless_trust_level` site setting to
`approve_unless_allowed_groups`.

See: https://meta.discourse.org/t/283408

- Adds the new site setting
- Adds a deprecation warning
- Updates core to use the new settings.
- Adds a migration to fill in the new setting of the old setting was
  changed
- Adds an entry to the site_setting.keywords section
- Updates many tests to account for the new change

After a couple of months we will remove the `approve_unless_trust_level`
setting entirely.

Internal ref: /t/115696
2023-11-21 11:31:42 -07:00
Roman Rizzi
75e2c6b506
DEV: Pass the user who requested the summary to the strategy. (#24489)
This change allows the `discourse-ai` plugin to log the user who requested the summary in the `AiApiAuditLog`.
2023-11-21 13:27:27 -03:00
Martin Brennan
e37fb3042d
FEATURE: Admin plugin list redesign (#24363)
* Remove checkmark for official plugins
* Add author for plugin, which is By Discourse for all discourse
  and discourse-org github plugins
* Link to meta topic instead of github repo
* Add experimental flag for plugin metadata and show this as a
  badge on the plugin list if present

---------

Co-authored-by: chapoi <101828855+chapoi@users.noreply.github.com>
2023-11-21 09:37:11 +10:00
Martin Brennan
146da75fd7
FEATURE: Add setting & preference for search sort default order (#24428)
This commit adds a new `search_default_sort_order` site setting,
set to "relevance" by default, that controls the default sort order
for the full page /search route.

If the user changes the order in the dropdown on that page, we remember
their preference automatically, and it takes precedence over the site
setting as a default from then on. This way people who prefer e.g.
Latest Post as their default can make it so.
2023-11-20 10:43:58 +10:00
Alan Guo Xiang Tan
e0ef88abca
DEV: Run QUnit tests for official Discourse themes (#24405)
Why this change?

As the number of themes which the Discourse team supports officially
grows, we want to ensure that changes made to Discourse core do not
break the plugins. As such, we are adding a step to our Github actions
test job to run the QUnit tests for all official themes.

What does this change do?

This change adds a new job to our tests Github actions workflow to run the QUnit
tests for all official plugins. This is achieved with the following
changes:

1. Update `testem.js` to rely on the `THEME_TEST_PAGES` env variable to set the
   `test_page` option when running theme QUnit tests with testem. The
   `test_page` option [allows an array to be specified](https://github.com/testem/testem#multiple-test-pages) such that tests for
   multiple pages can be run at the same time. We are relying on a ENV variable
   because  the `testem` CLI does not support passing a list of pages
   to the `--test_page` option.

2. Support a `/testem-theme-qunit/:testem_id/theme-qunit` Rails route in the development environment. This
   is done because testem prefixes the path with a unique ID to the configured `test_page` URL.
   This is problematic for us because we proxy all testem requests to the
   Rails server and testem's proxy configuration option does not allow us
   to easily rewrite the URL to remove the prefix. Therefore, we configure a proxy in testem to prefix `theme-qunit` requests with
  `/testem-theme-qunit` which can then be easily identified by the Rails server and routed accordingly. 

3. Update `qunit:test` to support a `THEME_IDS` environment variable
   which will allow it to run QUnit tests for multiple themes at the
   same time.

4. Support `bin/rake themes:qunit[ids,"<theme_id>|<theme_id>"]` to run
   the QUnit tests for multiple themes at the same time.

5. Adds a `themes:qunit_all_official` Rake task which runs the QUnit
   tests for all the official themes.
2023-11-17 07:17:32 +08:00
David Taylor
849002e90b
DEV: Silence successful db:migrate output in docker.rake (#24417)
Followup to 9449a0e0ed
2023-11-16 16:07:36 +00:00
Alan Guo Xiang Tan
416cef9ed1
DEV: Respect SKIP_TEST_DATABASE when running rake db:create (#24407)
Why this change?

By default the `db:create` Rake task in activerecord creates the
databases for both the development and test environment. This while
seemingly odd is by design from Rails. In order to avoid creating the
test database, Rails supports the `SKIP_TEST_DATABASE` environment
variable which we should respect when creating the multisite test
database.
2023-11-16 20:01:12 +08:00
Martin Brennan
68a912952c
FIX: min/max not passed to NumberField for site settings (#24402)
When we started using NumberField for integer site settings
in e113eff663, we did not end up
passing down a min/max value for the integer to the field, which
meant that for some fields where negative numbers were allowed
we were not accepting that as valid input.

This commit passes down the min/max options from the server for
integer settings then in turn passes them down to NumberField.

c.f. https://meta.discourse.org/t/delete-user-self-max-post-count-not-accepting-1-to-disable/285162
2023-11-16 12:37:05 +10:00
Alan Guo Xiang Tan
6ce55e5347
DEV: Run system tests for official themes (#24378)
Why this change?

As the number of themes which the Discourse team supports officially
grows, we want to ensure that changes made to Discourse core do not
break the plugins. As such, we are adding a step to our Github actions
test job to run the system tests for all official themes.

What does this change do?

This change adds a step to our Github actions test job to run the system
tests for all official plugins. This is achieved by the introduction of
the `themes:install_all_official` Rake task which installs all the
themes that are officially supported by the Discourse team.
2023-11-16 07:11:35 +08:00
Kris
797da5870b
FEATURE: remove category badge style options, set bullet style as default (#24198) 2023-11-13 10:46:15 -05:00
David Taylor
4982f95472
FIX: EmberCli cache clearance issue in production (#24343) 2023-11-13 10:34:06 +00:00
David Taylor
0878dde213
DEV: Modernise highlightjs loading (#24197)
- Remove vendored copy
- Update Rails implementation to look for language definitions in node_modules
- Use webpack-based dynamic import for hljs core
- Use browser-native dynamic import for site-specific language bundle (and fallback to webpack-based dynamic import in tests)
- Simplify markdown implementation to allow all languages into the `lang-{blah}` className
- Now that all languages are passed through, resolve aliases at runtime to avoid the need for the pre-built `highlightjs-aliases` index
2023-11-10 20:39:48 +00:00
David Taylor
ac896755bb
DEV: Simplify ember-cli proxy strategy (#24242)
Previously, the app HTML served by the Ember-CLI proxy was generated based on a 'bootstrap json' payload generated by Rails. This inevitably leads to differences between the Rails HTML and the Ember-CLI HTML.

This commit overhauls our proxying strategy. Now, we totally ignore the ember-cli `index.html` file. Instead, we take the full HTML from Rails and surgically replace script URLs based on a `data-discourse-entrypoint` attribute. This should be faster (only one request to Rails), more robust, and less confusing for developers.
2023-11-10 11:16:06 +00:00
Krzysztof Kotlarek
c0c525056f
UX: number site setting validation message (#24303)
Format big numbers validation to easy to read format like 1,000,000
2023-11-10 14:27:35 +11:00
Martin Brennan
731dffdf92
DEV: Align S3 transfer acceleration global settings (#24302)
Followup to fe05fdae24

For consistency with other S3 settings, make the global setting
the same name as the site setting and use SiteSetting.Upload
too so it reads from the correct place.
2023-11-10 09:50:23 +10:00
Andrei Prigorshnev
d91456fd53
DEV: Ability to collect stats without exposing them via API (#23933)
This adds the ability to collect stats without exposing them 
among other stats via API.

The most important thing I wanted to achieve is to provide 
an API where stats are not exposed by default, and a developer 
has to explicitly specify that they should be 
exposed (`expose_via_api: true`). Implementing an opposite 
solution would be simpler, but that's less safe in terms of 
potential security issues. 

When working on this, I had to refactor the current solution. 
I would go even further with the refactoring, but the next steps 
seem to be going too far in changing the solution we have, 
and that would also take more time. Two things that can be 
improved in the future:
1. Data structures for holding stats can be further improved
2. Core stats are hard-coded in the About template (it's hard 
to fix it without correcting data structures first, see point 1):
    63a0700d45/app/views/about/index.html.erb (L61-L101)

The most significant refactorings are:
1. Introducing the `Stat` model
2. Aligning the way the core and the plugin stats' are registered
2023-11-10 00:44:05 +04:00
Bianca Nenciu
bdb81b5346
DEV: Use a single registry for preloaded category custom fields (#24272)
There was a registry for preloaded site categories and a new one has
been introduced recently for categories serialized through a
CategoryList.

Having two registries created a lot of friction for developers and this
commit merges them into a single one, providing a unified API.
2023-11-09 18:23:24 +02:00
Martin Brennan
e772bb9fce
DEV: Allow new_features URL to be configurable (#24306)
This is so the new features plugin can be tested
easier locally.
2023-11-09 13:50:21 +10:00
Krzysztof Kotlarek
e5e7dd78c2 Bump version to v3.2.0.beta4-dev 2023-11-09 14:25:44 +11:00
Krzysztof Kotlarek
95b7cbd8a1 Bump version to v3.2.0.beta3 2023-11-09 14:25:44 +11:00
Ted Johansson
95a82d608d SECURITY: Prevent Onebox cache overflow by limiting downloads and URL lengths 2023-11-09 13:39:18 +11:00
Martin Brennan
3c5fb871c0 SECURITY: Filter unread bookmark reminders the user cannot see
There is an edge case where the following occurs:

1. The user sets a bookmark reminder on a post/topic
2. The post/topic is changed to a PM before or after the reminder
   fires, and the notification remains unread by the user
3. The user opens their bookmark reminder notification list
   and they can still see the notification even though they cannot
   access the topic anymore

There is a very low chance for information leaking here, since
the only thing that could be exposed is the topic title if it
changes to something sensitive.

This commit filters the bookmark unread notifications by using
the bookmarkable can_see? methods and also prevents sending
reminder notifications for bookmarks the user can no longer see.
2023-11-09 13:39:16 +11:00
Roman Rizzi
d78357917c SECURITY: Onebox templates' HTML injections.
The use of triple-curlies on Mustache templates opens the possibility for HTML injections.
2023-11-09 13:39:11 +11:00
Régis Hanol
7d484864fe SECURITY: escape display names
Ensure we escape the display names before passing it to the regexp used to update
quotes whenever a user change their display name.
2023-11-09 13:39:06 +11:00
Martin Brennan
67ac4c5616
DEV: Move min_trust_level_for_here_mention to group setting (#24263)
c.f. https://meta.discourse.org/t/-/283408
2023-11-09 12:07:01 +10:00
Martin Brennan
fe5383dbff
FIX: Prevent invalid tos and privacy URLs in cache (#24291)
Followup to 5fc1586abf

There are certain cases where the tos_url and privacy_policy_url
can end up with a "nil" value in the Discourse.urls_cache.

The cause of this is unclear, but it seems to behave differently
between doing this caching in the rails console and the running
server.

To avoid this we can just not store anything that looks like nil
in the cache; we can delete the cache keys entirely if we don't
need them anymore.
2023-11-09 10:33:38 +10:00
Andrei Prigorshnev
be2eb3df44
FIX: user got notified about a mention inside a chat message quote (#24229)
When quoting a chat message in a post, if that message contains a mention, 
that mention should be ignored. But we've been detecting them and sending 
notifications to users. This PR fixes the problem. Since this fix is for 
the chat plugin, I had to introduce a new API for plugins:

    # We strip posts before detecting mentions, oneboxes, attachments etc. 
    # We strip those elements that shouldn't be detected. For example, 
    # a mention inside a quote should be ignored, so we strip it off. 
    # Using this API plugins can register their own post strippers. 
    def register_post_stripper(&block) 
    end
2023-11-08 23:13:25 +04:00
Bianca Nenciu
277496b6e0
FIX: Replace watched words with wildcards (#24279)
These have been broken since fd07c943ad
because watched words were not correctly transformed to regexps.
This partially reverts the changes.
2023-11-08 18:51:11 +02:00
Mark VanLandingham
e3f8e9c0fb
DEV: Email notification filter plugin API (#24271) 2023-11-08 10:29:00 -06:00
David Taylor
0889f22a3b
FIX: Correctly map fingerprinted vendor.js file in production (#24267)
Followup to a0b94dca16
2023-11-07 11:34:19 +00:00
David Taylor
a0b94dca16
DEV: Use WebPack stats plugin to map entrypoints to chunks (#24239)
Previously, we were parsing webpack JS chunk filenames from the HTML files which ember-cli generates. This worked ok for simple entrypoints, but falls apart once we start using async imports(), which are not included in the HTML.

This commit uses the stats plugin to generate an assets.json file, and updates Rails to parse it instead of the HTML. Caching on the Rails side is also improved to avoid reading from the filesystem multiple times per request in develoment.

Co-authored-by: Godfrey Chan <godfreykfc@gmail.com>
2023-11-07 10:24:49 +00:00
Martin Brennan
b90b7ac705
DEV: Move shared_drafts_min_trust_level to group setting (#24257)
No plugins or themes rely on shared_drafts_min_trust_level so we
can just switch straight over to shared_drafts_allowed_groups

c.f. https://meta.discourse.org/t/changes-coming-to-settings-for-giving-access-to-features-from-trust-levels-to-groups/283408
2023-11-07 14:03:25 +10:00
Martin Brennan
fe05fdae24
DEV: Introduce S3 transfer acceleration for uploads behind hidden setting (#24238)
This commit adds an `enable_s3_transfer_acceleration` site setting,
which is hidden to begin with. We are adding this because in certain
regions, using https://aws.amazon.com/s3/transfer-acceleration/ can
drastically speed up uploads, sometimes as much as 70% in certain
regions depending on the target bucket region. This is important for
us because we have direct S3 multipart uploads enabled everywhere
on our hosting.

To start, we only want this on the uploads bucket, not the backup one.
Also, this will accelerate both uploads **and** downloads, depending
on whether a presigned URL is used for downloading. This is the case
when secure uploads is enabled, not anywhere else at this time. To
enable the S3 acceleration on downloads more generally would be a
more in-depth change, since we currently store S3 Upload record URLs
like this:

```
 url: "//test.s3.dualstack.us-east-2.amazonaws.com/original/2X/6/123456.png"
```

For acceleration, `s3.dualstack` would need to be changed to `s3-accelerate.dualstack`
here.

Note that for this to have any effect, Transfer Acceleration must be enabled
on the S3 bucket used for uploads per https://docs.aws.amazon.com/AmazonS3/latest/userguide/transfer-acceleration-examples.html.
2023-11-07 11:50:40 +10:00
Godfrey Chan
9a1695ccc1
DEV: remove markdown-it-bundle and custom build code (#23859)
With Embroider, we can rely on async `import()` to do the splitting
for us.

This commit extracts from `pretty-text` all the parts that are
meant to be loaded async into a new `discourse-markdown-it` package
that is also a V2 addon (meaning that all files are presumed unused
until they are imported, aka "static").

Mostly I tried to keep the very discourse specific stuff (accessing
site settings and loading plugin features) inside discourse proper,
while the new package aims to have some resembalance of a general
purpose library, a MarkdownIt++ if you will. It is far from perfect
because of how all the "options" stuff work but I think it's a good
start for more refactorings (clearing up the interfaces) to happen
later.

With this, pretty-text and app/lib/text are mostly a kitchen sink
of loosely related text processing utilities.

After the refactor, a lot more code related to setting up the
engine are now loaded lazily, which should be a pretty nice win. I
also noticed that we are currently pulling in the `xss` library at
initial load to power the "sanitize" stuff, but I suspect with a
similar refactoring effort those usages can be removed too. (See
also #23790).

This PR does not attempt to fix the sanitize issue, but I think it
sets things up on the right trajectory for that to happen later.

Co-authored-by: David Taylor <david@taylorhq.com>
2023-11-06 16:59:49 +00:00
David Taylor
c5e6e271a5
DEV: Remove legacy /brotli_asset workaround (#24243)
When Discourse first introduced brotli support, reverse-proxy/CDN support for passing through the accept-encoding header to our NGINX server was very poor. Therefore, a separate `/brotli_assets/...` path was introduced to serve the brotli assets. This worked well, but introduces additional complexity and inconsistencies.

Nowadays, Brotli encoding is well supported, so we don't need the separate paths any more. Requests can be routed to the asset `.js` URLs, and NGINX will serve the brotli/gzip version of the asset automatically.
2023-11-06 15:57:00 +00:00
David Taylor
82d6d691ee
DEV: Refactor discovery routes to remove use of 'named outlets' (#22622)
The motivation of this PR is to remove our dependence on Ember's 'named outlets', which are removed in Ember 4+.

At a high-level, the changes can be summarized as:

- The top-level `discovery` route is totally emptied of all logic. The HTML structure of the template is moved into the `<Discovery::Layout />` component for use by child routes.

- `AbstractTopicRoute` and `AbstractCategoryRoute` routes now both lean on the `DiscoverySortableController` and associated template. This controller is where most of the logic from the old top-level `discovery` controller has ended up.

- All navigation controllers/templates have been replaced with components. `navigation/categories`, `navigation/category` and `navigation/default` were very similar, and so they've all been combined into `<Navigation::Default>`. `navigation/filter` gets its own component.

- The `discovery/topics` controller/template have been moved into a new `<Discovery::Topics>` component.

Various other parts of the app have been tweaked to support these changes, but I've tried to keep that to a minimum.

Anything from `<TopicList>` down is untouched, which should hopefully mean that a large proportion of topic-list-customizing themes are unaffected.

For more information, see https://meta.discourse.org/t/282816
2023-11-06 10:39:31 +00:00
Martin Brennan
baeac8f105
FIX: Do not log client site setting deprecations (#24237)
For deprecated site settings, we log out a warning when
the old setting is used. However when we convert all the client
settings to JSON, we are creating a lot of log noise like this:

> Deprecation notice: `SiteSetting.anonymous_posting_min_trust_level` has been deprecated.

We don't need to do this because we are just dumping the JSON.
2023-11-06 16:36:20 +10:00
Ted Johansson
c3708c4276
DEV: Add support for custom retries for scheduled admin checks (#24224)
We updated scheduled admin checks to run concurrently in their own jobs. The main reason for this was so that we can implement re-check functionality for especially flaky checks (e.g. group e-mail credentials check.)

This works in the following way:

1. The check declares its retry policy using class methods.
2. A block can be yielded to if there are problems, but before they are committed to Redis.
3. The job uses this block to either a) schedule a retry if there are any remaining or b) do nothing and let the check commit.
2023-11-06 08:57:02 +08:00
Osama Sayegh
3cadd6769e
FEATURE: Theme settings migrations (#24071)
This commit introduces a new feature that allows theme developers to manage the transformation of theme settings over time. Similar to Rails migrations, the theme settings migration system enables developers to write and execute migrations for theme settings, ensuring a smooth transition when changes are required in the format or structure of setting values.

Example use cases for the theme settings migration system:

1. Renaming a theme setting.

2. Changing the data type of a theme setting (e.g., transforming a string setting containing comma-separated values into a proper list setting).

3. Altering the format of data stored in a theme setting.

All of these use cases and more are now possible while preserving theme setting values for sites that have already modified their theme settings.

Usage:

1. Create a top-level directory called `migrations` in your theme/component, and then within the `migrations` directory create another directory called `settings`.

2. Inside the `migrations/settings` directory, create a JavaScript file using the format `XXXX-some-name.js`, where `XXXX` is a unique 4-digit number, and `some-name` is a descriptor of your choice that describes the migration.

3. Within the JavaScript file, define and export (as the default) a function called `migrate`. This function will receive a `Map` object and must also return a `Map` object (it's acceptable to return the same `Map` object that the function received).

4. The `Map` object received by the `migrate` function will include settings that have been overridden or changed by site administrators. Settings that have never been changed from the default will not be included.

5. The keys and values contained in the `Map` object that the `migrate` function returns will replace all the currently changed settings of the theme.

6. Migrations are executed in numerical order based on the XXXX segment in the migration filenames. For instance, `0001-some-migration.js` will be executed before `0002-another-migration.js`.

Here's a complete example migration script that renames a setting from `setting_with_old_name` to `setting_with_new_name`:

```js
// File name: 0001-rename-setting.js

export default function migrate(settings) {
  if (settings.has("setting_with_old_name")) {
    settings.set("setting_with_new_name", settings.get("setting_with_old_name"));
  }
  return settings;
}
```

Internal topic: t/109980
2023-11-02 08:10:15 +03:00
Martin Brennan
d50fccfcaf
DEV: Do not auto-generate plugin routes for admin experimental sidebar (#24211)
Followup to b53449eac9, we cannot
generate the links to plugin admin pages in this way because it
depends on which plugins are installed; we would need to somehow
do it at runtime. Leaving it out for now, for people who need to
find these admin routes the Ember Inspector extension for Chrome
can be used in the meantime.
2023-11-02 12:13:20 +10:00
Martin Brennan
b53449eac9
DEV: Automatically generate all admin links for app for new sidebar (#24175)
NOTE: Most of this is experimental and will be removed at a later
time, which is why things like translations have not been added.

The new /admin-revamp UI uses a sidebar for admin nav. This initial
step adds a script to generate a map of all the current admin nav
into a format the sidebar to read. Then, people can experiment
with different changes to this structure.

The structure can then be edited from `/admin-revamp/config/sidebar-experiment`,
and it is saved to local storage so people can visually experiment with different ways
of showing the admin sidebar links.
2023-11-02 10:34:37 +10:00
Daniel Waterworth
5c92d7da22
FIX: Increase defer queue length (#24200)
It's important that there is a limit, but the current limit is too
restrictive.
2023-11-01 14:02:53 -05:00
David Taylor
85c2cc2f9e
FIX: Correctly handle materialized views during db restore (#24193) 2023-11-01 16:46:13 +00:00
Bianca Nenciu
fd07c943ad
DEV: Refactor watched words (#24163)
- Ignore only invalid words, not all words if one of them is invalid

- The naming scheme for methods was inconsistent

- Optimize regular expressions
2023-11-01 16:41:10 +02:00
Neil Lalonde
5dae0fdfb6
DEV: add a plugin modifier to change hidden site settings (#24160)
Plugins can use a new modifier to change which site settings are hidden using the :hidden_site_settings modifier. For example:

```
register_modifier(:hidden_site_settings) do |hidden|
  (hidden + [:invite_only, :login_required]).uniq
end
```
2023-10-31 10:09:51 -04:00
Martin Brennan
0a4b1b655d
FIX: Alter "Take Action" default behaviour to hide post (#24088)
This commit fixes an issue where clicking the default
"Take Action" option on a flag for a post doesn't always
end up with the post hidden.

This is because the "take_action" score bonus doesn’t take into account
the final score required to hide the post.

Especially with the `hide_post_sensitivity` site setting set to `low`
sensitivity, there is a likelihood the score needed to hide the post
won’t be reached.

Now, the default "Take Action" button has been changed to "Hide Post"
to reflect what is actually happening and the description has been
improved, and if "Take Action" is clicked we _always_ hide the post
regardless of score and sensitivity settings. This way the action reflects
expectations of the user.
2023-10-30 10:24:35 +10:00
Blake Erickson
1a78e8ec1b
FEATURE: Add keywords support for site_settings search (#24146)
* FEATURE: Add keywords support for site_settings search

This change allows for a new `keywords` field that can be added to site
settings in order to help with searching. Keywords are not visible in
the UI, but site settings matching one of the contained keywords will
appear when searching for that keyword.

Keywords can be added for site settings inside of the
`config/locales/server.en.yml` file under the new `keywords` key.

```
site_settings
  example_1: "fancy description"
  example_2: "another description"

  keywords:
    example_1: "capybara"
```

* Add keywords entry for a recently changed site setting and add system specs

* Use page.visit now that we have our own visit
2023-10-27 15:42:57 -06:00
David Taylor
983fd04f4b
FIX: Memoization in EmberCli ruby helper class (#24139)
Previously we were memoizing based on `defined?`, but the `clear_cache!` method was doing `@blah = nil`. That meant that after the cache was cleared, future calls to the memoized method would return `nil` instead of triggering a recalculation.
2023-10-27 13:35:33 +01:00
David Taylor
e2bb84757e
FIX: Ensure JS transpiler is available for multisite-migrate (#24136)
Previously done for the normal `db:migrate` command in 1e59e18ad2
2023-10-27 09:59:41 +01:00
Ted Johansson
f9f9cf0bf4
DEV: Remove unreachable IP address validation message (#24131)
The message: :signup_not_allowed option to the IP address validator does nothing, because the AllowedIpAddressValidator chooses one of either:

- ip_address.blocked or
- ip_address.max_new_accounts_per_registration_ip

internally. This means that the translation for this was also never used.

This PR removes the ineffectual option and the unused translation. It also moves the translated error messages for blocked and max_new_accounts_per_registration_ip into the correct location so we can pass a symbol to ActiveModel::Errors#add.

There is no actual change in behaviour.
2023-10-27 15:22:38 +08:00
Ted Johansson
9acdafe87c
DEV: Remove deprecated EmailValidator.email_regex (#24129)
The EmailValidator.email_regex method was moved to EmailAddressValidator.email_regex and marked for removal in 2.9.0. The method was proxied for backwards compatibility in plugins. This PR removes the method.
2023-10-27 15:11:10 +08:00
Ted Johansson
ac70ef929f
DEV: Remove deprecated #pluck_first freedom patch (#24130)
The #pluck_first method got a replacement in ActiveRecord core named #pick. After a bunch of replacements in core and plugins, we are now ready to retire this freedom patch.
2023-10-27 14:52:23 +08:00
Neil Lalonde
bd7cea7dd7
Revert "DEV: add a plugin modifier to change hidden site settings (#24104)" (#24117)
This reverts commit ddcee84e45.
2023-10-26 15:56:08 -04:00
Neil Lalonde
ddcee84e45
DEV: add a plugin modifier to change hidden site settings (#24104)
Plugins can use a new modifier to change which site settings are
hidden using the :hidden_site_settings modifier. For example:

register_modifier(:hidden_site_settings) do |hidden|
  (hidden + [:invite_only, :login_required]).uniq
end
2023-10-26 15:28:39 -04:00
David Taylor
c124c69833
DEV: Simplify sprockets configuration (#24111)
- Remove the wildcard crawler. This was already excluding almost all file types, but the exclude list was missing '.gjs' which meant those files were unnecessarily being hoisted into the `public/` directory during precompile

- Automatically include all ember-cli-generated assets without needing them to be listed. The main motivation for this change is to allow us to start using async imports via Embroider/Webpack. The filenames for those new async bundles will not be known in advance.

- Skips sprockets fingerprinting on Embroider/Webpack chunk JS files. Their filenames already include a fingerprint, and having sprockets change the filenames will cause problems for the async import feature (where filenames are included deep inside js bundles)

This commit also updates our ember-cli build so that it skips building plugin tests in the production environment. This should provide a slight build speed improvement.
2023-10-26 17:29:53 +01:00
Angus McLeod
2a75656ff2
DEV: Add category custom field preloading to CategoryList (#23969)
This commit also introduced a plugin API for preloading category custom
fields.
2023-10-26 16:34:23 +03:00
David Taylor
c88303bb27
DEV: Relax auth provider registration restrictions for plugins (#24095)
In the past we would build the stack of Omniauth providers at boot, which meant that plugins had to register any authenticators in the root of their plugin.rb (i.e. not in an `after_initialize` block). This could be frustrating because many features are not available that early in boot (e.g. Zeitwerk autoloading).

Now that we build the omniauth strategy stack 'just in time', it is safe for plugins to register their auth methods in an `after_initialize` block. This commit relaxes the old restrictions so that plugin authors have the option to move things around.
2023-10-26 10:54:30 +01:00
David Taylor
5c38e55dc9
DEV: Only run omniauth strategies for enabled authenticators (#24094)
Previously, we would build the stack of omniauth authenticators once on boot. That meant that all strategies had to be included, even if they were disabled. We then used the `before_request_phase` to ensure disabled strategies could not be used. This works well, but it means that omniauth is often doing unnecessary work running logic in disabled strategies.

This commit refactors things so that we build the stack of strategies on each request. That means we only need to include the enabled strategies in the stack - disabled strategies are totally ignored. Building the stack on-demand like this does add some overhead to auth requests, but on the majority of sites that will be significantly outweighed by the fact we're now skipping logic for disabled authenticators.

As well as the slight performance improvement, this new approach means that:

- Broken (i.e. exception-raising) strategies cannot cause issues on a site if they're disabled

- `other_phase` of disabled strategies will never appear in the backtrace of other authentication errors
2023-10-25 13:52:33 +01:00
Martin Brennan
9db4eaa870
DEV: Change anonymous_posting_min_trust_level to a group-based setting (#24072)
No plugins or themes rely on anonymous_posting_min_trust_level so we
can just switch straight over to anonymous_posting_allowed_groups

This also adds an AUTO_GROUPS const which can be imported in JS
tests which is analogous to the one defined in group.rb. This can be used
to set the current user's groups where JS tests call for checking these groups
against site settings.

Finally a AtLeastOneGroupValidator validator is added for group_list site
settings which ensures that at least one group is always selected, since if
you want to allow all users to use a feature in this way you can just use
the everyone group.
2023-10-25 11:45:10 +10:00
Neil Lalonde
8eda55e639
DEV: add a remove_step method to Wizard (#24063)
Using Wizard.exclude_steps applies to all sites in a multisite cluster.
In order to exclude steps for individual sites at run-time, a new
instance method `remove_step` is being added.
2023-10-24 13:22:55 -04:00
Blake Erickson
07360a96aa
DEV: Remove core plugins from plugin metadata file (#24070)
* DEV: Remove core plugins from plugin metadata file

* Add new plugin names
2023-10-23 15:26:59 -06:00
Jeff Wong
a2000a3559
FIX: missing quote (#24049) 2023-10-22 19:23:55 -07:00
Jeff Wong
441a330f1e
DEV: refactor rake asset precompile tasks (#24045)
* DEV: refactor rake asset precompile tasks

add a separate ember build task that does not depend on rails env
allowing us to compile assets without db+redis connections

rename EMBER_CLI_COMPILE_DONE to SKIP_EMBER_CLI_COMPILE
better semantics in build steps
2023-10-22 18:55:09 -07:00
David Taylor
c06b308895
DEV: Support RUN_SYSTEM_TESTS and SINGLE_PLUGIN in docker test task (#24040)
Previously, RUN_SYSTEM_TESTS would run system tests of all plugins. This commit makes it respect the SINGLE_PLUGIN env if it's set.
2023-10-23 07:41:21 +08:00
Sérgio Saquetim
0cfc42e0e6
FEATURE: Add dark mode option for category backgrounds (#24003)
Adds a new upload field for a dark mode category background that will be used as an alternative when Discourse is using a dark mode theme.
2023-10-20 12:48:06 +00:00
Ty Correll
700c636d29
UX: unify ai representing icon (#24029)
This PR addresses the push to unify the icon representing AI throughout Discourse, by using the discourse-sparkles icon.

The icon is being moved to core to make changes with dependencies included in core that were using the "magic" icon instead.

In 2 places "magic" -> "discourse-sparkles,
1. topic summaries
2. (unreleased) chat summaries example
2023-10-19 17:30:27 -05:00
Rafael dos Santos Silva
0604dc7d3e
FIX: Remove invalid chars from feed XMLs (#24001)
* FIX: Remove invalid chars from feed XMLs

See https://meta.discourse.org/t/rss-subscription-broken-by-post-content/282415?u=falco

* Adjust filter condition
2023-10-19 14:37:37 -03:00
Ted Johansson
47b2667099
DEV: Remove deprecated UrlHelper#escape_uri helper (#24002)
The UrlHelper#escape_uri helper has been deprecated and replaced by UrlHelper#normalized_encode, and was marked for removal in 3.0. This PR removes the method.
2023-10-19 10:37:14 +08:00
Martin Brennan
5dc45b5dcf
FIX: Secure upload post processing race condition (#23968)
* FIX: Secure upload post processing race condition

This commit fixes a couple of issues.

A little background -- when uploads are created in the composer
for posts, regardless of whether the upload will eventually be
marked secure or not, if secure_uploads is enabled we always mark
the upload secure at first. This is so the upload is by default
protected, regardless of post type (regular or PM) or category.

This was causing issues in some rare occasions though because
of the order of operations of our post creation and processing
pipeline. When creating a post, we enqueue a sidekiq job to
post-process the post which does various things including
converting images to lightboxes. We were also enqueuing a job
to update the secure status for all uploads in that post.

Sometimes the secure status job would run before the post process
job, marking uploads as _not secure_ in the background and changing
their ACL before the post processor ran, which meant the users
would see a broken image in their posts. This commit fixes that issue
by always running the upload security changes inline _within_ the
cooked_post_processor job.

The other issue was that the lightbox wrapper link for images in
the post would end up with a URL like this:

```
href="/secure-uploads/original/2X/4/4e1f00a40b6c952198bbdacae383ba77932fc542.jpeg"
```

Since we weren't actually using the `upload.url` to pass to
`UrlHelper.cook_url` here, we weren't converting this href to the CDN
URL if the post was not in a secure context (the UrlHelper does not
know how to convert a secure-uploads URL to a CDN one). Now we
always end up with the correct lightbox href. This was less of an issue
than the other one, since the secure-uploads URL works even when the
upload has become non-secure, but it was a good inconsistency to fix
anyway.
2023-10-18 23:48:01 +00:00
Jarek Radosz
16d16c8969
DEV: Precompile the transpiler before themes:update (#23997) 2023-10-19 01:00:23 +02:00
Jarek Radosz
75c9635d8b
DEV: Remove the transpilation message (#23998) 2023-10-19 01:00:15 +02:00
Godfrey Chan
c34f8b65cb
DEV: Rename I18n imports to discourse-i18n (#23915)
As of #23867 this is now a real package, so updating the imports to
use the real package name, rather than relying on the alias. The
name change in the package name is because `I18n` is not a valid
name as NPM packages must be all lowercase.

This commit also introduces an eslint rule to prevent importing from
the old I18n path.

For themes/plugins, the old 'i18n' name remains functional.
2023-10-18 11:07:09 +01:00
Martin Brennan
61c87fb59f
FIX: Properly attach secure images to email for non-secure uploads (#23865)
There are cases where a user can copy image markdown from a public
post (such as via the discourse-templates plugin) into a PM which
is then sent via an email. Since a PM is a secure context (via the
.with_secure_uploads? check on Post), the image will get a secure
URL in the PM post even though the backing upload is not secure.

This fixes the bug in that case where the image would be stripped
from the email (since it had a /secure-uploads/ URL) but not re-attached
further down the line using the secure_uploads_allow_embed_images_in_emails
setting because the upload itself was not secure.

The flow in Email::Sender for doing this is still not ideal, but
there are chicken and egg problems around when to strip the images,
how to fit in with other attachments and email size limits, and
when to apply the images inline via Email::Styles. It's convoluted,
but at least this fixes the Template use case for now.
2023-10-17 14:08:21 +10:00
Penar Musaraj
1400d4a8fd
Bump version to v3.2.0.beta3-dev 2023-10-16 11:20:22 -04:00
Penar Musaraj
be04154838
Bump version to v3.2.0.beta2 2023-10-16 11:20:20 -04:00
Alan Guo Xiang Tan
cbbe3a808b
SECURITY: Add a default limit as to when logs should be truncated
Why this change?

This ensures that malicious requests cannot end up causing the logs to
quickly fill up. The default chosen is sufficient for most legitimate
requests to the Discourse application.

When truncation happens, parsing of logs in supported format like
lograge may break down.
2023-10-16 10:34:38 -04:00
Alan Guo Xiang Tan
4cb7472376
SECURITY: Prevent arbitrary topic custom fields from being set
Why this change?

The `PostsController#create` action allows arbitrary topic custom fields
to be set by any user that can create a topic. Without any restrictions,
this opens us up to potential security issues where plugins may be using
topic custom fields in security sensitive areas.

What does this change do?

1. This change introduces the `register_editable_topic_custom_field` plugin
API which allows plugins to register topic custom fields that are
editable either by staff users only or all users. The registered
editable topic custom fields are stored in `DiscoursePluginRegistry` and
is called by a new method `Topic#editable_custom_fields` which is then
used in the `PostsController#create` controller action. When an unpermitted custom fields is present in the `meta_data` params,
a 400 response code is returned.

2. Removes all reference to `meta_data` on a topic as it is confusing
   since we actually mean topic custom fields instead.
2023-10-16 10:34:35 -04:00
Bianca Nenciu
76bdea5ce2
SECURITY: Hide user profiles from public
User profiles, including the summary, should be private to anonymous
users if hide_user_profiles_from_public is enabled.
2023-10-16 10:34:32 -04:00
Blake Erickson
2443446e62
DEV: Prevent videos from preloading metadata (#23807)
Preloading just metadata is not always respected by browsers, and
sometimes the whole video will be downloaded. This switches to using a
placeholder image for the video and only loads the video when the play
button is clicked.
2023-10-12 13:47:48 -06:00
Godfrey Chan
2e00482ac4
DEV: convert I18n pseudo package into real package (discourse-i18n) (#23867)
Currently, `window.I18n` is defined in an old school hand written
script, inlined into locale/*.js by the Rails asset pipeline, and
then the global variable is shimmed into a pseudo AMD module later
in `module-shims.js`.

This approach has some problems – for one thing, when we add a new
V2 addon (e.g. in #23859), Embroider/Webpack is stricter about its
dependencies and won't let you `import from "I18n";` when `"I18n"`
isn't listed as one of its `dependencies` or `peerDependencies`.

This moves `I18n` into a real package – `discourse-i18n`. (I was
originally planning to keep the `I18n` name since it's a private
package anyway, but NPM packages are supposed to have lower case
names and that may cause problems with other tools.)

This package defines and exports a regular class, but also defines
the default global instance for backwards compatibility. We should
use the exported class in tests to make one-off instances without
mutating the global instance and having to clean it up after the
test run. However, I did not attempt that refactor in this PR.

Since `discourse-i18n` is now included by the app, the locale
scripts needs to be loaded after the app chunks. Since no "real"
work happens until later on when we kick things off in the boot
script, the order in which the script tags appear shouldn't be a
problem. Alternatively, we can rework the locale bundles to be more
lazy like everything else, and require/import them into the app.

I avoided renaming the imports in this commit since that would be
quite noisy and drowns out the actual changes here. Instead, I used
a Webpack alias to redirect the current `"I18n"` import to the new
package for the time being. In a separate commit later on, I'll
rename all the imports in oneshot and remove the alias. As always,
plugins and the legacy bundles (admin/wizard) still relies on the
runtime AMD shims regardless.

For the most part, I avoided refactoring the actual I18n code too
much other than making it a class, and some light stuff like `var`
into `let`.

However, now that it is in a reasonable format to work with (no
longer inside the global script context!) it may also be a good
opportunity to refactor and make clear what is intended to be
public API vs internal implementation details.

Speaking of, I took the librety to make `PLACEHOLDER`, `SEPARATOR`
and `I18nMissingInterpolationArgument` actual constants since it
seemed pretty clear to me those were just previously stashed on to
the `I18n` global to avoid polluting the global namespace, rather
than something we expect the consumers to set/replace.
2023-10-12 14:44:01 +01:00
Penar Musaraj
e3e73a3091
DEV: Add routes and controller actions for passkeys (2/3) (#23587)
This is part 2 (of 3) for passkeys support.

This adds a hidden site setting plus routes and controller actions.

1. registering passkeys

Passkeys are registered in a two-step process. First, `create_passkey`
returns details for the browser to create a passkey. This includes
- a challenge
- the relying party ID and Origin
- the user's secure identifier
- the supported algorithms
- the user's existing passkeys (if any)

Then the browser creates a key with this information, and submits it to
the server via `register_passkey`.

2. authenticating passkeys

A similar process happens here as well. First, a challenge is created
and sent to the browser. Then the browser makes a public key credential
and submits it to the server via `passkey_auth_perform`.

3. renaming/deleting passkeys

These routes allow changing the name of a key and deleting it.

4. checking if session is trusted for sensitive actions

Since a passkey is a password replacement, we want to make sure to confirm the user's identity before allowing adding/deleting passkeys. The u/trusted-session GET route returns success if user has confirmed their session (and failed if user hasn't). In the frontend (in the next PR), we're using these routes to show the password confirmation screen. 

The `/u/confirm-session` route allows the user to confirm their session with a password. The latter route's functionality already existed in core, under the 2FA flow, but it has been abstracted into its own here so it can be used independently.


Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2023-10-11 14:36:54 -04:00
Mark VanLandingham
4c584f6e03
FIX: List parent/child tags correctly for categories restricted to tag groups (#23708)
Co-authored-by: Sérgio Saquetim <saquetim@discourse.org>
2023-10-10 17:30:24 +00:00
Mark VanLandingham
f29c476521
DEV: Add hooks to allow overriding notify_user behavior (#23850)
Adds new plugin registry `:post_action_notify_user_handlers` and more!
2023-10-10 12:21:57 -05:00
David Taylor
3f8a85ed49
DEV: Write ember exam execution file for plugin qunit in CI (#23877) 2023-10-10 16:29:28 +01:00