Commit Graph

6479 Commits

Author SHA1 Message Date
Osama Sayegh
e3ae57ea7a
FIX: Create directory items for new users when in bootstrap mode (#27020)
The users directory is updated on a daily cadence. However, when a site is new and doesn't have many users, it can be confusing that a user who has just joined doesn't show up in the users until a day after they join. To eliminate this confusion, this commit triggers a refresh for the users directory as soon as as a user joins, if the site is in bootstrap mode. The reason for the conditional trigger is that refreshing the users directory is an expensive operation and doing it often on a large site with many users could lead to performance problems.

Internal topic: t/126076.
2024-05-15 03:06:58 +03:00
Régis Hanol
1eec8c3fa6 FEATURE: add HTML replacements
This adds support for Watched Words to allow replacement with HTML content rather than always replacing with text.

Can be useful when automatically replacing with the '<abbr>' tag for example.

Discussion - https://meta.discourse.org/t/replace-text-with-more-than-just-links/305672
2024-05-14 10:41:27 +02:00
Bianca Nenciu
3e7601cada
DEV: Embed IDs in serializer for about page (#26679)
It used to embed the objects which could lead to duplicated objects
when the same user or category was used multiple times (user was admin,
moderator and category or category was parent for multiple categories).
2024-05-10 20:11:43 +03:00
Osama Sayegh
3be4924b99
DEV: Move array type custom fields to JSON type in automation (#26939)
The automation plugin has 4 custom field types that are array typed. However, array typed custom fields are deprecated and should be migrated to JSON type.

This commit does a couple of things:

1. Migrate all four custom fields to JSON
2. Fix a couple of small bugs that have been discovered while migrating the custom fields to JSON (see the comments on this commit's PR for details https://github.com/discourse/discourse/pull/26939)
2024-05-10 18:47:12 +03:00
David Taylor
97847f6cd8
Revert "DEV: @babel/plugin-proposal-decorators -> decorator-transforms (#25290)" (#26971)
This reverts commit 0f4520867b.

This has led to two problems:

1. An incompatibility with Cloudflare's "auto minify" feature. They've deprecated this feature because of incompatibility with modern JS syntax. But unfortunately it will remain enabled on existing properties until 2024-08-05.

2. Discourse fails to boot in Safari 15. This is strange, because Safari does support all the required features in our production JS bundles. Even more strangely, things start working as soon as you open the developer tools. That suggests the cause could be a Safari bug rather than a simple incompatibility.

Reverting while we work out a path forward on both those issues.
2024-05-10 12:48:16 +01:00
Natalie Tay
113e6fd274
DEV: Increase external avatar url limit (#26966) 2024-05-10 16:19:23 +08:00
David Taylor
0f4520867b
DEV: @babel/plugin-proposal-decorators -> decorator-transforms (#25290)
decorator-transforms (https://github.com/ef4/decorator-transforms) is a modern replacement for babel's plugin-proposal-decorators. It provides a decorator implementation using modern browser features, without needing to enable babel's full suite of class feature transformations. This improves the developer experience and performance.

In local testing with Google's 'tachometer' tool, this reduces Discourse's 'init-to-render' time by around 3-4% (230ms -> 222ms).

It reduces our initial gzip'd JS payloads by 3.2% (2.43MB -> 2.35MB), or 7.5% (14.5MB -> 13.4MB) uncompressed.
2024-05-08 10:40:51 +01:00
Alan Guo Xiang Tan
0b947b6aab
DEV: Improve code comment about when ignored columns can be removed (#26894)
Ignored columns can only be dropped when its associated post-deploy
migration has been promoted to a regular migration. This is so because
Discourse doesn't rely on a schema file system to setup a brand new
database and thus the column information will be loaded by the
application first before the post-deploy migration runs.
2024-05-07 11:06:31 +08:00
Jarek Radosz
378faf060d
Revert "DEV: Remove unused ignored_columns from ActiveRecord models (#26875)" (#26887)
This reverts commit 755ff43dc1.
2024-05-06 20:18:53 +02:00
Daniel Waterworth
e2ceea8815
FIX: Preload all ancestors of sidebar categories (#26715)
... instead of just the immediate parents.
2024-05-06 11:55:20 -05:00
Daniel Waterworth
a6b8051645
DEV: Use has_many and ArraySerializer for SidebarSectionsSerializer (#26716) 2024-05-06 11:32:18 -05:00
Régis Hanol
10f77556cd FIX: ensure no infinite category loop
If there's ever a circular reference in categories, don't go into an infinite loop when generating the category slug.

Instead, keep track of parent ids, and bail out as soon as we're encountering one more than once.
2024-05-06 18:02:22 +02:00
Ted Johansson
9655bf3e24
DEV: Delete upload references on draft cleanup (#26877)
In #22851 we added a dependent strategy for deleting upload references when a draft is destroyed. This, however, didn't catch all cases, because we still have some code that issues DELETE drafts queries directly to the database. Specifically in the weekly cleanup job handled by Draft#cleanup!.

This PR fixes that by turning the raw query into an ActiveRecord #destroy_all, which will invoke the dependent strategy that ultimately deletes the upload references. It also includes a post migration to clear orphaned upload references that are already in the database.
2024-05-06 14:08:10 +08:00
Alan Guo Xiang Tan
755ff43dc1
DEV: Remove unused ignored_columns from ActiveRecord models (#26875)
The columns have already been dropped in
beea8215d7.
2024-05-06 11:42:08 +10:00
Alan Guo Xiang Tan
243fcb6ffc
DEV: Introduce run_theme_migration spec helper in test environment (#26845)
This commit introduces the `run_theme_migration` spec helper to allow
theme developers to write RSpec tests for theme migrations. For example,
this allows the following RSpec test to be written in themes:

```
RSpec.describe "0003-migrate-small-links-setting migration" do
  let!(:theme) { upload_theme_component }

  it "should set target property to `_blank` if previous target component is not valid or empty" do
    theme.theme_settings.create!(
      name: "small_links",
      theme: theme,
      data_type: ThemeSetting.types[:string],
      value: "some text, #|some text 2, #, invalid target",
    )

    run_theme_migration(theme, "0003-migrate-small-links-setting")

    expect(theme.settings[:small_links].value).to eq(
      [
        { "text" => "some text", "url" => "#", "target" => "_blank" },
        { "text" => "some text 2", "url" => "#", "target" => "_blank" },
      ],
    )
  end
end
```

This change is being introduced because we realised that writting just
javascript tests for the migrations is insufficient since javascript
tests do not ensure that the migrated theme settings can actually be
successfully saved into the database. Hence, we are introduce this
helper as a way for theme developers to write "end-to-end" migrations
tests.
2024-05-03 06:29:18 +08:00
Régis Hanol
e7d0083dbe FIX: creating watched words...
... wasn't working because it wasn't storing the proper "action" value.

Issue was that we were using the "action" parameter which is being used by Rails to determine which controller action to call.

We need to use the "action_key" parameter instead.
2024-04-30 19:16:47 +02:00
Vinoth Kannan
143f06f2c6
FEATURE: Allow watched words to be created as a group (#26632)
At the moment, there is no way to create a group of related watched words together.  If a user needed a set of words to be created together, they'll have to create them individually one at a time.

This change attempts to allow related watched words to be created as a group. The idea here is to have a list of words be tied together via a common `WatchedWordGroup` record.  Given a list of words, a `WatchedWordGroup` record is created and assigned to each `WatchedWord` record. The existing WatchedWord creation behaviour remains largely unchanged.

Co-authored-by: Selase Krakani <skrakani@gmail.com>
Co-authored-by: Martin Brennan <martin@discourse.org>
2024-04-29 15:50:55 +05:30
David Battersby
0c8f531909
FEATURE: encourage users to set chat thread titles (#26617)
This change encourages users to title their threads to make it easier for other users to join in on conversations that matter to them.

The creator of the chat thread will receive a toast notification prompting them to add a thread title when on mobile and the thread has at least 5 sent replies.
2024-04-29 17:20:01 +08:00
Martin Brennan
94fe31e5b3
UX: Fix report color duplication (#26799)
In the "Consolidated Pageviews with Browser Detection (Experimental)"
report we used the same color for "Known Crawler" and "Other pageviews"
which makes the report confusing to look at, this commit makes them
different.
2024-04-29 15:12:47 +10:00
Martin Brennan
edec941a87
FIX: Better tracking of topic visibility changes (#26709)
This commit introduces a few changes as a result of
customer issues with finding why a topic was relisted.
In one case, if a user edited the OP of a topic that was
unlisted and hidden because of too many flags, the topic
would get relisted by directly changing topic.visible,
instead of going via TopicStatusUpdater.

To improve tracking we:

* Introduce a visibility_reason_id to topic which functions
  in a similar way to hidden_reason_id on post, this column is
  set from the various places we change topic visibility
* Fix Post#unhide! which was directly modifying topic.visible,
  instead we use TopicStatusUpdater which sets visibility_reason_id
  and also makes a small action post
* Show the reason topic visibility changed when hovering the
  unlisted icon in topic status on topic titles
2024-04-29 10:34:46 +10:00
Daniel Waterworth
e0e0e0506f
DEV: Limit the number of category sidebar links a user can have (#26756) 2024-04-25 13:21:39 -05:00
Bianca Nenciu
dbe923d26e
FIX: Fetch categories for "+subcategories" option (#26622)
Selecting the +subcategories option does not work sometimes when "lazy
load categories" is enabled because the subcategories may not be
fetched. This ensures that subcategories are loaded by requesting them
before being used.
2024-04-25 16:47:45 +03:00
Natalie Tay
00a9369ca2
FIX: Move user reindexing into a job (#26753)
In a large forum with millions of users and millions of user_fields
updating the list of dropdown user field options will result in a
502 now due to the large number of fields.

This commit moves the indexing into a job.
2024-04-25 20:58:34 +08:00
David Taylor
2f2da72747
FEATURE: Add experimental tracking of 'real browser' pageviews (#26647)
Our 'page_view_crawler' / 'page_view_anon' metrics are based purely on the User Agent sent by clients. This means that 'badly behaved' bots which are imitating real user agents are counted towards 'anon' page views.

This commit introduces a new method of tracking visitors. When an initial HTML request is made, we assume it is a 'non-browser' request (i.e. a bot). Then, once the JS application has booted, we notify the server to count it as a 'browser' request. This reliance on a JavaScript-capable browser matches up more closely to dedicated analytics systems like Google Analytics.

Existing data collection and graphs are unchanged. Data collected via the new technique is available in a new 'experimental' report.
2024-04-25 11:00:01 +01:00
Vinoth Kannan
859b55366f
DEV: don't send moderator welcome message to first admin. (#26719)
We already skipping the admin welcome message for the first admin user. We should also skip the moderator message.
2024-04-24 00:20:14 +05:30
Penar Musaraj
98d400f7b5
DEV: Refactor discover setting reporting (#26706) 2024-04-23 09:52:01 -04:00
Krzysztof Kotlarek
e1d9fd479f
FEATURE: after wizard admin is redirected to the guide page (#26696)
After the wizard is completed, the admin should be redirected to the admin guide topic.

Also tooltip from "Getting started" button was removed.
2024-04-23 10:04:15 +10:00
Sam
1c67917367
FIX: disable storing invalid post and topic timing when sent from client (#26683)
This ensures we only ever store correct post and topic timing when the client
notifies.

Previous to this change we would blindly trust the client.

Additionally this has error correction code that will correct the last seen
post number when you visit a topic with incorrect timings.
2024-04-19 18:10:50 +10:00
Krzysztof Kotlarek
c5dd50aa02
FIX: don't purge users who were deactivated by the system (#26656)
In a previous PR, we skipped users deactivated by admins from being automatically purged - https://github.com/discourse/discourse/pull/26478.

However, users deactivated by `discourse-auto-deactivate` plugin are deactivated by the system user. Those users should not be purged as well.

https://github.com/discourse/discourse-auto-deactivate/blob/main/plugin.rb#L62
2024-04-18 09:53:43 +10:00
Bianca Nenciu
9638ce17fa
FIX: Serialize categories for bookmarks (#26606)
This is necessary when "lazy load categories" feature is enabled to make
sure the categories are rendered for topics and posts.
2024-04-17 17:23:47 +03:00
Martin Brennan
7a083daf27
Revert "FIX: Post uploads setting access_control_post_id unnecessarily (#26627)" (#26643)
This reverts commit cdc8e9de1b.

It's made things worse internally and on meta.
2024-04-16 14:10:25 +10:00
Martin Brennan
cdc8e9de1b
FIX: Post uploads setting access_control_post_id unnecessarily (#26627)
This commit addresses an issue for sites where secure_uploads
is turned on after the site has been operating without it for
some time.

When uploads are linked when they are used inside a post,
we were setting the access_control_post_id unconditionally
if it was NULL to that post ID and secure_uploads was true.

However this causes issues if an upload has been used in a
few different places, especially if a post was previously
used in a PM and marked secure, so we end up with a case of
the upload using a public post for its access control, which
causes URLs to not use the /secure-uploads/ path in the post,
breaking things like image uploads.

We should only set the access_control_post_id if the post is the first time the
upload is referenced so it cannot hijack uploads from other places.
2024-04-16 10:37:57 +10:00
Blake Erickson
334a2f216f
FEATURE: User Notifications Links Page (#26609) 2024-04-11 15:54:28 -06:00
Bianca Nenciu
8ce836c039
FIX: Load categories with user activity and drafts (#26553)
When lazy load categories is enabled, categories should be loaded with
user activity items and drafts because the categories may not be
preloaded on the client side.
2024-04-10 17:35:42 +03:00
David Taylor
3733db866c
DEV: Introduce default 'auto' mode for glimmer header (#26467)
This will automatically enable the glimmer header when all installed themes/plugins are ready. This replaces the old group-based site setting.

In 'auto' mode, we check for calls to deprecated APIs (e.g. decorateWidget) which affect the old header. If any are present, we stick to the old header implementation and print a message to the console alongside the normal deprecation messages.

To override this automatic behavior, a new `glimmer_header_mode` site setting can be set to 'disabled' or 'enabled'.

This change also means that our test suite is running with the glimmer header. This unveiled a couple of small issues (e.g. some incorrect `aria-*` and `alt` text) which are now fixed. A number of selectors had to be updated to ensure the tests were clicking the actual `<button>` elements rather than the surrounding `<li>` elements.
2024-04-10 14:35:54 +01:00
Martin Brennan
b7a2d29b7b
DEV: Introduce post_should_secure_uploads? plugin modifier (#26508)
This modifier allows plugins to alter the outcome of
`should_secure_uploads?` on a Post record, for cases when
plugins need post-attached uploads to always be secure (or
not secure) in specific scenarios.
2024-04-10 12:02:44 +10:00
Penar Musaraj
a52b1d6b4a
FIX: Let users reset their homepage choice if custom homepage is from… (#26536)
Co-authored-by: Régis Hanol <regis@hanol.fr>
2024-04-09 15:54:44 -04:00
Blake Erickson
72ac675e4e
FEATURE: Consolidate link notifications (#26567)
Just like we have for consolidating likes this adds similar
functionality for consolidating links.
2024-04-09 11:53:37 -06:00
Martin Brennan
0d0dbd391a
DEV: Rename with_secure_uploads? to should_secure_uploads? on Post (#26549)
This method name is a bit confusing; with_secure_uploads implies
it may return a block or something with the uploads of the post,
and has_secure_uploads implies that it's checking whether the post
is linked to any secure uploads.

should_secure_uploads? communicates the true intent of this method --
which is to say whether uploads attached to this post should be
secure or not.
2024-04-09 13:23:11 +10:00
Bianca Nenciu
6c2c1a43ae
FIX: Load categories with post revisions (#26496)
When lazy load categories is enabled, categories should be loaded with
post revisions because the categories may not be preloaded on the client
side.
2024-04-08 11:33:33 +03:00
Sérgio Saquetim
175d2451a9
DEV: Add topic_embed_import_create_args plugin modifier (#26527)
* DEV: Add `topic_embed_import_create_args` plugin modifier

This modifier allows a plugin to change the arguments used when creating
 a new topic for an imported article.

For example: let's say you want to prepend "Imported: " to the title of
every imported topic. You could use this modifier like so:

```ruby
# In your plugin's code
plugin.register_modifier(:topic_embed_import_create_args) do |args|
  args[:title] = "Imported: #{args[:title]}"
  args
end
```

In this example, the modifier is prepending "Imported: " to the `title` in the `create_args` hash. This modified title would then be used when the new topic is created.
2024-04-05 12:37:53 -03:00
Bianca Nenciu
19eb0a7055
FIX: Load category info for about page (#26519) 2024-04-05 09:38:54 +03:00
Gerhard Schlager
82c62fe44f
DEV: Correctly pluralize error messages (#26469) 2024-04-04 15:02:09 +02:00
Alan Guo Xiang Tan
a440e15291
DEV: Remove experimental_objects_type_for_theme_settings site setting (#26507)
Why this change?

Objects type for theme settings is no longer considered experimental so
we are dropping the site setting.
2024-04-04 12:01:31 +08:00
Isaac Janzen
db10dd5319
PERF: Improve performance of most_replied_to_users (#26373)
This PR improves the performance of the `most_replied_to_users` method on the `UserSummary` model.

### Old Query
```ruby
    post_query
      .joins(
        "JOIN posts replies ON posts.topic_id = replies.topic_id AND posts.reply_to_post_number = replies.post_number",
      )
      # We are removing replies by @user, but we can simplify this by getting the using the user_id on the posts.
      .where("replies.user_id <> ?", @user.id)
      .group("replies.user_id")
      .order("COUNT(*) DESC")
      .limit(MAX_SUMMARY_RESULTS)
      .pluck("replies.user_id, COUNT(*)")
      .each { |r| replied_users[r[0]] = r[1] }
```
 
### Old Query with corrections

```ruby
post_query
  .joins(
    "JOIN posts replies ON posts.topic_id = replies.topic_id AND replies.reply_to_post_number = posts.post_number",
  )
  # Remove replies by @user but instead look on loaded posts (we do this so we don't count self replies)
  .where("replies.user_id <> posts.user_id")
  .group("replies.user_id")
  .order("COUNT(*) DESC")
  .limit(MAX_SUMMARY_RESULTS)
  .pluck("replies.user_id, COUNT(*)")
  .each { |r| replied_users[r[0]] = r[1] }
```

### New Query
```ruby
    post_query
      .joins(
        "JOIN posts replies ON posts.topic_id = replies.topic_id AND posts.reply_to_post_number = replies.post_number",
      )
      # Only include regular posts in our joins, this makes sure we don't have the bloat of loading private messages
      .joins(
        "JOIN topics ON replies.topic_id = topics.id AND topics.archetype <> 'private_message'",
      )
      # Only include visible post types, so exclude posts like whispers, etc
      .joins(
        "AND replies.post_type IN (#{Topic.visible_post_types(@user, include_moderator_actions: false).join(",")})",
      )
      .where("replies.user_id <> posts.user_id")
      .group("replies.user_id")
      .order("COUNT(*) DESC")
      .limit(MAX_SUMMARY_RESULTS)
      .pluck("replies.user_id, COUNT(*)")
      .each { |r| replied_users[r[0]] = r[1] }
```

# Conclusion

`most_replied_to_users` was untested, so I introduced a test for the logic, and have confirmed that it passes on both the new query **AND** the old query. 

Thank you @danielwaterworth for the debugging assistance.
2024-04-03 14:20:54 -06:00
Vinoth Kannan
9dc6325821
DEV: add logo URL and locale details to the Discover stats. (#26320)
We will be collecting the logo URL and the site's default locale values along with existing basic details to display the site on the Discourse Discover listing page. It will be included only if the site is opted-in by enabling the "`include_in_discourse_discover`" site setting.

Also, we no longer going to use `about.json` and `site/statistics.json` endpoints retrieve these data. We will be using only the `site/basic-info.json` endpoint.
2024-04-04 00:22:28 +05:30
Osama Sayegh
3d4faf3272
FEATURE: Merge discourse-automation (#26432)
Automation (previously known as discourse-automation) is now a core plugin.
2024-04-03 18:20:43 +03:00
Krzysztof Kotlarek
ba04fc6a01
FEATURE: ignore manually deactivated users when purging (#26478)
When a user is manually deactivated, they should not be deleted by our background job that purges inactive users.

In addition, site settings keywords should accept an array of keywords.
2024-04-03 14:06:31 +11:00
Penar Musaraj
1eb70973a2
DEV: allow themes to render their own custom homepage (#26291)
This PR adds a theme modifier and route so that custom themes can opt to show their own homepage. See PR description for example usage.
2024-04-02 11:05:08 -04:00
Bianca Nenciu
3b9e9354d6
DEV: Better categories pagination (#26421)
Pagination is enabled only when "lazy load categories" is enabled. For
those cases when it is not, the first page should return all the
results.
2024-03-28 18:19:09 +02:00
Martin Brennan
1ab2fe0a81
DEV: Add missing belongs_to to UserAction model (#26415) 2024-03-28 16:13:18 +10:00
Ted Johansson
0c875cb4d5
DEV: Make problem check registration more explicit (#26413)
Previously the problem check registry simply looked at the subclasses of ProblemCheck. This was causing some confusion in environments where eager loading is not enabled, as the registry would appear empty as a result of the classes never being referenced (and thus never loaded.)

This PR changes the approach to a more explicit one. I followed other implementations (bookmarkable and hashtag autocomplete.) As a bonus, this now has a neat plugin entry point as well.
2024-03-28 14:00:47 +08:00
Alan Guo Xiang Tan
69af29cc40
DEV: Add a test to ensure that our SMTP settings are correct (#26410)
Why this change?

This is a follow up to 897be75941.

When updating `net-smtp` from `0.4.x` to `0.5.x`, our test suite passed
but the error `ArgumentError: SMTP-AUTH requested but missing user name`
was being thrown in production leading to emails being failed to send
out via SMTP.

This commit adds a test to ensure that our production SMTP settings will
at least attemp to connect to an SMTP server.
2024-03-28 10:18:19 +08:00
Sérgio Saquetim
abf86271ff
DEV: Extract the query code from CategoryList.find_relevant_topics into a separate method (#26390)
## Why this change?
The previous implementation of the method generated the query to find the relevant topics and iterated over the results, processing them.

This behavior made difficult reusing or changing the query logic in classes extending `CategoryList`.

This commit extracts the query logic into another method called `relevant_topics_query ` which can be reused or overwritten in descendant classes.
2024-03-27 16:32:45 -03:00
Juan David Martínez Cubillos
fd83107674
FIX: Export false on confirm user fields when using user invites (#26332) 2024-03-27 09:12:14 -04:00
Angus McLeod
7dc552c9cc
DEV: Add import_embed_unlisted site setting (#26222) 2024-03-27 08:57:43 -04:00
Ted Johansson
5875b25e68
DEV: Add the ability for problem checks to specify 'max blips' (#26388)
This was originally introduced in #26071, but that PR was closed, because the requirements changed. This PR lifts only the relevant parts, since they are a prerequisite for the new admin notice system.
2024-03-27 10:07:56 +08:00
Daniel Waterworth
cc8f0a79e2
PERF: Replace posts reply_to_post_number index (#26385)
post_number is a sequence per topic, so it doesn't make sense to have an
index on only reply_to_post_number without also including the topic_id.
2024-03-27 09:56:29 +08:00
Sam
e3a0faefc5
FEATURE: allow re-scoping chat user search via a plugin (#26361)
This enables the following in Discourse AI

```
 plugin.register_modifier(:chat_allowed_bot_user_ids) do |user_ids, guardian|
  if guardian.user
    mentionables = AiPersona.mentionables(user: guardian.user)
    allowed_bot_ids = mentionables.map { |mentionable| mentionable[:user_id] }
    user_ids.concat(allowed_bot_ids)
  end
  user_ids
end
```

some bots that are id < 0 need to be discoverable in search otherwise people can not talk to them.

---------

Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
2024-03-27 08:55:53 +11:00
Daniel Waterworth
739a139bcf
DEV: Reduce duplication (#26329) 2024-03-25 11:56:32 -05:00
Daniel Waterworth
d52abe2324
FIX: Set has_children correctly in Category.preload_user_fields! (#26327) 2024-03-22 12:41:28 -05:00
Bianca Nenciu
4cdf5f2cea
FIX: Load subcategories through CategoryList (#26297)
When "lazy load categories" is enabled and parent_category_id was set,
the query fetching categories contained a contradiction filtering both
by parent_category_id and parent_category_id = NULL.
2024-03-21 21:39:14 +02:00
Bianca Nenciu
a6e06915c4
FIX: Serialize parent categories first (#26294)
When categories are loaded by the frontend, the parent category is
looked up by ID and the `parentCategory` is set with the result. If the
categories returned are not in order, the parent category may miss.
2024-03-21 19:51:41 +02:00
Loïc Guitaut
d2a730b8b5 DEV: Expose extra data from themes
This patch exposes a normalized repository URL and how many users are
using a given theme.
2024-03-21 15:06:36 +01:00
David Taylor
4e082c7390
FIX: Ensure sitemaps include all topics with no duplicates (#26289)
We were using `OFFSET`/`LIMIT` to query topics without an 'ORDER'. Without an explicit order, postgres makes no guarantees about which rows will be returned for each query. This commit adds `ORDER BY id ASC` so that our sitemaps behave consistently.
2024-03-21 13:19:53 +00:00
David Taylor
e3cfb1967d
FIX: Simplify sidebar custom link implementation (#26201)
All our link validation, and conversion from url -> route/model/query is expensive and prone to bugs. Instead, if people enter a link, we can just use it as-is.

Originally all this extra logic was added to handle unusual situations like `/safe-mode`, `/my/...`, etc. However, all of these are now handled correctly by our Ember router, so there is no need for it.

Now, we just pass the user-supplied `href` directly to the SectionLink component, and let Ember handle routing to it when clicked.

The only functional change here is that we no longer validate internal links by parsing them with the Ember router. But I'd argue this is fine, because the previous logic would cause both false positives (e.g. `/t/123` would be valid, even if topic 123 doesn't exist), and false negatives (for routes which are server-side only, like the new AI share pages).
2024-03-20 12:55:40 +00:00
Joffrey JAFFEUX
a884842fa5
FIX: do not use return in block (#26260)
We were incorrectly using `return` in a block which was causing exceptions at runtime. These exceptions were not causing much issues as they are in defer block.

While working on writing a test for this specific case, I noticed that our `upsert_custom_fields` function was using rails `update_all` which is not updating the `updated_at` timestamp. This commit also fixes it and adds a test for it.
2024-03-20 10:49:28 +01:00
Ted Johansson
4ca41e0af2
DEV: Promote block problem checks to ProblemCheck (#26193)
In #26122 we promoted all problem checks defined as class methods on AdminDashboardData to their own first-class ProblemCheck instances.

This PR continues that by promoting problem checks that are implemented as blocks as well. This includes updating a couple plugins that have problem checks.
2024-03-20 08:52:25 +08:00
Alan Guo Xiang Tan
239b70342f
PERF: Remove unnecessary <link rel="preload"> for theme javascript (#26220)
This is a follow up to e2da72b76c.

Why this change?

According to https://web.dev/articles/preload-critical-assets,

> By preloading a certain resource, you are telling the browser that you would like to fetch it sooner than the browser would otherwise discover it because you are certain that it is important for the current page.

The preload resource hint is meant to tell the browser to fetch
resources that it would not discover upfront or early. However, we are
not using it the right way because we are literally adding the resource
hint right before a `<script>` tag which means the browser would have
discovered the resource even without the resource hint.

What does this change do?

This commit removes the preload resource hint which are added right
before script tags since the optimization here is highly questionable at the expense of making
our initial DOM larger.
2024-03-19 07:03:49 +11:00
Régis Hanol
4e02bb5dd9
PERF: avoid publishing user actions to the user who did the action (#26225)
We never use that information and this also fixes an issue with the BCC plugin which ends up triggering a rate-limit because we were publishing a "NEW_PRIVATE_MESSAGE" to the user sending the BCC for every recipients 💥

Internal - t/118283
2024-03-18 18:05:46 +01:00
Penar Musaraj
531e33b303
DEV: Allow user api key scope for notifications#totals (#26205)
The `/notifications/totals` route is a stripped down version of `notifications#index`. This just allows the mobile app to use this new route.
2024-03-15 16:06:32 -04:00
Penar Musaraj
8cf2f909f5
DEV: Dedicated route for current user notification counts (#26106)
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2024-03-15 12:08:37 -04:00
Penar Musaraj
62ea382247
SECURITY: Limit invites params length 2024-03-15 14:24:07 +08:00
Daniel Waterworth
8cade1e825
SECURITY: Prevent large staff actions causing DoS
This commit operates at three levels of abstraction:

 1. We want to prevent user history rows from being unbounded in size.
    This commit adds rails validations to limit the sizes of columns on
    user_histories,

 2. However, we don't want to prevent certain actions from being
    completed if these columns are too long. In those cases, we truncate
    the values that are given and store the truncated versions,

 3. For endpoints that perform staff actions, we can further control
    what is permitted by explicitly validating the params that are given
    before attempting the action,
2024-03-15 14:24:04 +08:00
Daniel Waterworth
819361ba28
SECURITY: Don't disclose the existence of secret subcategories 2024-03-15 14:23:55 +08:00
Blake Erickson
70c23f11a9
DEV: Add API scopes for post revisions (#26183)
This commit adds API scopes for reading, modifying, and deleting post
revisions.
2024-03-14 15:24:54 -06:00
Ted Johansson
ea5c3a3bdc
DEV: Move non scheduled problem checks to classes (#26122)
In AdminDashboardData we have a bunch of problem checks implemented as methods on that class. This PR absolves it of the responsibility by promoting each of those checks to a first class ProblemCheck. This way each of them can have their own priority and arbitrary functionality can be isolated in its own class.

Think "extract class" refactoring over and over. Since they were all moved we can also get rid of the @@problem_syms class variable which was basically the old version of the registry now replaced by ProblemCheck.realtime.

In addition AdminDashboardData::Problem value object has been entirely replaced with the new ProblemCheck::Problem (with compatible API).

Lastly, I added some RSpec matchers to simplify testing of problem checks and provide helpful error messages when assertions fail.
2024-03-14 10:55:01 +08:00
Arpit Jalan
1bd803d360
FIX: store registration ip address when creating user via SSO (#26121) 2024-03-11 15:19:37 +05:30
Ted Johansson
2211ffa851
DEV: Move problem checks to app directory (#26120)
There are a couple of reasons for this.

The first one is practical, and related to eager loading. Since /lib is not eager loaded, when the application boots, ProblemCheck["identifier"] will be nil because the child classes aren't loaded.

The second one is more conceptual. There turns out to be a lot of inter-dependencies between the part of the problem check system that live in /app and the parts that live in /lib, which probably suggests it should all go in /app.
2024-03-11 13:36:22 +08:00
Blake Erickson
f71e9aad60
FEATURE: Silence Close Notifications User Setting (#26072)
This change creates a user setting that they can toggle if
they don't want to receive unread notifications when someone closes a
topic they have read and are watching/tracking it.
2024-03-08 15:14:46 -07:00
Alan Guo Xiang Tan
898b71da88
PERF: Add indexes to speed up notifications queries by user menu (#26048)
Why this change?

There are two problematic queries in question here when loading
notifications in various tabs in the user menu:

```
SELECT "notifications".*
FROM "notifications"
LEFT JOIN topics ON notifications.topic_id = topics.id
WHERE "notifications"."user_id" = 1338 AND (topics.id IS NULL OR topics.deleted_at IS NULL)
ORDER BY notifications.high_priority AND NOT notifications.read DESC,
  NOT notifications.read AND notifications.notification_type NOT IN (5,19,25) DESC,
  notifications.created_at DESC
LIMIT 30;
```

and

```
EXPLAIN ANALYZE SELECT "notifications".*
FROM "notifications"
LEFT JOIN topics ON notifications.topic_id = topics.id
WHERE "notifications"."user_id" = 1338
AND (topics.id IS NULL OR topics.deleted_at IS NULL)
AND "notifications"."notification_type" IN (5, 19, 25)
ORDER BY notifications.high_priority AND NOT notifications.read DESC, NOT notifications.read DESC, notifications.created_at DESC LIMIT 30;
```

For a particular user, the queries takes about 40ms and 26ms
respectively on one of our production instance where the user has 10K notifications while the site has 600K notifications in total.

What does this change do?

1. Adds the `index_notifications_user_menu_ordering` index to the `notifications` table which is
   indexed on `(user_id, (high_priority AND NOT read) DESC, (NOT read)
DESC, created_at DESC)`.

1. Adds a second index `index_notifications_user_menu_ordering_deprioritized_likes` to the `notifications`
   table which is indexed on `(user_id, (high_priority AND NOT read) DESC, (NOT read AND notification_type NOT IN (5,19,25)) DESC, created_at DESC)`. Note that we have to hardcode the like typed notifications type here as it is being used in an ordering clause.

With the two indexes above, both queries complete in roughly 0.2ms. While I acknowledge that there will be some overhead in insert,update or delete operations. I believe this trade-off is worth it since viewing notifications in the user menu is something that is at the core of using a Discourse forum so we should optimise this experience as much as possible.
2024-03-06 16:52:19 +08:00
Krzysztof Kotlarek
28af4031ae
FIX: active webhook types exclude inactive plugins (#26022)
Bug introduced when webhooks were granularized in this PR - c468110929

Only active webhooks should be available when webhooks is configured.

https://meta.discourse.org/t/i18n-keys-showing-on-webhooks-edit-page/297701
2024-03-05 12:47:04 +11:00
Michael Fitz-Payne
f880f1a42f
DEV: add topic_id index to incoming_emails table (#26004)
This table may grow to be large, and an index on this column improves
performance for SELECT queries.
2024-03-04 13:50:01 +10:00
Blake Erickson
2d890d73a2
FEATURE: Add recover api scopes (#25978)
This commit adds two new api scopes. One for recovering topics, and the
other for recovering posts.
2024-02-29 15:49:29 -07:00
Alan Guo Xiang Tan
f562da3150
PERF: Reduce ActiveRecord allocations in CategoryList#find_relevant_topics (#25950)
Why this change?

Prior to this change, the `CategoryList#find_relevant_topics` method was
loading and allocating all `CategoryFeaturedTopic` records in the
database to eventually only just use its `category_id` and `topic_id`
column. On a site with many `CategoryFeaturedTopic` records, the loading
of the ActiveRecord objects is a source of bottleneck.

The other problem with the `CategoryList#find_relevant_topics` method is
that it is unconditionally loading all records from the database even if
the user does not have access to the category. This again is wasteful.

What does this change do?

This commit makes it such that `CategoryList#find_relevant_topics` is
called only after `CategoryList#find_categories` in the `CategoryList#initialize`
method so that we can filter featured topics against categories that the
user has access to.

The second change is that Instead of loading `CategoryFeaturedTopic` records, we make an
inner join agains the `topics` table instead and skip any allocation of
`CatgoryFeaturedTopic` ActiveRecord objects.
2024-02-29 12:19:04 +08:00
Natalie Tay
a8a39d86b4
UX: Improve invite error message when a user uses an email that has already redeemed (#25695)
Improve invite error message when a user uses an email that has already redeemed
2024-02-27 18:24:20 +08:00
Ted Johansson
1bcb521fbf
DEV: Add DB backed problem checks to support perform_every config (#25834)
As part of problem checks refactoring, we're moving some data to be DB backed. In this PR it's the tracking of problem check execution. When was it last run, when was the last problem, when should it run next, how many consecutive checks had problems, etc.

This allows us to implement the perform_every feature in scheduled problem checks for checks that don't need to be run every 10 minutes.
2024-02-27 11:17:39 +08:00
Andrei Prigorshnev
b3a1199493
FEATURE: Hide user status when user is hiding public profile and presence (#24300)
Users can hide their public profile and presence information by checking 
“Hide my public profile and presence features” on the 
`u/{username}/preferences/interface` page. In that case, we also don't 
want to return user status from the server.

This work has been started in https://github.com/discourse/discourse/pull/23946. 
The current PR fixes all the remaining places in Core.

Note that the actual fix is quite simple – a5802f484d. 
But we had a fair amount of duplication in the code responsible for 
the user status serialization, so I had to dry that up first. The refactoring 
as well as adding some additional tests is the main part of this PR.
2024-02-26 17:40:48 +04:00
David Taylor
542cb22fd4 DEV: Drop Ember 3 feature flag 2024-02-26 12:22:05 +00:00
Isaac Janzen
21f23cc032
DEV: Convert header to glimmer (#25214)
Here is a breakdown of the changes that will be implemented in this PR.

# Widgets -> Glimmer

Obviously, the intention of the todo here is to convert the header from widgets to glimmer. This PR splits the respective widgets as so:

### widgets/site-header.js
```mermaid height=200
flowchart TB
    A[widgets/site-header.js] 
    A-->B[components/glimmer-site-header.gjs]
```

### widgets/header.js and children
```mermaid height=200
flowchart TB
    A[widgets/header.js] 
    A-->B[components/glimmer-header.gjs]
    B-->C[glimmer-header/contents.gjs]
    C-->D[./auth-buttons.gjs]
    C-->E[./icons.gjs]
    C-->F[./user-menu-wrapper.gjs]
    C-->G[./hamburger-dropdown-wrapper.gjs]
    C-->H[./user-menu-wrapper.gjs]
    C-->I[./sidebar-toggle.gjs]
    C-->J[./topic/info.gjs]
```

There are additional components rendered within the `glimmer-header/*` components, but I will leave those out for now. From this view you can see that we split apart the logic of `widgets/header.js` into 10+ components. Breaking apart these mega files has many benefits (readability, etc).

# Services

I have introduced a [header](cdb42caa04/app/assets/javascripts/discourse/app/services/header.js) service. This simplifies how we pass around data in the header, as well as fixes a bug we have with "swiping" menu panels.


# Modifiers
Added a [close-on-click-outside](cdb42caa04/app/assets/javascripts/discourse/app/modifiers/close-on-click-outside.js) modifier that is built upon the [close-on-click-outside modifier](https://github.com/discourse/discourse/blob/main/app/assets/javascripts/float-kit/addon/modifiers/close-on-click-outside.js) that @jjaffeux built for float-kit. I think we could replace float-kit's implementation with mine and have it in a centralized location as they are extremely similar.

# Tests
Rewrote the existing header tests ([1](https://github.com/discourse/discourse/blob/main/app/assets/javascripts/discourse/tests/integration/components/widgets/header-test.js), [2](https://github.com/discourse/discourse/blob/main/app/assets/javascripts/discourse/tests/integration/components/site-header-test.js)) as system tests. 

# Other
- Converted `widgets/user-status-bubble.js` to a gjs component
- Converted `widgets/sidebar-toggle.js` to a gjs component
- Converted `topicFeaturedLinkNode()` to a gjs component
- Deprecated the [docking mixin](https://github.com/discourse/discourse/blob/main/app/assets/javascripts/discourse/app/mixins/docking.js)
2024-02-23 11:08:15 -07:00
Vinoth Kannan
b3238bfc34
FEATURE: call hub API to update Discourse discover enrollment. (#25634)
Now forums can enroll their sites to be showcased in the Discourse [Discover](https://discourse.org/discover) directory. Once they enable the site setting `include_in_discourse_discover` to enroll their forum the `CallDiscourseHub` job will ping the `api.discourse.org/api/discover/enroll` endpoint. Then the Discourse Hub will fetch the basic details from the forum and add it to the review queue. If the site is approved then the forum details will be displayed in the `/discover` page.
2024-02-23 11:42:28 +05:30
Sam
207cb2052f
FIX: muted tags breaking hot page when filtered to tags (#25824)
Also, remove experimental setting and simply use top_menu for feature detection

This means that when people eventually enable the hot top menu, there will
be topics in it


Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2024-02-23 17:11:39 +11:00
Ted Johansson
a72dc2f420
DEV: Introduce a problem checks API (#25783)
Previously, problem checks were all added as either class methods or blocks in AdminDashboardData. Another set of class methods were used to add and run problem checks.

As of this PR, problem checks are promoted to first-class citizens. Each problem check receives their own class. This class of course contains the implementation for running the check, but also configuration items like retry strategies (for scheduled checks.)

In addition, the parent class ProblemCheck also serves as a registry for checks. For example we can get a list of all existing check classes through ProblemCheck.checks, or just the ones running on a schedule through ProblemCheck.scheduled.

After this refactor, the task of adding a new check is significantly simplified. You add a class that inherits ProblemCheck, you implement it, add a test, and you're good to go.
2024-02-23 11:20:32 +08:00
Daniel Waterworth
13291dc5ef
FIX: Cache keys should be strings (#25791)
* FIX: Cache keys should be strings

Otherwise, there are subtle bugs that don't show up with a single
process.
2024-02-21 10:55:48 -06:00
Alan Guo Xiang Tan
3e331b1725
DEV: Set a bytesize limit for ThemeSetting#json_value (#25761)
Why this change?

Firstly, note that this is not a security commit because this feature is
still in development and should not be used anywhere.

The reason we want to set a limit here is to greatly reduce the
possibility of a DoS attack in the future via `ThemeSetting` where
someone would set an arbituary large json string in
`ThemeSetting#json_value` and causing the server to run out of resources
trying to serialize/deserialize the value.

What does this change do?

Adds an ActiveRecord validation to ensure that the bytesize of the json
string being stored is smaller than or equal to 0.5mb. We believe 0.5mb
is a decent limit for now but we can review the limit in the future if
we believe it is too small.
2024-02-21 08:09:37 +08:00
Alan Guo Xiang Tan
6ca2396b12
DEV: Centralise logic for validating a theme setting value (#25764)
Why this change?

The logic for validating a theme setting's value and default value was
not consistent as each part of the code would implement its own logic.
This is not ideal as the default value may be validated differently than
when we are setting a new value. Therefore, this commit seeks to
refactor all the validation logic for a theme setting's value into a
single service class.

What does this change do?

Introduce the `ThemeSettingsValidator` service class which holds all the
necessary helper methods required to validate a theme setting's value
2024-02-21 08:08:26 +08:00
Daniel Waterworth
13083d03ae
DEV: Async category search for sidebar modal (#25686) 2024-02-20 11:24:30 -06:00
Bianca Nenciu
a24d110258
FIX: Preload parent categories for sidebar (#25726)
When "lazy load categories" is enabled, only the categories present in
the sidebar are preloaded. This is insufficient because the parent
categories are necessary too for the sidebar to be rendered properly.
2024-02-16 16:39:18 +02:00
David Taylor
b1f74ab59e
FEATURE: Add experimental option for strict-dynamic CSP (#25664)
The strict-dynamic CSP directive is supported in all our target browsers, and makes for a much simpler configuration. Instead of allowlisting paths, we use a per-request nonce to authorize `<script>` tags, and then those scripts are allowed to load additional scripts (or add additional inline scripts) without restriction.

This becomes especially useful when admins want to add external scripts like Google Tag Manager, or advertising scripts, which then go on to load a ton of other scripts.

All script tags introduced via themes will automatically have the nonce attribute applied, so it should be zero-effort for theme developers. Plugins *may* need some changes if they are inserting their own script tags.

This commit introduces a strict-dynamic-based CSP behind an experimental `content_security_policy_strict_dynamic` site setting.
2024-02-16 11:16:54 +00:00
Martin Brennan
5935148bd8
FIX: Respect homepage prefs on admin sidebar Back to Forum link (#25642) 2024-02-16 14:31:42 +10:00
Martin Brennan
ae24e04a5e
DEV: Add a plugin modifier for user_action_stream_builder (#25691)
Reactions needs this to be able to filter out likes received
actions, where there is also an associated reaction, since
now most reactions also count as a like.
2024-02-16 10:24:39 +10:00
Alan Guo Xiang Tan
cc9480b24a
PERF: Use -ping option to ImageMagick identify command (#25713)
Why this change?

This adds the `-ping` option to the spots we missed in
cfdb461e9a.
2024-02-16 07:39:49 +08:00
Osama Sayegh
cfdb461e9a
PERF: Pass the -ping option to the identify ImageMagick command to speed it up (#25697)
The `-ping` option significantly speeds up the ImageMagick `identify` command per our testing and the [documentation](https://imagemagick.org/script/command-line-options.php#ping):

> -ping
Efficiently determine these image characteristics: image number, the file name, the width and height of the image, whether the image is colormapped or not, the number of colors in the image, the number of bytes in the image, the format of the image (JPEG, PNM, etc.). Use +ping to ensure accurate image properties.

We already pass the `-ping` option in other places where the `identify` command is used, so it makes sense to use the option everywhere.

Internal topic: t/121431.
2024-02-15 18:55:39 +03:00
Sam
4346abe260
FEATURE: apply pinning to hot topic lists (#25690)
pinned topics should be pinned even on hot lists so it can be used as a
home page
2024-02-15 18:27:54 +11:00
Penar Musaraj
c1577019c8
DEV: Add post_id parameter to reset_bump_date route (#25372)
This would allow a theme component (or an API call) to reset the bump
date of a topic to a given post's created_at date.

I picked `post_id` as the parameter here because it provides a bit of
extra protection against accidentally resetting the bump date to a date
that doesn't make sense.
2024-02-15 16:42:42 +11:00
Blake Erickson
bb261094cf
FEATURE: Auto generate and display video preview image (#25633)
This change will allow auto generated video thumbnails to be used
instead of the black video thumbnail that overlays videos.

Follow up to: 2443446e62
2024-02-14 13:43:53 -07:00
Bianca Nenciu
9a6406d4bb
FIX: Preload user-specific category fields (#25663)
This is used when lazy_load_categories is enabled to fetch more info
about the category.
2024-02-13 20:00:44 +02:00
Martin Brennan
6b596151ff
DEV: Change Group.trusted_group_ids to use const (#25639)
We have AUTO_GROUPS, we can use this instead of the
hardcoded 10..19 (not even sure why it goes up to 19,
trust levels previously only went to 5 max).
2024-02-12 12:36:00 +10:00
Daniel Waterworth
6311a80724
FIX: Preload associations on subcategories when lazy loading categories (#25630) 2024-02-09 11:48:26 -06:00
Sam
c8410537c1
FIX: hot not adding recently bumped topics (#25619)
When we insert into the hot set we add things with a score of 0
This means that if hot has more than batch size items in it with a score, then the 0s don't get an initial score

This corrects the situation by always ensuring we re-score:

1. batch size high scoring topics
2. (new) batch size recently bumped topics

* Update spec/models/topic_hot_scores_spec.rb

Co-authored-by: Isaac Janzen <50783505+janzenisaac@users.noreply.github.com>

---------

Co-authored-by: Isaac Janzen <50783505+janzenisaac@users.noreply.github.com>
2024-02-09 07:45:47 +11:00
Penar Musaraj
6bd26e81c1
FIX: Respect date range in top traffic sources report (#25599)
See https://meta.discourse.org/t/reports-top-traffic-sources-topics-stat-issue/179850
2024-02-08 11:17:59 -05:00
Martin Brennan
7ce76143ac
FIX: Always trust admin and moderators with post edits (#25602)
Removes duplication from LimitedEdit to see who can edit
posts, and also removes the old trust level setting check
since it's no longer necessary.

Also make it so staff can always edit since can_edit_post?
already has a staff escape hatch.
2024-02-08 13:10:26 +10:00
Alan Guo Xiang Tan
9f884cdaab
DEV: Introduce experimental type: objects theme setting (#25538)
Why this change?

This commit introduces an experimental `type: objects` theme setting
which will allow theme developers to store a collection of objects as
JSON in the database. Currently, the feature is still in development and
this commit is simply setting up the ground work for us to introduce the
feature in smaller pieces.

What does this change do?

1. Adds a `json_value` column as `jsonb` data type to the `theme_settings` table.
2. Adds a `experimental_objects_type_for_theme_settings` site setting to
   determine whether `ThemeSetting` records of with the `objects` data
   type can be created.
3. Updates `ThemeSettingsManager` to support read/write access from the
   `ThemeSettings#json_value` column.
2024-02-08 10:20:59 +08:00
Martin Brennan
adb4eee153
DEV: Make more group-based settings client: false (#25585)
Affects the following settings:

* whispers_allowed_groups
* anonymous_posting_allowed_groups
* personal_message_enabled_groups
* shared_drafts_allowed_groups
* here_mention_allowed_groups
* uploaded_avatars_allowed_groups
* ignore_allowed_groups

This turns off `client: true` for these group-based settings,
because there is no guarantee that the current user gets all
their group memberships serialized to the client. Better to check
server-side first.
2024-02-08 09:43:34 +10:00
Ted Johansson
95a2d285d3
FEATURE: Add new 'illegal' flag reason (#25498)
To comply with Digital Services Act we need a way for users to flag a post as potentially illegal. This PR adds that functionality.
2024-02-07 10:12:22 +08:00
Gerhard Schlager
dd5ca6cc4c
FEATURE: Permalinks for users (#25552) 2024-02-05 17:31:31 +01:00
Alan Guo Xiang Tan
d460229ed8
FIX: Update themes javascript cache after running themes migrations (#25562)
Why this change?

This is caused by a regression in
59839e428f, where we stopped saving the
`Theme` object because it was unnecessary. However, it resulted in the
`after_save` callback not being called and hence
`Theme#update_javascript_cache!` not being called. As a result, some
sites were reporting that after runing a theme migration, the defaults
for the theme settings were used instead of the settings overrides
stored in the database.

What does this change do?

Add a call to `Theme#update_javascript_cache!` after running theme
migrations.
2024-02-05 14:35:11 +08:00
Bianca Nenciu
1d160702ad
FIX: Preload sidebar categories when lazy loading categories (#25332)
This fixes a bug where the sidebar categories would not be loaded when
the categories were lazy loaded because the sidebar uses the preloaded
category list, which was empty.
2024-02-02 10:35:15 +02:00
Ted Johansson
e071b74a79
DEV: Drop deprecated Badge#image column (#25536)
We just completed the 3.2 release, which marks a good time to drop some previously deprecated columns.

Since the column has been marked in ignored_columns, it has been inaccessible to application code since then. There's a tiny risk that this might break a Data Explorer query, but given the nature of the column, the years of disuse, and the fact that such a breakage wouldn't be critical, we accept it.
2024-02-02 14:09:55 +08:00
Sam
140b2118af
FEATURE: improvements to hot feature (#25533)
1. Don't show visited line for hot filter, it is in random order
2. Don't count likes on non regular posts (eg: whispers / small actions)
3. Don't count participants in non regular posts
2024-02-02 10:53:27 +11:00
Sam
690ff4499c
DEV: adjustments to hot algorithm (#25517)
1. Serial likers will just like a bunch of posts on the same topic, this will
heavily inflate hot score. To avoid artificial "heat" generated by one user only count
the first like on the topic within the recent_cutoff range per topic

2. When looking at recent topics prefer "unique likers", defer to total likes on
older topics cause we do not have an easy count for unique likers

3. Stop taking 1 off like_count, it is not needed - platforms like reddit
allow you to like own post so they need to remove it.
2024-02-01 17:11:40 +11:00
Alan Guo Xiang Tan
44f8418093
DEV: Refactor Theme#settings to return a hash instead of array (#25516)
Why this change?

Returning an array makes it hard to immediately retrieve a setting by
name and makes the retrieval an O(N) operation. By returning an array,
we make it easier for us to lookup a setting by name and retrieval is
O(1) as well.
2024-02-01 10:26:56 +08:00
chapoi
a695d85cbd
UX: Update selected colour var (#25500) 2024-01-31 09:32:38 +01:00
David Taylor
88305e3d96
DEV: Remove version-number-based logic (#25482)
The `deprecate_column` helper would change its behavior based on the current `Discourse::VERSION`. This means that 'finalizing' a stable release introduces a previously untested behavior change.

Much better to keep it as a deprecation until manual action is taken to introduce the breaking change.
2024-01-30 17:34:10 +00:00
Sam
27301ae5c7
FEATURE: support silent internal links (#25472)
Internal links always notify and add internal connections in topics.

This adds a special feature that lets you append `?silent=true` to a link
to have it excluded from:

1. Notifications - users will not be notified for these links
2. Post links below posts in the UI

This is specifically useful for large reports where adding all these connections
just results in noise.
2024-01-30 17:03:58 +11:00
marstall
5a00d1964f
DEV: add site setting to disable watched word checking in user fields (#25411)
adding a hidden sitesetting, `disable_watched_word_checking_in_user_fields` - false by default. if set to true, you can use any word at all in user profile fields.

meta: https://meta.discourse.org/t/watched-words-scope/282699/20
2024-01-29 12:44:32 -05:00
David Taylor
63f6bd5efe
DEV: Add admin warning for sites on Ember 3 (#25459)
Running Discourse 3.2 stable under Ember 3 will technically be possible, but is only intended as a short-term migration point. This commit adds an admin warning for sites which are using this configuration, to make it clear that themes and plugins are unlikely to support the configuration.

https://meta.discourse.org/t/287211
2024-01-29 14:09:07 +00:00
Ted Johansson
6ad34a0152
DEV: Exclude system users when calculating group user count (#25400)
We want to exclude the system user from group user counts, since intuitively admins wouldn't include them.

Originally this was accomplished by booting said system user from the groups, but this is causing problems, because the system user needs TL group membership to perform certain tasks.

After this PR, system user is still in the TL groups, but excluded when refreshing the user count.
2024-01-25 08:13:58 +08:00
Ted Johansson
32e2a1fd4a
DEV: Add delegated Group#human_users scope (#25398)
Some preparatory refactoring as we're working on TL groups for the system user. On User we have a scope #human_users to exclude the system user, DiscoBot, etc. This PR adds the same scope (delegated to User) on Group.
2024-01-24 13:33:05 +08:00
Penar Musaraj
4f901cae8f
PERF: Better query performance for user avatar consistency check. (#25342) 2024-01-22 18:33:39 +01:00
Roman Rizzi
57915d9edc
FIX: Radar chart not widely available. (#25368)
We added support for radar type charts in #24274. However, radar charts work with three variables, meaning we can't display any report that way.

Unfortunately, by adding `:radar` to the `Report#modes` variable, I made them widely available.

Related bug report: https://meta.discourse.org/t/report-radar-graph-uncaught-typeerror/292360
2024-01-22 11:21:28 -03:00
Martin Brennan
b3904eab45
FIX: User group check should return true for system user for auto groups (#25357)
This is a temporary fix to address an issue where the
system user is losing its automatic groups when the server
is running. If any auto groups are provided, and the user is
a system user, then we return true. The system user is admin,
moderator, and TL4, so they usually have all auto groups.

We can remove this when we get to the bottom of why the auto
groups are being deleted.
2024-01-22 14:40:29 +10:00
Blake Erickson
561851b104
FEATURE: Add hot as a homepage option (#25325) 2024-01-18 16:36:18 -07:00
Bianca Nenciu
6a205ea0a5
DEV: Replace lazy_load_categories site setting (#25302)
The site setting has been removed in a previous commit abad38c, but it
was merged at the same time with 4cfc0e2 which added it back.
2024-01-17 21:44:48 +02:00
Bianca Nenciu
abad38c2e7
DEV: Make lazy_load_categories setting use groups (#25282)
This allows certain users to test the new feature and avoid disruptions
in other's workflows.
2024-01-17 20:26:51 +02:00
Bianca Nenciu
4cfc0e231a
DEV: Change categories#index loading strategy (#25232)
The old strategy used to load 25 categories at a time, including the
subcategories. The new strategy loads 20 parent categories and at most
5 subcategories for each parent category, for a maximum of 120
categories in total.
2024-01-17 17:18:01 +02:00
Sam
df8bb947b2
FEATURE: improvements to hot algorithm (#25295)
- Decrease gravity, we come in too hot prioritizing too many new topics
- Remove all muted topics / categories and tags from the hot list
- Punish topics with zero likes in algorithm
2024-01-17 16:12:03 +11:00
Sam
ebd3971533
FEATURE: experiment with hot sort order (#25274)
This introduces a new experimental hot sort ordering. 

It attempts to float top conversations by first prioritizing a  topics with lots of recent activity (likes and users responding) 

The schedule that updates hot topics is disabled unless the hidden site setting: `experimental_hot_topics` is enabled. 

You can control "decay" with `hot_topic_gravity` and `recency` with `hot_topics_recent_days` 

Data is stored in the new `topic_hot_scores` table and you can check it out on the `/hot` route once 
enabled. 
---------

Co-authored-by: Penar Musaraj <pmusaraj@gmail.com>
2024-01-17 13:01:04 +11:00
Gerhard Schlager
241bf48497 DEV: Allow rebakes to generate optimized images at the same time
Previously only Sidekiq was allowed to generate more than one optimized image at the same time per machine. This adds an easy mechanism to allow the same in rake tasks and other tools.
2024-01-16 14:33:16 +01:00
Gerhard Schlager
bcb8b3fab9 REFACTOR: Reuse Discourse.store instance
Calling `Discourse.store` creates a new instance of a store each time.
2024-01-16 14:33:16 +01:00
Alan Guo Xiang Tan
22614ca85b
DEV: Compile theme migrations javascript files when running theme qunit (#25219)
Why this change?

Currently, is it hard to iteratively write a theme settings migrations
because our theme migrations system does not rollback. Therefore, we
want to allow theme developers to be able to write QUnit tests for their
theme migrations files enabling them to iteratively write their theme
migrations.

What does this change do?

1. Update `Theme#baked_js_tests_with_digest` to include all `ThemeField`
records of `ThemeField#target` equal to `migrations`. Note that we do
not include the `settings` and `themePrefix` variables for migration files.

2. Don't minify JavaScript test files becasue it makes debugging in
   development hard.
2024-01-16 09:50:44 +08:00
Alan Guo Xiang Tan
c76ca876a6
DEV: Add more debugging information to AR query logs on GitHub actions (#25237)
Why this change?

We have been chasing a problem with our flaky system test where the user
is logged out when it should never be.

What does this change do?

1. Logs the request path when lookup a user auth token.
2. Logs the request path and also the current thread's object id in
   ActiveRecord query logs.
2024-01-12 13:06:29 +08:00
Renato Atilio
f5f3742166
FIX: respect creation date when paginating group activity posts (#24993)
* FIX: respect creation date when paginating group activity posts

There are scenarios where the chronological order of posts doesn't match the order of their IDs. For instance, when moving the first post from one topic or PM to another, a new post (with a higher ID) will be created, but it will retain the original creation time.

This PR changes the group activity page and endpoint to paginate posts using created_at instead of relying on ID ordering.
2024-01-11 13:37:27 -03:00
Alan Guo Xiang Tan
59839e428f
DEV: Add skip_migrations param when importing remote theme (#25218)
Why this change?

Importing theme with the `bundle` params is used mainly by
`discourse_theme` CLI in the development environment. However, we do not
want migrations to automatically run in the development environment
and instead want the developer to be intentional about running theme
migrations. As such, this commit adds support for a
`skip_migrations` param when importing a theme with the `bundle` params.

This commit also adds a `migrated` attribute for migrations theme fields
to indicate whether a migrations theme field has been migrated or not.
2024-01-11 14:04:02 +08:00
Bianca Nenciu
be46acce8f
PERF: Prefer subquery instead of two queries (#25167)
The query that is now a subquery could return a long list of category
IDs, which slowed down the query considerably. This improvement reduces
the execution time from over 2 seconds down to about 100ms.
2024-01-09 08:19:37 +08:00
Daniel Waterworth
75c645453d
SECURITY: Store custom field values according to their registered type 2024-01-08 08:02:17 -07:00
Daniel Waterworth
4494d62531
SECURITY: Run custom field validations with save_custom_fields 2024-01-08 08:02:16 -07:00
Rafael dos Santos Silva
13735f35fb
FEATURE: Cache embed contents in the database (#25133)
* FEATURE: Cache embed contents in the database

This will be useful for features that rely on the semantic content of topics, like the many AI features



Co-authored-by: Roman Rizzi <rizziromanalejandro@gmail.com>
2024-01-05 10:09:31 -03:00
Isaac Janzen
1f94da349b
DEV: Make the Glimmer Search Menu the new default (#25092)
- Convert group based `experimental_search_menu_groups_enabled` site setting to be a _hidden_ boolean `experimental_search_menu` setting.
- Make default `true`
- Remove widget search menu tests

Discourse Encrypt Test Failure Fix - https://github.com/discourse/discourse-encrypt/pull/301
2024-01-03 09:07:27 -07:00
Martin Brennan
4e6d4193ea
DEV: Fix spec for post menu (#25100)
Followup to b92993fcee
I ran out of time to get this working for that fix,
also here I am making the post.url method have parity
with post.shareUrl in JS, which omits the post number
for the first post.
2024-01-03 16:55:08 +10:00
Natalie Tay
b4f36507e0
FIX: Allow the flags to be cleaned up (#25085)
Currently, the reviewable queue includes ReviewableFlaggedPost with posts that have already been hidden. This allows for such hidden posts to be cleared up by the auto-tool.
2024-01-02 18:32:50 +08:00
Bianca Nenciu
14269232ba
DEV: No longer preload categories (#24950)
Categories will no longer be preloaded when `lazy_load_categories` is
enabled through PreloadStore.

Instead, the list of site categories will continue to be populated
by `Site.updateCategory` as more and more categories are being loaded
from different sources (topic lists, category selectors, etc).
2023-12-28 14:36:33 +02:00
Martin Brennan
89705be722
DEV: Add auto map from TL -> group site settings in DeprecatedSettings (#24959)
When setting an old TL based site setting in the console e.g.:

SiteSetting.min_trust_level_to_allow_ignore = TrustLevel[3]

We will silently convert this to the corresponding Group::AUTO_GROUP. And vice-versa, when we read the value on the old setting, we will automatically get the lowest trust level corresponding to the lowest auto group for the new setting in the database.
2023-12-26 14:39:18 +08:00
Ted Johansson
25ccf6fab1
FIX: Update position on model when re-positioning record (#24997)
When updating the position of a category, the server correctly updates the position in the database, but the response sent back to the client still contains the old position, causing it to "flip back" in the UI when saving. Only reloading the page will reveal the new, correct value.

The Positionable concern correctly positions the record and updates the database, but we don't assign the new position to the already instantiated model.

This change just assigns self.position after the database update. 😎
2023-12-21 10:15:10 +08:00
Blake Erickson
3380d283c9
FEATURE: Add API scope for /logs route (#24956)
Adds an API scope for accessing Logster's routes. This one is a bit
different than routes from core because it is mounted like

```
mount Logster::Web => "/logs"
```

and doesn't have all the route info a traditional rails app/engine does.
2023-12-18 19:45:04 -07:00
Blake Erickson
a08691a599
FIX: Ensure file size restriction types are ints (#24947)
Settings that are using the new `file_size_restriction` types like the
`max_image_size_kb` setting need to have their values saved as integers.
This was a recent regression in 00209f03e6
that caused these values to be saved as strings.

This change also removes negatives from the validation regex because
file sizes can't be negative anyways.

Bug report: https://meta.discourse.org/t/289037
2023-12-18 09:22:50 -07:00
Bianca Nenciu
680cf443f4
FIX: Better infinite scrolling on categories page (#24831)
This commit refactor CategoryList to remove usage of EmberObject,
hopefully make the code more readable and fixes various edge cases with
lazy loaded categories (third level subcategories not being visible,
subcategories not being visible on category page, requesting for more
pages even if the last one did not return any results, etc).

The problems have always been here, but were not visible because a lot
of the processing was handled by the server and then the result was
serialized. With more of these being moved to the client side for the
lazy category loading, the problems became more obvious.
2023-12-18 16:46:09 +02:00
Martin Brennan
6de00f89c2
FEATURE: Initial admin sidebar navigation (#24789)
This is v0 of admin sidebar navigation, which moves
all of the top-level admin nav from the top of the page
into a sidebar. This is hidden behind a enable_admin_sidebar_navigation
site setting, and is opt-in for now.

This sidebar is dynamically shown whenever the user enters an
admin route in the UI, and is hidden and replaced with either
the:

* Main forum sidebar
* Chat sidebar

Depending on where they navigate to. For now, custom sections
are not supported in the admin sidebar.

This commit removes the experimental admin sidebar generation rake
task but keeps the experimental sidebar UI for now for further
testing; it just uses the real nav as the default now.
2023-12-18 11:48:25 +10:00
Kelv
2477bcc32e
DEV: lint against Layout/EmptyLineBetweenDefs (#24914) 2023-12-15 23:46:04 +08:00
Daniel Waterworth
d7a09fb08d
DEV: Add true_fields method for CustomFields (#24876)
This is useful for plugins that might otherwise rely on the
CUSTOM_FIELD_TRUE constant.
2023-12-14 11:06:21 -06:00
Daniel Waterworth
f7aefffea7
DEV: Concerns can use class_methods (#24875) 2023-12-13 14:12:03 -06:00
Ted Johansson
36057638ca
DEV: Convert min_trust_to_edit_post to groups (#24840)
We're changing the implementation of trust levels to use groups. Part of this is to have site settings that reference trust levels use groups instead. It converts the min_trust_to_edit_post site setting to edit_post_allowed_groups.

The old implementation will co-exist for a short period while I update any references in plugins and themes.
2023-12-13 13:25:13 +08:00
Jarek Radosz
8355664c82
FIX: Don't use :true/:false symbols (#24861)
on line 252 `has_children` is set to true/false (actual booleans) which means `has_children?` was in some cases returning incorrect value…
2023-12-13 00:27:14 +01:00
Angus McLeod
95c61b88dc
Apply embed unlisted setting consistently (#24294)
Applies the embed_unlisted site setting consistently across topic embeds, including those created via the WP Discourse plugin. Relatedly, adds a embed exception to can_create_unlisted_topic? check. Users creating embedded topics are not always staff.
2023-12-12 09:35:26 -05:00
Bianca Nenciu
81b0420614
FEATURE: Add pagination to categories page (#23976)
When `lazy_load_categories` is enabled, the categories are no longer
preloaded in the `Site` object, but instead they are being requested
on a need basis.

The categories page still loaded all categories at once, which was not
ideal for sites with many categories because ti would take a lot of
time to build and parse the response.

This commit adds pagination to the categories page using the LoadMore
helper. As the user scrolls through the categories page, more categories
are requested from the server and appended to the page.

<!-- NOTE: All pull requests should have tests (rspec in Ruby, qunit in JavaScript). If your code does not include test coverage, please include an explanation of why it was omitted. -->
2023-12-11 17:58:45 +02:00
Gerhard Schlager
d725b3ca9e DEV: Add script for preprocessing uploads as part of a migration
This script preprocesses all uploads within a intermediate DB (output of converters) and uploads those files to S3. It does the same for optimized images. This speeds up migrations when you have to run them multiple times, because you only have to preprocess and upload the files once.

This script is very hacky and mostly undocumented for now. That will change in the future.
2023-12-11 16:23:07 +01:00
Bianca Nenciu
dcd81d56c0
FIX: category selectors for lazy loaded categories (#24533)
A lot of work has been put in the select kits used for selecting
categories: CategorySelector, CategoryChooser, CategoryDrop, however
they still do not work as expected when these selectors already have
values set, because the category were still looked up in the list of
categories stored on the client-side Categrories.list().

This PR fixes that by looking up the categories when the selector is
initialized. This required altering the /categories/find.json endpoint
to accept a list of IDs that need to be looked up. The API is called
using Category.asyncFindByIds on the client-side.

CategorySelector was also updated to receive a list of category IDs as
attribute, instead of the list of categories, because the list of
categories may have not been loaded.

During this development, I noticed that SiteCategorySerializer did not
serializer all fields (such as permission and notification_level)
which are not a property of category, but a property of the relationship
between users and categories. To make this more efficient, the
preload_user_fields! method was implemented that can be used to
preload these attributes for a user and a list of categories.
2023-12-08 12:01:08 +02:00
Martin Brennan
d9a422cf61
FIX: Do not attempt S3 ACL call if secure status did not change (#24785)
When rebaking and in various other places for posts, we
run through the uploads and call `update_secure_status` on
each of them.

However, if the secure status didn't change, we were still
calling S3 to change the ACL, which would have been a noop
in many cases and takes ~1 second per call, slowing things
down a lot.

Also, we didn't account for the s3_acls_enabled site setting
being false here, and in the specs doing an assertion
that `Discourse.store.update_ACL` is not called doesn't
work; `Discourse.store` isn't a singleton, it re-initializes
`FileStore::S3Store.new` every single time.
2023-12-08 12:58:45 +10:00
Daniel Waterworth
611acaa6bc
FIX: Validate each value in an array custom field separately (#24659) 2023-12-07 14:24:04 -06:00
Mark VanLandingham
ee05f57e2d
FEATURE: Site setting to display user avatars in user menu (#24514) 2023-12-07 11:30:44 -06:00
Martin Brennan
7afb5fc481
DEV: Use Discourse::SYSTEM_USER_ID in fixtures/009_users (#24743)
I couldn't find where we created the system user and
this is why -- everywhere else in the app we reference
SYSTEM_USER_ID but here.
2023-12-07 09:04:45 +10:00
Jarek Radosz
694b5f108b
DEV: Fix various rubocop lints (#24749)
These (21 + 3 from previous PRs) are soon to be enabled in rubocop-discourse:

Capybara/VisibilityMatcher
Lint/DeprecatedOpenSSLConstant
Lint/DisjunctiveAssignmentInConstructor
Lint/EmptyConditionalBody
Lint/EmptyEnsure
Lint/LiteralInInterpolation
Lint/NonLocalExitFromIterator
Lint/ParenthesesAsGroupedExpression
Lint/RedundantCopDisableDirective
Lint/RedundantRequireStatement
Lint/RedundantSafeNavigation
Lint/RedundantStringCoercion
Lint/RedundantWithIndex
Lint/RedundantWithObject
Lint/SafeNavigationChain
Lint/SafeNavigationConsistency
Lint/SelfAssignment
Lint/UnreachableCode
Lint/UselessMethodDefinition
Lint/Void

Previous PRs:
Lint/ShadowedArgument
Lint/DuplicateMethods
Lint/BooleanSymbol
RSpec/SpecFilePathSuffix
2023-12-06 23:25:00 +01:00
Jarek Radosz
6a66dc1cfb
DEV: Fix Lint/BooleanSymbol (#24747) 2023-12-06 13:19:09 +01:00
Jarek Radosz
138bf486d3
DEV: Fix Lint/DuplicateMethods (#24746) 2023-12-06 13:18:34 +01:00
Daniel Waterworth
434ae5bbe7
FIX: Allow setting an array custom field to a singleton value (#24636)
Also, validation happens per item in an array field.
2023-11-29 14:18:47 -06:00
Daniel Waterworth
eef93ac926
DEV: Allow setting max_length for field types using the plugin API (#24635) 2023-11-29 14:17:12 -06:00
Sam
b09422428d
DEV: update syntax tree to latest (#24623)
update format to latest syntax tree
2023-11-29 16:38:07 +11:00
Bianca Nenciu
e85a81f33c
FIX: Make category-drop work with lazy_load_categories (#24187)
The category drop was rerendered after every category async change
because it updated the categories list. This is not necessary and
categories can be referenced indirectly by ID instead.
2023-11-28 17:58:47 +02:00
David Taylor
5783f231f8
DEV: Introduce DISCOURSE_ASSET_URL_SALT (#24596)
This value is included when generating static asset URLs. Updating the value will allow site operators to invalidate all asset urls to recover from configuration issues which may have been cached by CDNs/browsers.
2023-11-28 11:28:40 +00:00
Martin Brennan
1fc0ce1ac2
FIX: with_secure_uploads? could return nil in some cases (#24592)
When we check upload security, one of the checks is to
run `access_control_post.with_secure_uploads?`. The problem
here is that the `topic` for the post could be deleted,
which would make the check return `nil` sometimes instead
of false because of safe navigation. We just need to be
more explicit.
2023-11-28 13:12:28 +10:00
Krzysztof Kotlarek
5551a71c55
FEATURE: increase tag description limit to 1000 (#24561)
Admin can add tag description up to 1000 characters.

Full description is displayed on tag page, however on topic list it is truncated to 80 characters.
2023-11-28 08:45:40 +11:00
Martin Brennan
91232847e3
FIX: Video placeholders not auto-linking post uploads (#24559)
Followup to 2443446e62

We introduced video placeholders which prevent preloading
metadata for videos in posts. The structure looks like this
in HTML when the post is cooked:

```
<div class="video-placeholder-container" data-video-src="http://some-url.com/video.mp4" dir="ltr" style="cursor: pointer;">
  <div class="video-placeholder-wrapper">
    <div class="video-placeholder-overlay">
      <svg class="fa d-icon d-icon-play svg-icon svg-string" xmlns="http://www.w3.org/2000/svg">
        <use href="#play"></use>
      </svg>
    </div>
  </div>
</div>
```

However, we did not update the code that links post uploads
to the post via UploadReference, so any videos uploaded since
this change are essentially dangling and liable to be deleted.
This also causes some uploads to be marked secure when they
shouldn't be, because they are not picked up and analysed in the
CookedPostProcessor flow.
2023-11-27 12:38:52 +10:00
Bianca Nenciu
012541b045
FIX: Serialize parent categories first (#24530)
The parent category needs to be serialized before the child category
because they are parsed in order. Otherwise the client will not build
the parent-child relationship correctly.
2023-11-23 19:03:05 +02:00
Daniel Waterworth
6aa69bdaea
DEV: Allow setting different custom field length limits by key (#24505) 2023-11-22 12:00:42 -06:00
Daniel Waterworth
0059e6807c
DEV: Refactor save_custom_fields methods (#24495)
Operate a key at a time, to make it clearer what's going on.

This also fixes a bug where array integer fields would get re-written
even when there wasn't a change.
2023-11-21 12:40:15 -06:00
Daniel Waterworth
ced21fa5c1
DEV: Deprecate array custom fields (#24492)
Array custom fields use separate rows for each value, but whenever we
update an array, we have always destroy the existing rows and create new
ones. Therefore, there's no benefit over using the json type.
2023-11-21 11:05:49 -06:00
Daniel Waterworth
b3c67a78b8
FIX: Preserve custom field array order (#24491) 2023-11-21 10:55:22 -06:00
Jarek Radosz
8968887e24
DEV: Fix various typos (#24461)
November 2023 edition
2023-11-20 16:49:49 +01:00
Martin Brennan
146da75fd7
FEATURE: Add setting & preference for search sort default order (#24428)
This commit adds a new `search_default_sort_order` site setting,
set to "relevance" by default, that controls the default sort order
for the full page /search route.

If the user changes the order in the dropdown on that page, we remember
their preference automatically, and it takes precedence over the site
setting as a default from then on. This way people who prefer e.g.
Latest Post as their default can make it so.
2023-11-20 10:43:58 +10:00
Martin Brennan
186e415e38
DEV: Housekeeping for CleanUpUploads job (#24361)
Followup to 9db8f00b3d, we
don't need this dead code any more. Also made some minor
improvements and comments.
2023-11-20 09:50:09 +10:00
David Taylor
04a58a6e64
PERF: Only invalidate other translations when en changes (#24443)
en is the only fallback locale we use, so there's no need to invalidate everything when other languages change. Limiting this also helps to prevent circular dependent_field relations which could cause issues in some situations.

Followup to eda79186ee
2023-11-18 12:36:25 +00:00
Daniel Waterworth
0c712eaa3a
DEV: Don't define methods in an included block (#24433) 2023-11-17 12:22:32 -06:00
Alan Guo Xiang Tan
e0ef88abca
DEV: Run QUnit tests for official Discourse themes (#24405)
Why this change?

As the number of themes which the Discourse team supports officially
grows, we want to ensure that changes made to Discourse core do not
break the plugins. As such, we are adding a step to our Github actions
test job to run the QUnit tests for all official themes.

What does this change do?

This change adds a new job to our tests Github actions workflow to run the QUnit
tests for all official plugins. This is achieved with the following
changes:

1. Update `testem.js` to rely on the `THEME_TEST_PAGES` env variable to set the
   `test_page` option when running theme QUnit tests with testem. The
   `test_page` option [allows an array to be specified](https://github.com/testem/testem#multiple-test-pages) such that tests for
   multiple pages can be run at the same time. We are relying on a ENV variable
   because  the `testem` CLI does not support passing a list of pages
   to the `--test_page` option.

2. Support a `/testem-theme-qunit/:testem_id/theme-qunit` Rails route in the development environment. This
   is done because testem prefixes the path with a unique ID to the configured `test_page` URL.
   This is problematic for us because we proxy all testem requests to the
   Rails server and testem's proxy configuration option does not allow us
   to easily rewrite the URL to remove the prefix. Therefore, we configure a proxy in testem to prefix `theme-qunit` requests with
  `/testem-theme-qunit` which can then be easily identified by the Rails server and routed accordingly. 

3. Update `qunit:test` to support a `THEME_IDS` environment variable
   which will allow it to run QUnit tests for multiple themes at the
   same time.

4. Support `bin/rake themes:qunit[ids,"<theme_id>|<theme_id>"]` to run
   the QUnit tests for multiple themes at the same time.

5. Adds a `themes:qunit_all_official` Rake task which runs the QUnit
   tests for all the official themes.
2023-11-17 07:17:32 +08:00
Daniel Waterworth
46e2523177
DEV: Remove custom field regexes (#24390)
As far as I can tell, this isn't used
2023-11-16 11:37:10 -06:00
David Taylor
eda79186ee
FIX: Recompile theme translations when fallback data changes (#24371)
Previously we would only recompile a theme locale when its own data changes. However, the output also includes fallback data from other locales, so we need to invalidate all locales when fallback locale data is changed. Building a list of dependent locales is tricky, so let's just invalidate them all.
2023-11-14 19:53:27 +00:00
Kris
797da5870b
FEATURE: remove category badge style options, set bullet style as default (#24198) 2023-11-13 10:46:15 -05:00
David Battersby
4b78254065
FIX: Post moved small action links should respect subfolder installs (#24336)
This change fixes an issue with small action links (when post is moved) to add the subfolder path to the url.
2023-11-13 11:06:25 +08:00
Martin Brennan
731dffdf92
DEV: Align S3 transfer acceleration global settings (#24302)
Followup to fe05fdae24

For consistency with other S3 settings, make the global setting
the same name as the site setting and use SiteSetting.Upload
too so it reads from the correct place.
2023-11-10 09:50:23 +10:00
Andrei Prigorshnev
d91456fd53
DEV: Ability to collect stats without exposing them via API (#23933)
This adds the ability to collect stats without exposing them 
among other stats via API.

The most important thing I wanted to achieve is to provide 
an API where stats are not exposed by default, and a developer 
has to explicitly specify that they should be 
exposed (`expose_via_api: true`). Implementing an opposite 
solution would be simpler, but that's less safe in terms of 
potential security issues. 

When working on this, I had to refactor the current solution. 
I would go even further with the refactoring, but the next steps 
seem to be going too far in changing the solution we have, 
and that would also take more time. Two things that can be 
improved in the future:
1. Data structures for holding stats can be further improved
2. Core stats are hard-coded in the About template (it's hard 
to fix it without correcting data structures first, see point 1):
    63a0700d45/app/views/about/index.html.erb (L61-L101)

The most significant refactorings are:
1. Introducing the `Stat` model
2. Aligning the way the core and the plugin stats' are registered
2023-11-10 00:44:05 +04:00
Bianca Nenciu
bdb81b5346
DEV: Use a single registry for preloaded category custom fields (#24272)
There was a registry for preloaded site categories and a new one has
been introduced recently for categories serialized through a
CategoryList.

Having two registries created a lot of friction for developers and this
commit merges them into a single one, providing a unified API.
2023-11-09 18:23:24 +02:00
Martin Brennan
3c5fb871c0 SECURITY: Filter unread bookmark reminders the user cannot see
There is an edge case where the following occurs:

1. The user sets a bookmark reminder on a post/topic
2. The post/topic is changed to a PM before or after the reminder
   fires, and the notification remains unread by the user
3. The user opens their bookmark reminder notification list
   and they can still see the notification even though they cannot
   access the topic anymore

There is a very low chance for information leaking here, since
the only thing that could be exposed is the topic title if it
changes to something sensitive.

This commit filters the bookmark unread notifications by using
the bookmarkable can_see? methods and also prevents sending
reminder notifications for bookmarks the user can no longer see.
2023-11-09 13:39:16 +11:00
Krzysztof Kotlarek
5f20748e40 SECURITY: SSRF vulnerability in TopicEmbed
Block redirects when making the final request in TopicEmbed to prevent Server Side Request Forgery (SSRF)
2023-11-09 13:39:08 +11:00
Andrei Prigorshnev
be2eb3df44
FIX: user got notified about a mention inside a chat message quote (#24229)
When quoting a chat message in a post, if that message contains a mention, 
that mention should be ignored. But we've been detecting them and sending 
notifications to users. This PR fixes the problem. Since this fix is for 
the chat plugin, I had to introduce a new API for plugins:

    # We strip posts before detecting mentions, oneboxes, attachments etc. 
    # We strip those elements that shouldn't be detected. For example, 
    # a mention inside a quote should be ignored, so we strip it off. 
    # Using this API plugins can register their own post strippers. 
    def register_post_stripper(&block) 
    end
2023-11-08 23:13:25 +04:00
Roman Rizzi
3c29a84d98
DEV: Add support for radar charts when displaying reports. (#24274)
Reports can have the radar type, which will get rendered by the `admin-report-radar` component.
2023-11-08 10:48:12 -03:00
Ted Johansson
c3708c4276
DEV: Add support for custom retries for scheduled admin checks (#24224)
We updated scheduled admin checks to run concurrently in their own jobs. The main reason for this was so that we can implement re-check functionality for especially flaky checks (e.g. group e-mail credentials check.)

This works in the following way:

1. The check declares its retry policy using class methods.
2. A block can be yielded to if there are problems, but before they are committed to Redis.
3. The job uses this block to either a) schedule a retry if there are any remaining or b) do nothing and let the check commit.
2023-11-06 08:57:02 +08:00
Ted Johansson
47e58357b6
DEV: Parallel scheduled admin checks (#24190)
This PR does some preparatory refactoring of scheduled admin checks in order for us to be able to do custom retry strategies for some of them.

Instead of running all checks in sequence inside a single, scheduled job, the scheduled job spawns one new job per check.

In order to be concurrency-safe, we need to change the existing Redis data structure from a string (of serialized JSON) to a list of strings (of serialized JSON).
2023-11-03 09:05:29 +08:00
Osama Sayegh
3cadd6769e
FEATURE: Theme settings migrations (#24071)
This commit introduces a new feature that allows theme developers to manage the transformation of theme settings over time. Similar to Rails migrations, the theme settings migration system enables developers to write and execute migrations for theme settings, ensuring a smooth transition when changes are required in the format or structure of setting values.

Example use cases for the theme settings migration system:

1. Renaming a theme setting.

2. Changing the data type of a theme setting (e.g., transforming a string setting containing comma-separated values into a proper list setting).

3. Altering the format of data stored in a theme setting.

All of these use cases and more are now possible while preserving theme setting values for sites that have already modified their theme settings.

Usage:

1. Create a top-level directory called `migrations` in your theme/component, and then within the `migrations` directory create another directory called `settings`.

2. Inside the `migrations/settings` directory, create a JavaScript file using the format `XXXX-some-name.js`, where `XXXX` is a unique 4-digit number, and `some-name` is a descriptor of your choice that describes the migration.

3. Within the JavaScript file, define and export (as the default) a function called `migrate`. This function will receive a `Map` object and must also return a `Map` object (it's acceptable to return the same `Map` object that the function received).

4. The `Map` object received by the `migrate` function will include settings that have been overridden or changed by site administrators. Settings that have never been changed from the default will not be included.

5. The keys and values contained in the `Map` object that the `migrate` function returns will replace all the currently changed settings of the theme.

6. Migrations are executed in numerical order based on the XXXX segment in the migration filenames. For instance, `0001-some-migration.js` will be executed before `0002-another-migration.js`.

Here's a complete example migration script that renames a setting from `setting_with_old_name` to `setting_with_new_name`:

```js
// File name: 0001-rename-setting.js

export default function migrate(settings) {
  if (settings.has("setting_with_old_name")) {
    settings.set("setting_with_new_name", settings.get("setting_with_old_name"));
  }
  return settings;
}
```

Internal topic: t/109980
2023-11-02 08:10:15 +03:00
Bianca Nenciu
fd07c943ad
DEV: Refactor watched words (#24163)
- Ignore only invalid words, not all words if one of them is invalid

- The naming scheme for methods was inconsistent

- Optimize regular expressions
2023-11-01 16:41:10 +02:00
Ted Johansson
3f3d2ee2c0
DEV: Deprecate defunct User#flag_level column (#24134)
The User#flag_level column has not been in use for a very long time. The "new" reviewable system dynamically calculates flag scores based on past performance of the user.

This PR removes flag_level from the admin user serializer (since it isn't displayed anywhere in admin user lists) and marks the column as deprecated and targeted for removal in the next minor version.
2023-10-27 17:27:04 +08:00
Ted Johansson
f9f9cf0bf4
DEV: Remove unreachable IP address validation message (#24131)
The message: :signup_not_allowed option to the IP address validator does nothing, because the AllowedIpAddressValidator chooses one of either:

- ip_address.blocked or
- ip_address.max_new_accounts_per_registration_ip

internally. This means that the translation for this was also never used.

This PR removes the ineffectual option and the unused translation. It also moves the translated error messages for blocked and max_new_accounts_per_registration_ip into the correct location so we can pass a symbol to ActiveModel::Errors#add.

There is no actual change in behaviour.
2023-10-27 15:22:38 +08:00
Martin Brennan
219b071994
FIX: Revise and reject post breaks on new topics queued (#24109)
Followup to 9762e65758. This
original commit did not take into account the fact that
new topics can end up in the approval queue as a
ReviewableQueuedPost, and so there was a 500 error raised
when accessing `self.topic` when sending a PM to the user.
2023-10-27 13:05:41 +10:00
Ted Johansson
d0915027a8
DEV: Remove deprecated queue_jobs site setting (#24127)
Using SiteSetting.queue_jobs= to configure job asynchronicity was deprecated here four years ago and marked for removal in version 2.9.0. This PR removes the fallback method we kept since then. The method was there because it was still being used in a bunch of plugin tests (now fixed.)
2023-10-27 11:05:02 +08:00
Ted Johansson
3ad8e1fbde
DEV: Remove deprecated PostAction.remove_act method (#24126)
The PostAction.remove_act class method has been deprecated and replaced by PostActionDestroyer. It was marked for removal in version 2.9.0. This PR removes the method.
2023-10-27 10:02:14 +08:00
Angus McLeod
2a75656ff2
DEV: Add category custom field preloading to CategoryList (#23969)
This commit also introduced a plugin API for preloading category custom
fields.
2023-10-26 16:34:23 +03:00
Alan Guo Xiang Tan
f2a90afa4c
DEV: Introduce Theme#get_setting (#24032)
Why this change?

Currently, we do not have a method to easily retrieve a theme setting's
value on the server side. Such a method can be useful in the test
environment where we need to retrieve the theme's setting and use its
value in assertions.

What does this change do?

This change introduces the `Theme#get_setting` instance method.
2023-10-23 07:41:40 +08:00
Sérgio Saquetim
0cfc42e0e6
FEATURE: Add dark mode option for category backgrounds (#24003)
Adds a new upload field for a dark mode category background that will be used as an alternative when Discourse is using a dark mode theme.
2023-10-20 12:48:06 +00:00
Bianca Nenciu
e7afd18155
DEV: Fix lazy_load_categories for uncategorized topic lists (#24028) 2023-10-20 13:31:20 +03:00
David Battersby
75441e063a
DEV: create new_post_moved event trigger when moving posts (#24005)
This change adds a new event trigger (new_post_moved) when the first post in a topic is moved to a new topic.

Plugins that listen for the new_post_moved event now have an easy way to update old data based on the post id.
2023-10-20 17:56:50 +08:00
Régis Hanol
33715ccc57
FEATURE: Add all user update API scopes (#24016)
There are a few PUT requests that users can do in their preferences tab that aren't going through the standard `user#update` action.

This commit adds all the "trivial" ones (aka. except the security-related one, username and email changes) so you can now change the badge title, the avatar or featured topic of a user via the API.
2023-10-19 15:37:25 +02:00
Martin Brennan
9ef3a18ce4
DEV: Add new experimental admin UI route and sidebar (#23952)
This commit adds a new admin UI under the route `/admin-revamp`, which is
only accessible if the user is in a group defined by the new `enable_experimental_admin_ui_groups` site setting. It
also adds a special `admin` sidebar panel that is shown instead of the `main`
forum one when the admin is in this area.

![image](https://github.com/discourse/discourse/assets/920448/fa0f25e1-e178-4d94-aa5f-472fd3efd787)

We also add an "Admin Revamp" sidebar link to the community section, which
will only appear if the user is in the setting group:

![image](https://github.com/discourse/discourse/assets/920448/ec05ca8b-5a54-442b-ba89-6af35695c104)

Within this there are subroutes defined like `/admin-revamp/config/:area`,
these areas could contain any UI imaginable, this is just laying down an
initial idea of the structure and how the sidebar will work. Sidebar links are
currently hardcoded.

Some other changes:

* Changed the `main` and `chat` panels sidebar panel keys to use exported const values for reuse
* Allowed custom sidebar sections to hide their headers with the `hideSectionHeader` option
* Add a `groupSettingArray` setting on `this.siteSettings` in JS, which accepts a group site setting name
  and splits it by `|` then converts the items in the array to integers, similar to the `_map` magic for ruby
  group site settings
* Adds a `hidden` option for sidebar panels which prevents them from showing in separated mode and prevents
  the switch button from being shown

---------

Co-authored-by: Krzysztof Kotlarek <kotlarek.krzysztof@gmail.com>
2023-10-19 14:23:41 +10:00
Martin Brennan
5dc45b5dcf
FIX: Secure upload post processing race condition (#23968)
* FIX: Secure upload post processing race condition

This commit fixes a couple of issues.

A little background -- when uploads are created in the composer
for posts, regardless of whether the upload will eventually be
marked secure or not, if secure_uploads is enabled we always mark
the upload secure at first. This is so the upload is by default
protected, regardless of post type (regular or PM) or category.

This was causing issues in some rare occasions though because
of the order of operations of our post creation and processing
pipeline. When creating a post, we enqueue a sidekiq job to
post-process the post which does various things including
converting images to lightboxes. We were also enqueuing a job
to update the secure status for all uploads in that post.

Sometimes the secure status job would run before the post process
job, marking uploads as _not secure_ in the background and changing
their ACL before the post processor ran, which meant the users
would see a broken image in their posts. This commit fixes that issue
by always running the upload security changes inline _within_ the
cooked_post_processor job.

The other issue was that the lightbox wrapper link for images in
the post would end up with a URL like this:

```
href="/secure-uploads/original/2X/4/4e1f00a40b6c952198bbdacae383ba77932fc542.jpeg"
```

Since we weren't actually using the `upload.url` to pass to
`UrlHelper.cook_url` here, we weren't converting this href to the CDN
URL if the post was not in a secure context (the UrlHelper does not
know how to convert a secure-uploads URL to a CDN one). Now we
always end up with the correct lightbox href. This was less of an issue
than the other one, since the secure-uploads URL works even when the
upload has become non-secure, but it was a good inconsistency to fix
anyway.
2023-10-18 23:48:01 +00:00
Martin Brennan
e91d8feab3
Revert "FEATURE: Count only approved flagged posts in user pages (#22799)" (#23962)
This reverts commit 5f0bc4557f.

Through extensive internal discussion we have decided to revert
this change, as it significantly impacted moderation flow for
some Discourse site moderators, especially around "something else"
flags. We need to re-approach how flags are counted holistically,
so to that end this change is being reverted.
2023-10-18 11:38:17 +10:00
Bianca Nenciu
bf97899029
DEV: Limit preloaded categories (#23958)
Site data is preloaded on the first page load, which includes categories
data. For sites with many categories, site data takes a long time to
serialize and to transfer.

In the future, preloaded category data will be completely removed.
2023-10-17 22:04:56 +03:00
Blake Erickson
60ae69027c
DEV: Add category style deprecation check warning (#23951)
The category style site setting is being deprecated. This commit will
show a warning on the admin dashboard if a site isn't using the default
category style (bullet).
2023-10-17 10:40:31 -06:00
Bianca Nenciu
c95ffb98ef
DEV: Serialize categories in topic lists (#23597)
At this moment, this feature is under a site setting named
lazy_load_categories.

In the future, categories will no longer be preloaded through site data.
This commit add information about categories in topic list and ensures
that data is used to display topic list items.

Parent categories are serialized too because they are necessary to
render {{category-link}}.
2023-10-17 19:06:01 +03:00
Martin Brennan
61c87fb59f
FIX: Properly attach secure images to email for non-secure uploads (#23865)
There are cases where a user can copy image markdown from a public
post (such as via the discourse-templates plugin) into a PM which
is then sent via an email. Since a PM is a secure context (via the
.with_secure_uploads? check on Post), the image will get a secure
URL in the PM post even though the backing upload is not secure.

This fixes the bug in that case where the image would be stripped
from the email (since it had a /secure-uploads/ URL) but not re-attached
further down the line using the secure_uploads_allow_embed_images_in_emails
setting because the upload itself was not secure.

The flow in Email::Sender for doing this is still not ideal, but
there are chicken and egg problems around when to strip the images,
how to fit in with other attachments and email size limits, and
when to apply the images inline via Email::Styles. It's convoluted,
but at least this fixes the Template use case for now.
2023-10-17 14:08:21 +10:00
Krzysztof Kotlarek
09eca87c76
FIX: synonym tags are not considered as unused (#23950)
Currently, `Tag.unused` scope is used to delete unused tags on `/tags` and by CleanUpTags job. Synonym tags, should not be included and treated as unused. Synonyms are only deleted when main tag is deleted:

https://github.com/discourse/discourse/blob/main/app/models/tag.rb#L57
2023-10-16 23:53:02 +00:00
Alan Guo Xiang Tan
4cb7472376
SECURITY: Prevent arbitrary topic custom fields from being set
Why this change?

The `PostsController#create` action allows arbitrary topic custom fields
to be set by any user that can create a topic. Without any restrictions,
this opens us up to potential security issues where plugins may be using
topic custom fields in security sensitive areas.

What does this change do?

1. This change introduces the `register_editable_topic_custom_field` plugin
API which allows plugins to register topic custom fields that are
editable either by staff users only or all users. The registered
editable topic custom fields are stored in `DiscoursePluginRegistry` and
is called by a new method `Topic#editable_custom_fields` which is then
used in the `PostsController#create` controller action. When an unpermitted custom fields is present in the `meta_data` params,
a 400 response code is returned.

2. Removes all reference to `meta_data` on a topic as it is confusing
   since we actually mean topic custom fields instead.
2023-10-16 10:34:35 -04:00
Martin Brennan
9762e65758
FEATURE: Add Revise... option for queued post reviewable (#23454)
This commit adds a new Revise... action that can be taken
for queued post reviewables. This will open a modal where
the user can select a Reason from a preconfigured list
(or by choosing Other..., a custom reason) and provide feedback
to the user about their post.

The post will be rejected still, but a PM will also be sent to
the user so they have an opportunity to improve their post when
they resubmit it.
2023-10-13 11:28:31 +10:00
David Taylor
525cfcbe0e
FIX: Ensure nested ember components can be used with mustache syntax (#23912)
We run the ember-this-fallback transformation on plugin and theme code so that they can continue omitting `this.` in `.hbs` templates. A bug in the implementation meant that it was incorrectly transforming things like `{{dir/some-component}}` into `<DirSomeComponent />` (rather than `<Dir::SomeComponent />`).

This commit uses patch-package to apply the fix from https://github.com/tildeio/ember-this-fallback/pull/56
2023-10-12 11:08:57 +01:00
Martin Brennan
542f77181a
FIX: Update upload security on post rebake from UI (#23861)
When a user creates or edits a post, we already were updating
the security of uploads in the post based on site settings and
their access control post, which is important since these uploads
may be switched from secure/not secure based on configuration.
The `with_secure_uploads?` method on a post is used to determine
whether to use the secure-uploads URL for all uploads in the post,
regardless of their individual security, so if this is false and
some of the posts are still secure when rebaking, we end up with
broken URLs.

This commit just makes it so rebaking via the UI also re-evaluates
upload security so that when the post is loaded again after processing,
all of the uploads have the correct security.
2023-10-10 11:15:51 +10:00
Krzysztof Kotlarek
c468110929
FEATURE: granular webhooks (#23070)
Before this change, webhooks could be only configured for specific groups like for example, all topic events.

We would like to have more granular control like for example topic_created or topic_destroyed.

Test are failing because plugins changed has to be merged as well:
discourse/discourse-assign#498
discourse/discourse-solved#248
discourse/discourse-topic-voting#159
2023-10-09 03:35:31 +00:00
Alan Guo Xiang Tan
832b3b9e60
FEATURE: Remove support for legacy navigation menu (#23752)
Why this change?

Back in May 17 2023 along with the release of Discourse 3.1, we announced
on meta that the legacy hamburger dropdown navigation menu is
deprecated and will be dropped in Discourse 3.2. This is the link to the announcement
on meta: https://meta.discourse.org/t/removing-the-legacy-hamburger-navigation-menu-option/265274

## What does this change do?

This change removes the `legacy` option from the `navigation_menu` site
setting and migrates existing sites on the `legacy` option to the
`header dropdown` option.

All references to the `legacy` option in code and tests have been
removed as well.
2023-10-09 07:24:10 +08:00
Sam
f21a4a6cb3
Revert "FIX: Allow category moderators to move topics to their categories" (#23810)
This reverts commit 70be873b9c.
2023-10-06 09:00:22 +08:00
Penar Musaraj
0af6c5efdc
DEV: Refactor webauthn to support passkeys (1/3) (#23586)
This is part 1 of 3, split up of PR #23529. This PR refactors the
webauthn code to support passkey authentication/registration.

Passkeys aren't used yet, that is coming in PRs 2 and 3.

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2023-10-03 14:59:28 -04:00
Natalie Tay
70be873b9c
FIX: Allow category moderators to move topics to their categories (#20896) 2023-10-03 17:59:16 +08:00
KThompson-Lane-Unity
607f700c8c
FEATURE: Add API key scopes for tag_groups (#23634) 2023-10-03 16:20:17 +08:00
Jarek Radosz
5a904949b2
DEV: Add gjs support for themes (#23473) 2023-10-02 12:36:06 +02:00
Ella E
b860c6ec17
A11Y: Improve contrast on the WCAG color schemes (#23692)
A11Y: Improve contrast on WCAG color palette

---------

Co-authored-by: Penar Musaraj <pmusaraj@gmail.com>
2023-09-29 10:03:08 -06:00
David Taylor
fc2e92d423
DEV: Drop post_uploads table (#23673)
The post_uploads table has not been used since 9db8f00b3d
2023-09-27 12:43:19 +01:00
Arpit Jalan
3669723a86
FEATURE: allow filtering posts report by multiple categories (#23669) 2023-09-26 21:56:47 +05:30
Bianca Nenciu
3700514819
DEV: Prefer nested queries (#23464)
Some sites have a large number of categories and fetching the category
IDs or category topic IDs just to build another query can take a long
time or resources (i.e. memory).
2023-09-25 19:38:54 +03:00
Ted Johansson
950357391a
DEV: Remove deprecated PostAction.act method (#23641)
The PostAction.act class method was deprecated four years ago and marked for removal in 2.9.0. This PR removes it.
2023-09-24 08:16:32 +01:00
Osama Sayegh
83621ccbe7
FIX: Parse the digest_suppress_tags setting correctly (#23623)
Meta topic: https://meta.discourse.org/t/suppress-these-tags-from-summary-emails-settings-is-not-working-in-preview-digest-email/279196?u=osama

Follow-up to 477a5dd371

The `digest_suppress_tags` setting is designed to be a list of pipe-delimited tag names, but the tag-based topic suppression logic assumes (incorrectly) that the setting contains pipe-delimited tag IDs. This mismatch in expectations led to the setting not working as expected.

This PR adds a step that converts the list of tag names in the setting to their corresponding IDs, which is then used to suppress topics tagged with those specific tags.
2023-09-18 10:45:43 +03:00
Renato Atilio
d93c2cb3d2
FEATURE: site settings to revoke api keys older than a number of days (#23595)
* FEATURE: site settings to revoke api keys older than a number of days
2023-09-15 16:31:29 -03:00
Sérgio Saquetim
e03dd76dc6
FEATURE: add outgoing web hooks for Chat messages 2023-09-13 17:31:42 -03:00
tshenry
c163634ff9
DEV: Bump theme compiler version for max svg sprite size change (#23567)
This should have been included in f6326d0
2023-09-13 10:02:08 -07:00
Bianca Nenciu
6f782d8e45
SECURITY: Add limits for themes and theme assets
This commit adds limits to themes and theme components on the:

- file size of about.json and .discourse-compatibility
- file size of theme assets
- number of files in a theme
2023-09-12 15:31:31 -03:00
Daniel Waterworth
290306a932
SECURITY: Reduce maximum size of SVG sprite cache to prevent DoS
Co-authored-by: Penar Musaraj <pmusaraj@gmail.com>
2023-09-12 15:31:28 -03:00
Gerhard Schlager
e3a2446874
SECURITY: Limit number of drafts per user and length of draft_key
The hidden site setting max_drafts_per_user defaults to 10_000 drafts per user.
The longest key should be "topic_<MAX_BIG_INT>" which is 25 characters.
2023-09-12 15:31:26 -03:00
OsamaSayegh
c1b5faa5fd
SECURITY: Limit name field length of TOTP authenticators and security keys 2023-09-12 15:31:17 -03:00
Ted Johansson
f08c6d2756
DEV: Switch over category settings to new table - Part 3 (#20657)
In #20135 we prevented invalid inputs from being accepted in category setting form fields on the front-end. We didn't do anything on the back-end at that time, because we were still discussing which path we wanted to take. Eventually we decided we want to move this to a new CategorySetting model.

This PR moves the require_topic_approval and require_reply_approval from custom fields to the new CategorySetting model.

This PR is nearly identical to #20580, which migrated num_auto_bump_daily, but since these are slightly more sensitive, they are moved after the previous one is verified.
2023-09-12 09:51:49 +08:00
Alan Guo Xiang Tan
d2e4b32c87
DEV: Add support for uploading a theme from a directory in system tests (#23402)
Why this change?

Currently, we do not have an easy way to test themes and theme components
using Rails system tests. While we support QUnit acceptance tests for
themes and theme components, QUnit acceptance tests stubs out the server
and setting up the fixtures for server responses is difficult and can lead to a
frustrating experience. System tests on the other hand allow authors to
set up the test fixtures using our fabricator system which is much
easier to use.

What does this change do?

In order for us to allow authors to run system tests with their themes
installed, we are adding a `upload_theme` helper that is made available
when writing system tests. The `upload_theme` helper requires a single
`directory` parameter where `directory` is the directory of the theme
locally and returns a `Theme` record.
2023-09-12 07:38:47 +08:00
Bianca Nenciu
4db5310135
DEV: Remove unused topic_create_allowed_category_ids (#23463) 2023-09-08 12:03:22 +03:00
Martin Brennan
c532f6eb3d
FEATURE: Secure uploads in PMs only (#23398)
This adds a new secure_uploads_pm_only site setting. When secure_uploads
is true with this setting, only uploads created in PMs will be marked
secure; no uploads in secure categories will be marked as secure, and
the login_required site setting has no bearing on upload security
either.

This is meant to be a stopgap solution to prevent secure uploads
in a single place (private messages) for sensitive admin data exports.
Ideally we would want a more comprehensive way of saying that certain
upload types get secured which is a hybrid/mixed mode secure uploads,
but for now this will do the trick.
2023-09-06 09:39:09 +10:00
Arpit Jalan
e5f3c26d20
FEATURE: add group filter for admin reports (#23381)
FEATURE: add group filter for admin reports

DEV: add plugin outlet for admin dashboard tabs
2023-09-05 11:17:18 +05:30
Joffrey JAFFEUX
f1d8cd529e
Revert "Revert "PERF: Cache each theme field value once (#23192)" (#23354)" (#23356)
This reverts commit 9821ca9413.
2023-08-31 14:12:03 -05:00
Joffrey JAFFEUX
9821ca9413
Revert "PERF: Cache each theme field value once (#23192)" (#23354)
This reverts commit 82a56334a3.
2023-08-31 19:04:43 +02:00
Daniel Waterworth
82a56334a3
PERF: Cache each theme field value once (#23192)
Previously, theme fields from components would be cached for each of
their parent themes.
2023-08-31 11:24:02 -05:00
Penar Musaraj
006a5166e5
DEV: Refactor rp_id and rp_name (#23339)
They're both constant per-instance values, there is no need to store them
in the session. This also makes the code a bit more readable by moving
the `session_challenge_key` method up to the `DiscourseWebauthn` module.
2023-08-31 09:11:23 -04:00
Renato Atilio
58b49bce41
FEATURE: support to initial values for form templates through /new-topic (#23313)
* FEATURE: adds support for initial values through /new-topic to form templates
2023-08-29 18:41:33 -03:00
Ted Johansson
4b52269827
DEV: Move option to delete user under reviewable reject menu (#23257)
Follow-up to #23199 in which we moved the "delete user" options under the relevant action menu for flagged post. This change does the same, but to queued posts.
2023-08-27 10:05:05 +08:00
Ted Johansson
12cdd7db1a
DEV: Move option to delete user under reviewable agree menu (#23199)
As per discussion, we want to move the options to delete the user under the "Yes" menu of the review queue, since these options are often the most recommended, but also frequently missed because they are tucked away under their own menu.

In addition to the move, I removed the title case of the options so that it matches the other options in the "Yes" menu.
2023-08-25 10:53:56 +08:00
Jarek Radosz
70f1cc5552
DEV: Use esbuild to make DiscourseJsProcessor (#23223)
Reverts e2705df and re-lands #23187 and #23219.

The issue was incorrect order of execution of Rails' `assets:precompile` task in our own precompilation stack.

Co-authored-by: David Taylor <david@taylorhq.com>
2023-08-24 16:36:22 +02:00
David Taylor
e2705df0f4
Revert "DEV: Use esbuild to make DiscourseJsProcessor (#23187)" (#23221)
This reverts commit 4dfe25d062 and 4fdeb6281e. We are investigating an issue related to asset compilation and S3 assets
2023-08-24 13:25:44 +01:00
Jarek Radosz
4dfe25d062
DEV: Use esbuild to make DiscourseJsProcessor (#23187)
Co-authored-by: David Taylor <david@taylorhq.com>
2023-08-24 12:43:59 +02:00
Mark VanLandingham
730f652255
DEV: Add plugin modifier locations for user search locations (#23169) 2023-08-21 12:23:42 -05:00
Juan David Martínez Cubillos
477a5dd371
FEATURE: Digest suppression by tags (#23089)
* FEATURE: Digest suppression by tags

* fixed stree issues

* fixed code so untagged topics are not suppressed when suppressing certain tags
2023-08-18 14:28:20 -05:00
Joffrey JAFFEUX
b2b84cc957
FEATURE: implements user based sidebar mode (#23078) 2023-08-18 20:33:07 +02:00
David Taylor
82b16f4f47
DEV: Do not manipulate theme module paths at build-time (#23148)
Manipulating theme module paths means that the paths you author are not the ones used at runtime. This can lead to some very unexpected behavior and potential module name clashes. It also meant that the refactor in 16c6ab8661 was unable to correctly match up theme connector js/templates.

While this could technically be a breaking change, I think it is reasonably safe because:

1. Themes are already forced to use relative paths when referencing their own modules (since they're namespaced based on the site-specific id). The only time this might be problematic is when theme tests reference modules in the theme's main `javascripts` directory

2. For things like components/services/controllers/etc. our custom Ember resolver works backwards from the end of the path, so adding `discourse/` in the middle will not affect resolution.
2023-08-18 18:15:23 +01:00
marstall
0dd1ee2e09
FIX: correct bulk invite expire time for DST (#23073)
This is a bug that happens only when the current date is less than 90 days from a date on which the time zone transitions into or out of Daylight Savings Time.

In these conditions, bulk invites show the time of day of their expiration as being 1 hour later than the current time.

Whereas it should match the time of day the invite was generated.

This is because the server has not been using the user's timezone in calculating the expiration time of day. This PR fixes issue by considering the user's timezone when doing the date math.

https://meta.discourse.org/t/bulk-invite-logic-to-generate-expire-date-bug/274689
2023-08-18 12:33:40 -04:00
Daniel Waterworth
91f560a521
DEV: Make every DistributedCache lazily instantiated (#23147) 2023-08-18 10:59:11 -05:00
Selase Krakani
87ebbec9b2
FIX: Pending post deletion by creator (#23130)
`ReviewableQueuedPost` got refactored a while back to use the more
appropriate `target_created_by` for the user of the post being queued
instead of `created_by`. The change was not extended to the `DELETE
/review/:id` endpoint leading to error responses for a user attempting
to deleting their own queued post.

This fix extends the `Reviewable` lookup implementation in
`ReviewablesController#destroy` and Guardian implementation to account
for this change.
2023-08-18 15:30:59 +00:00
Penar Musaraj
10c6b2a0c2
WIP: Rename Webauthn to DiscourseWebauthn (#23077) 2023-08-18 08:39:10 -04:00
David Taylor
16c6ab8661
DEV: Allow plugin outlets to be defined using gjs (#23142)
Previously we were discovering plugin outlets by checking first for dedicated template files, and then looking for classes to match them. This doesn't work for components which are entirely defined in JS (e.g. those authored with gjs, or those which are re-exports of a colocated component).

This commit refactors our detection logic to look for both class and template modules in a single pass. It also refactors things so that the modules themselves are required lazily when needd, rather than all being loaded during app boot.
2023-08-18 12:07:10 +01:00
Ted Johansson
79e3d4e2bd
FIX: Don't run post validations when hiding post (#23139)
When hiding a post (essentially updating hidden, hidden_at, and hidden_reason_id) our callbacks are running the whole battery of post validations. This can cause the hiding to fail in a number of edge cases. The issue is similar to the one fixed in #11680, but applies to all post validations, none of which should apply when hiding a post.

After some code reading and discussion, none of the validations in PostValidator seem to be relevant when hiding posts, so instead of just skipping unique check, we skip all post validator checks.
2023-08-18 10:55:17 +08:00
Daniel Waterworth
80a0d88e4c
FIX: Ensure javascript caches are unique per theme/theme_field (#23126)
We were assuming that this was the case before, but not enforcing it.
2023-08-17 13:57:04 -05:00
Krzysztof Kotlarek
f8cd1da92a
FIX: increase sidebar URL limit to 1000 (#23120)
Before this change, sidebar URL had a limit of 200 characters. In some cases it is not enough, therefore it was increased to 1000.
2023-08-17 14:46:24 +10:00
Andreas Klöckner
9e568df316
FIX: reference to non-existent groups#remove_members in API key scope (#23042) 2023-08-17 07:20:55 +10:00
marstall
77626c088e
UX: support links in tag descriptions (#22994)
* scrub non-a html tags from tag descriptions on create, strips all tags from tag description when displayed in tag hover

* test for tag description links

* UX: basic render-tag test

* UX: fix linting

* UX: fix linting

* fix broken tests

* Update spec/models/tag_spec.rb

Co-authored-by: Penar Musaraj <pmusaraj@gmail.com>

* UX: use has_sanitizable_fields instead of has_scrubbable_fields to ensafen tag.description

---------

Co-authored-by: Penar Musaraj <pmusaraj@gmail.com>
2023-08-16 11:43:54 -04:00
Renato Atilio
840bea3c51
FEATURE: add topic voting webhook event type (#23072)
* FEATURE: add topic upvote webhook event type

* DEV: use a generic event type name for other actions in the same plugin
2023-08-11 13:42:28 -03:00
Ted Johansson
8053cb0c21
DEV: Add a HasDeprecatedColumns concern for better deprecation messages (#22930)
Currently when we decide we're going to drop a column in the future we just mark it with a TODO comment and add it to ignored_columns. This makes it instantly unavailable, and we mostly forget about the TODO in the end. 😬

This change adds a HasDeprecatedColumns concern which offers a little bit more flexibility. We can still simulate the old behaviour by setting drop_from to the current version, but we can also set it to a future version, causing it to raise a deprecation warning until then if used.
2023-08-11 15:25:44 +08:00
Jarek Radosz
94649565ce
DEV: Correct Style/RedundantReturn rubocop issues (#23052) 2023-08-10 02:03:38 +02:00
Gerhard Schlager
b2fee68b3f DEV: Add rake task for generating avatars from SSO 2023-08-09 20:56:14 +02:00
Martin Brennan
09223e5ae7
DEV: Remove enable_experimental_hashtag_autocomplete logic (#22820)
This commit removes any logic in the app and in specs around
enable_experimental_hashtag_autocomplete and deletes some
old category hashtag code that is no longer necessary.

It also adds a `slug_ref` category instance method, which
will generate a reference like `parent:child` for a category,
with an optional depth, which hashtags use. Also refactors
PostRevisor which was using CategoryHashtagDataSource directly
which is a no-no.

Deletes the old hashtag markdown rule as well.
2023-08-08 11:18:55 +10:00
David Taylor
e76e0ad592
DEV: In development, refresh client when theme changes are made (#22978)
This brings the theme development experience (via the discourse_theme cli) closer to the experience of making javascript changes in Discourse core/plugins via Ember CLI. Whenever a change is made to a non-css theme field, all clients will be instructed to immediately refresh via message-bus.
2023-08-04 11:02:26 +01:00
Ted Johansson
1f7e5e8e75
DEV: Switch over category settings to new table - Part 2 (#20580)
In #20135 we prevented invalid inputs from being accepted in category setting form fields on the front-end. We didn't do anything on the back-end at that time, because we were still discussing which path we wanted to take. Eventually we decided we want to move this to a new CategorySetting model.

This PR moves the num_auto_bump_daily from custom fields to the new CategorySetting model.

In addition it sets the default value to 0, which exhibits the same behaviour as when the value is NULL.
2023-08-04 10:53:22 +08:00
Renato Atilio
701ae8764e
FIX: keep first post edit history when moving/merging (#22966) 2023-08-03 22:04:35 -03:00
TheJammiestDodger
e680437030
FIX: Add 'Ignored' flags to Moderator Activity report (#22041)
* FIX Add 'Ignored' flags to Moderator Activity report

The Moderator Activity query didn’t include the number of deferred flags in the Flags Reviewed totals. As this number is designed to reflect how many flags a moderator has seen, reviewed, and made a judgement on, the Ignored ones should also be included.

* Apply suggestions from code review

---------

Co-authored-by: Jarek Radosz <jradosz@gmail.com>
2023-08-02 12:27:31 +01:00
Daniel Waterworth
50d527b80c
DEV: Remove db_timeout setting (#22912)
It doesn't actually do anything.
2023-08-01 14:17:43 -05:00
Martin Brennan
6286e790b2
DEV: Remove unread_private_messages and deprecation (#22893)
This was added all the way back in 2020 in b79ea986ac,
enough time has passed, we can delete this now.
2023-08-01 14:44:39 +10:00
Kelv
5f0bc4557f
FEATURE: Count only approved flagged posts in user pages (#22799)
FEATURE: Only approved flags for post counters

* Why was this change necessary?
The counters for flagged posts in the user's profile and user index from
the admin view include flags that were rejected, ignored or pending
review. This introduces unnecessary noise. Also the flagged posts
counter in the user's profile includes custom flags which add further
noise to this signal.

* How does it address the problem?

* Modifying User#flags_received_count to return posts with only approved
  standard flags
* Refactoring User#number_of_flagged_posts to alias to
  User#flags_received_count
* Updating the flagged post staff counter hyperlink to navigate to a
  filtered view of that user's approved flagged posts to maintain
  consistency with the counter
* Adding system tests for the profile page to cover the flagged posts
  staff counter
2023-07-31 13:33:10 +08:00
Alan Guo Xiang Tan
2f5e66b6f8
PERF: Optimise TopicTrackingState.report query to speed up query (#22871)
In the query generated by `TopicTrackingState.report`, there are two
subqueies being executed. The first subquery fetches all the topics
that are new for a given user while the second subquery fetches all the topics with
unread posts for a given user. For the second subquery, there is a
filter `topics.updated_at >= user_stats.first_unread_at` which is used
as a performance optimisation to reduce the number of rows that PG has
to scan through the `topics` table.

However, we started to notice in production that the PG planner doesn't
always execute the filter first to reduce the number of rows that it has
to scan through. Running the following query in one of our production
instance,

```
EXPLAIN ANALYZE
SELECT
           DISTINCT topics.id as topic_id,
           u.id as user_id,
           topics.created_at,
           topics.updated_at,
           topics.highest_staff_post_number AS highest_post_number,
           last_read_post_number,
           c.id as category_id,
           c.topic_id AS category_topic_id,
           tu.notification_level,
           us.first_unread_at,
           GREATEST(
              CASE
              WHEN COALESCE(uo.new_topic_duration_minutes, 2880) = -1 THEN u.created_at
              WHEN COALESCE(uo.new_topic_duration_minutes, 2880) = -2 THEN COALESCE(
                u.previous_visit_at,u.created_at
              )
              ELSE ('2023-07-31 03:29:45.737630'::timestamp - INTERVAL '1 MINUTE' * COALESCE(uo.new_topic_duration_minutes, 2880))
              END, u.created_at, '2023-07-25 15:06:44'
           ) AS treat_as_new_topic_start_date
FROM topics
JOIN users u on u.id = 13455
JOIN user_stats AS us ON us.user_id = u.id
JOIN user_options AS uo ON uo.user_id = u.id
JOIN categories c ON c.id = topics.category_id
LEFT JOIN topic_users tu ON tu.topic_id = topics.id AND tu.user_id = u.id

WHERE u.id = 13455 AND
       topics.updated_at >= us.first_unread_at AND
      topics.archetype <> 'private_message' AND
      (("topics"."deleted_at" IS NULL AND (tu.last_read_post_number < topics.highest_staff_post_number) AND (COALESCE(tu.notification_level, 1) >= 2)) OR (1=0)) AND

      NOT (
  COALESCE((select array_agg(tag_id) from topic_tags where topic_tags.topic_id = topics.id), ARRAY[]::int[]) && ARRAY[451,452,453]
) AND

      topics.deleted_at IS NULL AND

      NOT (
        last_read_post_number IS NULL AND
        (
          topics.category_id IN (SELECT "categories"."id" FROM "categories" LEFT JOIN categories categories2 ON categories2.id = categories.parent_category_id LEFT JOIN category_users ON category_users.category_id = categories.id AND category_users.user_id = 13455 LEFT JOIN category_users category_users2 ON category_users2.category_id = categories2.id AND category_users2.user_id = 13455 WHERE ((category_users.id IS NULL AND COALESCE(category_users2.notification_level, 1) = 0) OR COALESCE(category_users.notification_level, 1) = 0))
          AND tu.notification_level <= 1
        )
      )
```

we get the following

```
                                                                                                                                                                                                                                                                                                                          QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Unique  (cost=201606.06..201608.15 rows=76 width=60) (actual time=91.279..91.294 rows=14 loops=1)
   ->  Sort  (cost=201606.06..201606.25 rows=76 width=60) (actual time=91.278..91.284 rows=14 loops=1)
         Sort Key: topics.id, topics.created_at, topics.updated_at, topics.highest_staff_post_number, tu.last_read_post_number, c.id, c.topic_id, tu.notification_level, us.first_unread_at, (GREATEST(CASE WHEN (COALESCE(uo.new_topic_duration_minutes, 2880) = '-1'::integer) THEN u.created_at WHEN (COALESCE(uo.new_topic_duration_minutes, 2880) = '-2'::integer) THEN COALESCE(u.previous_visit_at, u.created_at) ELSE ('2023-07-31 03:29:45.73763'::timestamp without time zone - ('00:01:00'::interval * (COALESCE(uo.new_topic_duration_minutes, 2880))::double precision)) END, u.created_at, '2023-07-25 15:06:44'::timestamp without time zone))
         Sort Method: quicksort  Memory: 26kB
         ->  Hash Join  (cost=97519.51..201603.69 rows=76 width=60) (actual time=87.662..91.268 rows=14 loops=1)
               Hash Cond: (topics.id = tu.topic_id)
               Join Filter: ((tu.last_read_post_number < topics.highest_staff_post_number) AND ((tu.last_read_post_number IS NOT NULL) OR (NOT (hashed SubPlan 2)) OR (tu.notification_level > 1)))
               Rows Removed by Join Filter: 10
               ->  Nested Loop  (cost=1.54..104075.36 rows=3511 width=68) (actual time=0.055..3.609 rows=548 loops=1)
                     ->  Nested Loop  (cost=1.13..25.20 rows=1 width=32) (actual time=0.027..0.033 rows=1 loops=1)
                           ->  Nested Loop  (cost=0.71..16.76 rows=1 width=28) (actual time=0.020..0.023 rows=1 loops=1)
                                 ->  Index Scan using users_pkey on users u  (cost=0.42..8.44 rows=1 width=20) (actual time=0.010..0.012 rows=1 loops=1)
                                       Index Cond: (id = 13455)
                                 ->  Index Scan using user_stats_pkey on user_stats us  (cost=0.29..8.31 rows=1 width=12) (actual time=0.008..0.010 rows=1 loops=1)
                                       Index Cond: (user_id = 13455)
                           ->  Index Scan using index_user_options_on_user_id_and_default_calendar on user_options uo  (cost=0.42..8.44 rows=1 width=8) (actual time=0.007..0.008 rows=1 loops=1)
                                 Index Cond: (user_id = 13455)
                     ->  Nested Loop  (cost=0.41..104015.12 rows=3504 width=36) (actual time=0.026..3.503 rows=548 loops=1)
                           ->  Seq Scan on categories c  (cost=0.00..13.73 rows=73 width=8) (actual time=0.003..0.039 rows=73 loops=1)
                           ->  Index Only Scan using index_topics_on_updated_at_public on topics  (cost=0.41..1424.20 rows=48 width=28) (actual time=0.012..0.046 rows=8 loops=73)
                                 Index Cond: ((updated_at >= us.first_unread_at) AND (category_id = c.id))
                                 Filter: (NOT (COALESCE((SubPlan 1), '{}'::integer[]) && '{451,452,453}'::integer[]))
                                 Heap Fetches: 553
                                 SubPlan 1
                                   ->  Aggregate  (cost=4.31..4.32 rows=1 width=32) (actual time=0.002..0.002 rows=1 loops=548)
                                         ->  Index Only Scan using index_topic_tags_on_topic_id_and_tag_id on topic_tags  (cost=0.29..4.31 rows=1 width=4) (actual time=0.002..0.002 rows=1 loops=548)
                                               Index Cond: (topic_id = topics.id)
                                               Heap Fetches: 178
               ->  Hash  (cost=97222.14..97222.14 rows=19914 width=16) (actual time=87.545..87.546 rows=42884 loops=1)
                     Buckets: 65536 (originally 32768)  Batches: 1 (originally 1)  Memory Usage: 2387kB
                     ->  Bitmap Heap Scan on topic_users tu  (cost=1217.47..97222.14 rows=19914 width=16) (actual time=14.419..78.286 rows=42884 loops=1)
                           Recheck Cond: (user_id = 13455)
                           Filter: (COALESCE(notification_level, 1) >= 2)
                           Rows Removed by Filter: 15839
                           Heap Blocks: exact=45285
                           ->  Bitmap Index Scan on index_topic_users_on_user_id_and_topic_id  (cost=0.00..1212.49 rows=59741 width=0) (actual time=6.448..6.448 rows=58723 loops=1)
                                 Index Cond: (user_id = 13455)
               SubPlan 2
                 ->  Nested Loop Left Join  (cost=0.74..46.90 rows=1 width=4) (never executed)
                       Join Filter: (category_users2.category_id = categories2.id)
                       Filter: (((category_users.id IS NULL) AND (COALESCE(category_users2.notification_level, 1) = 0)) OR (COALESCE(category_users.notification_level, 1) = 0))
                       ->  Nested Loop Left Join  (cost=0.45..32.31 rows=73 width=16) (never executed)
                             Join Filter: (category_users.category_id = categories.id)
                             ->  Nested Loop Left Join  (cost=0.15..18.45 rows=73 width=8) (never executed)
                                   ->  Seq Scan on categories  (cost=0.00..13.73 rows=73 width=8) (never executed)
                                   ->  Memoize  (cost=0.15..0.28 rows=1 width=4) (never executed)
                                         Cache Key: categories.parent_category_id
                                         Cache Mode: logical
                                         ->  Index Only Scan using categories_pkey on categories categories2  (cost=0.14..0.27 rows=1 width=4) (never executed)
                                               Index Cond: (id = categories.parent_category_id)
                                               Heap Fetches: 0
                             ->  Materialize  (cost=0.29..11.69 rows=2 width=12) (never executed)
                                   ->  Index Scan using idx_category_users_user_id_category_id on category_users  (cost=0.29..11.68 rows=2 width=12) (never executed)
                                         Index Cond: (user_id = 13455)
                       ->  Materialize  (cost=0.29..11.69 rows=2 width=8) (never executed)
                             ->  Index Scan using idx_category_users_user_id_category_id on category_users category_users2  (cost=0.29..11.68 rows=2 width=8) (never executed)
                                   Index Cond: (user_id = 13455)
 Planning Time: 1.740 ms
 Execution Time: 91.414 ms
(59 rows)
```

From the execution plan, we can see the most of the time is spent
joining about 42888 rows in the `topics` table to the `topic_users` table.
However, we know that we only have to scan through a
subset of the `topics` table because the user's last unread at is '2023-07-20 11:33:05'.
If we filter the `topics` table with `topics.updated_at >= '2023-07-20 11:33:05'`, this would only
return about 1500 rows.

From our testing in production, the PG planner is able to execute a
better query plan when we avoid the unnecessary joins on `user_stats` just to be
able to get the user's `UserStat#first_unread_at`. Instead, we can just
pass the value of `UserStat#first_unread_at` directly as a query
parameter.

```
EXPLAIN ANALYZE
SELECT
           DISTINCT topics.id as topic_id,
           u.id as user_id,
           topics.created_at,
           topics.updated_at,
           topics.highest_staff_post_number AS highest_post_number,
           last_read_post_number,
           c.id as category_id,
           c.topic_id AS category_topic_id,
           tu.notification_level,
           GREATEST(
              CASE
              WHEN COALESCE(uo.new_topic_duration_minutes, 2880) = -1 THEN u.created_at
              WHEN COALESCE(uo.new_topic_duration_minutes, 2880) = -2 THEN COALESCE(
                u.previous_visit_at,u.created_at
              )
              ELSE ('2023-07-31 03:29:45.737630'::timestamp - INTERVAL '1 MINUTE' * COALESCE(uo.new_topic_duration_minutes, 2880))
              END, u.created_at, '2023-07-25 15:06:44'
           ) AS treat_as_new_topic_start_date
FROM topics
JOIN users u on u.id = 13455
JOIN user_options AS uo ON uo.user_id = u.id
JOIN categories c ON c.id = topics.category_id
LEFT JOIN topic_users tu ON tu.topic_id = topics.id AND tu.user_id = u.id

WHERE u.id = 13455 AND
       topics.updated_at >= '2023-07-20 11:33:05' AND
      topics.archetype <> 'private_message' AND
      (("topics"."deleted_at" IS NULL AND (tu.last_read_post_number < topics.highest_staff_post_number) AND (COALESCE(tu.notification_level, 1) >= 2)) OR (1=0)) AND

      NOT (
  COALESCE((select array_agg(tag_id) from topic_tags where topic_tags.topic_id = topics.id), ARRAY[]::int[]) && ARRAY[451,452,453]
) AND

      topics.deleted_at IS NULL AND

      NOT (
        last_read_post_number IS NULL AND
        (
          topics.category_id IN (SELECT "categories"."id" FROM "categories" LEFT JOIN categories categories2 ON categories2.id = categories.parent_category_id LEFT JOIN category_users ON category_users.category_id = categories.id AND category_users.user_id = 13455 LEFT JOIN category_users category_users2 ON category_users2.category_id = categories2.id AND category_users2.user_id = 13455 WHERE ((category_users.id IS NULL AND COALESCE(category_users2.notification_level, 1) = 0) OR COALESCE(category_users.notification_level, 1) = 0))
          AND tu.notification_level <= 1
        )
      );
```

Note how the filter is now `topics.updated_at >= '2023-07-20 11:33:05'`
instead of `topics.updated_at >= us.first_unread_at`. The modified query
above generates the following execution plan.

```
                                                                                                                                                                                                                                                                                                                QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Unique  (cost=5189.86..5189.88 rows=1 width=52) (actual time=4.991..5.002 rows=14 loops=1)
   ->  Sort  (cost=5189.86..5189.86 rows=1 width=52) (actual time=4.990..4.994 rows=14 loops=1)
         Sort Key: topics.id, topics.created_at, topics.updated_at, topics.highest_staff_post_number, tu.last_read_post_number, c.id, c.topic_id, tu.notification_level, (GREATEST(CASE WHEN (COALESCE(uo.new_topic_duration_minutes, 2880) = '-1'::integer) THEN u.created_at WHEN (COALESCE(uo.new_topic_duration_minutes, 2880) = '-2'::integer) THEN COALESCE(u.previous_visit_at, u.created_at) ELSE ('2023-07-31 03:29:45.73763'::timestamp without time zone - ('00:01:00'::interval * (COALESCE(uo.new_topic_duration_minutes, 2880))::double precision)) END, u.created_at, '2023-07-25 15:06:44'::timestamp without time zone))
         Sort Method: quicksort  Memory: 26kB
         ->  Nested Loop  (cost=52.11..5189.85 rows=1 width=52) (actual time=0.093..4.974 rows=14 loops=1)
               ->  Nested Loop  (cost=51.70..5181.39 rows=1 width=60) (actual time=0.084..4.931 rows=14 loops=1)
                     ->  Nested Loop  (cost=51.28..5172.94 rows=1 width=44) (actual time=0.076..4.887 rows=14 loops=1)
                           ->  Nested Loop  (cost=0.41..1698.46 rows=59 width=36) (actual time=0.029..3.537 rows=548 loops=1)
                                 ->  Seq Scan on categories c  (cost=0.00..13.73 rows=73 width=8) (actual time=0.005..0.039 rows=73 loops=1)
                                 ->  Index Only Scan using index_topics_on_updated_at_public on topics  (cost=0.41..23.07 rows=1 width=28) (actual time=0.012..0.047 rows=8 loops=73)
                                       Index Cond: ((updated_at >= '2023-07-20 11:33:05'::timestamp without time zone) AND (category_id = c.id))
                                       Filter: (NOT (COALESCE((SubPlan 1), '{}'::integer[]) && '{451,452,453}'::integer[]))
                                       Heap Fetches: 552
                                       SubPlan 1
                                         ->  Aggregate  (cost=4.31..4.32 rows=1 width=32) (actual time=0.002..0.002 rows=1 loops=548)
                                               ->  Index Only Scan using index_topic_tags_on_topic_id_and_tag_id on topic_tags  (cost=0.29..4.31 rows=1 width=4) (actual time=0.002..0.002 rows=1 loops=548)
                                                     Index Cond: (topic_id = topics.id)
                                                     Heap Fetches: 178
                           ->  Index Scan using index_topic_users_on_user_id_and_topic_id on topic_users tu  (cost=50.86..58.88 rows=1 width=16) (actual time=0.002..0.002 rows=0 loops=548)
                                 Index Cond: ((user_id = 13455) AND (topic_id = topics.id))
                                 Filter: ((COALESCE(notification_level, 1) >= 2) AND (last_read_post_number < topics.highest_staff_post_number) AND ((last_read_post_number IS NOT NULL) OR (NOT (hashed SubPlan 2)) OR (notification_level > 1)))
                                 Rows Removed by Filter: 0
                                 SubPlan 2
                                   ->  Nested Loop Left Join  (cost=0.74..50.43 rows=1 width=4) (never executed)
                                         Join Filter: (category_users2.category_id = categories2.id)
                                         Filter: (((category_users.id IS NULL) AND (COALESCE(category_users2.notification_level, 1) = 0)) OR (COALESCE(category_users.notification_level, 1) = 0))
                                         ->  Nested Loop Left Join  (cost=0.45..35.84 rows=73 width=16) (never executed)
                                               Join Filter: (category_users.category_id = categories.id)
                                               ->  Nested Loop Left Join  (cost=0.15..21.97 rows=73 width=8) (never executed)
                                                     ->  Seq Scan on categories  (cost=0.00..13.73 rows=73 width=8) (never executed)
                                                     ->  Memoize  (cost=0.15..0.61 rows=1 width=4) (never executed)
                                                           Cache Key: categories.parent_category_id
                                                           Cache Mode: logical
                                                           ->  Index Only Scan using categories_pkey on categories categories2  (cost=0.14..0.60 rows=1 width=4) (never executed)
                                                                 Index Cond: (id = categories.parent_category_id)
                                                                 Heap Fetches: 0
                                               ->  Materialize  (cost=0.29..11.69 rows=2 width=12) (never executed)
                                                     ->  Index Scan using idx_category_users_user_id_category_id on category_users  (cost=0.29..11.68 rows=2 width=12) (never executed)
                                                           Index Cond: (user_id = 13455)
                                         ->  Materialize  (cost=0.29..11.69 rows=2 width=8) (never executed)
                                               ->  Index Scan using idx_category_users_user_id_category_id on category_users category_users2  (cost=0.29..11.68 rows=2 width=8) (never executed)
                                                     Index Cond: (user_id = 13455)
                     ->  Index Scan using users_pkey on users u  (cost=0.42..8.44 rows=1 width=20) (actual time=0.003..0.003 rows=1 loops=14)
                           Index Cond: (id = 13455)
               ->  Index Scan using index_user_options_on_user_id_and_default_calendar on user_options uo  (cost=0.42..8.44 rows=1 width=8) (actual time=0.002..0.002 rows=1 loops=14)
                     Index Cond: (user_id = 13455)
 Planning Time: 1.281 ms
 Execution Time: 5.092 ms
(48 rows)
```

With the new query, PG first does an index scan using the `index_topics_on_updated_at_public` index to filter away most of the topics making the subsequent joins much cheaper. Total query time has been reduced from ~90ms to ~5ms.

This optimisation will mostly affect users with very few/recent unread topics since a large `UserStat#firsts_unread_at` value will still mean scanning through a large portion of the `topics` table.
2023-07-31 12:21:41 +08:00
Ted Johansson
c4d0bbce62
DEV: Delete upload references upon deleting draft (#22851)
We currently are accumulating orphaned upload references whenever drafts are deleted.

This change deals with future cases by adding a dependent strategy of delete_all on the Draft#upload_references association. (We don't really need destroy strategy here, since UploadReference is a simple data bag and there are no validations or callbacks on the model.)

It deals with existing cases through a migration that deletes all existing, orphaned draft upload references.
2023-07-31 10:16:23 +08:00
Selase Krakani
81cf481b16
FIX: Missing pending queued posts from topic view (#22838)
A previous change updated `ReviewableQueuedPost`'s `created_by`
to be consistent with other reviewable types. It assigns
the the creator of the post being queued to `target_created_by` and sets
the `created_by` to the creator of the reviewable itself.

This fix updates some of the `created_by` references missed during the
intial fix.
2023-07-28 16:16:23 +00:00
Bianca Nenciu
0736611423
SECURITY: Hide restricted tags in noscript view
The hidden tags are usually filtered out by the serializer, but the
noscript view uses the topic objects instead of the serialized objects.
2023-07-28 12:53:50 +01:00
Penar Musaraj
dcc825bda5
SECURITY: Limit length of edit reason column 2023-07-28 12:53:49 +01:00
Blake Erickson
62a609ea2d
SECURITY: Handle concurrent invite accepts
Raise an error on concurrent invite accept attempts.
2023-07-28 12:53:48 +01:00
Alan Guo Xiang Tan
bfc3132bb2
SECURITY: Impose a upper bound on limit params in various controllers
What is the problem here?

In multiple controllers, we are accepting a `limit` params but do not
impose any upper bound on the values being accepted. Without an upper
bound, we may be allowing arbituary users from generating DB queries
which may end up exhausing the resources on the server.

What is the fix here?

A new `fetch_limit_from_params` helper method is introduced in
`ApplicationController` that can be used by controller actions to safely
get the limit from the params as a default limit and maximum limit has
to be set. When an invalid limit params is encountered, the server will
respond with the 400 response code.
2023-07-28 12:53:46 +01:00
Natalie Tay
173de8afe6
DEV: Add logging and rescue when user already exists and connecting via DiscourseConnect (#22833)
This is happening because despite the user already existing in the forum, the `SingleSignOnRecord` doesn't exist and "require_activation" is set on the provider, causing us to skip looking for the email, and resulting in us creating a new User then seeing Validation failed: Primary email has already been taken when DiscourseConnect is attempting to make a new account.
2023-07-28 02:53:33 +08:00