Commit Graph

3288 Commits

Author SHA1 Message Date
Krzysztof Kotlarek
99e27c4fd5
FIX: change around to after in flag spec (#27168) 2024-05-24 12:55:21 +10:00
Krzysztof Kotlarek
a4c5f85b10
FIX: flaky post action counts specs (#27165)
After flags were moved to the database, with each save they are changing available PostActionTypes. Therefore, flag specs should clear the state before and after each example not just before.

In addition, we need to clear `nil` counts for dynamically created flags from serializer.
2024-05-24 11:55:32 +10:00
Jeff Wong
755f8de6d4
FEATURE: add agree and edit (#27088)
* FEATURE: add agree and edit

adds agree and edit - an alias for agree and keep -- but with a client action to
edit the post in the composer before the flag is agreed with

---------

Co-authored-by: Juan David Martinez <juan@discourse.org>
2024-05-23 11:21:42 -07:00
Ted Johansson
7b437c9401
FEATURE: Implement new required options in admin user fields UI (#27079)
We're planning to implement a feature that allows adding required fields for existing users. This PR does some preparatory refactoring to make that possible. There should be no changes to existing behaviour. Just a small update to the admin UI.
2024-05-23 19:18:25 +08:00
Alan Guo Xiang Tan
f84eda7c8d
FIX: Post#each_upload_url yielding external URLs (#27149)
This commit updates `Post#each_upload_url` to reject URLs that do not
have a host which matches `Discourse.current_hostname` but follows the
`/uploads/short-url` uploads URL format. This situation most commonly
happen when users copy upload URL link between different Discourse
sites.
2024-05-23 15:15:16 +10:00
Krzysztof Kotlarek
cfbbfd177c
DEV: move post flags into database (#27125)
This is preparation for a feature that will allow admins to define their custom flags. Current behaviour should stay untouched.
2024-05-23 12:19:07 +10:00
Ted Johansson
3137e60653
DEV: Database backed admin notices (#26192)
This PR introduces a basic AdminNotice model to store these notices. Admin notices are categorized by their source/type (currently only notices from problem check.) They also have a priority.
2024-05-23 09:29:08 +08:00
Krzysztof Kotlarek
40d65dddf8
Revert "DEV: move post flags into database (#26951)" (#27102)
This reverts commit 7aff9806eb.
2024-05-21 16:21:07 +10:00
Krzysztof Kotlarek
7aff9806eb
DEV: move post flags into database (#26951)
This is preparation for a feature that will allow admins to define their custom flags. Current behaviour should stay untouched.
2024-05-21 13:15:32 +10:00
Régis Hanol
b908abe35a
FIX: keep topic.word_count in sync (#27065)
Whenever one creates, updates, or deletes a post, we should keep the `topic.word_count` counter in sync.

Context - https://meta.discourse.org/t/-/308062
2024-05-17 17:05:49 +02:00
Jean
63b7a36fac
FEATURE: Extend embeddable hosts with Individual tags and author assignments (#26868)
* FEATURE: Extend embeddable hosts with tags and author assignments
2024-05-16 15:47:01 -04:00
Osama Sayegh
e3ae57ea7a
FIX: Create directory items for new users when in bootstrap mode (#27020)
The users directory is updated on a daily cadence. However, when a site is new and doesn't have many users, it can be confusing that a user who has just joined doesn't show up in the users until a day after they join. To eliminate this confusion, this commit triggers a refresh for the users directory as soon as as a user joins, if the site is in bootstrap mode. The reason for the conditional trigger is that refreshing the users directory is an expensive operation and doing it often on a large site with many users could lead to performance problems.

Internal topic: t/126076.
2024-05-15 03:06:58 +03:00
Daniel Waterworth
e2ceea8815
FIX: Preload all ancestors of sidebar categories (#26715)
... instead of just the immediate parents.
2024-05-06 11:55:20 -05:00
Régis Hanol
10f77556cd FIX: ensure no infinite category loop
If there's ever a circular reference in categories, don't go into an infinite loop when generating the category slug.

Instead, keep track of parent ids, and bail out as soon as we're encountering one more than once.
2024-05-06 18:02:22 +02:00
Ted Johansson
9655bf3e24
DEV: Delete upload references on draft cleanup (#26877)
In #22851 we added a dependent strategy for deleting upload references when a draft is destroyed. This, however, didn't catch all cases, because we still have some code that issues DELETE drafts queries directly to the database. Specifically in the weekly cleanup job handled by Draft#cleanup!.

This PR fixes that by turning the raw query into an ActiveRecord #destroy_all, which will invoke the dependent strategy that ultimately deletes the upload references. It also includes a post migration to clear orphaned upload references that are already in the database.
2024-05-06 14:08:10 +08:00
Alan Guo Xiang Tan
243fcb6ffc
DEV: Introduce run_theme_migration spec helper in test environment (#26845)
This commit introduces the `run_theme_migration` spec helper to allow
theme developers to write RSpec tests for theme migrations. For example,
this allows the following RSpec test to be written in themes:

```
RSpec.describe "0003-migrate-small-links-setting migration" do
  let!(:theme) { upload_theme_component }

  it "should set target property to `_blank` if previous target component is not valid or empty" do
    theme.theme_settings.create!(
      name: "small_links",
      theme: theme,
      data_type: ThemeSetting.types[:string],
      value: "some text, #|some text 2, #, invalid target",
    )

    run_theme_migration(theme, "0003-migrate-small-links-setting")

    expect(theme.settings[:small_links].value).to eq(
      [
        { "text" => "some text", "url" => "#", "target" => "_blank" },
        { "text" => "some text 2", "url" => "#", "target" => "_blank" },
      ],
    )
  end
end
```

This change is being introduced because we realised that writting just
javascript tests for the migrations is insufficient since javascript
tests do not ensure that the migrated theme settings can actually be
successfully saved into the database. Hence, we are introduce this
helper as a way for theme developers to write "end-to-end" migrations
tests.
2024-05-03 06:29:18 +08:00
Vinoth Kannan
143f06f2c6
FEATURE: Allow watched words to be created as a group (#26632)
At the moment, there is no way to create a group of related watched words together.  If a user needed a set of words to be created together, they'll have to create them individually one at a time.

This change attempts to allow related watched words to be created as a group. The idea here is to have a list of words be tied together via a common `WatchedWordGroup` record.  Given a list of words, a `WatchedWordGroup` record is created and assigned to each `WatchedWord` record. The existing WatchedWord creation behaviour remains largely unchanged.

Co-authored-by: Selase Krakani <skrakani@gmail.com>
Co-authored-by: Martin Brennan <martin@discourse.org>
2024-04-29 15:50:55 +05:30
Martin Brennan
edec941a87
FIX: Better tracking of topic visibility changes (#26709)
This commit introduces a few changes as a result of
customer issues with finding why a topic was relisted.
In one case, if a user edited the OP of a topic that was
unlisted and hidden because of too many flags, the topic
would get relisted by directly changing topic.visible,
instead of going via TopicStatusUpdater.

To improve tracking we:

* Introduce a visibility_reason_id to topic which functions
  in a similar way to hidden_reason_id on post, this column is
  set from the various places we change topic visibility
* Fix Post#unhide! which was directly modifying topic.visible,
  instead we use TopicStatusUpdater which sets visibility_reason_id
  and also makes a small action post
* Show the reason topic visibility changed when hovering the
  unlisted icon in topic status on topic titles
2024-04-29 10:34:46 +10:00
Natalie Tay
00a9369ca2
FIX: Move user reindexing into a job (#26753)
In a large forum with millions of users and millions of user_fields
updating the list of dropdown user field options will result in a
502 now due to the large number of fields.

This commit moves the indexing into a job.
2024-04-25 20:58:34 +08:00
Sam
1c67917367
FIX: disable storing invalid post and topic timing when sent from client (#26683)
This ensures we only ever store correct post and topic timing when the client
notifies.

Previous to this change we would blindly trust the client.

Additionally this has error correction code that will correct the last seen
post number when you visit a topic with incorrect timings.
2024-04-19 18:10:50 +10:00
Krzysztof Kotlarek
c5dd50aa02
FIX: don't purge users who were deactivated by the system (#26656)
In a previous PR, we skipped users deactivated by admins from being automatically purged - https://github.com/discourse/discourse/pull/26478.

However, users deactivated by `discourse-auto-deactivate` plugin are deactivated by the system user. Those users should not be purged as well.

https://github.com/discourse/discourse-auto-deactivate/blob/main/plugin.rb#L62
2024-04-18 09:53:43 +10:00
Martin Brennan
7a083daf27
Revert "FIX: Post uploads setting access_control_post_id unnecessarily (#26627)" (#26643)
This reverts commit cdc8e9de1b.

It's made things worse internally and on meta.
2024-04-16 14:10:25 +10:00
Martin Brennan
cdc8e9de1b
FIX: Post uploads setting access_control_post_id unnecessarily (#26627)
This commit addresses an issue for sites where secure_uploads
is turned on after the site has been operating without it for
some time.

When uploads are linked when they are used inside a post,
we were setting the access_control_post_id unconditionally
if it was NULL to that post ID and secure_uploads was true.

However this causes issues if an upload has been used in a
few different places, especially if a post was previously
used in a PM and marked secure, so we end up with a case of
the upload using a public post for its access control, which
causes URLs to not use the /secure-uploads/ path in the post,
breaking things like image uploads.

We should only set the access_control_post_id if the post is the first time the
upload is referenced so it cannot hijack uploads from other places.
2024-04-16 10:37:57 +10:00
Bianca Nenciu
8ce836c039
FIX: Load categories with user activity and drafts (#26553)
When lazy load categories is enabled, categories should be loaded with
user activity items and drafts because the categories may not be
preloaded on the client side.
2024-04-10 17:35:42 +03:00
Penar Musaraj
a52b1d6b4a
FIX: Let users reset their homepage choice if custom homepage is from… (#26536)
Co-authored-by: Régis Hanol <regis@hanol.fr>
2024-04-09 15:54:44 -04:00
Martin Brennan
0d0dbd391a
DEV: Rename with_secure_uploads? to should_secure_uploads? on Post (#26549)
This method name is a bit confusing; with_secure_uploads implies
it may return a block or something with the uploads of the post,
and has_secure_uploads implies that it's checking whether the post
is linked to any secure uploads.

should_secure_uploads? communicates the true intent of this method --
which is to say whether uploads attached to this post should be
secure or not.
2024-04-09 13:23:11 +10:00
Sérgio Saquetim
175d2451a9
DEV: Add topic_embed_import_create_args plugin modifier (#26527)
* DEV: Add `topic_embed_import_create_args` plugin modifier

This modifier allows a plugin to change the arguments used when creating
 a new topic for an imported article.

For example: let's say you want to prepend "Imported: " to the title of
every imported topic. You could use this modifier like so:

```ruby
# In your plugin's code
plugin.register_modifier(:topic_embed_import_create_args) do |args|
  args[:title] = "Imported: #{args[:title]}"
  args
end
```

In this example, the modifier is prepending "Imported: " to the `title` in the `create_args` hash. This modified title would then be used when the new topic is created.
2024-04-05 12:37:53 -03:00
Bianca Nenciu
19eb0a7055
FIX: Load category info for about page (#26519) 2024-04-05 09:38:54 +03:00
Gerhard Schlager
82c62fe44f
DEV: Correctly pluralize error messages (#26469) 2024-04-04 15:02:09 +02:00
Alan Guo Xiang Tan
a440e15291
DEV: Remove experimental_objects_type_for_theme_settings site setting (#26507)
Why this change?

Objects type for theme settings is no longer considered experimental so
we are dropping the site setting.
2024-04-04 12:01:31 +08:00
Isaac Janzen
db10dd5319
PERF: Improve performance of most_replied_to_users (#26373)
This PR improves the performance of the `most_replied_to_users` method on the `UserSummary` model.

### Old Query
```ruby
    post_query
      .joins(
        "JOIN posts replies ON posts.topic_id = replies.topic_id AND posts.reply_to_post_number = replies.post_number",
      )
      # We are removing replies by @user, but we can simplify this by getting the using the user_id on the posts.
      .where("replies.user_id <> ?", @user.id)
      .group("replies.user_id")
      .order("COUNT(*) DESC")
      .limit(MAX_SUMMARY_RESULTS)
      .pluck("replies.user_id, COUNT(*)")
      .each { |r| replied_users[r[0]] = r[1] }
```
 
### Old Query with corrections

```ruby
post_query
  .joins(
    "JOIN posts replies ON posts.topic_id = replies.topic_id AND replies.reply_to_post_number = posts.post_number",
  )
  # Remove replies by @user but instead look on loaded posts (we do this so we don't count self replies)
  .where("replies.user_id <> posts.user_id")
  .group("replies.user_id")
  .order("COUNT(*) DESC")
  .limit(MAX_SUMMARY_RESULTS)
  .pluck("replies.user_id, COUNT(*)")
  .each { |r| replied_users[r[0]] = r[1] }
```

### New Query
```ruby
    post_query
      .joins(
        "JOIN posts replies ON posts.topic_id = replies.topic_id AND posts.reply_to_post_number = replies.post_number",
      )
      # Only include regular posts in our joins, this makes sure we don't have the bloat of loading private messages
      .joins(
        "JOIN topics ON replies.topic_id = topics.id AND topics.archetype <> 'private_message'",
      )
      # Only include visible post types, so exclude posts like whispers, etc
      .joins(
        "AND replies.post_type IN (#{Topic.visible_post_types(@user, include_moderator_actions: false).join(",")})",
      )
      .where("replies.user_id <> posts.user_id")
      .group("replies.user_id")
      .order("COUNT(*) DESC")
      .limit(MAX_SUMMARY_RESULTS)
      .pluck("replies.user_id, COUNT(*)")
      .each { |r| replied_users[r[0]] = r[1] }
```

# Conclusion

`most_replied_to_users` was untested, so I introduced a test for the logic, and have confirmed that it passes on both the new query **AND** the old query. 

Thank you @danielwaterworth for the debugging assistance.
2024-04-03 14:20:54 -06:00
Krzysztof Kotlarek
ba04fc6a01
FEATURE: ignore manually deactivated users when purging (#26478)
When a user is manually deactivated, they should not be deleted by our background job that purges inactive users.

In addition, site settings keywords should accept an array of keywords.
2024-04-03 14:06:31 +11:00
Ted Johansson
0c875cb4d5
DEV: Make problem check registration more explicit (#26413)
Previously the problem check registry simply looked at the subclasses of ProblemCheck. This was causing some confusion in environments where eager loading is not enabled, as the registry would appear empty as a result of the classes never being referenced (and thus never loaded.)

This PR changes the approach to a more explicit one. I followed other implementations (bookmarkable and hashtag autocomplete.) As a bonus, this now has a neat plugin entry point as well.
2024-03-28 14:00:47 +08:00
Juan David Martínez Cubillos
fd83107674
FIX: Export false on confirm user fields when using user invites (#26332) 2024-03-27 09:12:14 -04:00
Angus McLeod
7dc552c9cc
DEV: Add import_embed_unlisted site setting (#26222) 2024-03-27 08:57:43 -04:00
Sam
e3a0faefc5
FEATURE: allow re-scoping chat user search via a plugin (#26361)
This enables the following in Discourse AI

```
 plugin.register_modifier(:chat_allowed_bot_user_ids) do |user_ids, guardian|
  if guardian.user
    mentionables = AiPersona.mentionables(user: guardian.user)
    allowed_bot_ids = mentionables.map { |mentionable| mentionable[:user_id] }
    user_ids.concat(allowed_bot_ids)
  end
  user_ids
end
```

some bots that are id < 0 need to be discoverable in search otherwise people can not talk to them.

---------

Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
2024-03-27 08:55:53 +11:00
Jarek Radosz
4c860995e0
DEV: Remove unnecessary rails_helper requiring (#26364) 2024-03-26 11:32:01 +01:00
Ted Johansson
5ee23fc394
DEV: Make all admins TL4 in tests (#25435)
Make admins TL4 by default in tests, foregoing the need to call refresh_auto_groups on them.
2024-03-26 11:41:12 +08:00
Bianca Nenciu
4cdf5f2cea
FIX: Load subcategories through CategoryList (#26297)
When "lazy load categories" is enabled and parent_category_id was set,
the query fetching categories contained a contradiction filtering both
by parent_category_id and parent_category_id = NULL.
2024-03-21 21:39:14 +02:00
Loïc Guitaut
d2a730b8b5 DEV: Expose extra data from themes
This patch exposes a normalized repository URL and how many users are
using a given theme.
2024-03-21 15:06:36 +01:00
Ted Johansson
4ca41e0af2
DEV: Promote block problem checks to ProblemCheck (#26193)
In #26122 we promoted all problem checks defined as class methods on AdminDashboardData to their own first-class ProblemCheck instances.

This PR continues that by promoting problem checks that are implemented as blocks as well. This includes updating a couple plugins that have problem checks.
2024-03-20 08:52:25 +08:00
Régis Hanol
4e02bb5dd9
PERF: avoid publishing user actions to the user who did the action (#26225)
We never use that information and this also fixes an issue with the BCC plugin which ends up triggering a rate-limit because we were publishing a "NEW_PRIVATE_MESSAGE" to the user sending the BCC for every recipients 💥

Internal - t/118283
2024-03-18 18:05:46 +01:00
Martin Brennan
78bafb331a
FEATURE: Allow site settings to be edited throughout admin UI (#26154)
This commit makes it so the site settings filter controls and
the list of settings input editors themselves can be used elsewhere
in the admin UI outside of /admin/site_settings

This allows us to provide more targeted groups of settings in different
UI areas where it makes sense to provide them, such as on plugin pages.
You could open a single page for a plugin where you can see information
about that plugin, change settings, and configure it with custom UIs
in the one place.

In future we will do this in "config areas" for other parts of the
admin UI.
2024-03-18 08:50:39 +10:00
Penar Musaraj
8cf2f909f5
DEV: Dedicated route for current user notification counts (#26106)
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2024-03-15 12:08:37 -04:00
Daniel Waterworth
819361ba28
SECURITY: Don't disclose the existence of secret subcategories 2024-03-15 14:23:55 +08:00
Ted Johansson
ea5c3a3bdc
DEV: Move non scheduled problem checks to classes (#26122)
In AdminDashboardData we have a bunch of problem checks implemented as methods on that class. This PR absolves it of the responsibility by promoting each of those checks to a first class ProblemCheck. This way each of them can have their own priority and arbitrary functionality can be isolated in its own class.

Think "extract class" refactoring over and over. Since they were all moved we can also get rid of the @@problem_syms class variable which was basically the old version of the registry now replaced by ProblemCheck.realtime.

In addition AdminDashboardData::Problem value object has been entirely replaced with the new ProblemCheck::Problem (with compatible API).

Lastly, I added some RSpec matchers to simplify testing of problem checks and provide helpful error messages when assertions fail.
2024-03-14 10:55:01 +08:00
Arpit Jalan
1bd803d360
FIX: store registration ip address when creating user via SSO (#26121) 2024-03-11 15:19:37 +05:30
Ted Johansson
2211ffa851
DEV: Move problem checks to app directory (#26120)
There are a couple of reasons for this.

The first one is practical, and related to eager loading. Since /lib is not eager loaded, when the application boots, ProblemCheck["identifier"] will be nil because the child classes aren't loaded.

The second one is more conceptual. There turns out to be a lot of inter-dependencies between the part of the problem check system that live in /app and the parts that live in /lib, which probably suggests it should all go in /app.
2024-03-11 13:36:22 +08:00
Alan Guo Xiang Tan
898b71da88
PERF: Add indexes to speed up notifications queries by user menu (#26048)
Why this change?

There are two problematic queries in question here when loading
notifications in various tabs in the user menu:

```
SELECT "notifications".*
FROM "notifications"
LEFT JOIN topics ON notifications.topic_id = topics.id
WHERE "notifications"."user_id" = 1338 AND (topics.id IS NULL OR topics.deleted_at IS NULL)
ORDER BY notifications.high_priority AND NOT notifications.read DESC,
  NOT notifications.read AND notifications.notification_type NOT IN (5,19,25) DESC,
  notifications.created_at DESC
LIMIT 30;
```

and

```
EXPLAIN ANALYZE SELECT "notifications".*
FROM "notifications"
LEFT JOIN topics ON notifications.topic_id = topics.id
WHERE "notifications"."user_id" = 1338
AND (topics.id IS NULL OR topics.deleted_at IS NULL)
AND "notifications"."notification_type" IN (5, 19, 25)
ORDER BY notifications.high_priority AND NOT notifications.read DESC, NOT notifications.read DESC, notifications.created_at DESC LIMIT 30;
```

For a particular user, the queries takes about 40ms and 26ms
respectively on one of our production instance where the user has 10K notifications while the site has 600K notifications in total.

What does this change do?

1. Adds the `index_notifications_user_menu_ordering` index to the `notifications` table which is
   indexed on `(user_id, (high_priority AND NOT read) DESC, (NOT read)
DESC, created_at DESC)`.

1. Adds a second index `index_notifications_user_menu_ordering_deprioritized_likes` to the `notifications`
   table which is indexed on `(user_id, (high_priority AND NOT read) DESC, (NOT read AND notification_type NOT IN (5,19,25)) DESC, created_at DESC)`. Note that we have to hardcode the like typed notifications type here as it is being used in an ordering clause.

With the two indexes above, both queries complete in roughly 0.2ms. While I acknowledge that there will be some overhead in insert,update or delete operations. I believe this trade-off is worth it since viewing notifications in the user menu is something that is at the core of using a Discourse forum so we should optimise this experience as much as possible.
2024-03-06 16:52:19 +08:00
Krzysztof Kotlarek
28af4031ae
FIX: active webhook types exclude inactive plugins (#26022)
Bug introduced when webhooks were granularized in this PR - c468110929

Only active webhooks should be available when webhooks is configured.

https://meta.discourse.org/t/i18n-keys-showing-on-webhooks-edit-page/297701
2024-03-05 12:47:04 +11:00
Loïc Guitaut
f7d7092a7a DEV: Update rubocop-discourse to latest version
The lastest version of rubocop-discourse enables rules regarding
plugins.
2024-03-04 15:08:35 +01:00
Krzysztof Kotlarek
41f78b31a9
FIX: down downgrade trust level if all requirements are met. (#25953)
Currently, the trust level method  is calculating trust level based on maximum value from:
- locked trust level
- group automatic trust level
- previously granted trust level by admin

https://github.com/discourse/discourse/blob/main/lib/trust_level.rb#L33

Let's say the user belongs to groups with automatic trust level 1 and in the meantime meets all criteria to get trust level 2.

Each time, a user is removed from a group with automatic trust_level 1, they will be downgraded to trust_level 1 and promoted to trust_level 2

120a2f70a9/lib/promotion.rb (L142)

This will cause duplicated promotion messages.

Therefore, we have to check if the user meets the criteria, before downgrading.
2024-03-04 09:30:30 +11:00
Martin Brennan
6bcbe56116
DEV: Use freeze_time_safe in more places (#25949)
Followup to 120a2f70a9,
uses new method to avoid time-based spec flakiness
2024-03-01 10:07:35 +10:00
Jarek Radosz
5c54fbfdb1
DEV: Fix random typos (#25957)
February 2024 edition
2024-02-29 12:24:37 +01:00
Alan Guo Xiang Tan
f562da3150
PERF: Reduce ActiveRecord allocations in CategoryList#find_relevant_topics (#25950)
Why this change?

Prior to this change, the `CategoryList#find_relevant_topics` method was
loading and allocating all `CategoryFeaturedTopic` records in the
database to eventually only just use its `category_id` and `topic_id`
column. On a site with many `CategoryFeaturedTopic` records, the loading
of the ActiveRecord objects is a source of bottleneck.

The other problem with the `CategoryList#find_relevant_topics` method is
that it is unconditionally loading all records from the database even if
the user does not have access to the category. This again is wasteful.

What does this change do?

This commit makes it such that `CategoryList#find_relevant_topics` is
called only after `CategoryList#find_categories` in the `CategoryList#initialize`
method so that we can filter featured topics against categories that the
user has access to.

The second change is that Instead of loading `CategoryFeaturedTopic` records, we make an
inner join agains the `topics` table instead and skip any allocation of
`CatgoryFeaturedTopic` ActiveRecord objects.
2024-02-29 12:19:04 +08:00
Martin Brennan
120a2f70a9
DEV: Fix hot topic flaky spec (#25948)
It's February 29th, you know what that means...date-based flaky specs! If today is
February 29th 2024:

```
freeze_time(1.year.ago) -> Tue, 28 Feb 2023 01:38:42.732875000 UTC +00:00
```

Then

```
freeze_time(1.year.from_now) -> Wed, 28 Feb 2024 01:38:42.732875000 UTC +00:00
```

So then our "now" for the insert query ends up being "yesterday"

```
WHERE topic_hot_scores.topic_id IS NULL
  AND topics.deleted_at IS NULL
  AND topics.archetype <> :private_message
  AND topics.created_at <= :now
```
2024-02-29 11:54:36 +10:00
Natalie Tay
a8a39d86b4
UX: Improve invite error message when a user uses an email that has already redeemed (#25695)
Improve invite error message when a user uses an email that has already redeemed
2024-02-27 18:24:20 +08:00
Ted Johansson
1bcb521fbf
DEV: Add DB backed problem checks to support perform_every config (#25834)
As part of problem checks refactoring, we're moving some data to be DB backed. In this PR it's the tracking of problem check execution. When was it last run, when was the last problem, when should it run next, how many consecutive checks had problems, etc.

This allows us to implement the perform_every feature in scheduled problem checks for checks that don't need to be run every 10 minutes.
2024-02-27 11:17:39 +08:00
Alan Guo Xiang Tan
7bcfe60a76
DEV: Validate default value for type: objects theme settings (#25833)
Why this change?

This change adds validation for the default value for `type: objects` theme
settings when a setting theme field is uploaded. This helps the theme
author to ensure that the objects which they specifc in the default
value adhere to the schema which they have declared.

When an error is encountered in one of the objects, the error
message will look something like:

`"The property at JSON Pointer '/0/title' must be at least 5 characters
long."`

We use a JSON Pointer to reference the property in the object which is
something most json-schema validator uses as well.

What does this change do?

1. This commit once again changes the shape of hash returned by
   `ThemeSettingsObjectValidator.validate`. Instead of using the
   property name as the key previously, we have decided to avoid
   multiple levels of nesting and instead use a JSON Pointer as the key
   which helps to simplify the implementation.

2 Introduces `ThemeSettingsObjectValidator.validate_objects` which
  returns an array of validation error messages for all the objects
  passed to the method.
2024-02-27 09:16:37 +08:00
Ted Johansson
a72dc2f420
DEV: Introduce a problem checks API (#25783)
Previously, problem checks were all added as either class methods or blocks in AdminDashboardData. Another set of class methods were used to add and run problem checks.

As of this PR, problem checks are promoted to first-class citizens. Each problem check receives their own class. This class of course contains the implementation for running the check, but also configuration items like retry strategies (for scheduled checks.)

In addition, the parent class ProblemCheck also serves as a registry for checks. For example we can get a list of all existing check classes through ProblemCheck.checks, or just the ones running on a schedule through ProblemCheck.scheduled.

After this refactor, the task of adding a new check is significantly simplified. You add a class that inherits ProblemCheck, you implement it, add a test, and you're good to go.
2024-02-23 11:20:32 +08:00
Alan Guo Xiang Tan
3e331b1725
DEV: Set a bytesize limit for ThemeSetting#json_value (#25761)
Why this change?

Firstly, note that this is not a security commit because this feature is
still in development and should not be used anywhere.

The reason we want to set a limit here is to greatly reduce the
possibility of a DoS attack in the future via `ThemeSetting` where
someone would set an arbituary large json string in
`ThemeSetting#json_value` and causing the server to run out of resources
trying to serialize/deserialize the value.

What does this change do?

Adds an ActiveRecord validation to ensure that the bytesize of the json
string being stored is smaller than or equal to 0.5mb. We believe 0.5mb
is a decent limit for now but we can review the limit in the future if
we believe it is too small.
2024-02-21 08:09:37 +08:00
Alan Guo Xiang Tan
6ca2396b12
DEV: Centralise logic for validating a theme setting value (#25764)
Why this change?

The logic for validating a theme setting's value and default value was
not consistent as each part of the code would implement its own logic.
This is not ideal as the default value may be validated differently than
when we are setting a new value. Therefore, this commit seeks to
refactor all the validation logic for a theme setting's value into a
single service class.

What does this change do?

Introduce the `ThemeSettingsValidator` service class which holds all the
necessary helper methods required to validate a theme setting's value
2024-02-21 08:08:26 +08:00
Daniel Waterworth
13083d03ae
DEV: Async category search for sidebar modal (#25686) 2024-02-20 11:24:30 -06:00
Bianca Nenciu
a24d110258
FIX: Preload parent categories for sidebar (#25726)
When "lazy load categories" is enabled, only the categories present in
the sidebar are preloaded. This is insufficient because the parent
categories are necessary too for the sidebar to be rendered properly.
2024-02-16 16:39:18 +02:00
David Taylor
b1f74ab59e
FEATURE: Add experimental option for strict-dynamic CSP (#25664)
The strict-dynamic CSP directive is supported in all our target browsers, and makes for a much simpler configuration. Instead of allowlisting paths, we use a per-request nonce to authorize `<script>` tags, and then those scripts are allowed to load additional scripts (or add additional inline scripts) without restriction.

This becomes especially useful when admins want to add external scripts like Google Tag Manager, or advertising scripts, which then go on to load a ton of other scripts.

All script tags introduced via themes will automatically have the nonce attribute applied, so it should be zero-effort for theme developers. Plugins *may* need some changes if they are inserting their own script tags.

This commit introduces a strict-dynamic-based CSP behind an experimental `content_security_policy_strict_dynamic` site setting.
2024-02-16 11:16:54 +00:00
Martin Brennan
ae24e04a5e
DEV: Add a plugin modifier for user_action_stream_builder (#25691)
Reactions needs this to be able to filter out likes received
actions, where there is also an associated reaction, since
now most reactions also count as a like.
2024-02-16 10:24:39 +10:00
David Battersby
d7dd871d9f
FIX: quoted private topic url respects subfolder install (#25643)
Fixes an issue where private topics that are quoted have an incorrectly formatted url when using a subfolder install.

This update returns a relative url that includes the base_path rather than a combination of base_url + base_path.
2024-02-13 13:20:24 +08:00
Martin Brennan
cf4d92f686
FIX: Change max_image_megapixels logic (#25625)
This commit changes `max_image_megapixels` to be used
as is without multiplying by 2 to give extra leway.
We found in reality this was just causing confusion
for admins, especially with the already permissive
40MP default.
2024-02-12 09:56:43 +10:00
Sam
c8410537c1
FIX: hot not adding recently bumped topics (#25619)
When we insert into the hot set we add things with a score of 0
This means that if hot has more than batch size items in it with a score, then the 0s don't get an initial score

This corrects the situation by always ensuring we re-score:

1. batch size high scoring topics
2. (new) batch size recently bumped topics

* Update spec/models/topic_hot_scores_spec.rb

Co-authored-by: Isaac Janzen <50783505+janzenisaac@users.noreply.github.com>

---------

Co-authored-by: Isaac Janzen <50783505+janzenisaac@users.noreply.github.com>
2024-02-09 07:45:47 +11:00
Penar Musaraj
6bd26e81c1
FIX: Respect date range in top traffic sources report (#25599)
See https://meta.discourse.org/t/reports-top-traffic-sources-topics-stat-issue/179850
2024-02-08 11:17:59 -05:00
Alan Guo Xiang Tan
9f884cdaab
DEV: Introduce experimental type: objects theme setting (#25538)
Why this change?

This commit introduces an experimental `type: objects` theme setting
which will allow theme developers to store a collection of objects as
JSON in the database. Currently, the feature is still in development and
this commit is simply setting up the ground work for us to introduce the
feature in smaller pieces.

What does this change do?

1. Adds a `json_value` column as `jsonb` data type to the `theme_settings` table.
2. Adds a `experimental_objects_type_for_theme_settings` site setting to
   determine whether `ThemeSetting` records of with the `objects` data
   type can be created.
3. Updates `ThemeSettingsManager` to support read/write access from the
   `ThemeSettings#json_value` column.
2024-02-08 10:20:59 +08:00
Ted Johansson
95a2d285d3
FEATURE: Add new 'illegal' flag reason (#25498)
To comply with Digital Services Act we need a way for users to flag a post as potentially illegal. This PR adds that functionality.
2024-02-07 10:12:22 +08:00
Gerhard Schlager
dd5ca6cc4c
FEATURE: Permalinks for users (#25552) 2024-02-05 17:31:31 +01:00
Alan Guo Xiang Tan
d460229ed8
FIX: Update themes javascript cache after running themes migrations (#25562)
Why this change?

This is caused by a regression in
59839e428f, where we stopped saving the
`Theme` object because it was unnecessary. However, it resulted in the
`after_save` callback not being called and hence
`Theme#update_javascript_cache!` not being called. As a result, some
sites were reporting that after runing a theme migration, the defaults
for the theme settings were used instead of the settings overrides
stored in the database.

What does this change do?

Add a call to `Theme#update_javascript_cache!` after running theme
migrations.
2024-02-05 14:35:11 +08:00
Sam
140b2118af
FEATURE: improvements to hot feature (#25533)
1. Don't show visited line for hot filter, it is in random order
2. Don't count likes on non regular posts (eg: whispers / small actions)
3. Don't count participants in non regular posts
2024-02-02 10:53:27 +11:00
Sam
690ff4499c
DEV: adjustments to hot algorithm (#25517)
1. Serial likers will just like a bunch of posts on the same topic, this will
heavily inflate hot score. To avoid artificial "heat" generated by one user only count
the first like on the topic within the recent_cutoff range per topic

2. When looking at recent topics prefer "unique likers", defer to total likes on
older topics cause we do not have an easy count for unique likers

3. Stop taking 1 off like_count, it is not needed - platforms like reddit
allow you to like own post so they need to remove it.
2024-02-01 17:11:40 +11:00
Alan Guo Xiang Tan
44f8418093
DEV: Refactor Theme#settings to return a hash instead of array (#25516)
Why this change?

Returning an array makes it hard to immediately retrieve a setting by
name and makes the retrieval an O(N) operation. By returning an array,
we make it easier for us to lookup a setting by name and retrieval is
O(1) as well.
2024-02-01 10:26:56 +08:00
Sam
27301ae5c7
FEATURE: support silent internal links (#25472)
Internal links always notify and add internal connections in topics.

This adds a special feature that lets you append `?silent=true` to a link
to have it excluded from:

1. Notifications - users will not be notified for these links
2. Post links below posts in the UI

This is specifically useful for large reports where adding all these connections
just results in noise.
2024-01-30 17:03:58 +11:00
marstall
5a00d1964f
DEV: add site setting to disable watched word checking in user fields (#25411)
adding a hidden sitesetting, `disable_watched_word_checking_in_user_fields` - false by default. if set to true, you can use any word at all in user profile fields.

meta: https://meta.discourse.org/t/watched-words-scope/282699/20
2024-01-29 12:44:32 -05:00
Ted Johansson
f0a46f8b6f
DEV: Automatically update groups for test users with explicit TL (#25415)
For performance reasons we don't automatically add fabricated users to trust level auto-groups. However, when explicitly passing a trust level to the fabricator, in 99% of cases it means that trust level is relevant for the test, and we need the groups.

This change makes it so that when a trust level is explicitly passed to the fabricator, the auto-groups are refreshed. There's no longer a need to also pass refresh_auto_groups: true, which means clearer tests, fewer mistakes, and less confusion.
2024-01-29 17:52:02 +08:00
Ted Johansson
7e5d2a95ee
DEV: Convert min_trust_level_to_tag_topics to groups (#25273)
We're changing the implementation of trust levels to use groups. Part of this is to have site settings that reference trust levels use groups instead. It converts the min_trust_level_to_tag_topics site setting to tag_topic_allowed_groups.
2024-01-26 13:25:03 +08:00
Ted Johansson
57ea56ee05
DEV: Remove full group refreshes from tests (#25414)
We have all these calls to Group.refresh_automatic_groups! littered throughout the tests. Including tests that are seemingly unrelated to groups. This is because automatic group memberships aren't fabricated when making a vanilla user. There are two places where you'd want to use this:

You have fabricated a user that needs a certain trust level (which is now based on group membership.)
You need the system user to have a certain trust level.
In the first case, we can pass refresh_auto_groups: true to the fabricator instead. This is a more lightweight operation that only considers a single user, instead of all users in all groups.

The second case is no longer a thing after #25400.
2024-01-25 14:28:26 +08:00
Ted Johansson
6ad34a0152
DEV: Exclude system users when calculating group user count (#25400)
We want to exclude the system user from group user counts, since intuitively admins wouldn't include them.

Originally this was accomplished by booting said system user from the groups, but this is causing problems, because the system user needs TL group membership to perform certain tasks.

After this PR, system user is still in the TL groups, but excluded when refreshing the user count.
2024-01-25 08:13:58 +08:00
Martin Brennan
0e50f88212
DEV: Move min_trust_to_post_embedded_media to group setting (#25238)
c.f. https://meta.discourse.org/t/we-are-changing-giving-access-to-features/283408
2024-01-25 09:50:59 +10:00
Ted Johansson
32e2a1fd4a
DEV: Add delegated Group#human_users scope (#25398)
Some preparatory refactoring as we're working on TL groups for the system user. On User we have a scope #human_users to exclude the system user, DiscoBot, etc. This PR adds the same scope (delegated to User) on Group.
2024-01-24 13:33:05 +08:00
Martin Brennan
b3904eab45
FIX: User group check should return true for system user for auto groups (#25357)
This is a temporary fix to address an issue where the
system user is losing its automatic groups when the server
is running. If any auto groups are provided, and the user is
a system user, then we return true. The system user is admin,
moderator, and TL4, so they usually have all auto groups.

We can remove this when we get to the bottom of why the auto
groups are being deleted.
2024-01-22 14:40:29 +10:00
Ted Johansson
fb087b7ff6
DEV: Convert min_trust_to_post_links to groups (#25298)
We're changing the implementation of trust levels to use groups. Part of this is to have site settings that reference trust levels use groups instead. It converts the min_trust_to_post_links  site setting to post_links_allowed_groups.

This isn't used by any of our plugins or themes, so very little fallout.
2024-01-18 14:08:40 +08:00
Bianca Nenciu
abad38c2e7
DEV: Make lazy_load_categories setting use groups (#25282)
This allows certain users to test the new feature and avoid disruptions
in other's workflows.
2024-01-17 20:26:51 +02:00
Bianca Nenciu
4cfc0e231a
DEV: Change categories#index loading strategy (#25232)
The old strategy used to load 25 categories at a time, including the
subcategories. The new strategy loads 20 parent categories and at most
5 subcategories for each parent category, for a maximum of 120
categories in total.
2024-01-17 17:18:01 +02:00
Sam
df8bb947b2
FEATURE: improvements to hot algorithm (#25295)
- Decrease gravity, we come in too hot prioritizing too many new topics
- Remove all muted topics / categories and tags from the hot list
- Punish topics with zero likes in algorithm
2024-01-17 16:12:03 +11:00
Sam
ebd3971533
FEATURE: experiment with hot sort order (#25274)
This introduces a new experimental hot sort ordering. 

It attempts to float top conversations by first prioritizing a  topics with lots of recent activity (likes and users responding) 

The schedule that updates hot topics is disabled unless the hidden site setting: `experimental_hot_topics` is enabled. 

You can control "decay" with `hot_topic_gravity` and `recency` with `hot_topics_recent_days` 

Data is stored in the new `topic_hot_scores` table and you can check it out on the `/hot` route once 
enabled. 
---------

Co-authored-by: Penar Musaraj <pmusaraj@gmail.com>
2024-01-17 13:01:04 +11:00
Alan Guo Xiang Tan
22614ca85b
DEV: Compile theme migrations javascript files when running theme qunit (#25219)
Why this change?

Currently, is it hard to iteratively write a theme settings migrations
because our theme migrations system does not rollback. Therefore, we
want to allow theme developers to be able to write QUnit tests for their
theme migrations files enabling them to iteratively write their theme
migrations.

What does this change do?

1. Update `Theme#baked_js_tests_with_digest` to include all `ThemeField`
records of `ThemeField#target` equal to `migrations`. Note that we do
not include the `settings` and `themePrefix` variables for migration files.

2. Don't minify JavaScript test files becasue it makes debugging in
   development hard.
2024-01-16 09:50:44 +08:00
Penar Musaraj
f2cf5434f3
Revert "DEV: Convert min_trust_level_to_tag_topics to groups (#25258)" (#25262)
This reverts commit c7e3d27624 due to
test failures. This is temporary.
2024-01-15 11:33:47 -05:00
Ted Johansson
c7e3d27624
DEV: Convert min_trust_level_to_tag_topics to groups (#25258)
We're changing the implementation of trust levels to use groups. Part of this is to have site settings that reference trust levels use groups instead. It converts the min_trust_level_to_tag_topics site setting to tag_topic_allowed_groups.
2024-01-15 20:59:08 +08:00
Renato Atilio
f5f3742166
FIX: respect creation date when paginating group activity posts (#24993)
* FIX: respect creation date when paginating group activity posts

There are scenarios where the chronological order of posts doesn't match the order of their IDs. For instance, when moving the first post from one topic or PM to another, a new post (with a higher ID) will be created, but it will retain the original creation time.

This PR changes the group activity page and endpoint to paginate posts using created_at instead of relying on ID ordering.
2024-01-11 13:37:27 -03:00
Alan Guo Xiang Tan
59839e428f
DEV: Add skip_migrations param when importing remote theme (#25218)
Why this change?

Importing theme with the `bundle` params is used mainly by
`discourse_theme` CLI in the development environment. However, we do not
want migrations to automatically run in the development environment
and instead want the developer to be intentional about running theme
migrations. As such, this commit adds support for a
`skip_migrations` param when importing a theme with the `bundle` params.

This commit also adds a `migrated` attribute for migrations theme fields
to indicate whether a migrations theme field has been migrated or not.
2024-01-11 14:04:02 +08:00
Blake Erickson
6ebe61ecec
FIX: Logs api scope not working (#25215) 2024-01-10 19:30:10 -07:00
Rafael dos Santos Silva
13735f35fb
FEATURE: Cache embed contents in the database (#25133)
* FEATURE: Cache embed contents in the database

This will be useful for features that rely on the semantic content of topics, like the many AI features



Co-authored-by: Roman Rizzi <rizziromanalejandro@gmail.com>
2024-01-05 10:09:31 -03:00
Ted Johansson
a5f0935307
DEV: Convert min_trust_level_to_create_tag to groups (#24899)
We're changing the implementation of trust levels to use groups. Part of this is to have site settings that reference trust levels use groups instead. It converts the min_trust_level_to_create_tag  site setting to create_tag_allowed_groups.

This PR maintains backwards compatibility until we can update plugins and themes using this.
2024-01-05 10:19:43 +08:00
Jarek Radosz
b9f6e6d637
DEV: Skip flaky specs (#25111)
Dynamically adding and removing deprecated or TL-related site settings in specs is leaky
2024-01-03 12:32:26 +01:00