In #20135 we prevented invalid inputs from being accepted in category setting form fields on the front-end. We didn't do anything on the back-end at that time, because we were still discussing which path we wanted to take. Eventually we decided we want to move this to a new CategorySetting model.
This PR moves the require_topic_approval and require_reply_approval from custom fields to the new CategorySetting model.
This PR is nearly identical to #20580, which migrated num_auto_bump_daily, but since these are slightly more sensitive, they are moved after the previous one is verified.
Why this change?
Currently, we do not have an easy way to test themes and theme components
using Rails system tests. While we support QUnit acceptance tests for
themes and theme components, QUnit acceptance tests stubs out the server
and setting up the fixtures for server responses is difficult and can lead to a
frustrating experience. System tests on the other hand allow authors to
set up the test fixtures using our fabricator system which is much
easier to use.
What does this change do?
In order for us to allow authors to run system tests with their themes
installed, we are adding a `upload_theme` helper that is made available
when writing system tests. The `upload_theme` helper requires a single
`directory` parameter where `directory` is the directory of the theme
locally and returns a `Theme` record.
This adds a new secure_uploads_pm_only site setting. When secure_uploads
is true with this setting, only uploads created in PMs will be marked
secure; no uploads in secure categories will be marked as secure, and
the login_required site setting has no bearing on upload security
either.
This is meant to be a stopgap solution to prevent secure uploads
in a single place (private messages) for sensitive admin data exports.
Ideally we would want a more comprehensive way of saying that certain
upload types get secured which is a hybrid/mixed mode secure uploads,
but for now this will do the trick.
They're both constant per-instance values, there is no need to store them
in the session. This also makes the code a bit more readable by moving
the `session_challenge_key` method up to the `DiscourseWebauthn` module.
Follow-up to #23199 in which we moved the "delete user" options under the relevant action menu for flagged post. This change does the same, but to queued posts.
As per discussion, we want to move the options to delete the user under the "Yes" menu of the review queue, since these options are often the most recommended, but also frequently missed because they are tucked away under their own menu.
In addition to the move, I removed the title case of the options so that it matches the other options in the "Yes" menu.
Reverts e2705df and re-lands #23187 and #23219.
The issue was incorrect order of execution of Rails' `assets:precompile` task in our own precompilation stack.
Co-authored-by: David Taylor <david@taylorhq.com>
Manipulating theme module paths means that the paths you author are not the ones used at runtime. This can lead to some very unexpected behavior and potential module name clashes. It also meant that the refactor in 16c6ab8661 was unable to correctly match up theme connector js/templates.
While this could technically be a breaking change, I think it is reasonably safe because:
1. Themes are already forced to use relative paths when referencing their own modules (since they're namespaced based on the site-specific id). The only time this might be problematic is when theme tests reference modules in the theme's main `javascripts` directory
2. For things like components/services/controllers/etc. our custom Ember resolver works backwards from the end of the path, so adding `discourse/` in the middle will not affect resolution.
This is a bug that happens only when the current date is less than 90 days from a date on which the time zone transitions into or out of Daylight Savings Time.
In these conditions, bulk invites show the time of day of their expiration as being 1 hour later than the current time.
Whereas it should match the time of day the invite was generated.
This is because the server has not been using the user's timezone in calculating the expiration time of day. This PR fixes issue by considering the user's timezone when doing the date math.
https://meta.discourse.org/t/bulk-invite-logic-to-generate-expire-date-bug/274689
`ReviewableQueuedPost` got refactored a while back to use the more
appropriate `target_created_by` for the user of the post being queued
instead of `created_by`. The change was not extended to the `DELETE
/review/:id` endpoint leading to error responses for a user attempting
to deleting their own queued post.
This fix extends the `Reviewable` lookup implementation in
`ReviewablesController#destroy` and Guardian implementation to account
for this change.
Previously we were discovering plugin outlets by checking first for dedicated template files, and then looking for classes to match them. This doesn't work for components which are entirely defined in JS (e.g. those authored with gjs, or those which are re-exports of a colocated component).
This commit refactors our detection logic to look for both class and template modules in a single pass. It also refactors things so that the modules themselves are required lazily when needd, rather than all being loaded during app boot.
When hiding a post (essentially updating hidden, hidden_at, and hidden_reason_id) our callbacks are running the whole battery of post validations. This can cause the hiding to fail in a number of edge cases. The issue is similar to the one fixed in #11680, but applies to all post validations, none of which should apply when hiding a post.
After some code reading and discussion, none of the validations in PostValidator seem to be relevant when hiding posts, so instead of just skipping unique check, we skip all post validator checks.
* scrub non-a html tags from tag descriptions on create, strips all tags from tag description when displayed in tag hover
* test for tag description links
* UX: basic render-tag test
* UX: fix linting
* UX: fix linting
* fix broken tests
* Update spec/models/tag_spec.rb
Co-authored-by: Penar Musaraj <pmusaraj@gmail.com>
* UX: use has_sanitizable_fields instead of has_scrubbable_fields to ensafen tag.description
---------
Co-authored-by: Penar Musaraj <pmusaraj@gmail.com>
Currently when we decide we're going to drop a column in the future we just mark it with a TODO comment and add it to ignored_columns. This makes it instantly unavailable, and we mostly forget about the TODO in the end. 😬
This change adds a HasDeprecatedColumns concern which offers a little bit more flexibility. We can still simulate the old behaviour by setting drop_from to the current version, but we can also set it to a future version, causing it to raise a deprecation warning until then if used.
This commit removes any logic in the app and in specs around
enable_experimental_hashtag_autocomplete and deletes some
old category hashtag code that is no longer necessary.
It also adds a `slug_ref` category instance method, which
will generate a reference like `parent:child` for a category,
with an optional depth, which hashtags use. Also refactors
PostRevisor which was using CategoryHashtagDataSource directly
which is a no-no.
Deletes the old hashtag markdown rule as well.
This brings the theme development experience (via the discourse_theme cli) closer to the experience of making javascript changes in Discourse core/plugins via Ember CLI. Whenever a change is made to a non-css theme field, all clients will be instructed to immediately refresh via message-bus.
In #20135 we prevented invalid inputs from being accepted in category setting form fields on the front-end. We didn't do anything on the back-end at that time, because we were still discussing which path we wanted to take. Eventually we decided we want to move this to a new CategorySetting model.
This PR moves the num_auto_bump_daily from custom fields to the new CategorySetting model.
In addition it sets the default value to 0, which exhibits the same behaviour as when the value is NULL.
* FIX Add 'Ignored' flags to Moderator Activity report
The Moderator Activity query didn’t include the number of deferred flags in the Flags Reviewed totals. As this number is designed to reflect how many flags a moderator has seen, reviewed, and made a judgement on, the Ignored ones should also be included.
* Apply suggestions from code review
---------
Co-authored-by: Jarek Radosz <jradosz@gmail.com>
FEATURE: Only approved flags for post counters
* Why was this change necessary?
The counters for flagged posts in the user's profile and user index from
the admin view include flags that were rejected, ignored or pending
review. This introduces unnecessary noise. Also the flagged posts
counter in the user's profile includes custom flags which add further
noise to this signal.
* How does it address the problem?
* Modifying User#flags_received_count to return posts with only approved
standard flags
* Refactoring User#number_of_flagged_posts to alias to
User#flags_received_count
* Updating the flagged post staff counter hyperlink to navigate to a
filtered view of that user's approved flagged posts to maintain
consistency with the counter
* Adding system tests for the profile page to cover the flagged posts
staff counter
In the query generated by `TopicTrackingState.report`, there are two
subqueies being executed. The first subquery fetches all the topics
that are new for a given user while the second subquery fetches all the topics with
unread posts for a given user. For the second subquery, there is a
filter `topics.updated_at >= user_stats.first_unread_at` which is used
as a performance optimisation to reduce the number of rows that PG has
to scan through the `topics` table.
However, we started to notice in production that the PG planner doesn't
always execute the filter first to reduce the number of rows that it has
to scan through. Running the following query in one of our production
instance,
```
EXPLAIN ANALYZE
SELECT
DISTINCT topics.id as topic_id,
u.id as user_id,
topics.created_at,
topics.updated_at,
topics.highest_staff_post_number AS highest_post_number,
last_read_post_number,
c.id as category_id,
c.topic_id AS category_topic_id,
tu.notification_level,
us.first_unread_at,
GREATEST(
CASE
WHEN COALESCE(uo.new_topic_duration_minutes, 2880) = -1 THEN u.created_at
WHEN COALESCE(uo.new_topic_duration_minutes, 2880) = -2 THEN COALESCE(
u.previous_visit_at,u.created_at
)
ELSE ('2023-07-31 03:29:45.737630'::timestamp - INTERVAL '1 MINUTE' * COALESCE(uo.new_topic_duration_minutes, 2880))
END, u.created_at, '2023-07-25 15:06:44'
) AS treat_as_new_topic_start_date
FROM topics
JOIN users u on u.id = 13455
JOIN user_stats AS us ON us.user_id = u.id
JOIN user_options AS uo ON uo.user_id = u.id
JOIN categories c ON c.id = topics.category_id
LEFT JOIN topic_users tu ON tu.topic_id = topics.id AND tu.user_id = u.id
WHERE u.id = 13455 AND
topics.updated_at >= us.first_unread_at AND
topics.archetype <> 'private_message' AND
(("topics"."deleted_at" IS NULL AND (tu.last_read_post_number < topics.highest_staff_post_number) AND (COALESCE(tu.notification_level, 1) >= 2)) OR (1=0)) AND
NOT (
COALESCE((select array_agg(tag_id) from topic_tags where topic_tags.topic_id = topics.id), ARRAY[]::int[]) && ARRAY[451,452,453]
) AND
topics.deleted_at IS NULL AND
NOT (
last_read_post_number IS NULL AND
(
topics.category_id IN (SELECT "categories"."id" FROM "categories" LEFT JOIN categories categories2 ON categories2.id = categories.parent_category_id LEFT JOIN category_users ON category_users.category_id = categories.id AND category_users.user_id = 13455 LEFT JOIN category_users category_users2 ON category_users2.category_id = categories2.id AND category_users2.user_id = 13455 WHERE ((category_users.id IS NULL AND COALESCE(category_users2.notification_level, 1) = 0) OR COALESCE(category_users.notification_level, 1) = 0))
AND tu.notification_level <= 1
)
)
```
we get the following
```
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Unique (cost=201606.06..201608.15 rows=76 width=60) (actual time=91.279..91.294 rows=14 loops=1)
-> Sort (cost=201606.06..201606.25 rows=76 width=60) (actual time=91.278..91.284 rows=14 loops=1)
Sort Key: topics.id, topics.created_at, topics.updated_at, topics.highest_staff_post_number, tu.last_read_post_number, c.id, c.topic_id, tu.notification_level, us.first_unread_at, (GREATEST(CASE WHEN (COALESCE(uo.new_topic_duration_minutes, 2880) = '-1'::integer) THEN u.created_at WHEN (COALESCE(uo.new_topic_duration_minutes, 2880) = '-2'::integer) THEN COALESCE(u.previous_visit_at, u.created_at) ELSE ('2023-07-31 03:29:45.73763'::timestamp without time zone - ('00:01:00'::interval * (COALESCE(uo.new_topic_duration_minutes, 2880))::double precision)) END, u.created_at, '2023-07-25 15:06:44'::timestamp without time zone))
Sort Method: quicksort Memory: 26kB
-> Hash Join (cost=97519.51..201603.69 rows=76 width=60) (actual time=87.662..91.268 rows=14 loops=1)
Hash Cond: (topics.id = tu.topic_id)
Join Filter: ((tu.last_read_post_number < topics.highest_staff_post_number) AND ((tu.last_read_post_number IS NOT NULL) OR (NOT (hashed SubPlan 2)) OR (tu.notification_level > 1)))
Rows Removed by Join Filter: 10
-> Nested Loop (cost=1.54..104075.36 rows=3511 width=68) (actual time=0.055..3.609 rows=548 loops=1)
-> Nested Loop (cost=1.13..25.20 rows=1 width=32) (actual time=0.027..0.033 rows=1 loops=1)
-> Nested Loop (cost=0.71..16.76 rows=1 width=28) (actual time=0.020..0.023 rows=1 loops=1)
-> Index Scan using users_pkey on users u (cost=0.42..8.44 rows=1 width=20) (actual time=0.010..0.012 rows=1 loops=1)
Index Cond: (id = 13455)
-> Index Scan using user_stats_pkey on user_stats us (cost=0.29..8.31 rows=1 width=12) (actual time=0.008..0.010 rows=1 loops=1)
Index Cond: (user_id = 13455)
-> Index Scan using index_user_options_on_user_id_and_default_calendar on user_options uo (cost=0.42..8.44 rows=1 width=8) (actual time=0.007..0.008 rows=1 loops=1)
Index Cond: (user_id = 13455)
-> Nested Loop (cost=0.41..104015.12 rows=3504 width=36) (actual time=0.026..3.503 rows=548 loops=1)
-> Seq Scan on categories c (cost=0.00..13.73 rows=73 width=8) (actual time=0.003..0.039 rows=73 loops=1)
-> Index Only Scan using index_topics_on_updated_at_public on topics (cost=0.41..1424.20 rows=48 width=28) (actual time=0.012..0.046 rows=8 loops=73)
Index Cond: ((updated_at >= us.first_unread_at) AND (category_id = c.id))
Filter: (NOT (COALESCE((SubPlan 1), '{}'::integer[]) && '{451,452,453}'::integer[]))
Heap Fetches: 553
SubPlan 1
-> Aggregate (cost=4.31..4.32 rows=1 width=32) (actual time=0.002..0.002 rows=1 loops=548)
-> Index Only Scan using index_topic_tags_on_topic_id_and_tag_id on topic_tags (cost=0.29..4.31 rows=1 width=4) (actual time=0.002..0.002 rows=1 loops=548)
Index Cond: (topic_id = topics.id)
Heap Fetches: 178
-> Hash (cost=97222.14..97222.14 rows=19914 width=16) (actual time=87.545..87.546 rows=42884 loops=1)
Buckets: 65536 (originally 32768) Batches: 1 (originally 1) Memory Usage: 2387kB
-> Bitmap Heap Scan on topic_users tu (cost=1217.47..97222.14 rows=19914 width=16) (actual time=14.419..78.286 rows=42884 loops=1)
Recheck Cond: (user_id = 13455)
Filter: (COALESCE(notification_level, 1) >= 2)
Rows Removed by Filter: 15839
Heap Blocks: exact=45285
-> Bitmap Index Scan on index_topic_users_on_user_id_and_topic_id (cost=0.00..1212.49 rows=59741 width=0) (actual time=6.448..6.448 rows=58723 loops=1)
Index Cond: (user_id = 13455)
SubPlan 2
-> Nested Loop Left Join (cost=0.74..46.90 rows=1 width=4) (never executed)
Join Filter: (category_users2.category_id = categories2.id)
Filter: (((category_users.id IS NULL) AND (COALESCE(category_users2.notification_level, 1) = 0)) OR (COALESCE(category_users.notification_level, 1) = 0))
-> Nested Loop Left Join (cost=0.45..32.31 rows=73 width=16) (never executed)
Join Filter: (category_users.category_id = categories.id)
-> Nested Loop Left Join (cost=0.15..18.45 rows=73 width=8) (never executed)
-> Seq Scan on categories (cost=0.00..13.73 rows=73 width=8) (never executed)
-> Memoize (cost=0.15..0.28 rows=1 width=4) (never executed)
Cache Key: categories.parent_category_id
Cache Mode: logical
-> Index Only Scan using categories_pkey on categories categories2 (cost=0.14..0.27 rows=1 width=4) (never executed)
Index Cond: (id = categories.parent_category_id)
Heap Fetches: 0
-> Materialize (cost=0.29..11.69 rows=2 width=12) (never executed)
-> Index Scan using idx_category_users_user_id_category_id on category_users (cost=0.29..11.68 rows=2 width=12) (never executed)
Index Cond: (user_id = 13455)
-> Materialize (cost=0.29..11.69 rows=2 width=8) (never executed)
-> Index Scan using idx_category_users_user_id_category_id on category_users category_users2 (cost=0.29..11.68 rows=2 width=8) (never executed)
Index Cond: (user_id = 13455)
Planning Time: 1.740 ms
Execution Time: 91.414 ms
(59 rows)
```
From the execution plan, we can see the most of the time is spent
joining about 42888 rows in the `topics` table to the `topic_users` table.
However, we know that we only have to scan through a
subset of the `topics` table because the user's last unread at is '2023-07-20 11:33:05'.
If we filter the `topics` table with `topics.updated_at >= '2023-07-20 11:33:05'`, this would only
return about 1500 rows.
From our testing in production, the PG planner is able to execute a
better query plan when we avoid the unnecessary joins on `user_stats` just to be
able to get the user's `UserStat#first_unread_at`. Instead, we can just
pass the value of `UserStat#first_unread_at` directly as a query
parameter.
```
EXPLAIN ANALYZE
SELECT
DISTINCT topics.id as topic_id,
u.id as user_id,
topics.created_at,
topics.updated_at,
topics.highest_staff_post_number AS highest_post_number,
last_read_post_number,
c.id as category_id,
c.topic_id AS category_topic_id,
tu.notification_level,
GREATEST(
CASE
WHEN COALESCE(uo.new_topic_duration_minutes, 2880) = -1 THEN u.created_at
WHEN COALESCE(uo.new_topic_duration_minutes, 2880) = -2 THEN COALESCE(
u.previous_visit_at,u.created_at
)
ELSE ('2023-07-31 03:29:45.737630'::timestamp - INTERVAL '1 MINUTE' * COALESCE(uo.new_topic_duration_minutes, 2880))
END, u.created_at, '2023-07-25 15:06:44'
) AS treat_as_new_topic_start_date
FROM topics
JOIN users u on u.id = 13455
JOIN user_options AS uo ON uo.user_id = u.id
JOIN categories c ON c.id = topics.category_id
LEFT JOIN topic_users tu ON tu.topic_id = topics.id AND tu.user_id = u.id
WHERE u.id = 13455 AND
topics.updated_at >= '2023-07-20 11:33:05' AND
topics.archetype <> 'private_message' AND
(("topics"."deleted_at" IS NULL AND (tu.last_read_post_number < topics.highest_staff_post_number) AND (COALESCE(tu.notification_level, 1) >= 2)) OR (1=0)) AND
NOT (
COALESCE((select array_agg(tag_id) from topic_tags where topic_tags.topic_id = topics.id), ARRAY[]::int[]) && ARRAY[451,452,453]
) AND
topics.deleted_at IS NULL AND
NOT (
last_read_post_number IS NULL AND
(
topics.category_id IN (SELECT "categories"."id" FROM "categories" LEFT JOIN categories categories2 ON categories2.id = categories.parent_category_id LEFT JOIN category_users ON category_users.category_id = categories.id AND category_users.user_id = 13455 LEFT JOIN category_users category_users2 ON category_users2.category_id = categories2.id AND category_users2.user_id = 13455 WHERE ((category_users.id IS NULL AND COALESCE(category_users2.notification_level, 1) = 0) OR COALESCE(category_users.notification_level, 1) = 0))
AND tu.notification_level <= 1
)
);
```
Note how the filter is now `topics.updated_at >= '2023-07-20 11:33:05'`
instead of `topics.updated_at >= us.first_unread_at`. The modified query
above generates the following execution plan.
```
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Unique (cost=5189.86..5189.88 rows=1 width=52) (actual time=4.991..5.002 rows=14 loops=1)
-> Sort (cost=5189.86..5189.86 rows=1 width=52) (actual time=4.990..4.994 rows=14 loops=1)
Sort Key: topics.id, topics.created_at, topics.updated_at, topics.highest_staff_post_number, tu.last_read_post_number, c.id, c.topic_id, tu.notification_level, (GREATEST(CASE WHEN (COALESCE(uo.new_topic_duration_minutes, 2880) = '-1'::integer) THEN u.created_at WHEN (COALESCE(uo.new_topic_duration_minutes, 2880) = '-2'::integer) THEN COALESCE(u.previous_visit_at, u.created_at) ELSE ('2023-07-31 03:29:45.73763'::timestamp without time zone - ('00:01:00'::interval * (COALESCE(uo.new_topic_duration_minutes, 2880))::double precision)) END, u.created_at, '2023-07-25 15:06:44'::timestamp without time zone))
Sort Method: quicksort Memory: 26kB
-> Nested Loop (cost=52.11..5189.85 rows=1 width=52) (actual time=0.093..4.974 rows=14 loops=1)
-> Nested Loop (cost=51.70..5181.39 rows=1 width=60) (actual time=0.084..4.931 rows=14 loops=1)
-> Nested Loop (cost=51.28..5172.94 rows=1 width=44) (actual time=0.076..4.887 rows=14 loops=1)
-> Nested Loop (cost=0.41..1698.46 rows=59 width=36) (actual time=0.029..3.537 rows=548 loops=1)
-> Seq Scan on categories c (cost=0.00..13.73 rows=73 width=8) (actual time=0.005..0.039 rows=73 loops=1)
-> Index Only Scan using index_topics_on_updated_at_public on topics (cost=0.41..23.07 rows=1 width=28) (actual time=0.012..0.047 rows=8 loops=73)
Index Cond: ((updated_at >= '2023-07-20 11:33:05'::timestamp without time zone) AND (category_id = c.id))
Filter: (NOT (COALESCE((SubPlan 1), '{}'::integer[]) && '{451,452,453}'::integer[]))
Heap Fetches: 552
SubPlan 1
-> Aggregate (cost=4.31..4.32 rows=1 width=32) (actual time=0.002..0.002 rows=1 loops=548)
-> Index Only Scan using index_topic_tags_on_topic_id_and_tag_id on topic_tags (cost=0.29..4.31 rows=1 width=4) (actual time=0.002..0.002 rows=1 loops=548)
Index Cond: (topic_id = topics.id)
Heap Fetches: 178
-> Index Scan using index_topic_users_on_user_id_and_topic_id on topic_users tu (cost=50.86..58.88 rows=1 width=16) (actual time=0.002..0.002 rows=0 loops=548)
Index Cond: ((user_id = 13455) AND (topic_id = topics.id))
Filter: ((COALESCE(notification_level, 1) >= 2) AND (last_read_post_number < topics.highest_staff_post_number) AND ((last_read_post_number IS NOT NULL) OR (NOT (hashed SubPlan 2)) OR (notification_level > 1)))
Rows Removed by Filter: 0
SubPlan 2
-> Nested Loop Left Join (cost=0.74..50.43 rows=1 width=4) (never executed)
Join Filter: (category_users2.category_id = categories2.id)
Filter: (((category_users.id IS NULL) AND (COALESCE(category_users2.notification_level, 1) = 0)) OR (COALESCE(category_users.notification_level, 1) = 0))
-> Nested Loop Left Join (cost=0.45..35.84 rows=73 width=16) (never executed)
Join Filter: (category_users.category_id = categories.id)
-> Nested Loop Left Join (cost=0.15..21.97 rows=73 width=8) (never executed)
-> Seq Scan on categories (cost=0.00..13.73 rows=73 width=8) (never executed)
-> Memoize (cost=0.15..0.61 rows=1 width=4) (never executed)
Cache Key: categories.parent_category_id
Cache Mode: logical
-> Index Only Scan using categories_pkey on categories categories2 (cost=0.14..0.60 rows=1 width=4) (never executed)
Index Cond: (id = categories.parent_category_id)
Heap Fetches: 0
-> Materialize (cost=0.29..11.69 rows=2 width=12) (never executed)
-> Index Scan using idx_category_users_user_id_category_id on category_users (cost=0.29..11.68 rows=2 width=12) (never executed)
Index Cond: (user_id = 13455)
-> Materialize (cost=0.29..11.69 rows=2 width=8) (never executed)
-> Index Scan using idx_category_users_user_id_category_id on category_users category_users2 (cost=0.29..11.68 rows=2 width=8) (never executed)
Index Cond: (user_id = 13455)
-> Index Scan using users_pkey on users u (cost=0.42..8.44 rows=1 width=20) (actual time=0.003..0.003 rows=1 loops=14)
Index Cond: (id = 13455)
-> Index Scan using index_user_options_on_user_id_and_default_calendar on user_options uo (cost=0.42..8.44 rows=1 width=8) (actual time=0.002..0.002 rows=1 loops=14)
Index Cond: (user_id = 13455)
Planning Time: 1.281 ms
Execution Time: 5.092 ms
(48 rows)
```
With the new query, PG first does an index scan using the `index_topics_on_updated_at_public` index to filter away most of the topics making the subsequent joins much cheaper. Total query time has been reduced from ~90ms to ~5ms.
This optimisation will mostly affect users with very few/recent unread topics since a large `UserStat#firsts_unread_at` value will still mean scanning through a large portion of the `topics` table.
We currently are accumulating orphaned upload references whenever drafts are deleted.
This change deals with future cases by adding a dependent strategy of delete_all on the Draft#upload_references association. (We don't really need destroy strategy here, since UploadReference is a simple data bag and there are no validations or callbacks on the model.)
It deals with existing cases through a migration that deletes all existing, orphaned draft upload references.
A previous change updated `ReviewableQueuedPost`'s `created_by`
to be consistent with other reviewable types. It assigns
the the creator of the post being queued to `target_created_by` and sets
the `created_by` to the creator of the reviewable itself.
This fix updates some of the `created_by` references missed during the
intial fix.
What is the problem here?
In multiple controllers, we are accepting a `limit` params but do not
impose any upper bound on the values being accepted. Without an upper
bound, we may be allowing arbituary users from generating DB queries
which may end up exhausing the resources on the server.
What is the fix here?
A new `fetch_limit_from_params` helper method is introduced in
`ApplicationController` that can be used by controller actions to safely
get the limit from the params as a default limit and maximum limit has
to be set. When an invalid limit params is encountered, the server will
respond with the 400 response code.
This is happening because despite the user already existing in the forum, the `SingleSignOnRecord` doesn't exist and "require_activation" is set on the provider, causing us to skip looking for the email, and resulting in us creating a new User then seeing Validation failed: Primary email has already been taken when DiscourseConnect is attempting to make a new account.
Why this change?
In `PostDestroyer#make_previous_post_the_last_one` and
`Topic.reset_highest`, we have a query that looks something like this:
```
SELECT user_id FROM posts
WHERE topic_id = :topic_id AND
deleted_at IS NULL AND
post_type <> 4
#{post_type}
ORDER BY created_at desc
LIMIT 1
```
However, we currently don't have an index that caters directly to this
query. As a result, we have seen this query performing poorly on large
sites if the PG planner ends up using an index that is suboptimal for
the query.
This commit adds an index to the `posts` table on `topic_id` and then
`created_at`. For the query above, PG will be able to do a backwards
index scan efficiently.
Context of this change:
There are two site settings which an admin can configured to set the
default categories and tags that are shown for a new user. `default_navigation_menu_categories`
is used to determine the default categories while
`default_navigation_menu_tags` is used to determine the default tags.
Prior to this change when seeding the defaults, we will filter out the
categories/tags that the user do not have permission to see. However,
this means that when the user does eventually gain permission down the
line, the default categories and tags do not appear.
What does this change do?
With this commit, we have changed it such that all the categories and tags
configured in the `default_navigation_menu_categories` and
`default_navigation_menu_tags` site settings are seeded regardless of
whether the user's visibility of the categories or tags. During
serialization, we will then filter out the categories and tags which the
user does not have visibility of.
These methods were deprecated and marked for removal in 2.6. This change deletes them.
These deprecations use raise_error: true, so the fallbacks are at this point unreachable and can't be used anyway.
Recently we started giving admins a notice in the advice panel when their translations have become outdated due to changes in core. However, we didn't include any additional information.
This PR adds more information about the outdated translation inside the site text edit page, together with an option to dismiss the warning.
Performing a `Delete User`/`Delete and Block User` reviewable actions for a
queued post reviewable from the `review.show` route results in an error
popup even if the action completes successfully.
This happens because unlike other reviewable types, a user delete action
on a queued post reviewable results in the deletion of the reviewable
itself. A subsequent attempt to reload the reviewable record results in
404. The deletion happens as part of the call to `UserDestroyer` which
includes a step for destroying reviewables created by the user being
destroyed. At the root of this is the creator of the queued post
being set as the creator of the reviewable as instead of the system
user.
This change assigns the creator of the reviewable to the system user and
uses the more approapriate `target_created_by` column for the creator of the
post being queued.
Adding a filter without a type parameter has been deprecated for the last three years, and was marked for removal in 2.9.0.
During this time we have had a few deprecation warnings in logs coming from Reports::Bookmarks.
The fallback was to set the type to the name of the filter. This change just passes the type (same as name) explicitly instead, and removes the deprecation fallback.
We have a number of raw comments indicating that certain methods and classes are deprecated and marked for removal. This change turn those comments into deprecation warnings so that we can 1) see them in the logs of our own hosting and 2) give some warning to self hosters.
Why this change?
The `legacy` navigation menu option for the `navigation_menu` site
setting will be removed shortly after the release of Discourse 3.1 in
the first beta release of Discourse 3.2. Therefore, we're adding an
admin dashboard warning to give sites on the `legacy` navigation menu a
heads up.
We recently introduced this advice to admins when some translation overrides are outdated or using unknown interpolation keys:
However we missed the case where the original translation key has been renamed or altogether removed. When this happens they are no longer visible in the admin interface, leading to the confusing situation where we say there are outdated translations, but none are shown.
Because we don't explicitly handle this case, some deleted translations were incorrectly marked as having unknown interpolation keys. (This is because I18n.t will return a string like "Translation missing: foo", which obviously has no interpolation keys inside.)
This change adds an additional status, deprecated for TranslationOverride, and the job that checks them will check for this status first, taking precedence over invalid_interpolation_keys. Since the advice only checks for the outdated and invalid_interpolation_keys statuses, this fixes the problem.
The attribute Reviewable#post_options was deprecated (and replaced by #payload) four years ago, and marked for deletion in 2.9.0. This commit removes it.
Why this change?
Prior to this change, dismissing unreads posts did not publish the
changes across clients for the same user. As a result, users can end up
seeing an unread count being present but saw no topics being loaded when
visiting the `/unread` route.
Simplified query based on SiteSettings to join only relevant user_options rows.
In addition, index was added to 'watched_precedence_over_muted` column in `user_options` table to speed up query
Instead of having to remember every time, just always wait until the
current transaction (if it exists) has committed before clearing any
DistributedCache.
The only exception to this is caches that aren't caching things from
postgres.
This means we have to do the test setup after setting the test
transaction, because doing the test setup involves clearing caches.
Reapplying this - it now doesn't use after_commit if skip_db is set
This PR adds a feature to help admins stay up-to-date with their translations. We already have protections preventing admins from problems when they update their overrides. This change adds some protection in the other direction (where translations change in core due to an upgrade) by creating a notice for admins when defaults have changed.
Terms:
- In the case where Discourse core changes the default translation, the translation override is considered "outdated".
- In the case above where interpolation keys were changed from the ones the override is using, it is considered "invalid".
- If none of the above applies, the override is considered "up to date".
How does it work?
There are a few pieces that makes this work:
- When an admin creates or updates a translation override, we store the original translation at the time of write. (This is used to detect changes later on.)
- There is a background job that runs once every day and checks for outdated and invalid overrides, and marks them as such.
- When there are any outdated or invalid overrides, a notice is shown in admin dashboard with a link to the text customization page.
Known limitations
The link from the dashboard links to the default locale text customization page. Given there might be invalid overrides in multiple languages, I'm not sure what we could do here. Consideration for future improvement.
Instead of having to remember every time, just always wait until the
current transaction (if it exists) has committed before clearing any
DistributedCache.
The only exception to this is caches that aren't caching things from
postgres.
This means we have to do the test setup after setting the test
transaction, because doing the test setup involves clearing caches.
Follow-up to b27e12445d
This commit adds 2 new site settings `default_sidebar_link_to_filtered_list` and `default_sidebar_show_count_of_new_items` to control the default values for the navigation menu preferences that were added in the linked commit (`sidebar_link_to_filtered_list` and `sidebar_show_count_of_new_items` respectively).
Recently, site setting watched_precedence_over_muted was introduced - https://github.com/discourse/discourse/pull/22252
In this PR, we are allowing users to override it. The option is only displayed when the user has watched categories and muted tags, or vice versa.
Recently, we added the option for watched tag/categories to take precedence over muted tag/categories. Therefore, `remove_muted_tags` is using `category_users` to check if categories are not watched. There was missing join in CategoryList which was causing an error.
This is the first of a number of PRs aimed at helping admins manage their translation overrides. It simply adds a list of available interpolation keys below the input field when editing an override.
It also includes custom interpolation key.
Why this change?
We are currently not fully satisfied with the current way to edit the
categories and tags that appears in the sidebar where the user is
redirected to the tracking preferences tab in the user's profile causing
the user to lose context of the current page. In addition, the dropdown
to select categories or tags limits the amount of information we can
display.
Since editing or adding a custom categories section is already using a
modal, we have decided to switch editing the categories and tags that
appear in the sidebar to use a modal as well.
This commit removes the `new_edit_sidebar_categories_tags_interface_groups` site setting and
make the modals the default for all users.
Updates the interface for implementing summarization strategies and adds a cache layer to summarize topics once.
The cache stores the final summary and each chunk used to build it, which will be useful when we have to extend or rebuild it.
While we are unable to support OAUTH2 with pop3 (due to upstream dependency ruby/net-pop#16), we are adding the support for mail pollers plugin. Doing so, it would be possible to write a plugin which then uses other ways (microsoft graph sdk for example) to poll emails from a mailbox.
The idea is that a plugin would define a class which inherits from Email::Poller and defines a poll_mailbox static method which returns an array of strings. Then the plugin could call register_mail_poller(<class_name>) to have it registered. All the configuration (oauth2 tokens, email, etc) could be managed by sitesettings defined in the plugin.
This change adds support retroactively updating display names in the new quote format when the user's name is changed. It happens through a background job that is triggered by a callback when a user is saved with a new name.
This PR splits up the preference that controls the count vs dot and destination of sidebar links, which is really hard to understand, into 2 simpler checkboxes:
The new preferences/checkboxes are off by default, but there are database migrations to switch the old preference to the new ones so that existing users don't have to update their preferences to keep their preferred behavior of sidebar links when this changed is rolled out.
Internal topic: t/103529.
Don't cache user_fields on users separately from custom_fields, since they can get out of sync.
---------
Co-authored-by: Daniel Waterworth <me@danielwaterworth.com>
https://meta.discourse.org/t/markdown-preview-and-result-differ/263878
The result of this markdown had different results in the composer preview and the post. This is solved by updating Loofah to the latest version and using html5 fragments like our user had reported. While the change was only needed in cooked_post_processor.rb for this fix, other areas also had to be updated due to various side effects.
# Top level view
This PR is the first version of converting the search menu and its logic from (deprecated) widgets to glimmer components. The changes are hidden behind a group based feature flag. This will give us the ability to test the new implementation in a production setting before fully committing to the new search menu.
# What has changed
The majority of the logic from the widget implementation has been updated to fit within the context of a glimmer component, but it has not fundamentally changed. Instead of having a single widget - [search-menu.js](https://github.com/discourse/discourse/blob/main/app/assets/javascripts/discourse/app/widgets/search-menu.js) - that built the bulk of the search menu logic, we split the logic into (20+) bite size components. This greatly increases the readability and makes extending a component in the search menu much more straightforward.
That being said, certain pieces needed to be rewritten from scratch as they did not translate from widget -> glimmer, or there was a general code upgraded needed. There are a few of these changes worth noting:
### Search Service
**Search Term** -> In the widget implementation we had a overly complex way of managing the current search term. We tracked the search term across multiple different states (`term`, `opts.term`, `searchData.term`) causing headaches. This PR introduces a single source of truth:
```js
this.search.activeGlobalSearchTerm
```
This tracked value is available anywhere the `search` service is injected. In the case the search term should be needs to be updated you can call
```js
this.search.activeGlobalSearchTerm = "foo"
```
**event listeners** -> In the widget implementation we defined event listeners **only** on the search input to handle things such as
- keyboard navigation / shortcuts
- closing the search menu
- performing a search with "enter"
Having this in one place caused a lot of bloat in our logic as we had to handle multiple different cases in one location. Do _x_ if it is this element, but do _y_ if it is another. This PR updates the event listeners to be attached to individual components, allowing for a more fine tuned set of actions per element. To not duplicate logic across multiple components, we have condensed shared logic to actions on the search service to be reused. For example - `this.search.handleArrowUpOrDown` - to handle keyboard navigation.
### Search Context
We have unique logic based on the current search context (topic / tag / category / user / etc). This context is set within a models route file. We have updated the search service with a tracked value `searchContext` that can be utilized and updated from any component where the search service is injected.
```js
# before
this.searchService.set("searchContext", user.searchContext);
# after
this.searchService.searchContext = user.searchContext;
```
# Views
<img width="434" alt="Screenshot 2023-06-15 at 11 01 01 AM" src="https://github.com/discourse/discourse/assets/50783505/ef57e8e6-4e7b-4ba0-a770-8f2ed6310569">
<img width="418" alt="Screenshot 2023-06-15 at 11 04 11 AM" src="https://github.com/discourse/discourse/assets/50783505/2c1e0b38-d12c-4339-a1d5-04f0c1932b08">
<img width="413" alt="Screenshot 2023-06-15 at 11 04 34 AM" src="https://github.com/discourse/discourse/assets/50783505/b871d164-88cb-405e-9b78-d326a6f63686">
<img width="419" alt="Screenshot 2023-06-15 at 11 07 51 AM" src="https://github.com/discourse/discourse/assets/50783505/c7309a19-f541-47f4-94ef-10fa65658d8c">
<img width="424" alt="Screenshot 2023-06-15 at 11 04 48 AM" src="https://github.com/discourse/discourse/assets/50783505/f3dba06e-b029-431c-b3d0-36727b9e6dce">
<img width="415" alt="Screenshot 2023-06-15 at 11 08 57 AM" src="https://github.com/discourse/discourse/assets/50783505/ad4e7250-040c-4d06-bf06-99652f4c7b7c">
Communities can use sidebar or header dropdown, therefore navigation menu is a better name settings in 2 places:
- Old user sidebar preferences;
- Site setting about default tags and categories.
Currently, groups owned by moderators are not visible to them on the
groups page. This happens because, the group visibility queries don't
account for non-admin staff user group ownership.
This change updates the group visibility scope queries to account for a
moderator(non-admin staff user) group ownership.
* FEATURE: Content custom summarization strategies.
This PR establishes a pattern for plugins to register alternative ways of summarizing content by extending a class that defines an interface.
Core controls which strategy we'll use and who has access to it through the `summarization_strategy` and `custom_summarization_allowed_groups`. It also defines the UI for summarizing topics.
Other plugins can access this summarization mechanism and implement their features, removing cross-plugin customizations, as it currently happens between chat and the discourse-ai plugin.
* Group membership validation and rate limiting
* Work with objects instead of classes
* Port summarization feature from discourse-ai to chat
* Rename available summaries to 'Top Replies' and 'Summary'
This patch sets some limits on custom fields:
- an entity can’t have more than 100 custom fields defined on it
- a custom field can’t hold a value greater than 10,000,000 characters
The current implementation of custom fields is relatively complex and
does an upsert in SQL at some point, thus preventing to simply add an
`ActiveRecord` validation on the custom field model without having to
rewrite a part of the existing logic.
That’s one of the reasons this patch is implementing validations in the
`HasCustomField` module adding them to the model including the module.
* DEV: Implement staff logs for user columns edits
* deleted extra space in staff logger detail string, deleted string when no changes are made, added basic test coverage for EditDirectoryColumnsController
* fixed change made to #self.staff_actions un UserHistory
* implemented a method that builds the details, previous_values and new_values in a dynamic way
* removed details of changes
* refactored small merge
What this change?
We are currently not fully satisfied with the current way to edit the
categories and tags that appears in the sidebar where the user is
redirected to the tracking preferences tab in the user's profile causing
the user to lose context of the current page. In addition, the dropdown
to select categories or tags limits the amount of information we can
display.
Since editing or adding a custom categories section is already using a
modal, we have decided to switch editing the categories and tags that
appear in the sidebar to use a modal as well.
This commit ships a first pass of the edit categories modal such that we
can keep the commit small and reviewable. The incomplete nature of the
feature is also reflected in the fact that the feature is hidden behind
a new `new_edit_sidebar_categories_tags_interface_groups` site setting.
* FEATURE: reduce avatar sizes to 6 from 20
This PR introduces 3 changes:
1. SiteSetting.avatar_sizes, now does what is says on the tin.
previously it would introduce a large number of extra sizes, to allow for
various DPIs. Instead we now trust the admin with the size list.
2. When `avatar_sizes` changes, we ensure consistency and remove resized
avatars that are not longer allowed per site setting. This happens on the
12 hourly job and limited out of the box to 20k cleanups per cycle, given
this may reach out to AWS 20k times to remove things.
3.Our default avatar sizes are now "24|48|72|96|144|288" these sizes were
very specifically picked to limit amount of bluriness introduced by webkit.
Our avatars are already blurry due to 1px border, so this corrects old blur.
This change heavily reduces storage required by forums which simplifies
site moves and more.
Co-authored-by: David Taylor <david@taylorhq.com>
This commit adds modifiers that allow plugins to change how categories and groups are prefetched into the application and listed in the respective controllers.
Possible use cases:
- prevent some categories/groups from being prefetched when the application loads for performance reasons.
- prevent some categories/groups from being listed in their respective index pages.
When a user chooses to move a topic/message to an existing topic/message, they can now opt to merge the posts chronologically (using a checkbox in the UI).
What is this change required?
I noticed that actions in `SidebarSectionsController` resulted in
lots of N+1 queries problem and I wanted a solution to
prevent such problems without having to write N+1 queries tests. I have
also used strict loading for `SidebarSection` queries in performance
sensitive spots.
Note that in this commit, I have also set `config.active_record.action_on_strict_loading_violation = :log`
for the production environment so that we have more visibility of
potential N+1 queries problem in the logs. In development and test
environment, we're sticking with the default of raising an error.
What is the problem?
In the test environement, we were calling `SiteSetting.setting` directly
to introduce new site settings. However, this leads to changes in state of the SiteSettings
hash that is stored in memory as test runs. Changing or leaking states
when running tests is one of the major contributors of test flakiness.
An example of how this resulted in test flakiness is our `spec/integrity/i18n_spec.rb` spec file which
had a test case that would fail because a new "plugin_setting" site
setting was registered in another test case but the site setting did not
have translations for the site setting set.
What is the fix?
There are a couple of changes being introduced in this commit:
1. Make `SiteSetting.setting` a private method as it is not safe to be
exposed as a public method of the `SiteSetting` class
2. Change test cases to use existing site settings in Discourse instead
of creating custom site settings. Existing site settings are not
removed often so we don't really need to dynamically add new site
settings in test cases. Even if the site settings being used in test
cases are removed, updating the test cases to rely on other site
settings is a very easy change.
3. Set up a plugin instance in the test environment as a "fixture"
instead of having each test create its own plugin instance.
This patch is a followup of
https://github.com/discourse/discourse/pull/21504 where limits on custom
message for an invite were introduced.
This had a side effect of making some existing invites invalid and with
the current code, they can’t be invalidated anymore.
This patch takes the approach of skipping the validations when invites
are invalidated since the important thing here is to mark the invite as
invalidated regardless of its actual state in the DB. (no other
attributes are updated at the same time anyway)
* FIX: Video thumbnails can have duplicates
It's possible that a duplicate video or even a very similar video could
generate the same video thumbnail. Because video thumbnails are mapped
to their corresponding video by using the video sha1 in the thumbnail
filename we need to allow for duplicate thumbnails otherwise even when a
thumbnail has been generated for a topic it will not be mapped
correctly.
This will also allow you to re-upload a video on the same topic to
regenerate the thumbnail.
* fix typo
The welcome topic user tip was for admins only, but in general, user
tips should be used for guiding new users through the features that
Discourse offers. For this reason, we decided to remove the user tip.
This commit also includes a few more copy tweaks to the welcome topic.
The old method updated only existing records, without considering that
new tags might have been created or some tags might not exist anymore.
This was usually not a problem because the stats were also updated by
other code paths.
However, the ensure consistency job should be more solid and help when
other code paths fail or after importing data.
Also, update category tag stats too should happen when updating other
category stats as well.
### What is this change?
The lounge category was replaced with the general category in https://github.com/discourse/discourse/pull/18097.
However, there are still a few references to the lounge category in code. In particular, `Category#seeded?` is erroring out in production looking for `SiteSetting.lounge_category_id`.
In #21498, we split `BaseStore#download` into a "safe" version which returns nil on errors, and an "unsafe" version which raises an exception, which was the old behaviour of `#download`.
This change updates call sites that used the old `#download`, which raised exceptions, to use the new `#download!` to preserve behaviour (and silence deprecation warnings.)
It also silences the deprecation warning in tests.
- Update welcome topic copy
- Edit the welcome topic automatically when the title or description changes
- Remove “Create your Welcome Topic” banner/CTA
- Add "edit welcome topic" user tip
### Background
When SSRF detection fails, the exception bubbles all the way up, causing a log alert. This isn't actionable, and should instead be ignored. The existing `rescue` does already ignore network errors, but fails to account for SSRF exceptions coming from `FinalDestination`.
### What is this change?
This PR does two things.
---
Firstly, it introduces a common root exception class, `FinalDestination::SSRFError` for SSRF errors. This serves two functions: 1) it makes it easier to rescue both errors at once, which is generally what one wants to do and 2) prevents having to dig deep into the class hierarchy for the constant.
This change is fully backwards compatible thanks to how inheritance and exception handling works.
---
Secondly, it rescues this new exception in `UserAvatar.import_url_for_user`, which is causing sporadic errors to be logged in production. After this SSRF errors are handled the same as network errors.
This fixes a bug in the create invite API where if you passed in an
integer for the group_ids field it would fail to add the user to the
specified group.
This commit fixes an issue where the Likes Received notification
count in the user digest email was not affected by the
since/last_seen date for the user, which meant that no matter
how long it had been since the user visited the count was
always constant.
Now instead for the Likes Received count, we only count the
unread notifications of that type since the user was last
seen.
The current limit (250 characters) is too low, as we have some
translations used for our badge descriptions that result in a
description length of 264 characters.
To be on the safe side, the limit is now set to 500 characters.
A category's slug can be encoded when
`SiteSetting.slug_generation_method` has been set to "encoded". As a
result, we have to support non ASCII characters as well.
This amends it so our cached counting reliant specs run in synchronize mode
When running async there are situations where data is left over in the table
after a transactional test. This means that repeat runs of the test suite
fail.
* DEV: move sidebar community section to database
Before, community section was hard-coded. In the future, we are planning to allow admins to edit it. Therefore, it has to be moved to database to `custom_sections` table.
Few steps and simplifications has to be made:
- custom section was hidden behind `enable_custom_sidebar_sections` feature flag. It has to be deleted so all forums, see community section;
- migration to add `section_type` column to sidebar section to show it is a special type;
- migration to add `segment` column to sidebar links to determine if link should be displayed in primary section or in more section;
- simplify more section to have one level only (secondary section links are merged);
- ensure that links like `everything` are correctly tracking state;
- make user an anonymous links position consistence. For example, from now on `faq` link for user and anonymous is visible in more tab;
- delete old community-section template.
Before this fix if the underlying model of a reviewable was changed, the filter wouldn't work anymore as it was expecting a 1:1 relation between filter type and model name.
This commit also relies on the `Reviewable.types` array to check against valid types instead of a regex not checking much.
Finally this commit adds a spec to ensure chat reviewables are listable from the review index page.
Many blog posts use these to illustrate and images were previously omitted
Additionally strip superfluous HTML and BODY tags from embed HTML.
This was incorrectly returned from server.
Group user event webhooks filtered by group fail silently
because the `group_ids` job arg wasn't being passed into the job.
This change add's `group_ids` to the `EmitWebHookEvent` jobs queued for
`user_added_to_group` and `user_removed_from_group` events.
Currently, only user badge grants emit webhook events. This change
extends the `user_badge` webhook to emit user badge revocation events.
A new `user_badge_revoked` event has been introduced instead of relying
on the existing `user_badge_removed` event. `user_badge_removed` emitted
just the `badge_id` and `user_id` which aren't helpful for generating a
meaningful webhook payload for revoked(deleted) user badges.
The new event emits the user badge object.
When revising a post, if the topic that post belonged to did not have a category attached it would error with
> NoMethodError (undefined method `read_restricted' for nil:NilClass)
* FIX: Do not overwrite existing thumbnails
When auto generating video thumbnails they should not overwrite any
existing topic thumbnails.
This also addresses an issue with capitalized file extensions like .MOV
that were being excluded.
* Update app/models/post.rb
Remove comment
Co-authored-by: Penar Musaraj <pmusaraj@gmail.com>
---------
Co-authored-by: Penar Musaraj <pmusaraj@gmail.com>
Responding to negative behaviour tends to solicit more of the same. Common wisdom states: "don't feed the trolls".
This change codifies that advice by introducing a new nudge when hitting the reply button on a flagged post. It will be shown if either the current user, or two other users (configurable via a site setting) have flagged the post.
This PR adds the ability to destroy drafts for a passed user via the API. This was not possible before as this action was reserved for only your personal drafts.
If a user is an admin and calls the `#destroy` action from the API they are able to destroy a draft for a passed user. A user can be targeted by passed either their:
- username
- external_id (for SSO)
to the request.
In the case you attempt to destroy a non-personal draft and
- You are not an admin
- You do not access the `#destroy` action via the API
you will raise a `Discourse::InvalidAccess` (403) and will not succeed in destroying the draft.
This feature will allow sites to define which emoji are not allowed. Emoji in this list should be excluded from the set we show in the core emoji picker used in the composer for posts when emoji are enabled. And they should not be allowed to be chosen to be added to messages or as reactions in chat.
This feature prevents denied emoji from appearing in the following scenarios:
- topic title and page title
- private messages (topic title and body)
- inserting emojis into a chat
- reacting to chat messages
- using the emoji picker (composer, user status etc)
- using search within emoji picker
It also takes into account the various ways that emojis can be accessed, such as:
- emoji autocomplete suggestions
- emoji favourites (auto populates when adding to emoji deny list for example)
- emoji inline translations
- emoji skintones (ie. for certain hand gestures)
Previously, Discourse's password hashing was hard-coded to a specific algorithm and parameters. Any changes to the algorithm or parameters would essentially invalidate all existing user passwords.
This commit introduces a new `password_algorithm` column on the `users` table. This persists the algorithm/parameters which were use to generate the hash for a given user. All existing rows in the users table are assumed to be using Discourse's current algorithm/parameters. With this data stored per-user in the database, we'll be able to keep existing passwords working while adjusting the algorithm/parameters for newly hashed passwords.
Passwords which were hashed with an old algorithm will be automatically re-hashed with the new algorithm when the user next logs in.
Values in the `password_algorithm` column are based on the PHC string format (https://github.com/P-H-C/phc-string-format/blob/master/phc-sf-spec.md). Discourse's existing algorithm is described by the string `$pbkdf2-sha256$i=64000,l=32$`
To introduce a new algorithm and start using it, make sure it's implemented in the `PasswordHasher` library, then update `User::TARGET_PASSWORD_ALGORITHM`.
Previously, public custom sections were only visible to logged-in users. In this PR, we are making them visible to anonymous as well.
The reason is that Community Section will be moved into custom section model to be easily editable by admins.
This commit adds support for filtering for topics in specific
subcategories via the categories filter query language.
For example: `category:documentation:admins` will filter for topics and
subcategory topics in
the category with slug "admins" whose parent category has the slug
"documentation".
The `=` prefix can also be used such that
`=category:documentation:admins` will exclude subcategory topics of the
category with slug "admins" whose parent category has the slug
"documentation".
`Rails.application.routes.recognize_path(value)` was not working for /admin paths because StaffConstraint.new requires user to check permission.
This validation is not bringing much value, and the easiest way is to drop it. In the worse case scenario, a user will have an incorrect link in their sidebar.
Bug reported: https://meta.discourse.org/t/custom-sidebar-sections-being-tested-on-meta/255303/66
There is no need to validate the user's emails when
promoting/demoting their trust level, this can cause
issues in things like Jobs::Tl3Promotions, we don't
need to fail in that case when all we are doing is changing
trust level.
`default_categories_*` site settings will update the category preferences on user creation. But it shouldn't update the user's category preference if a group's setting already updated it for that user.
We perform lookups on sidebar section links based on sidebar_section_id
totally ignoring user. This ensures we have an index to work with.
This removes the previous index `links_user_id_section_id_position` which
partially doubled up `idx_unique_sidebar_section_links`
Before, incorrectly filled fields were marked with red border. Now, additional information under the field is displayed to notify the user what is incorrect.
/t/93696
There are many situations that may cause users to lose permission to
send messages in a chat channel. Until now we have relied on security
checks in `Chat::ChatChannelFetcher` to remove channels which the
user may have a `UserChatChannelMembership` record for but which
they do not have access to.
This commit takes a more proactive approach. Now any of these following
`DiscourseEvent` triggers may cause `UserChatChannelMembership`
records to be deleted:
* `category_updated` - Permissions of the category changed
(i.e. CategoryGroup records changed)
* `user_removed_from_group` - Means the user may not be able to access the
channel based on `GroupUser` or also `chat_allowed_groups`
* `site_setting_changed` - The `chat_allowed_groups` was updated, some
users may no longer be in groups that can access chat.
* `group_destroyed` - Means the user may not be able to access the
channel based on `GroupUser` or also `chat_allowed_groups`
All of these are handled in a distinct service run in a background
job. Users removed are logged via `StaffActionLog` and then we
publish messages on a per-channel basis to users who had their
memberships deleted.
When the user has a channel they are kicked from open, we show
a dialog saying "You no longer have access to this channel".
When they click OK we redirect them either:
* To their first other public channel, if they have any followed
* The chat browse page if they don't
This is to save on tons of requests from kicked out users getting messages
from other channels.
When the user does not have the kicked channel open, we can just
silently yoink it out of their sidebar and turn off subscriptions.
Followup to 184ce647ea,
this just implements Bianca's suggestion on the original
PR and catches the NameError, which was not necessary
before as we were not actually resolving any class from
bookmarkable_type.
Prior to this change `registered_bookmarkable` would return `nil` as `type` in `Bookmark.registered_bookmarkable_from_type(type)` would be `ChatMessage` and we registered a `Chat::Message` class.
This commit will now properly rely on each model `polymorphic_class_for(name)` to help us infer the proper type from a a `bookmarkable_type`.
Tests have also been added to ensure that creating/destroying chat message bookmarks is working correctly.
---
Longer explanation
Currently when you save a bookmark in the database, it's associated to another object through a polymorphic relationship, which will is represented by two columns: `bookmarkable_id` and `bookmarkable_type`. The `bookmarkable_id` contains the id of the relationship (a post ID for example) and the `bookmarkable_type` contains the type of the object as a string by default, (`"Post"` for example).
Chat plugin just started namespacing objects, as a result a model named `ChatMessage` is now named `Chat::Message`, to avoid complex and risky migrations we rely on methods provided by rails to alter the `bookmarkable_type` when we save it: we want to still save it as `"ChatMessage"` and not `"Chat::Message"`. And, to retrieve the correct model when we load the bookmark from the database: we want `"ChatMessage"` to load the `Chat::Message` model and not the `ChatMessage`model which doesn't exist anymore.
On top of this the bookmark codepath is allowing plugins to register types and will check against these types, so we alter this code path to be able to do a similar ChatMessage <-> Chat::Message dance and allow to check the type is valid. In the specific case of this commit, we were retrieving a `"ChatMessage"` bookmarkable_type from the DB and looking for it in the registered bookmarkable types which contain `Chat::Message` and not `ChatMessage`.
This commit main goal was to comply with Zeitwerk and properly rely on autoloading. To achieve this, most resources have been namespaced under the `Chat` module.
- Given all models are now namespaced with `Chat::` and would change the stored types in DB when using polymorphism or STI (single table inheritance), this commit uses various Rails methods to ensure proper class is loaded and the stored name in DB is unchanged, eg: `Chat::Message` model will be stored as `"ChatMessage"`, and `"ChatMessage"` will correctly load `Chat::Message` model.
- Jobs are now using constants only, eg: `Jobs::Chat::Foo` and should only be enqueued this way
Notes:
- This commit also used this opportunity to limit the number of registered css files in plugin.rb
- `discourse_dev` support has been removed within this commit and will be reintroduced later
<!-- NOTE: All pull requests should have tests (rspec in Ruby, qunit in JavaScript). If your code does not include test coverage, please include an explanation of why it was omitted. -->
Using `create_or_find_by!`, followed by `update_all!` requires two or three queries (two when the row doesn't already exist, three when it does). Instead, we can use postgres's native `INSERT ... ON CONFLICT ... DO UPDATE SET` feature to do the logic in a single atomic call.
If you happen to delete the general category before editing the welcome
topic, the banner will still display. This fix adds a after destroy hook
that will clear the entries for the welcome topic banner in the redis
cache.
Currently the auto-bump cooldown is hard-coded to 24 hours.
This change makes the highlighted 24 hours part configurable (defaulting to 24 hours), and the rest of the process remains the same.
This uses the new CategorySetting model associated with Category. We decided to add this because we want to move away from custom fields due to the lack of type casting and validations, but we want to keep the loading of these optional as they are not needed for almost all of the flows.
Category settings will be back-filled to all categories as part of this change, and creating a new category will now also create a category setting.
* FEATURE: Generate thumbnail images for uploaded videos
Topics in Discourse have a topic thumbnail feature which allows themes
to show a preview image before viewing the actual Topic.
This PR allows for the ability to generate a thumbnail image from an
uploaded video that can be use for the topic preview.
Currently, when a suspended user belongs to a group PM (private message
with more than two people in it) and a staff member sends a message to
this group PM, then the suspended user will receive an email.
This happens because a suspended user can only receive emails from staff
members. But in this case, this can be seen as a bug as the expected
behavior would be instead to not send any email to the suspended user. A
staff member can participate in active discussions like any other
member and so their messages in this context shouldn’t be treated
differently than the ones from regular users.
This patch addresses this issue by checking if a suspended user receives
a message from a group PM or not. If that’s the case then an email won’t
be sent no matter if the post originated from a staff member or not.
That column is obsolete since we added the `granted_title_badge_id` column in 2019 (56d3e29a69). Having both columns can lead to inconsistencies (mostly due to old data from before 2019).
For example, `BadgeGranter.revoke_ungranted_titles!` doesn't work correctly if `badge_granted_title` is `false` while `granted_title_badge_id` points to the badge that is used as title.
Similar spirit to e195e6f614,
this moves the Bookmarkable registration to DiscoursePluginRegistry
so plugins which are not enabled do not register additional
bookmarkable classes.
What is the problem?
We have a hidden site setting `show_category_definitions_in_topic_lists`
which is set to false by default. What this means is that category
definition topics are not shown in the topic list by default. Only the
category definition topic for the category being viewed will be shown.
However, we have a bug where we would show that a category has new
topics when a new child category along with its category definition
topic is created even though the topic list does not list the child
category's category definition topic.
What is the fix here?
This commit fixes the problem by shipping down an additional
`is_category_topic` attribute in `TopicTrackingStateItemSerializer` when
the `show_category_definitions_in_topic_lists` site setting has been set
to false. With the new attribute, we can then exclude counting child
categories' category definition topics when counting new and unread
counts for a category.
* UX: add type tag and design update
* UX: clarify status copy in reviewQ
* DEV: switch to selectKit
* UX: color approve/reject buttons in RQ
* DEV: regroup actions
* UX: add type tag and design update
* UX: clarify status copy in reviewQ
* Join questions for flagged post with "or" with new I18n function
* Move ReviewableScores component out of context
* Add CSS classes to reviewable-item based on human type
* UX: add table header for scoring
* UX: don't display % score
* UX: prefix modifier class with dash
* UX: reviewQ flag table styling
* UX: consistent use of ignore icon
* DEV: only show context question on pending status
* UX: only show table headers on pending status
* DEV: reviewQ regroup actions for hidden posts
* UX: reviewQ > approve/reject buttons
* UX: reviewQ add fadeout
* UX: reviewQ styling
* DEV: move scores back into component
* UX: reviewQ mobile styling
* UX: score table on mobile
* UX: reviewQ > move meta info outside table
* UX: reviewQ > score layout fixes
* DEV: readd `agree_and_keep` and fix the spec tests.
* Fix the spec tests
* fix the quint test
* DEV: readd deleting replies
* UX: reviewQ copy tweaks
* DEV: readd test for ignore + delete replies
* Remove old
* FIX: Add perform_ignore back in for backwards compat
* DEV: add an action alias `ignore` for `ignore_and_do_nothing`.
---------
Co-authored-by: Martin Brennan <martin@discourse.org>
Co-authored-by: Vinoth Kannan <svkn.87@gmail.com>
Fixes a small issue where allowed user removes themselves from a private message before the post activity (small action) is created.
I also added some test coverage to prevent regression.
/t/92811
Follow up to 098ab29d41. Since
we just used a `cattr_reader` on `About` this was not safe
for multisite, since some sites could have the chat plugin
enabled and some may not. Using `DiscoursePluginRegistry` gets
around this issue, and makes it so the chat stats only show
for a site if `chat_enabled` is true.
We currently apply type: :link watched words to custom user fields. This makes the user card pretty ugly because we don't allow html / links there. Additionally, the admin UI also does not say that we apply this to custom user fields, but only words in posts.
So this PR is to remove the replacement of link-type watch words for custom user fields.
This commit allows the user to set their preference vis-a-vis
the chat icon in the header of the page. There are three options:
- All New (default) - This maintains the existing behaviour where
all new messages in the channel show a blue dot on the icon
- Direct Messages and Mentions - Only show the green dot on the
icon when you are directly messaged or mentioned, the blue dot
is never shown
- Never - Never show any dot on the chat icon, for those who
want tractor-beam-laser-focus
This commit implements many changes to topic and comments embedding. It
deprecates the class_name field from EmbeddableHost and suggests using
the className parameter. discourse_username parameter has been
deprecated and it will fetch it from embedded site from the author or
discourse-username meta.
See the updated code sample from Admin > Customize > Embedding page.
* FEATURE: Add className parameter for Discourse embed
* DEV: Hide class_name from EmbeddableHost
* DEV: Deprecate class_name field of EmbeddableHost
* FEATURE: Use either author or discourse-username meta tag
* DEV: Deprecate discourse_username parameter
* DEV: Improve embed code sample
This commit introduces a few experimental changes to the New topics list and "Everything" link in the sidebar:
1. Make the New topics list include unread topics
2. Make the Everything section in the sidebar link to the New topics list (`/new`)
3. Remove "unread" or "new" text next to the count and keep the count
4. The count is a sum of new and unread topics counts
All of these of changes are behind an off-by-default feature flag. I've not written extensive tests for these changes because they're highly experimental.
Internal topic: t/77234.
Original solution to use `description` instead of `text_description` was wrong: https://github.com/discourse/discourse/pull/20436
Problem is that we have to escape HTML tags.
However, we would like to use escape method which is keep `/` intact. Expected behavior is given by ERB::Util.html_escape instead of Rack::Utils.escape_html
/t/92015
* FIX: Use pluralized string
* REFACTOR: Fix misuse of pluralized string
* REFACTOR: Fix misuse of pluralized string
* DEV: Remove linting of `one` key in MessageFormat string, it doesn't work
* REFACTOR: Fix misuse of pluralized string
This also ensures that the URL works on subfolder and shows the site setting link only for admins instead of staff. The string is quite complicated, so the best option was to switch to MessageFormat.
* REFACTOR: Fix misuse of pluralized string
* FIX: Use pluralized string
This also ensures that the URL works on subfolder and shows the site setting link only for admins instead of staff.
* REFACTOR: Correctly pluralize reaction tooltips in chat
This also ensures that maximum 5 usernames are shown and fixes the number of "others" which was off by 1 if the current user reacted on a message.
* REFACTOR: Use translatable string as comma separator
* DEV: Add comment to translation to clarify the meaning of `%{identifier}`
* REFACTOR: Use translatable comma separator and use explicit interpolation keys
* REFACTOR: Don't interpolate lowercase channel status
* REFACTOR: Fix misuse of pluralized string
* REFACTOR: Don't interpolate channel status
* REFACTOR: Use %{count} interpolation key
* REFACTOR: Fix misuse of pluralized string
* REFACTOR: Correctly pluralize DM chat channel titles
As of ba3f62f576, handlebars templates are colocated with js files so the path to hbs templates referenced by this rake task is no longer valid. This commit fixes the path to hbs templates and updates a couple of files that are generated by the rake task.
We call `post.update_uploads_secure_status` in both
`PostCreator` and `PostRevisor`. Only the former was checking
if `SiteSetting.secure_uploads?` was enabled, but the latter
was not. There is no need to enqueue the job
`UpdatePostUploadsSecureStatus` if secure_uploads is not
enabled for the site.
* Add username and name_or_username variables to SystemMessage defaults
* Allow username and name variables on welcome_user email template overrides
* Satisfy linting
* Add test
16 bit images were not returning the correct dominant color due truncation
The routine expected an 8bit color eg: #FFAA00, but ended up getting a 16bit one eg: #FFFAAA000. This caused a truncation, which leads to wildly off colors.
The #pluck_first freedom patch, first introduced by @danielwaterworth has served us well, and is used widely throughout both core and plugins. It seems to have been a common enough use case that Rails 6 introduced it's own method #pick with the exact same implementation. This allows us to retire the freedom patch and switch over to the built-in ActiveRecord method.
There is no replacement for #pluck_first!, but a quick search shows we are using this in a very limited capacity, and in some cases incorrectly (by assuming a nil return rather than an exception), which can quite easily be replaced with #pick plus some extra handling.
This is the first in a multi-part change to move the custom fields to a new table. It includes:
- Adding a new CategorySetting model and corresponding table.
- Populating it with data from the category_custom_fields table.
Improvements for this PR: https://github.com/discourse/discourse/pull/20057
What was fixed:
- [x] Use ember transitions instead of full reload
- [x] Link was inaccurately kept active
- [x] "+ save" renamed to just "save"
- [x] Render emojis in link name
- [x] UI to set icon
- [x] Delete link is trash icon instead of "x"
- [x] Add another link to on the left and rewording
- [x] Raname "link name" -> "name", "points to" -> link
- [x] Add limits to fields
- [x] Move add section button to the bottom
This PR is a major change to Sass compilation in Discourse.
The new version of sass-ruby moves to dart-sass putting we back on the supported version of Sass. It does so while keeping compatibility with the existing method signatures, so minimal change is needed in Discourse for this change.
This moves us
From:
- sassc 2.0.1 (Feb 2019)
- libsass 3.5.2 (May 2018)
To:
- dart-sass 1.58
This update applies the following breaking changes:
>
> These breaking changes are coming soon or have recently been released:
>
> [Functions are stricter about which units they allow](https://sass-lang.com/documentation/breaking-changes/function-units) beginning in Dart Sass 1.32.0.
>
> [Selectors with invalid combinators are invalid](https://sass-lang.com/documentation/breaking-changes/bogus-combinators) beginning in Dart Sass 1.54.0.
>
> [/ is changing from a division operation to a list separator](https://sass-lang.com/documentation/breaking-changes/slash-div) beginning in Dart Sass 1.33.0.
>
> [Parsing the special syntax of @-moz-document will be invalid](https://sass-lang.com/documentation/breaking-changes/moz-document) beginning in Dart Sass 1.7.2.
>
> [Compound selectors could not be extended](https://sass-lang.com/documentation/breaking-changes/extend-compound) in Dart Sass 1.0.0 and Ruby Sass 4.0.0.
SCSS files have been migrated automatically using `sass-migrator division app/assets/stylesheets/**/*.scss`
We recently had a bug which caused auto-bumping to "not work". The problem was that the value had been set to 0.5, which when coerced to an integer turned into 0. So the feature is "working as intended", but there's a possibility of misconfiguration.
When looking into this, I noticed that the inputs on the category settings page doesn't have any particular sanitisation in the front-end, and also one or two validations missing in the back-end.
This change:
- Takes an existing component, NumberField and enhances that by only allowing numeric input, essentially turning it into a managed input using the same approach as our PasswordField.
- Changes the numeric inputs on category settings page to use this component.
- Adds appropriate min constraints to the fields to disallow out-of-range values.
- Adds missing back-end validations to relevant fields.
* FIX: Remove action buttons if post has already been reviewed
* Change the approve to reject test to expect an error
* Adds a controller spec to ensure you can't edit a non-pending review item
* Remove unnessary conditional
When a CUSTOM_SCHEME is missing a color (e.g. 'Dracula' is missing a 'highlight' color), we need to fallback to `ColorScheme.base_colors`. This regressed in 66256c15bd
This commits adds the ability to add a header to the embedded comments
view. One use case for this is to allow `postMessage` communication
between the comments iframe and the parent frame, for example, when
toggling the theme of the parent webpage.
We caught it in logs, race condition led to this error:
ActiveRecord::RecordNotUnique
(PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "user_statuses_pkey"
DETAIL: Key (user_id)=(15) already exists.)
The reason the problem happened was that we were checking if a user has status and if not inserting status:
if user_status
...
else
self.user_status = UserStatus.create!(status)
end
The problem is that it's possible that another request will insert status just after we check if status exists and just before our request call `UserStatus.create!(status)`. Using `upsert` fixes the problem because under the hood `upsert` generates the only SQL request that uses "INSERT ... ON CONFLICT DO UPDATE". So we do everything in one SQL query, and that query takes care of resolving possible conflicts.
The algorithm failed to find the correct category by slug when there are multiple sub-sub-categories with the same child-category name and the first child doesn't have the correct grandchild.
So, searching for "child / grandchild" worked in the following case, it found (3):
- (1) parent 1
- (2) child
- (3) grandchild
- (4) parent 2
- (5) child
- (6) grandchild
But it failed to find the grandchild in the following case:
- (1) parent 1
- (2) child
- (4) parent 2
- (5) child
- (6) grandchild
And this also fixes a flaky spec by forcing categories to always order by by `parent_category_id` and `id`.
This makes it possible to partly revert 60990aab55
Allows users to configure their own custom sidebar sections with links withing Discourse instance. Links can be passed as relative path, for example "/tags" or full URL.
Only path is saved in DB, so when Discourse domain is changed, links will be still valid.
Feature is hidden behind SiteSetting.enable_custom_sidebar_sections. This hidden setting determines the group which members have access to this new feature.
Currently, when doing `@mention` for users we have 0 tolerance for typos and misspellings.
With this patch, if a user search doesn't return enough results we go and use `pg_trgm` features to try and find more matches based on trigrams of usernames and names.
It also introduces GiST indexes on those fields in order to improve performance of this search, going from 130ms down to 15ms in my tests.
This is all gated in a feature flag and can be enabled by running `SiteSetting.user_search_similar_results = true` in the rails console.
`--d-hover` is calculated to be equivalent to primary-100 in light mode, or primary-low in dark mode
`--d-selected` is calculated to be equivalent to primary-low in light mode, or primary-100 in dark mode
`lib/color_math` is introduced to provide some utilities for making these calculations.
## Why do we need this change?
When loading the ember app, [MessageBus does not start polling immediately](f31f0b70f8/app/assets/javascripts/discourse/app/initializers/message-bus.js (L71-L81)) and instead waits for `document.readyState` to be `complete`. What this means is that if there are new messages being created while we have yet to start polling, those messages will not be received by the client.
With sidebar being the default navigation menu, the counts derived from `topic-tracking-state.js` on the client side is prominently displayed on every page. Therefore, we want to ensure that we are not dropping any messages on the channels that `topic-tracking-state.js` subscribes to.
## What does this change do?
This includes the `MessageBus.last_id`s for the MessageBus channels which `topic-tracking-state.js` subscribes to as part of the preloaded data when loading a page. The last ids are then used when we subscribe the MessageBus channels so that messages which are published before MessageBus starts polling will not be missed.
## Review Notes
1. See https://github.com/discourse/message_bus#client-support for documentation about subscribing from a given message id.
This commits adds a database migration to limit the user status to 100
characters, limits the user status in the UI and makes sure that the
emoji is valid.
Follow up to commit b6f75e231c.
When the `tags_listed_by_group` site setting is disable, we were seeing
the N+1 queries problem when multiple `CategoryTag` records are listed.
This commit fixes that by ensuring that we are not filtering through the
category `tags` association after the association has been eager loaded.
This will allow us more granular control over changing a topic status.
For example you can now force the scope to only allow closing topics in
a specific category. This means that the same scope can't be used to
re-open topics, or close topics in a different category.
This TODO is irrelevant -- in reality this has not been a
perf issue, and there is not actually an N1 here. Furthermore,
this is only used in a single plugin, not in core.
The check used to be necessary because we validated the referrer too and
this bypass was a workaround a bug that is present in some browsers that
do not send the correct referrer.
When creating a group membership request, there is no character
limit on the 'reason' field. This can be potentially be used by
an attacker to create enormous amount of data in the database.
Co-authored-by: Ted Johansson <ted@discourse.org>
Currently we don’t have an association between reviewables and posts.
This sometimes leads to inconsistencies in the DB as a post can have
been deleted but an associated reviewable is still present.
This patch addresses this issue simply by adding a new association to
the `Post` model and by using the `dependent: :destroy` option.
This is just cleaning up a TODO I had to add more specs
to this controller -- there are more thorough tests on the
actual HashtagService class and the type-specific hashtag
classes.
We've had the UploadReference table for some time now in core,
but it was added after ChatUpload was and chat was just never
moved over to this new system.
This commit changes all chat code dealing with uploads to create/
update/delete/query UploadReference records instead of ChatUpload
records for consistency. At a later date we will drop the ChatUpload
table, but for now keeping it for data backup.
The migration + post migration are the same, we need both in case
any chat uploads are added/removed during deploy.
When EmbeddableHost is configured for a specific category and that category is deleted, then EmbeddableHost should be deleted as well.
In addition, migration was added to fix existing data.
Currently, `Tag#topic_count` is a count of all regular topics regardless of whether the topic is in a read restricted category or not. As a result, any users can technically poll a sensitive tag to determine if a new topic is created in a category which the user has not excess to. We classify this as a minor leak in sensitive information.
The following changes are introduced in this commit:
1. Introduce `Tag#public_topic_count` which only count topics which have been tagged with a given tag in public categories.
2. Rename `Tag#topic_count` to `Tag#staff_topic_count` which counts the same way as `Tag#topic_count`. In other words, it counts all topics tagged with a given tag regardless of the category the topic is in. The rename is also done so that we indicate that this column contains sensitive information.
3. Change all previous spots which relied on `Topic#topic_count` to rely on `Tag.topic_column_count(guardian)` which will return the right "topic count" column to use based on the current scope.
4. Introduce `SiteSetting.include_secure_categories_in_tag_counts` site setting to allow site administrators to always display the tag topics count using `Tag#staff_topic_count` instead.
When the `tags_listed_by_group` site setting is enabled, we were seeing
the N+1 queries problem when multiple `TagGroup` records are listed.
This commit fixes that by ensuring that we are not filtering through the
`tags` association after the association has been eager loaded.
The `enable_new_notifications_menu` site setting allows sites that have
`navigation_menu` set to `legacy` to use the redesigned notifications
menu before switching to the new sidebar navigation menu.
0403cda1d1 introduced a regression where
topics in non read-restricted categories have its TopicTrackingState
MessageBus messages published with the `group_ids: [nil]` option. This
essentially means that no one would be able to view the message.
* Firefox now finally returns PerformanceMeasure from performance.measure
* Some TODOs were really more NOTE or FIXME material or no longer relevant
* retain_hours is not needed in ExternalUploadsManager, it doesn't seem like anywhere in the UI sends this as a param for uploads
* https://github.com/discourse/discourse/pull/18413 was merged so we can remove JS test workaround for settings
When a topic belongs to category that is read restricted but permission
has not been granted to any groups, publishing ceratin topic tracking state
updates for the topic will result in the `MessageBus::InvalidMessageTarget` error being raised
because we're passing `nil` to `group_ids` which is not support by
MessageBus.
This commit ensures that for said category above, we will publish the
updates to the admin groups.
This change adds `target` to the set of attributes allowed by the
HTML sanitizer which is applied to the description of a user_field.
The rationale for this change:
* If one puts a link (<a>...</a>) in the description of a user_field
that is present and/or required at sign-up, the expectation is that
a prospective new user will click on that link during sign-up.
* Without an appropriate `target` attribute on the link, the new page
will be loaded in the same window/tab as the sign-up form, but this
will obliterate any fields that the user had already filled-out on
the form. (E.g., hitting the back-button will return to an
empty form.)
* Such UX behavior is incredibly aggravating to new users.
This change allows an admin to add a `target` attribute to links, to
instruct the browser to open them in a different window/tab, leaving
a sign-up form intact.
We previously used post creator's guardian permissions which will raise an error if the reviewer added a staff-only (restricted) tag.
Co-authored-by: Natalie Tay <natalie.tay@discourse.org>
When sending emails out via group SMTP, if we
are sending them to non-staged users we want
to mask those emails with BCC, just so we don't
expose them to anyone we shouldn't. Staged users
are ones that have likely only interacted with
support via email, and will likely include other
people who were CC'd on the original email to the
group.
Co-authored-by: Martin Brennan <martin@discourse.org>
Using a shared channel means that every user receives an update to the 'last_id' when *any* other user is logged out. If many users are being programmatically logged out at the same time, this can cause a very large number of message-bus polls.
This commit switches to use a user-specific channel, which means that each user has its own 'last id' which will only increment when they are logged out
When loading posts in a topic, the topic level guardian
checks are run multiple times even though all the posts belong to the
same topic. Profiling in production revealed that this accounted for a
significant amount of request time for a user that is not staff or anon.
Therefore, we're optimizing this by adding memoizing the topic level
calls in `PostGuardian`. Speficifally, the result of
`TopicGuardian#can_see_topic?` and `PostGuardian#can_create_post?`
method calls are memoized per topic.
Locally profiling shows a significant improvement for normal users
loading a topic with 100 posts.
Benchmark script command: `ruby script/bench.rb --unicorn --skip-bundle-assets --iterations 100`
Before:
```
topic user:
50: 114
75: 117
90: 122
99: 209
topic.json user:
50: 67
75: 69
90: 72
99: 162
```
After:
```
topic user:
50: 101
75: 104
90: 107
99: 184
topic.json user:
50: 53
75: 53
90: 56
99: 138
```
Instead of relying on the `ILIKE` operator to filter out image links, we
can instead rely on the `TopicLink#extension` column which allows us to
more efficiently filter out image links.
This optimization mainly affects topics that are link heavy which is
common in topics with alot of replies. When profiling a production
instance for a topic with 10K replies and 2.5K `topic_links`, this
optimization reduces the query time from ~18ms to around ~4ms.
Featured topics are eventually serialized by `ListableTopicSerializer`
which calls `Topic#image_url` which requires us to preload
`Topic#topic_thumbnails`.
* DEV: Remove enable_whispers site setting
Whispers are enabled as long as there is at least one group allowed to
whisper, see whispers_allowed_groups site setting.
* DEV: Always enable whispers for admins if at least one group is allowed.
This commit adds a new notification that gets sent to admins when the site gets new features after an upgrade/deploy. Clicking on the notification takes the admin to the admin dashboard at `/admin` where they can see the new features under the "New Features" section.
Internal topic: t/87166.
When user is watching category or tag (watching or watching first post) notifications are moved to other tab.
To achieve that and distinguish between post create to directly watched topics and indirectly watched topics, new notification type called `watching_category_or_tag` was introduced.
The guardian is useful for plugins to determine if the callback should
do anything. A common use case is to not do anything in the callback if
the user is anonymous.
* FIX: Use Category.secured(guardian) for hashtag datasource
Follow up to comments in #19219, changing the category
hashtag datasource to use Category.secured(guardian) instead
of Site.new(guardian).categories here since the latter does
more work for not much benefit, and the query time is the
same. Also eliminates some Hash -> Model back and forth
busywork. Add some more specs too.
* FIX: Server-side hashtag lookup cooking user loading
When we were using the PrettyText.options.currentUser
and parsing back and forth with JSON for the hashtag
lookups server-side, we had a bug where the user's
secure categories were not loaded since we never actually
loaded a User model from the database, only parsed it
from JSON.
This commit fixes the issue by instead using the
PretyText.options.userId and looking up the user directly
from the database when calling hashtag_lookup via the
PrettyText::Helpers code when cooking server-side. Added
the missing spec to check for this as well.
This new site setting replaces the
`enable_experimental_sidebar_hamburger` and `enable_sidebar` site
settings as the sidebar feature exits the experimental phase.
Note that we're replacing this without depreciation since the previous
site setting was considered experimental.
Internal Ref: /t/86563
In this PR, we introduced an option, that when all authenticators are disabled, but backup codes still exists, user can authenticate with those backup codes. This was reverted as this is not expected behavior.
https://github.com/discourse/discourse/pull/18982
Instead, when the last authenticator is deleted, backup codes should be deleted as well. Because this disables 2fa, user is asked to confirm that action by typing text.
In addition, UI for 2fa preferences was refreshed.
When finding the candidates for `Topic.similar_to`, we will now ignore
topics in categories where `Category#search_priority` has been set to
ignore and also topics in categories which the user has specifically
muted.
Internal Ref: /t/87132
Note that we don't have a database table and a model for post mentions yet, and I decided to implement it without adding one to avoid heavy data migrations. Still, we may want to add such a model later, that would be convenient, we have such a model for mentions in chat.
Note that status appears on all mentions on all posts in a topic except of the case when you just posted a new post, and it appeared on the bottom of the topic. On such posts, status won't be shown immediately for now (you'll need to reload the page to see the status). I'll take care of it in one of the following PRs.
When a screened IP address is matched because it is either blocked or
allowed it should update match_count. This did not work because it
tried to validate the IP address and it failed as it matched with
itself.
Follow-up to 6357a3ce33
where we allowed a general API key scope for user status
GET/PUT/DELETE, this commit allows the same for the
UserApiKey system.
Previously we would trigger the event before the `Topic#deleted_at`
column has been updated making it hard for plugins to correctly work
with the model when its new state has not been persisted in the
database.
* FIX: Only modify secured sidebar links on user promotion/demotion
If a user is created populate their sidebar with the default
categories/tags that they have access to.
If a user is promoted to admin populate any new categories/tags that
they now have access to.
If an admin is demoted remove any categories/tags that they no longer
have access to.
This will only apply for "secured" categories. For example if these are
the default sitebar categories:
- general
- site feedback
- staff
and a user only has these sidebar categories:
- general
when they are promoted to admin they will only receive the "staff"
category. As this is a default category they didn't previously have
access to.
* Add spec, remove tag logic on update
Change it so that if a user becomes unstaged it used the "add" method
instead of the "update" method because it is essentially following the
on_create path.
On admin promotion/demotion remove the logic for updating sidebar tags because
we don't currently have the tag equivalent like we do for User.secure_categories.
Added the test case for when a user is promoted to admin it should
receive *only* the new sidebar categories they didn't previously have
access to. Same for admin demotion.
* Add spec for suppress_secured_categories_from_admin site setting
* Update tags as well on admin promotion/demotion
* only update tags when they are enabled
* Use new SidebarSectionLinkUpdater
We now have a SidebarSectionLinkUpdater
that was introduced in: fb2507c6ce
* remove empty line
The centralization helps in reducing code duplication in our code base
and more importantly, centralizing logic for guardian checks into a
single spot.
Users who can access the review queue can claim a pending reviewable(s) which means that the claimed reviewable(s) can only be handled by the user who claimed it. Currently, we show claimed reviewables in the user menu, but this can be annoying for other reviewers because they can't do anything about a reviewable claimed by someone. So this PR makes sure that we only show in the user menu reviewables that are claimed by nobody or claimed by the current user.
Internal topic: t/77235.
This PR adds separate notification indicators for PMs and reviewables that have arrived since the last time the user opened the notifications menu.
The PM indicator is the strongest one of all three indicators followed by the reviewable indicator and then finally the blue indicator. This means that if there's a new PM and a new reviewable, then the PM indicator will be shown.
Meta topic: https://meta.discourse.org/t/no-green-or-red-notification-bubbles/242783?u=osama.
Internal topic: t/82995.
While load testing our user creation code path in production, we
identified that executing the DB statement to update the `Group#user_count` column within a
transaction is creating a bottleneck for us. This is because the
creation of a user and addition of the user to the relevant groups are
done in a transaction. When we execute the DB statement to update
`Group#user_count` for the relevant group, a row level lock is held
until the transaction completes. This row level lock acts like a global
lock when the server is creating users that will be added to the same
group in quick succession.
Instead of updating the counter cache within a transaction which the
default ActiveRecord `counter_cache` option does, we simply update the
counter cache outside of the committing transaction.
Co-authored-by: Rafael dos Santos Silva <xfalcox@gmail.com>
Co-authored-by: Rafael dos Santos Silva <xfalcox@gmail.com>
Adds stats for API and user API requests similar to regular page views.
This comes with a new report to visualize API requests per day like the
consolidated page views one.
This commit introduce a new API for registering callbacks, which we'll execute when a user gets destroyed, and the `delete_posts` opt is true. The chat plugin registers one callback and queues a job to destroy every message from that user in batches.
Follow up to 40e8912395
In this previous commit I introduced a bug that prevented
a legitimate case for an existing user to redeem an invite,
where the email/domain were both blank and the invite was
still redeemable by the user. Fixes the issue and adds more
specs for that case.
This adds API scope for the user status. This also adds a get method to the user status controller. We didn't need a dedicated method that returns status before because the server returns status with user objects, but I think we need to provide this method for API clients.
Update `release_notes_link` to current version
<!-- NOTE: All pull requests should have tests (rspec in Ruby, qunit in
JavaScript). If your code does not include test coverage, please include
an explanation of why it was omitted. -->
This commit fleshes out and adds functionality for the new `#hashtag` search and
lookup system, still hidden behind the `enable_experimental_hashtag_autocomplete`
feature flag.
**Serverside**
We have two plugin API registration methods that are used to define data sources
(`register_hashtag_data_source`) and hashtag result type priorities depending on
the context (`register_hashtag_type_in_context`). Reading the comments in plugin.rb
should make it clear what these are doing. Reading the `HashtagAutocompleteService`
in full will likely help a lot as well.
Each data source is responsible for providing its own **lookup** and **search**
method that returns hashtag results based on the arguments provided. For example,
the category hashtag data source has to take into account parent categories and
how they relate, and each data source has to define their own icon to use for the
hashtag, and so on.
The `Site` serializer has two new attributes that source data from `HashtagAutocompleteService`.
There is `hashtag_icons` that is just a simple array of all the different icons that
can be used for allowlisting in our markdown pipeline, and there is `hashtag_context_configurations`
that is used to store the type priority orders for each registered context.
When sending emails, we cannot render the SVG icons for hashtags, so
we need to change the HTML hashtags to the normal `#hashtag` text.
**Markdown**
The `hashtag-autocomplete.js` file is where I have added the new `hashtag-autocomplete`
markdown rule, and like all of our rules this is used to cook the raw text on both the clientside
and on the serverside using MiniRacer. Only on the server side do we actually reach out to
the database with the `hashtagLookup` function, on the clientside we just render a plainer
version of the hashtag HTML. Only in the composer preview do we do further lookups based
on this.
This rule is the first one (that I can find) that uses the `currentUser` based on a passed
in `user_id` for guardian checks in markdown rendering code. This is the `last_editor_id`
for both the post and chat message. In some cases we need to cook without a user present,
so the `Discourse.system_user` is used in this case.
**Chat Channels**
This also contains the changes required for chat so that chat channels can be used
as a data source for hashtag searches and lookups. This data source will only be
used when `enable_experimental_hashtag_autocomplete` is `true`, so we don't have
to worry about channel results suddenly turning up.
------
**Known Rough Edges**
- Onebox excerpts will not render the icon svg/use tags, I plan to address that in a follow up PR
- Selecting a hashtag + pressing the Quote button will result in weird behaviour, I plan to address that in a follow up PR
- Mixed hashtag contexts for hashtags without a type suffix will not work correctly, e.g. #ux which is both a category and a channel slug will resolve to a category when used inside a post or within a [chat] transcript in that post. Users can get around this manually by adding the correct suffix, for example ::channel. We may get to this at some point in future
- Icons will not show for the hashtags in emails since SVG support is so terrible in email (this is not likely to be resolved, but still noting for posterity)
- Additional refinements and review fixes wil
The hidden site setting `suppress_secured_categories_from_admin` will
suppress visibility of categories without explicit access from admins
in a few key areas (category drop downs and topic lists)
It is not intended to be a security wall since admins can amend any site
setting. Instead it is feature that allows hiding the categories from the
UI.
Admins will still be able to see topics in categories without explicit
access using direct URLs or flags.
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
When opening the invite acceptance page when the user
was already logged in, we were still showing the Accept
Invitation prompt even if the user had already redeemed
the invitation and was present in the `InvitedUser` table.
This would lead to errors when the user clicked on the button.
This commit fixes the issue by hiding the Accept Invitation
button and showing an error message instead indicating that
the user had already redeemed the invitation. This only applies
to multi-use invite links.
When this report in the admin dashboard has lots of data ( > 75 days of activity), the dates were ordered incorrectly. This is apparently expected behaviour; when using GROUP BY without specifying the ordering, PG decides to order, and it so happens that it works under some conditions but not others. Explicit ordering fixes the problem.
However, because this works in some conditions but not others, we can't really add a useful test.
* DEV: Add utility to hide all user tips
* DEV: Add UserTip Glimmer component
* DEV: Add tests for existing user tips
* FEATURE: Add user tip for post menu
* FEATURE: Add user tip for topic notification level
* FEATURE: Add user tip for suggested topics
* FEATURE: Hide new popups for existing users
We were doing get on Redis two times for each emoji while building the custom/standard/all lists which where resulting in ~3710 Redis calls. Given the emoji DB file is loaded in memory while we build/cache the emojis list this is unnecessary and slow.
As a simplification in pseudo code here is an explanation of what we were doing:
```ruby
emojis.each |emoji_name|
aliases = get_aliases_from_redis_cache(emoji_name)
is_tonable = get_is_tonable_from_redis_cache(emoji_name)
build_emoji(emoji_name, aliases, is_tonable)
end
```
The two redis calls are now simplified to a simple hash access: `@db[emoji_name]`
This commit adds some protections in InviteRedeemer to ensure that email
can never be nil, which could cause issues with inviting the invited
person to private topics since there was an incorrect inner join.
If the email is nil and the invite is scoped to an email, we just use
that invite.email unconditionally. If a redeeming_user (an existing
user) is passed in when redeeming an email, we use their email to
override the passed in email. Otherwise we just use the passed in
email. We now raise an error after all this if the email is still nil.
This commit also adds some tests to catch the private topic fix, and
some general improvements and comments around the invite code.
This commit also includes a migration to delete TopicAllowedUser records
for users who were mistakenly added to topics as part of the invite
redemption process.
Currently, we have available three 2fa methods:
- Token-Based Authenticators
- Physical Security Keys
- Two-Factor Backup Codes
If the first two are deleted, user lose visibility of their backup codes, which suggests that 2fa is disabled.
However, when they try to authenticate, the account is locked, and they have to ask admin to fix that problem.
This PR is fixing the issue. User still sees backup codes in their panel and can use them to authenticate.
In next PR, I will improve UI to clearly notify the user when 2fa is fully disabled and when it is still active.
This commit automatically ensures that category channels
have slugs when they are created or updated based on the
channel name, category name, or existing slug. The behaviour
has been copied from the Category model.
We also include a backfill here with a simplified version
of Slug.for with deduplication to fill the slugs for already
created Category chat channels.
The channel slug is also now used for chat notifications,
and for the UI and navigation for chat. `slugifyChannel`
is still used, but now does the following fallback:
* Uses channel.slug if it is present
* Uses channel.escapedTitle if it is present
* Uses channel.title if it is present
In future we may want to remove this altogether
and always rely on the slug being present, but this
is currently not possible because we are not generating
slugs for DM channels at this point.
* Remove old bookmark column ignores to follow up b22450c7a8
* Change some group site setting checks to use the _map helper
* Remove old secure_media helper stub for chat
* Change attr_accessor to attr_reader for preloaded_custom_fields to follow up 70af45055a
It is likely that a new admin user was created as just a regular user
before being promoted to admin so this change will update the sidebar
link records for any users that are promoted to admin. This way if any
of the default side bar categories or tags are restricted to admins
these new admins will have those added to their sidebar as well.
You can easily replicate this issue locally (prior to this fix) by using
`rails admin:create` where it creates a user first, then it is promoted
to admin. This means it would receive the default categories of regular
user, but never receive the ones they should have access to as an admin.
As part of this change I did drop the `!` from
`SidebarSectionLink.insert_all` so that it would add any new records
that were missing, but not throw a unique constraint error trying to add
any existing records.
Follow up to: 1b56a55f50
And: e320bbe513
Previously, we didn't have a site-wide setting to set the default behavior for user profile visibility and user presence features. But we already have a user preference for that.
This commit fixes a regression introduced in 8979adc where under certain conditions the groups syncing logic in Discourse Connect would try to add users to groups they're already members of and cause errors when users try to sign in using Discourse Connect.
The previous sidebar default tags and categories implementation did not
allow for a user to configure their sidebar to have no categories or
tags. This commit changes how the defaults are applied. When a user is being created,
we create the SidebarSectionLink records based on the `default_sidebar_categories` and
`default_sidebar_tags` site settings. SidebarSectionLink records are
only created for categories and tags which the user has visibility on at
the point of user creation.
With this change, we're also adding the ability for admins to apply
changes to the `default_sidebar_categories` and `default_sidebar_tags`
site settings historically when changing their site setting. When a new
category/tag has been added to the default, the new category/tag will be
added to the sidebar for all users if the admin elects to apply the changes historically.
Like wise when a tag/category is removed, the tag/category will be
removed from the sidebar for all users if the admin elects to apply the
changes historically.
Internal Ref: /t/73500
Before this commit, there was no way for us to efficiently check an
array of topics for which a user can see. Therefore, this commit
introduces the `TopicGuardian#can_see_topic_ids` method which accepts an
array of `Topic#id`s and filters out the ids which the user is not
allowed to see. The `TopicGuardian#can_see_topic_ids` method is meant to
maintain feature parity with `TopicGuardian#can_see_topic?` at all
times so a consistency check has been added in our tests to ensure that
`TopicGuardian#can_see_topic_ids` returns the same result as
`TopicGuardian#can_see_topic?`. In the near future, the plan is for us
to switch to `TopicGuardian#can_see_topic_ids` completely but I'm not
doing that in this commit as we have to be careful with the performance
impact of such a change.
This method is currently not being used in the current commit but will
be relied on in a subsequent commit.
Discourse Connect can be used to manage group memberships of users by including a `add_groups`, `remove_groups` or `groups` attribute in the Discourse Connect payload. However, additions/deletions of users from groups aren't logged to the groups logs (available at `/g/<group>/manage/logs`) which can cause confusions to admins they try to figure out when/how users were added or removed from a group. This commit makes Discourse Connect add entries to the groups logs when it makes changes to users' group memberships.
Our theme system injects a magical `settings` object at the top of themes JS modules to allow theme authors to access the settings as configured by admins in the UI. Within this `settings` object, there are a couple of special objects `theme_uploads` and `theme_uploads_local` that contain URLs for all the assets/uploads that the theme has.
For test modules/files, the theme system also injects a `settings` object at the top of tests modules, but it's not the same object as the object that's injected in non-test files. The difference is that in tests we want the settings to have their default values as opposed to any custom values that may exist in the site's database. This ensures that test results are consistent no matter the site that runs them.
However, the `settings` object in tests files currently doesn't have the special objects `theme_uploads` and `theme_uploads_local` which means that if a theme includes an asset that's lazy-loaded, it's not possible to write tests for anything that depends on the lazy-loaded asset because the theme will not be able to load the asset during the tests since `theme_uploads_local` and `theme_uploads` don't exist. This PR adds these special objects inside the `settings` object for test files.
Internal topic: t/71825/52.
Theme javascript is now minified using Terser, just like our core/plugin JS bundles. This reduces the amount of data sent over the network.
This commit also introduces sourcemaps for theme JS. Browser developer tools will now be able show each source file separately when browsing, and also in backtraces.
For theme test JS, the sourcemap is inlined for simplicity. Network load is not a concern for tests.
Previously, compiling theme 'extra_js' was done with a number of steps. Each theme_field would be compiled into its own value_baked column, and then the JavascriptCache content would be built by concatenating all of those compiled values.
This commit streamlines things by removing the value_baked step. The raw value of all extra_js theme_fields are passed directly to the ThemeJavascriptCompiler, and then the result is stored in the JavascriptCache.
In itself, this commit should not cause any behavior change. It is designed to open the door to more advanced compilation features which have interdependencies between different source files (e.g. template colocation, sourcemaps).
This commit fixes an issue where we had a typo in the
UserAction.stream query which meant that action_code_path
was not loaded correctly. Once that was fixed, we were also
not actually using the action_code_path in the user-stream-item,
so that has been fixed here too.
The bug this caused was that, when the link for the action was
clicked within the user-stream-item, the user would be redirected
to a URL ending with `[missing%20%%7Bpath%7D%20value]` because
the I18n call did not have the path present.
* FIX: Reset related site settings on general category delete
If the new seeded General category is deleted we also need to delete the
corresponding site setting for it so that we don't try and reference it.
This fixes a bug in the category dropdown composer.
This change creates the `clear_related_site_settings` after destroy
hook that could also be used by other features in the future, like maybe
when we have a `default_category_id` site_setting.
Looks like if `nil` out a site setting it is set to `0`?
```
[9] pry(main)> SiteSetting.general_category_id = nil
SiteSetting Load (0.4ms) SELECT "site_settings".* FROM "site_settings" WHERE "site_settings"."name" = 'general_category_id' LIMIT 1
=> nil
[10] pry(main)> SiteSetting.general_category_id
=> 0
```
That is why the tests check if the value is `< 1` and not `nil`.
* Use -1 instead of nil because it is the default
This commit introduces a new framework for building user tutorials as
popups using the Tippy JS library. Currently, the new framework is used
to replace the old notification spotlight and tips and show a new one
related to the topic timeline.
All popups follow the same structure and have a title, a description and
two buttons for either dismissing just the current tip or all of them
at once.
The state of all seen popups is stored in a user option. Updating
skip_new_user_tips will automatically update the list of seen popups
accordingly.
The previous fix in e83d35d6 was incorrect, and the stub in the test was never actually hit. This commit moves the error handling to the right place and updates the specs to ensure the stub is always used.
This can no longer be used from the user interface and could be used to
generate useless topic invites notifications. This commit adds site
setting max_topic_invitations_per_minute to prevent invite spam.
Adds a new upload field for a second dark mode category logo.
This alternative will be used when the browser is in dark mode (similar to the global site setting for a dark logo).
These errors tend to indicate that the upload is missing on the remote store. This is bad, but we don't want it to block the dominant-color calculation process. This commit catches errors when there is an HTTP error, and fixes the `base_store.rb` implementation when `FileHelper.download` returns nil.
Previously, when categories were not muted by default, we were sending message about unmuted topics (topics which user explicitly set notification level to watching)
The same mechanism can be used to fix a bug. When the user was explicitly watching topic, but category was muted, then the user was not informed about new reply.
This commit makes pending reviewables show up in the main tab (a.k.a. "all notifications" tab). Pending reviewables along with unread notifications are always shown first and they're sorted based on their creation date (most recent comes first).
The dismiss button currently only shows up if there are unread notifications and it doesn't dismiss pending reviewables. We may follow up with another change soon that allows makes the dismiss button work with reviewables and remove them from the list without taking any action on them.
Follow-up to 079450c9e4.
This commit adds non-archived group messages and `group_message_summary` notifications in the messages tab in the user menu. With this change, the messages tab in the user menu now includes 3 types of items:
1. Unread `private_message` notifications (notifications when you receive a reply in a PM)
2. Unread and read `group_message_summary` notifications (notifications when there's a new message in a group inbox that you track)
3. Non-archived personal and group messages
Unread `private_message` notifications are always shown first, followed by unread `group_message_summary` notifications, and then everything else (messages and read `group_message_summary` notifications) sorted by recency (most recent first).
Internal topic: t/72976.
Currently, the reviewables tab in the user menu shows pending reviewables at the top of the menu and fills the remaining space in the menu with old/handled reviewables. This PR makes the revieables tab show only pending reviewables and hides the tab altogether from the menu if there are no pending reviewables. We're going to follow-up with another change soon that will show pending reviewables in the main tab of the user menu.
Internal topic: t/73220.
This commit renames all secure_media related settings to secure_uploads_* along with the associated functionality.
This is being done because "media" does not really cover it, we aren't just doing this for images and videos etc. but for all uploads in the site.
Additionally, in future we want to secure more types of uploads, and enable a kind of "mixed mode" where some uploads are secure and some are not, so keeping media in the name is just confusing.
This also keeps compatibility with the `secure-media-uploads` path, and changes new
secure URLs to be `secure-uploads`.
Deprecated settings:
* secure_media -> secure_uploads
* secure_media_allow_embed_images_in_emails -> secure_uploads_allow_embed_images_in_emails
* secure_media_max_email_embed_image_size_kb -> secure_uploads_max_email_embed_image_size_kb
In the past, CategoryFeaturedTopic.feature_topics raised an exception
sometimes because it tried to create multiple CategoryFeaturedTopic
records for the same topic.
This code should not raise any exceptions as long as the list of new
topic IDs is unique because all previous records are deleted first,
then recreated and everything happens inside a transaction. The previous
rescue block was dead code anyway because it tried to catch
PG::UniqueViolation instead of ActiveRecord::RecordNotUnique. This
commit includes a fix to ensure that the topic IDs are unique.
This will replace `enable_personal_messages` and
`min_trust_to_send_messages`, this commit introduces
the setting `personal_message_enabled_groups`
and uses it in all places that `enable_personal_messages`
and `min_trust_to_send_messages` currently apply.
A migration is included to set `personal_message_enabled_groups`
based on the following rules:
* If `enable_personal_messages` was false, then set
`personal_message_enabled_groups` to `3`, which is
the staff auto group
* If `min_trust_to_send_messages` is not default (1)
and the above condition is false, then set the
`personal_message_enabled_groups` setting to
the appropriate auto group based on the trust level
* Otherwise just set `personal_message_enabled_groups` to
11 which is the TL1 auto group
After follow-up PRs to plugins using these old settings, we will be
able to drop the old settings from core, in the meantime I've added
DEPRECATED notices to their descriptions and added them
to the deprecated site settings list.
This commit also introduces a `_map` shortcut method definition
for all `group_list` site settings, e.g. `SiteSetting.personal_message_enabled_groups`
also has `SiteSetting.personal_message_enabled_groups_map` available,
which automatically splits the setting by `|` and converts it into
an array of integers.
See https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499
for thorough reasoning.
This commit changes how we generate Message-IDs and do email
threading for emails sent from Discourse. The main changes are
as follows:
* Introduce an outbound_message_id column on Post that
is either a) filled with a Discourse-generated Message-ID
the first time that post is used for an outbound email
or b) filled with an original Message-ID from an external
mail client or service if the post was created from an
incoming email.
* Change Discourse-generated Message-IDs to be more consistent
and static, in the format `discourse/post/:post_id@:host`
* Do not send References or In-Reply-To headers for emails sent
for the OP of topics.
* Make sure that In-Reply-To is filled with either a) the OP's
Message-ID if the post is not a direct reply or b) the parent
post's Message-ID
* Make sure that In-Reply-To has all referenced post's Message-IDs
* Make sure that References is filled with a chain of Message-IDs
from the OP down to the parent post of the new post.
We also are keeping X-Discourse-Post-Id and X-Discourse-Topic-Id,
headers that we previously removed, for easier visual debugging
of outbound emails.
Finally, we backfill the `outbound_message_id` for posts that have
a linked `IncomingEmail` record, using the `message_id` of that record.
We do not need to do that for posts that don't have an incoming email
since they are backfilled at runtime if `outbound_message_id` is missing.
The `add_column` `limit` parameter has no effect on a postgres `text` column. Instead we can perform the check in ActiveRecord.
We never expect this condition to be hit - users cannot control this value. It's just a safety net.
Adds limits to location and website fields at model and DB level
to match the bio_raw field limits. A limit cannot be added at the
DB level for bio_raw because it is a postgres text field.
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
We previously had a system which would generate a 10x10px preview of images and add their URLs in a data-small-upload attribute. The client would then use that as the background-image of the `<img>` element. This works reasonably well on fast connections, but on slower connections it can take a few seconds for the placeholders to appear. The act of loading the placeholders can also break or delay the loading of the 'real' images.
This commit replaces the placeholder logic with a new approach. Instead of a 10x10px preview, we use imagemagick to calculate the average color of an image and store it in the database. The hex color value then added as a `data-dominant-color` attribute on the `<img>` element, and the client can use this as a `background-color` on the element while the real image is loading. That means no extra HTTP request is required, and so the placeholder color can appear instantly.
Dominant color will be calculated:
1. When a new upload is created
2. During a post rebake, if the dominant color is missing from an upload, it will be calculated and stored
3. Every 15 minutes, 25 old upload records are fetched and their dominant color calculated and stored. (part of the existing PeriodicalUpdates job)
Existing posts will continue to use the old 10x10px placeholder system until they are next rebaked
Right now the experimental user menu sorts notifications the same way that the old menu does: unread high-priority notifications are shown first in reverse-chronological order followed by everything else also in reverse-chronological order. However, since the experimental user menu has dedicated tabs for some notification types and each tab displays a badge with the count of unread notifications in the tab, we feel like it makes sense to change how notifications are sorted in the experimental user menu to this:
1. unread high-priority notifications
2. unread regular notifications
3. all read notifications (both high-priority and regular)
4. within each group, notifications are sorted in reverse-chronological order (i.e. newest is shown first).
This new sorting logic applies to all tabs in the experimental user menu, however it doesn't change anything in the old menu. With this change, if a tab in the experimental user menu shows an unread notification badge for a really old notification, it will be surfaced to the top and prevents confusing scenarios where a user sees an unread notification badge on a tab, but the tab doesn't show the unread notification because it's too old to make it to the list.
Internal topic: t72199.
Previously we were relying on a highly-customized version of the unmaintained Barber gem for theme template compilation. This commit switches us to use our own DiscourseJsProcessor, which makes use of more modern patterns and will be easier to maintain going forward.
In summary:
- Refactors DiscourseJsProcessor to move multiline JS heredocs into a companion `discourse-js-processor.js` file
- Use MiniRacer's `.call` method to avoid manually escaping JS strings
- Move Theme template AST transformers into DiscourseJsProcessor, and formalise interface for extending RawHandlebars AST transformations
- Update Ember template compilation to use a babel-based approach, just like Ember CLI. This gives each template its own ES6 module rather than directly assigning `Ember.TEMPLATES` values
- Improve testing of template compilation (and move some tests from `theme_javascript_compiler_spec.rb` to `discourse_js_processor_spec.rb`
It used to return the next URL anyway which lead to an additional
request. On the frontend, if the result set was empty, it kept retrying
until at least one result was returned. This bug is fixed in this commit
too.
* FIX: hide welcome topic banner as soon as the welcome topic is edited
This commit adds a message bus listener on client to hide the welcome
topic banner as soon as the welcome topic is edited.
* update test
* only subscribe when show_welcome_topic_banner is true
* Do not lookup for messageBus service if it's not required
* Remove unneeded code
* Cache result for Site.show_welcome_topic_banner
* Update tests per latest changes
* Changes per PR review
Each new user menu notifications should have their own count. Therefore, we need to include all types to serializer and not only `grouped_unread_high_priority_notifications`
Additional PR will be created for chat and assign plugin, as they will have to switch to `grouped_unread_notifications` as well.
The source-of-truth for our ember version is now the installed node_module. The `ember_source` gem carries an old version of Ember and so the constant is no longer useful. We'll be dropping the gem soon.
This commit makes a number of improvements to the DiscourseJsProcessor:
1. Remove dependence on the out-of-date Ember template compiler from the ember-rails gem; switch to modern template compiler
2. Refactor to make use of a proper module system with `define`/`require`
3. Introduce `babel-plugin-ember-template-compilation` to enable inline hbs compilation
The `mini-loader` is upgraded to support relative lookup and `require.has`, so that these new JS packages work correctly.
If a topic is converted to a private message, all posters were invited
to the new private message. This included users who only whispered or
posted small actions.
Topic allowed user records were created for small actions, which lead to
the system user being invited in many private topics when the user
removed themselves or if a group was invited but some members already
had access.
This commits skips creating topic allowed user. They are already skipped
for the whisper posts.
If a user was granted a trust level, joined a group that granted a trust
level and left the group, the trust level was reset. This commit tries
to restore the last known trust level before joining the group by
looking into staff logs.
This commit also migrates old :change_trust_level user history records
to use previous_value and new_value fields.
When preloading topic_list data we were giving it a 'preload key' which was loosely based on the parameters of the list. However, it did not include all parameters, and mismatches between client/server-side logic would cause the preloaded data to be ignored.
This commit simplifies things by using a single key for all topic_list preloading. This works on the assumption that "The first topic_list the JS app will load is the one which was preloaded". That assumption also existed to some extent in the old design, so we don't expect any regressions here.
Default sidebar tags for not authenticated users can be defined in admin panel. Otherwise, top 5 categories and tags are taken.
Optionally, if categories are set up in permanent order, then the first 5 categories are taken.
```
1) CurrentUserSerializer#sidebar_category_ids includes visible default sidebar categories
Failure/Error: expect(json[:sidebar_category_ids]).to eq([category.id, category_2.id])
expected: [378, 379]
got: [379, 378]
```
Note that in the Ruby doc it says "The order is preserved from the original array". In this case, we want to preserve the order of the site setting.
Handles edge-case when a user is an admin and has an associated reviewable. Hitting this exception should be rare since we clear the reviewable when
granting staff to the user.
A public key must be added to GitHub when installing private themes.
When the process happens asynchronously (for example if the admin does
not have admin permissions to the GitHub repository), installing
private themes becomes very difficult.
In this case, the Discourse admin can partially install the theme by
letting Discourse save the private key, create a placeholder theme and
give the admin a public key to be used as a deploy key. After the key
is installed, the admin can finish theme installation by pressing a
button on the theme page.
Hard deleting topics that contained soft deleted posts or small actions
used to create orphan posts because only the first post was hard
deleted. This commit adds an error message if there are still posts left
in the topic that must be hard deleted first or hard deletes all small
actions too immediately (there is no other way of hard deleting a small
action because there is no wrench menu).
Some of the changes in this PR are extracted from https://github.com/discourse/discourse/pull/17379.
Similar to the bookmarks tab in the new user menu, the messages tab also displays a mix of notifications and messages. When there are unread message notifications, the tab displays all of these notifications at the top and fills the remaining space in the menu with a list of the user's messages. The bubble/badge count on the messages tab indicates how many unread message notifications there are.
This is a much better description of its function. It performs idempotent normalization of a URL. If consumers truly need to `encode` a URL (including double-encoding of existing encoded entities), they can use the existing `.encode` method.
Some of the changes in this commit are extracted from https://github.com/discourse/discourse/pull/17379.
The bookmarks tab in the new user menu is different from the other tabs in that it can display a mixture of notifications and bookmarks. When there are unread bookmark reminder notifications, the tab displays all of these notifications at the top and fills the remaining space in the menu with the rest of the bookmarks. The bubble/badge count on the bookmarks tab indicates how many unread bookmark reminder notifications there are.
On the technical aspect, since this commit introduces a new `bookmark-item` component, we've done some refactoring so that all 3 "item" components (`notification-item`, `reviewable-item` and the new `bookmark-item`) inherit from a base component and get identical HTML structure so they all look consistent.
Internal tickets: t70584 and t65045.
Currently we can’t add a case-sensitive watched word if another one
exists with a different case. For example, the existing watched word
`Meta` has been created and is case-sensitive. Now an admin tries to add
`metA` while marking it as case-sensitive too, this won’t work and the
word won’t be added.
This patch changes this behavior by allowing to add same words that have
different cases, so the example above will now work as expected.
We still check for uniqueness but case-sensitivy is now taken
into account. It means that if the watched word `meta` already exists
and is not case-sensitive then it will not be possible to add `Meta`
(case-sensitive or not) as `meta` already matches every possible
variations of this word.
Follow-up to ce9eec8606.
I did a last-minute refactoring before merging the commit above where I extracted the Message Bus publish call into a new method, but forgot to delete the publish call after adding a call to the new method.
The content from the remote site and the footer get cached for 10 minutes, so Discourse should use the default locale instead of the user locale for the footer. Otherwise Discourse might cache the message in a different language.
* FEATURE: Add case-sensitivity flag to watched_words
Currently, all watched words are matched case-insensitively. This flag
allows a watched word to be flagged for case-sensitive matching.
To allow allow for backwards compatibility the flag is set to false by
default.
* FEATURE: Support case-sensitive creation of Watched Words via API
Extend admin creation and upload of Watched Words to support case
sensitive flag. This lays the ground work for supporting
case-insensitive matching of Watched Words.
Support for an extra column has also been introduced for the Watched
Words upload CSV file. The new column structure is as follows:
word,replacement,case_sentive
* FEATURE: Enable case-sensitive matching of Watched Words
WordWatcher's word_matcher_regexp now returns a list of regular
expressions instead of one case-insensitive regular expression.
With the ability to flag a Watched Word as case-sensitive, an action
can have words of both sensitivities.This makes the use of the global
Regexp::IGNORECASE flag added to all words problematic.
To get around platform limitations around the use of subexpression level
switches/flags, a list of regular expressions is returned instead, one for each
case sensitivity.
Word matching has also been updated to use this list of regular expressions
instead of one.
* FEATURE: Use case-sensitive regular expressions for Watched Words
Update Watched Words regular expressions matching and processing to handle
the extra metadata which comes along with the introduction of
case-sensitive Watched Words.
This allows case-sensitive Watched Words to matched as such.
* DEV: Simplify type casting of case-sensitive flag from uploads
Use builtin semantics instead of a custom method for converting
string case flags in uploaded Watched Words to boolean.
* UX: Add case-sensitivity details to Admin Watched Words UI
Update Watched Word form to include a toggle for case-sensitivity.
This also adds support for, case-sensitive testing and matching of Watched Word
in the admin UI.
* DEV: Code improvements from review feedback
- Extract watched word regex creation out to a utility function
- Make JS array presence check more explicit and readable
* DEV: Extract Watched Word regex creation to utility function
Clean-up work from review feedback. Reduce code duplication.
* DEV: Rename word_matcher_regexp to word_matcher_regexp_list
Since a list is returned now instead of a single regular expression,
change `word_matcher_regexp` to `word_matcher_regexp_list` to better communicate
this change.
* DEV: Incorporate WordWatcher updates from upstream
Resolve conflicts and ensure apply_to_text does not remove non-word characters in matches
that aren't at the beginning of the line.
Relative URLs will work just fine for Web Workers, which were the original reason for introducing the `theme_uploads_local` feature
Making them relative will mean that `loadScript()` automatically uses the CDN (when enabled), which is doubly important because our CSP doesn't allow loading theme-javascripts from the host domain when the CDN is enabled.
When viewing a topic, we execute two queries to fetch the topic's
public topic timer and slow mode timer. The former query happens to be
able to use a unique index but the latter has to do a seq scan which is
slow. The query itself is not expensive but since viewing a topic is a
hot path, the little cuts add up overtime and the query itself
contributes significantly to the load of the database.
This commit removes the ability to enable/disable the Sidebar on a per
user basis and introduces a site wide setting. For testing purposes, sidebar can be enabled/disabled via the `enable_sidebar=1` or `enable_sidebar=0` query param.
The previous method for reused the PrettyText logic which applied the
watched word logic, but had the unwanted effect of cooking the text too.
This meant that regular text values were converted to HTML.
Follow up to commit 5a4c35f627.
Our theme system is very complex and it can take a while to figure out how to invalidate the various types of caches that are used throughout the theme system. So, having a single helper method that invalidates everything can be useful in emergency situations where there is no time to read through the code and figure out how to clear the various caches.
Internal ticket: t64732.
The invites should be redeemed during the signup process. This was a
problem because when user tried to redeem an admin invite it tried to
authenticate the user using information from the session that was not
available.
This commit introduces a new plugin API to register
a group of stats that will be included in about.json
and also conditionally in the site about UI at /about.
The usage is like this:
```ruby
register_about_stat_group("chat_messages", show_in_ui: true) do
{
last_day: 1,
"7_days" => 10,
"30_days" => 100,
count: 1000,
previous_30_days: 120
}
end
```
In reality the stats will be generated any way the implementer
chooses within the plugin. The `last_day`, `7_days`, `30_days,` and `count`
keys must be present but apart from that additional stats may be added.
Only those core 4 stat keys will be shown in the UI, but everything will be shown
in about.json.
The stat group name is used to prefix the stats in about.json like so:
```json
"chat_messages_last_day": 2322,
"chat_messages_7_days": 2322,
"chat_messages_30_days": 2322,
"chat_messages_count": 2322,
```
The `show_in_ui` option (default false) is used to determine whether the
group of stats is shown on the site About page in the Site Statistics
table. Some stats may be needed purely for reporting purposes and thus
do not need to be shown in the UI to admins/users. An extension to the Site
serializer, `displayed_about_plugin_stat_groups`, has been added so this
can be inspected on the client-side.
* FIX: properly validate multiselect user fields on user creation
* Add test cases
* FIX: don't check multiselect user fields for watched words
* Clarifiy/simplify tests
* Roll back apply_watched_words changes
Since this method no longer needs to deal with arrays for now. If/when
we add new user fields which uses them, we can deal with it then.
All `DistributedCache` instances in Discourse are automatically keyed on the `Discourse.git_version`. Normally the theme compiler version is updated via a commit, and everything is fine. However, in some situations, it's possible for the BASE_COMPILER_VERSION to change without a change to the git_version (e.g. when applying patches directly to the codebase).
This commit adds the `BASE_COMPILER_VERSION` to the DistributedCache key to ensure that content from different compiler versions does not leak into other processes.
The `unread_not_too_old` attribute is a little odd because there should never be a case where
the user's first_unread_at column is less than the `Topic#updated_at`
column of an unread topic. The `unread_not_too_old` attribute is causing
a bug where topic states synced into `TopicTrackingState` do not appear
as unread because the attribute does not exsist on a normal `Topic`
object and hence never set.
It makes more sense to use user_ids for the UserCommScreener
introduced in fa5f3e228c since
in most cases the ID will be available, not the username. This
was discovered while starting work on a plugin that will
use this. In the cases where only usernames are available
the extra query is negligble.
The idea behind this refactor is to centralise all of the user ignoring / muting / disallow PM checks in a single place, so they can be used consistently in core as well as for plugins like chat, while improving the main bulk of the checks to run in a single fast non-AR query.
Also fixed up the invite error when someone is muting/ignoring the user that is trying to invite them to the topic.
Currently we only apply watched words of the `Block` type to custom user
fields and user profile fields.
This patch enables all rules to be applied such as `Censor` or
`Replace`.
Before, whispers were only available for staff members.
Config has been changed to allow to configure privileged groups with access to whispers. Post migration was added to move from the old setting into the new one.
I considered having a boolean column `whisperer` on user model similar to `admin/moderator` for performance reason. Finally, I decided to keep looking for groups as queries are only done for current user and didn't notice any N+1 queries.
Updates automatically data on the stats section of the topic.
It will update automatically the following information: likes, replies and last reply (timestamp and user)
We have a `cleanup!` class method on bookmarks that deletes
bookmarks X days after their related record (post/topic) are
deleted. This commit changes this method to use the
registered_bookmarkables for this instead, and each bookmarkable
type can delete related bookmarks in their own way.
In certain situations, a logged in user can redeem an invite with an email that
either doesn't match the invite's email or does not adhere to the email domain
restriction of an invite link. The impact of this flaw is aggrevated
when the invite has been configured to add the user that accepts the
invite into restricted groups.
At some point in the past we decided to rename the 'regular' notification state of topics/categories to 'normal'. However, some UI copy was missed when the initial renaming was done so this commit changes the spots that were missed to the new name.
This is pre-request work to introduce a splash screen while site assets load.
The only change this commit introduces is that it ensures we add the defer attribute to core/plugin/theme .JS files. This will allow us to insert markup before the browser starts evaluating those scripts later on. It has no visual or functional impact on core.
This will not have any impact on how themes and plugins work. The only exception is themes loading external scripts in the </head> theme field directly via script tags. Everything will work the same but those would need to add the defer attribute if they want to keep the benefits introduced in this PR.
This reverts commit 94c3bbc2d1.
At this current point in time, we do not have enough data on whether
this centralisation is the trade-offs of coupling features into a single
channel.
When sending emails with delivery_method_options -> return_response
set to true, the SMTP sending code inside Mail will return the SMTP
response when calling deliver! for mail within the app. This commit
ensures that Email::Sender captures this response if it is returned
and stores it against the EmailLog created for the sent email.
A follow up PR will make this visible within the admin email UI.
The query is very inefficient without any constraints on large sites and
the average of all time to first response since the beginning of time is
not useful as well.
We do not zero-pad our base62 short URLs, so there is no guarantee that the length is 27. Instead, let's greedily match all consecutive base62 characters and look for a matching upload.
This reverts bd32656157 and 36f5d5eada.
This table holds associations between uploads and other models. This can be used to prevent removing uploads that are still in use.
* DEV: Create upload_references
* DEV: Use UploadReference instead of PostUpload
* DEV: Use UploadReference for SiteSetting
* DEV: Use UploadReference for Badge
* DEV: Use UploadReference for Category
* DEV: Use UploadReference for CustomEmoji
* DEV: Use UploadReference for Group
* DEV: Use UploadReference for ThemeField
* DEV: Use UploadReference for ThemeSetting
* DEV: Use UploadReference for User
* DEV: Use UploadReference for UserAvatar
* DEV: Use UploadReference for UserExport
* DEV: Use UploadReference for UserProfile
* DEV: Add method to extract uploads from raw text
* DEV: Use UploadReference for Draft
* DEV: Use UploadReference for ReviewableQueuedPost
* DEV: Use UploadReference for UserProfile's bio_raw
* DEV: Do not copy user uploads to upload references
* DEV: Copy post uploads again after deploy
* DEV: Use created_at and updated_at from uploads table
* FIX: Check if upload site setting is empty
* DEV: Copy user uploads to upload references
* DEV: Make upload extraction less strict
This reverts one of the changes introduced just now in:
27d7b0c6de
I don't think we need this `activated_not_suspended_not_staged` scope
because we can just compose it ourselves via method chaining like
`User.activated.not_suspended.not_staged`.
This commit introduces a new site setting: `block_hotlinked_media`. When enabled, all attempts to hotlink media (images, videos, and audio) will fail, and be replaced with a linked placeholder. Exceptions to the rule can be added via `block_hotlinked_media_exceptions`.
`download_remote_image_to_local` can be used alongside this feature. In that case, hotlinked images will be blocked immediately when the post is created, but will then be replaced with the downloaded version a few seconds later.
This implementation is purely server-side, and does not impact the composer preview.
Technically, there are two stages to this feature:
1. `PrettyText.sanitize_hotlinked_media` is called during `PrettyText.cook`, and whenever new images are introduced by Onebox. It will iterate over all src/srcset attributes in the post HTML and check if they're allowed. If not, the attributes will be removed and replaced with a `data-blocked-hotlinked-src(set)` attribute
2. In the `CookedPostProcessor`, we iterate over all `data-blocked-hotlinked-src(set)` attributes and check whether we have a downloaded version of the media. If yes, we update the src to use the downloaded version. If not, the entire media element is replaced with a placeholder. The placeholder is labelled 'external media', and is a link to the offsite media.
* FIX: Email Send post has already been taken error
Adding a failing test first before coming up with a good solution.
Related: 357011eb3b
The above commit changed
```
PostReplyKey.find_or_create_by_safe!
```
to
```
PostReplyKey.create_or_find_by!
```
But I don't think it is working as a 1-1 replacement because of the
`Validation failed: Post has already been taken` error we are receiving
with this change. Also we need to make sure we don't re-introduce any
concurrency issues.
Reported: https://meta.discourse.org/t/224706/13
* Remove rails unique constraint and rely on db index
I believe this is what is causing `create_or_find_by!` to fail. Because
we have a unique constraint in the db I think we can remove this rails
unique constraint?
* clean up spec wording
This commit resolves a bug where users are not auto approved based on
`SiteSetting.auto_approve_email_domains` when
`SiteSetting.must_approve_users` has been enabled.
When a site has `SiteSetting.invite_only` enabled, we create a
`ReviewableUser`record when activating a user if the user is not
approved. Therefore, we need to approve the user when redeeming an
invite.
There are some uncertainties surrounding why a `ReviewableRecord` is
created for a user in an invites only site but this commit does not seek
to address that.
Follow-up to 7c4e2d33fa
This security fix affects sites which have `SiteSetting.must_approve_users`
enabled. There are intentional and unintentional cases where invited
users can be auto approved and are deemed to have skipped the staff approval process.
Instead of trying to reason about when auto-approval should happen, we have decided that
enabling the `must_approve_users` setting going forward will just mean that all new users
must be explicitly approved by a staff user in the review queue. The only case where users are auto
approved is when the `auto_approve_email_domains` site setting is used.
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
Previously this mapping of **cooked** images was only being run for oneboxes. Now it runs for all images, so we can transform hotlinked images without needing to immediately update `raw`
This commit migrates all bookmarks to be polymorphic (using the
bookmarkable_id and bookmarkable_type) columns. It also deletes
all the old code guarded behind the use_polymorphic_bookmarks setting
and changes that setting to true for all sites and by default for
the sake of plugins.
No data is deleted in the migrations, the old post_id and for_topic
columns for bookmarks will be dropped later on.
- Only validate if custom_fields are loaded, so that we don't trigger a db query
- Only validate public user fields, not all custom_fields
This commit also reverts the unrelated spec changes in ba148e08, which were required to work around these issues
Currently we don’t apply watched words to custom user fields nor user
profile fields.
This led to users being able to use blocked words in their bio, location
or some custom user fields.
This patch addresses this issue by adding some validations so it’s not
possible anymore to save the User model or the UserProfile model if they
contain blocked words.
This allows the category_id filter for the bookmark
report to work with polymorphic bookmarks. Honestly this
is a little hardcode-y at the moment but until we go and
make this report a lot more flexible with more filters
I don't think it's worth the work to add extra interfaces
to RegisteredBookmarkable and BaseBookmarkable to make
this more flexible. This is enough for now.
We have not used anything related to bookmarks for PostAction
or UserAction records since 2020, bookmarks are their own thing
now. Deleting all this is just cleaning up old cruft.
Latest redis interoduces a block form of multi / pipelined, this was incorrectly
passed through and not namespaced.
Fix also updates logster, we held off on upgrading it due to missing functions
A bit of a mixed bag, this addresses several edge areas of bookmarks and makes them compatible with polymorphic bookmarks (hidden behind the `use_polymorphic_bookmarks` site setting). The main ones are:
* ExportUserArchive compatibility
* SyncTopicUserBookmarked job compatibility
* Sending different notifications for the bookmark reminders based on the bookmarkable type
* Import scripts compatibility
* BookmarkReminderNotificationHandler compatibility
This PR also refactors the `register_bookmarkable` API so it accepts a class descended from a `BaseBookmarkable` class instead. This was done because we kept having to add more and more lambdas/properties inline and it was very messy, so a factory pattern is cleaner. The classes can be tested independently as well.
Some later PRs will address some other areas like the discourse narrative bot, advanced search, reports, and the .ics endpoint for bookmarks.
We're adding this column now in preparation for a future commit(s) that will
redesign the avatar/notifications menu. The reason the column is added in a
separate commit is because the redesign changes are going to be complex with a
high risk of getting (temporarily) reverted and if they included a database
migration, they wouldn't revert cleanly/easily.
Internal ticket: t65045.
When a user views a topic that contains a topic timer to publish to a
restricted category, an error occurs on the client side because the user
does not have access to information about the category.
This commit fixes it such that the topic timer is not shown to the user
if the user does not have access to the category.
On some installations, this would fail with 'index row size exceeds btree version 4 maximum'. This commit replaces the (post_id, url)` index with a `(post_id, md5(url))` index, which is much more space efficient.
This will make future changes to the 'pull hotlinked images' system easier. This commit should not introduce any functional change.
For now, the old post_custom_field data is kept in the database. This will be dropped in a future commit.
This commit introduces a new site setting: `use_name_for_username_suggestions` (default true)
Admins can disable it if they want to stop using Name values when generating usernames for users. This can be useful if you want to keep real names private-by-default or, when used in conjunction with the `use_email_for_username_and_name_suggestions` setting, you would prefer to use email-based username suggestions.
* hidden siteSetting to enable experimental sidebar
* user preference to enable experimental sidebar
* `experimental_sidebar_enabled` attribute for current user
* Empty glimmer component for Sidebar
We were calling `dup` on the hash and using that to check for changes. However, we were not duplicating the values, so changes to arrays or nested hashes would not be detected.
This pull request follows on from https://github.com/discourse/discourse/pull/16308. This one does the following:
* Changes `BookmarkQuery` to allow for querying more than just Post and Topic bookmarkables
* Introduces a `Bookmark.register_bookmarkable` method which requires a model, serializer, fields and preload includes for searching. These registered `Bookmarkable` types are then used when validating new bookmarks, and also when determining which serializer to use for the bookmark list. The `Post` and `Topic` bookmarkables are registered by default.
* Adds new specific types for Post and Topic bookmark serializers along with preloading of associations in `UserBookmarkList`
* Changes to the user bookmark list template to allow for more generic bookmarkable types alongside the Post and Topic ones which need to display in a particular way
All of these changes are gated behind the `use_polymorphic_bookmarks` site setting, apart from the .hbs changes where I have updated the original `UserBookmarkSerializer` with some stub methods.
Following this PR will be several plugin PRs (for assign, chat, encrypt) that will register their own bookmarkable types or otherwise alter the bookmark serializers in their own way, also gated behind `use_polymorphic_bookmarks`.
This commit also removes `BookmarkQuery.preloaded_custom_fields` and the functionality surrounding it. It was added in 0cd502a558 but only used by one plugin (discourse-assign) where it has since been removed, and is now used by no plugins. We don't need it anymore.
Previously, 'crop' would resize the image to have the requested width, then crop the height to the requested value. This works when cropping images vertically, but not when cropping them horizontally.
For example, trying to crop a 500x500 image to 200x500 was actually resulting in a 200x200 image. Having an OptimizedImage with width/height columns mismatching the actual OptimizedImage width/height causes some unusual issues.
This commit ensures that a call to `OptimizedImage.crop(from, to, width, height)` will always return an image of the requested width/height. The `w x h^` syntax defines minimum width/height, while maintaining aspect ratio.
This commit fixes two issues at play. The first was introduced
in f6c852b (or maybe not introduced
but rather revealed). When a user posted a new message in a topic,
they received the unread topic tracking state MessageBus message,
and the Unread (X) indicator was incremented by one, because with the
aforementioned perf commit we "guess" the correct last read post
for the user, because we no longer calculate individual users' read
status there. This meant that every time a user posted in a topic
they tracked, the unread indicator was incremented. To get around
this, we can just exclude the user who created the post from the
target users of the unread state message.
The second issue was related to the private message topic tracking
state, and was somewhat similar. Whenever a user created a new private
message, the New (X) indicator was incremented, and could not be
cleared until the page was refreshed. To solve this, we just don't
update the topic state for the user when the new_topic tracking state
message comes through if the user who created the topic is the
same as the current user.
cf. https://meta.discourse.org/t/bottom-of-topic-shows-there-is-1-unread-remaining-when-there-are-actually-0-unread-topics-remaining/220817
Discourse has the Discourse Connect Provider protocol that makes it possible to
use a Discourse instance as an identity provider for external sites. As a
natural extension to this protocol, this PR adds a new feature that makes it
possible to use Discourse as a 2FA provider as well as an identity provider.
The rationale for this change is that it's very difficult to implement 2FA
support in a website and if you have multiple websites that need to have 2FA,
it's unrealistic to build and maintain a separate 2FA implementation for each
one. But with this change, you can piggyback on Discourse to take care of all
the 2FA details for you for as many sites as you wish.
To use Discourse as a 2FA provider, you'll need to follow this guide:
https://meta.discourse.org/t/-/32974. It walks you through what you need to
implement on your end/site and how to configure your Discourse instance. Once
you're done, there is only one additional thing you need to do which is to
include `require_2fa=true` in the payload that you send to Discourse.
When Discourse sees `require_2fa=true`, it'll prompt the user to confirm their
2FA using whatever methods they've enabled (TOTP or security keys), and once
they confirm they'll be redirected back to the return URL you've configured and
the payload will contain `confirmed_2fa=true`. If the user has no 2FA methods
enabled however, the payload will not contain `confirmed_2fa`, but it will
contain `no_2fa_methods=true`.
You'll need to be careful to re-run all the security checks and ensure the user
can still access the resource on your site after they return from Discourse.
This is very important because there's nothing that guarantees the user that
will come back from Discourse after they confirm 2FA is the same user that
you've redirected to Discourse.
Internal ticket: t62183.
This commit improves the logic for rolling up IPv4 screened IP
addresses and extending it for IPv6. IPv4 addresses will roll up only
up to /24. IPv6 can rollup to /48 at most. The log message that is
generated contains the list of original IPs and new subnet.
* FEATURE: Let sites add a sitemap.xml file.
This PR adds the same features discourse-sitemap provides to core. Sitemaps are only added to the robots.txt file if the `enable_sitemap` setting is enabled and `login_required` disabled.
After merging discourse/discourse-sitemap#34, this change will take priority over the sitemap plugin because it will disable itself. We're also using the same sitemaps table, so our migration won't try to create it
again using `if_not_exists: true`.
It's possible in Rails to map a single route to multiple controller
actions with different constraints. We do this in at least 1 place in
our application for the root route (/) to make it possible to change the
page that root route displays.
This means that if you get the list of routes of your application,
you'll get the same route for each time the route is defined. And if
there's an API scope for 2 (or more) controller actions that map to the
same route, the route will be listed twice in the Allowed URLs list of
the scope.
To prevent this, this PR adds the allowed URLs in a set so that
duplicate routes are automatically removed.
The Allowed URLs list of an API scope only includes routes that
constraint the format for the route to JSON. However, some routes define
no format constraints, but that doesn't mean they can't be used by an
API key.
This commit amends the logic for the Allowed URLs list so that it
includes routes that have no format constraints or the format
constraints include JSON.
Due to default CSP web workers instantiated from CDN based assets are still
treated as "same-origin" meaning that we had no way of safely instansiating
a web worker from a theme.
This limits the theme system and adds the arbitrary restriction that WASM
based components can not be safely used.
To resolve this limitation all js assets in about.json are also cached on
local domain.
{
"name": "Header Icons",
"assets" : {
"worker" : "assets/worker.js"
}
}
This can then be referenced in JS via:
settings.theme_uploads_local.worker
local_js_assets are unconditionally served from the site directly and
bypass the entire CDN, using the pre-existing JavascriptCache
Previous to this change this code was completely dormant on sites which
used s3 based uploads, this reuses the very well tested and cached asset
system on s3 based sites.
Note, when creating local_js_assets it is highly recommended to keep the
assets lean and keep all the heavy working in CDN based assets. For example
wasm files can still live on the CDN but the lean worker that loads it can
live on local.
This change unlocks wasm in theme components, so wasm is now also allowed
in `theme_authorized_extensions`
* more usages of upload.content
* add a specific test for upload.content
* Adjust logic to ensure that after upgrades we still get a cached local js
on save
Previously we only supported a single 'required tag group' for a category. This commit allows admins to specify multiple required tag groups, each with their own minimum tag count.
A new category_required_tag_groups database table replaces the existing columns on the categories table. Data is automatically migrated.
This patch removes some of our freedom patches that have been deprecated
for some time now.
Some of them have been updated so we’re not shipping code based on an
old version of Rails.
Via the API it is possible to create a user with an integer username. So
123 instead of "123". This causes the following 500 error:
```
NoMethodError (undefined method `unicode_normalize' for 1:Integer)
app/models/user.rb:276:in `normalize_username'
```
See: https://meta.discourse.org/t/222281
Previous to this change if any of the assets were not allowed extensions
they would simply be silently ignored, this could lead to broken themes
that are very hard to debug
Our group fabrication creates groups with name "my_group_#{n}" where n
is the sequence number of the group being created. However, this can
cause the test to be flaky if and when a group with name `my_group_10`
is created as it will be ordered before
`my_group_9`. This commits makes the group names determinstic to
eliminate any flakiness.
This reverts commit 558bc6b746.
This commit introduces a new use_polymorphic_bookmarks site setting
that is default false and hidden, that will be used to help continuous
development of polymorphic bookmarks. This setting **should not** be
enabled anywhere in production yet, it is purely for local development.
This commit uses the setting to enable create/update/delete actions
for polymorphic bookmarks on the server and client side. The bookmark
interactions on topics/posts are all usable. Listing, searching,
sending bookmark reminders, and other edge cases will be handled
in subsequent PRs.
Comprehensive UI tests will be added in the final PR -- we already
have them for regular bookmarks, so it will just be a matter of
changing them to be for polymorphic bookmarks.
Tags (and tag groups) can be configured so that they can only be used in specific categories and (optionally) restrict topics in these categories to be able to add/use only these tags. These restrictions work as expected when a topic is created without going through the review queue; however, if the topic has to be reviewed by a moderator then these restrictions currently aren't checked before the topic is sent to the review queue, but they're checked later when a moderator tries to approve the topic. This is because if a user manages to submit a topic that doesn't meet the restrictions, moderators won't be able to approve and it'll be stuck in the review queue.
This PR prevents topics that don't meet the tags requirements from being sent to the review queue and shows the poster an error message that indicates which tags that cannot be used.
Internal ticket: t60562.
As we are gradually moving to having a polymorphic
bookmarkable relationship on the Bookmark table,
we need to make the post_id column nullable to be
able to develop and test the new columns, and
for cutover/migration purposes later as well.
PostAnalyzer and CookedPostProcessor both replace URLs with oneboxes.
PostAnalyzer did not use the max_oneboxes_per_post site and setting and
CookedPostProcessor replaced at most max_oneboxes_per_post URLs ignoring
the oneboxes that were replaced already by PostAnalyzer.
This commit is a redo of2f1ddadff7dd47f824070c8a3f633f00a27aacde
which we reverted because it blew up an internal CI check. I looked
into it, and it happened because the old migration to add the bookmark
columns still existed, and those columns were dropped in a post migrate,
so the two migrations to add the columns were conflicting before
the post migrate was run.
------
This commit only includes the creation of the new columns and index,
and does not add any triggers, backfilling, or new data.
A backfill will be done in the final PR when we switch this over.
Intermediate PRs will look something like this:
Add an experimental site setting for using polymorphic bookmarks,
and make sure in the places where bookmarks are created or updated
we fill in the columns. This setting will be used in subsequent
PRs as well.
Listing and searching bookmarks based on polymorphic associations
Creating post and topic bookmarks using polymorphic associations,
and changing special for_topic logic to just rely on the Topic
bookmarkable_type
Querying bookmark reminders based on polymorphic associations
Make sure various other areas like importers, bookmark guardian,
and others all rely on the associations
Prepare plugins that rely on the Bookmark model to use polymorphic
associations
The final core PR will remove all the setting gates and switch over
to using the polymorphic associations, backfill the bookmarks
table columns, and ignore the old post_id and for_topic colummns.
Then it will just be a matter of dropping the old columns down the
line.
This commit is a redo of e21c640a3c
which we reverted to not include half-done work in a release.
This commit is slightly different though, in that it only includes
the creation of the new columns and index, and does not add any
triggers, backfilling, or new data.
A backfill will be done in the final PR when we switch this over.
Intermediate PRs will look something like this:
1. Add an experimental site setting for using polymorphic bookmarks,
and make sure in the places where bookmarks are created or updated
we fill in the columns. This setting will be used in subsequent
PRs as well.
2. Listing and searching bookmarks based on polymorphic associations
3. Creating post and topic bookmarks using polymorphic associations,
and changing special for_topic logic to just rely on the Topic
bookmarkable_type
4. Querying bookmark reminders based on polymorphic associations
5. Make sure various other areas like importers, bookmark guardian,
and others all rely on the associations
6. Prepare plugins that rely on the Bookmark model to use polymorphic
associations
The final core PR will remove all the setting gates and switch over
to using the polymorphic associations, backfill the bookmarks
table columns, and ignore the old post_id and for_topic colummns.
Then it will just be a matter of dropping the old columns down the
line.
Plugins like chat add custom score type to override the title in the UI, but that should be reserved for situations when you need to manage the flag priority separately, which is configurable in the queue settings page.
Currently, if a plugin creates a custom score type, it won't be able to associate a priority, so there's no real gain from doing so. Priorities are tightly related to post-action types, which is something we might want to revise. For now, this change lets plugins move away from custom score types without compromises.
The meaning of reminder_at and reminder_last_sent_at changed after
commit 6d422a8033. A bookmark reminder
will fire only if reminder_last_sent_at is null, but before that it
fired everytime reminder_at was set. This is no longer true because
sometimes reminder_at continues to exist even after a reminder fired.
* FEATURE: use canonical links in posts.rss feed
Previously we used non canonical links in posts.rss
These links get crawled frequently by crawlers when discovering new
content forcing crawlers to hop to non canonical pages just to end up
visiting canonical pages
This uses up expensive crawl time and adds load on Discourse sites
Old links were of the form:
`https://DOMAIN/t/SLUG/43/21`
New links are of the form
`https://DOMAIN/t/SLUG/43?page=2#post_21`
This also adds a post_id identified element to crawler view that was
missing.
Note, to avoid very expensive N+1 queries required to figure out the
page a post is on during rss generation, we cache that information.
There is a smart "cache breaker" which ensures worst case scenario is
a "page drift" - meaning we would publicize a post is on page 11 when
it is actually on page 10 due to post deletions. Cache holds for up to
12 hours.
Change only impacts public post RSS feeds (`/posts.rss`)
