The guardian is useful for plugins to determine if the callback should
do anything. A common use case is to not do anything in the callback if
the user is anonymous.
* FIX: Use Category.secured(guardian) for hashtag datasource
Follow up to comments in #19219, changing the category
hashtag datasource to use Category.secured(guardian) instead
of Site.new(guardian).categories here since the latter does
more work for not much benefit, and the query time is the
same. Also eliminates some Hash -> Model back and forth
busywork. Add some more specs too.
* FIX: Server-side hashtag lookup cooking user loading
When we were using the PrettyText.options.currentUser
and parsing back and forth with JSON for the hashtag
lookups server-side, we had a bug where the user's
secure categories were not loaded since we never actually
loaded a User model from the database, only parsed it
from JSON.
This commit fixes the issue by instead using the
PretyText.options.userId and looking up the user directly
from the database when calling hashtag_lookup via the
PrettyText::Helpers code when cooking server-side. Added
the missing spec to check for this as well.
This new site setting replaces the
`enable_experimental_sidebar_hamburger` and `enable_sidebar` site
settings as the sidebar feature exits the experimental phase.
Note that we're replacing this without depreciation since the previous
site setting was considered experimental.
Internal Ref: /t/86563
In this PR, we introduced an option, that when all authenticators are disabled, but backup codes still exists, user can authenticate with those backup codes. This was reverted as this is not expected behavior.
https://github.com/discourse/discourse/pull/18982
Instead, when the last authenticator is deleted, backup codes should be deleted as well. Because this disables 2fa, user is asked to confirm that action by typing text.
In addition, UI for 2fa preferences was refreshed.
When finding the candidates for `Topic.similar_to`, we will now ignore
topics in categories where `Category#search_priority` has been set to
ignore and also topics in categories which the user has specifically
muted.
Internal Ref: /t/87132
Note that we don't have a database table and a model for post mentions yet, and I decided to implement it without adding one to avoid heavy data migrations. Still, we may want to add such a model later, that would be convenient, we have such a model for mentions in chat.
Note that status appears on all mentions on all posts in a topic except of the case when you just posted a new post, and it appeared on the bottom of the topic. On such posts, status won't be shown immediately for now (you'll need to reload the page to see the status). I'll take care of it in one of the following PRs.
When a screened IP address is matched because it is either blocked or
allowed it should update match_count. This did not work because it
tried to validate the IP address and it failed as it matched with
itself.
Follow-up to 6357a3ce33
where we allowed a general API key scope for user status
GET/PUT/DELETE, this commit allows the same for the
UserApiKey system.
Previously we would trigger the event before the `Topic#deleted_at`
column has been updated making it hard for plugins to correctly work
with the model when its new state has not been persisted in the
database.
* FIX: Only modify secured sidebar links on user promotion/demotion
If a user is created populate their sidebar with the default
categories/tags that they have access to.
If a user is promoted to admin populate any new categories/tags that
they now have access to.
If an admin is demoted remove any categories/tags that they no longer
have access to.
This will only apply for "secured" categories. For example if these are
the default sitebar categories:
- general
- site feedback
- staff
and a user only has these sidebar categories:
- general
when they are promoted to admin they will only receive the "staff"
category. As this is a default category they didn't previously have
access to.
* Add spec, remove tag logic on update
Change it so that if a user becomes unstaged it used the "add" method
instead of the "update" method because it is essentially following the
on_create path.
On admin promotion/demotion remove the logic for updating sidebar tags because
we don't currently have the tag equivalent like we do for User.secure_categories.
Added the test case for when a user is promoted to admin it should
receive *only* the new sidebar categories they didn't previously have
access to. Same for admin demotion.
* Add spec for suppress_secured_categories_from_admin site setting
* Update tags as well on admin promotion/demotion
* only update tags when they are enabled
* Use new SidebarSectionLinkUpdater
We now have a SidebarSectionLinkUpdater
that was introduced in: fb2507c6ce
* remove empty line
The centralization helps in reducing code duplication in our code base
and more importantly, centralizing logic for guardian checks into a
single spot.
Users who can access the review queue can claim a pending reviewable(s) which means that the claimed reviewable(s) can only be handled by the user who claimed it. Currently, we show claimed reviewables in the user menu, but this can be annoying for other reviewers because they can't do anything about a reviewable claimed by someone. So this PR makes sure that we only show in the user menu reviewables that are claimed by nobody or claimed by the current user.
Internal topic: t/77235.
This PR adds separate notification indicators for PMs and reviewables that have arrived since the last time the user opened the notifications menu.
The PM indicator is the strongest one of all three indicators followed by the reviewable indicator and then finally the blue indicator. This means that if there's a new PM and a new reviewable, then the PM indicator will be shown.
Meta topic: https://meta.discourse.org/t/no-green-or-red-notification-bubbles/242783?u=osama.
Internal topic: t/82995.
While load testing our user creation code path in production, we
identified that executing the DB statement to update the `Group#user_count` column within a
transaction is creating a bottleneck for us. This is because the
creation of a user and addition of the user to the relevant groups are
done in a transaction. When we execute the DB statement to update
`Group#user_count` for the relevant group, a row level lock is held
until the transaction completes. This row level lock acts like a global
lock when the server is creating users that will be added to the same
group in quick succession.
Instead of updating the counter cache within a transaction which the
default ActiveRecord `counter_cache` option does, we simply update the
counter cache outside of the committing transaction.
Co-authored-by: Rafael dos Santos Silva <xfalcox@gmail.com>
Co-authored-by: Rafael dos Santos Silva <xfalcox@gmail.com>
Adds stats for API and user API requests similar to regular page views.
This comes with a new report to visualize API requests per day like the
consolidated page views one.
This commit introduce a new API for registering callbacks, which we'll execute when a user gets destroyed, and the `delete_posts` opt is true. The chat plugin registers one callback and queues a job to destroy every message from that user in batches.
Follow up to 40e8912395
In this previous commit I introduced a bug that prevented
a legitimate case for an existing user to redeem an invite,
where the email/domain were both blank and the invite was
still redeemable by the user. Fixes the issue and adds more
specs for that case.
This adds API scope for the user status. This also adds a get method to the user status controller. We didn't need a dedicated method that returns status before because the server returns status with user objects, but I think we need to provide this method for API clients.
Update `release_notes_link` to current version
<!-- NOTE: All pull requests should have tests (rspec in Ruby, qunit in
JavaScript). If your code does not include test coverage, please include
an explanation of why it was omitted. -->
This commit fleshes out and adds functionality for the new `#hashtag` search and
lookup system, still hidden behind the `enable_experimental_hashtag_autocomplete`
feature flag.
**Serverside**
We have two plugin API registration methods that are used to define data sources
(`register_hashtag_data_source`) and hashtag result type priorities depending on
the context (`register_hashtag_type_in_context`). Reading the comments in plugin.rb
should make it clear what these are doing. Reading the `HashtagAutocompleteService`
in full will likely help a lot as well.
Each data source is responsible for providing its own **lookup** and **search**
method that returns hashtag results based on the arguments provided. For example,
the category hashtag data source has to take into account parent categories and
how they relate, and each data source has to define their own icon to use for the
hashtag, and so on.
The `Site` serializer has two new attributes that source data from `HashtagAutocompleteService`.
There is `hashtag_icons` that is just a simple array of all the different icons that
can be used for allowlisting in our markdown pipeline, and there is `hashtag_context_configurations`
that is used to store the type priority orders for each registered context.
When sending emails, we cannot render the SVG icons for hashtags, so
we need to change the HTML hashtags to the normal `#hashtag` text.
**Markdown**
The `hashtag-autocomplete.js` file is where I have added the new `hashtag-autocomplete`
markdown rule, and like all of our rules this is used to cook the raw text on both the clientside
and on the serverside using MiniRacer. Only on the server side do we actually reach out to
the database with the `hashtagLookup` function, on the clientside we just render a plainer
version of the hashtag HTML. Only in the composer preview do we do further lookups based
on this.
This rule is the first one (that I can find) that uses the `currentUser` based on a passed
in `user_id` for guardian checks in markdown rendering code. This is the `last_editor_id`
for both the post and chat message. In some cases we need to cook without a user present,
so the `Discourse.system_user` is used in this case.
**Chat Channels**
This also contains the changes required for chat so that chat channels can be used
as a data source for hashtag searches and lookups. This data source will only be
used when `enable_experimental_hashtag_autocomplete` is `true`, so we don't have
to worry about channel results suddenly turning up.
------
**Known Rough Edges**
- Onebox excerpts will not render the icon svg/use tags, I plan to address that in a follow up PR
- Selecting a hashtag + pressing the Quote button will result in weird behaviour, I plan to address that in a follow up PR
- Mixed hashtag contexts for hashtags without a type suffix will not work correctly, e.g. #ux which is both a category and a channel slug will resolve to a category when used inside a post or within a [chat] transcript in that post. Users can get around this manually by adding the correct suffix, for example ::channel. We may get to this at some point in future
- Icons will not show for the hashtags in emails since SVG support is so terrible in email (this is not likely to be resolved, but still noting for posterity)
- Additional refinements and review fixes wil
The hidden site setting `suppress_secured_categories_from_admin` will
suppress visibility of categories without explicit access from admins
in a few key areas (category drop downs and topic lists)
It is not intended to be a security wall since admins can amend any site
setting. Instead it is feature that allows hiding the categories from the
UI.
Admins will still be able to see topics in categories without explicit
access using direct URLs or flags.
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
When opening the invite acceptance page when the user
was already logged in, we were still showing the Accept
Invitation prompt even if the user had already redeemed
the invitation and was present in the `InvitedUser` table.
This would lead to errors when the user clicked on the button.
This commit fixes the issue by hiding the Accept Invitation
button and showing an error message instead indicating that
the user had already redeemed the invitation. This only applies
to multi-use invite links.
When this report in the admin dashboard has lots of data ( > 75 days of activity), the dates were ordered incorrectly. This is apparently expected behaviour; when using GROUP BY without specifying the ordering, PG decides to order, and it so happens that it works under some conditions but not others. Explicit ordering fixes the problem.
However, because this works in some conditions but not others, we can't really add a useful test.
* DEV: Add utility to hide all user tips
* DEV: Add UserTip Glimmer component
* DEV: Add tests for existing user tips
* FEATURE: Add user tip for post menu
* FEATURE: Add user tip for topic notification level
* FEATURE: Add user tip for suggested topics
* FEATURE: Hide new popups for existing users
We were doing get on Redis two times for each emoji while building the custom/standard/all lists which where resulting in ~3710 Redis calls. Given the emoji DB file is loaded in memory while we build/cache the emojis list this is unnecessary and slow.
As a simplification in pseudo code here is an explanation of what we were doing:
```ruby
emojis.each |emoji_name|
aliases = get_aliases_from_redis_cache(emoji_name)
is_tonable = get_is_tonable_from_redis_cache(emoji_name)
build_emoji(emoji_name, aliases, is_tonable)
end
```
The two redis calls are now simplified to a simple hash access: `@db[emoji_name]`
This commit adds some protections in InviteRedeemer to ensure that email
can never be nil, which could cause issues with inviting the invited
person to private topics since there was an incorrect inner join.
If the email is nil and the invite is scoped to an email, we just use
that invite.email unconditionally. If a redeeming_user (an existing
user) is passed in when redeeming an email, we use their email to
override the passed in email. Otherwise we just use the passed in
email. We now raise an error after all this if the email is still nil.
This commit also adds some tests to catch the private topic fix, and
some general improvements and comments around the invite code.
This commit also includes a migration to delete TopicAllowedUser records
for users who were mistakenly added to topics as part of the invite
redemption process.
Currently, we have available three 2fa methods:
- Token-Based Authenticators
- Physical Security Keys
- Two-Factor Backup Codes
If the first two are deleted, user lose visibility of their backup codes, which suggests that 2fa is disabled.
However, when they try to authenticate, the account is locked, and they have to ask admin to fix that problem.
This PR is fixing the issue. User still sees backup codes in their panel and can use them to authenticate.
In next PR, I will improve UI to clearly notify the user when 2fa is fully disabled and when it is still active.
This commit automatically ensures that category channels
have slugs when they are created or updated based on the
channel name, category name, or existing slug. The behaviour
has been copied from the Category model.
We also include a backfill here with a simplified version
of Slug.for with deduplication to fill the slugs for already
created Category chat channels.
The channel slug is also now used for chat notifications,
and for the UI and navigation for chat. `slugifyChannel`
is still used, but now does the following fallback:
* Uses channel.slug if it is present
* Uses channel.escapedTitle if it is present
* Uses channel.title if it is present
In future we may want to remove this altogether
and always rely on the slug being present, but this
is currently not possible because we are not generating
slugs for DM channels at this point.
* Remove old bookmark column ignores to follow up b22450c7a8
* Change some group site setting checks to use the _map helper
* Remove old secure_media helper stub for chat
* Change attr_accessor to attr_reader for preloaded_custom_fields to follow up 70af45055a
It is likely that a new admin user was created as just a regular user
before being promoted to admin so this change will update the sidebar
link records for any users that are promoted to admin. This way if any
of the default side bar categories or tags are restricted to admins
these new admins will have those added to their sidebar as well.
You can easily replicate this issue locally (prior to this fix) by using
`rails admin:create` where it creates a user first, then it is promoted
to admin. This means it would receive the default categories of regular
user, but never receive the ones they should have access to as an admin.
As part of this change I did drop the `!` from
`SidebarSectionLink.insert_all` so that it would add any new records
that were missing, but not throw a unique constraint error trying to add
any existing records.
Follow up to: 1b56a55f50
And: e320bbe513
Previously, we didn't have a site-wide setting to set the default behavior for user profile visibility and user presence features. But we already have a user preference for that.
This commit fixes a regression introduced in 8979adc where under certain conditions the groups syncing logic in Discourse Connect would try to add users to groups they're already members of and cause errors when users try to sign in using Discourse Connect.
The previous sidebar default tags and categories implementation did not
allow for a user to configure their sidebar to have no categories or
tags. This commit changes how the defaults are applied. When a user is being created,
we create the SidebarSectionLink records based on the `default_sidebar_categories` and
`default_sidebar_tags` site settings. SidebarSectionLink records are
only created for categories and tags which the user has visibility on at
the point of user creation.
With this change, we're also adding the ability for admins to apply
changes to the `default_sidebar_categories` and `default_sidebar_tags`
site settings historically when changing their site setting. When a new
category/tag has been added to the default, the new category/tag will be
added to the sidebar for all users if the admin elects to apply the changes historically.
Like wise when a tag/category is removed, the tag/category will be
removed from the sidebar for all users if the admin elects to apply the
changes historically.
Internal Ref: /t/73500
Before this commit, there was no way for us to efficiently check an
array of topics for which a user can see. Therefore, this commit
introduces the `TopicGuardian#can_see_topic_ids` method which accepts an
array of `Topic#id`s and filters out the ids which the user is not
allowed to see. The `TopicGuardian#can_see_topic_ids` method is meant to
maintain feature parity with `TopicGuardian#can_see_topic?` at all
times so a consistency check has been added in our tests to ensure that
`TopicGuardian#can_see_topic_ids` returns the same result as
`TopicGuardian#can_see_topic?`. In the near future, the plan is for us
to switch to `TopicGuardian#can_see_topic_ids` completely but I'm not
doing that in this commit as we have to be careful with the performance
impact of such a change.
This method is currently not being used in the current commit but will
be relied on in a subsequent commit.
Discourse Connect can be used to manage group memberships of users by including a `add_groups`, `remove_groups` or `groups` attribute in the Discourse Connect payload. However, additions/deletions of users from groups aren't logged to the groups logs (available at `/g/<group>/manage/logs`) which can cause confusions to admins they try to figure out when/how users were added or removed from a group. This commit makes Discourse Connect add entries to the groups logs when it makes changes to users' group memberships.
Our theme system injects a magical `settings` object at the top of themes JS modules to allow theme authors to access the settings as configured by admins in the UI. Within this `settings` object, there are a couple of special objects `theme_uploads` and `theme_uploads_local` that contain URLs for all the assets/uploads that the theme has.
For test modules/files, the theme system also injects a `settings` object at the top of tests modules, but it's not the same object as the object that's injected in non-test files. The difference is that in tests we want the settings to have their default values as opposed to any custom values that may exist in the site's database. This ensures that test results are consistent no matter the site that runs them.
However, the `settings` object in tests files currently doesn't have the special objects `theme_uploads` and `theme_uploads_local` which means that if a theme includes an asset that's lazy-loaded, it's not possible to write tests for anything that depends on the lazy-loaded asset because the theme will not be able to load the asset during the tests since `theme_uploads_local` and `theme_uploads` don't exist. This PR adds these special objects inside the `settings` object for test files.
Internal topic: t/71825/52.
Theme javascript is now minified using Terser, just like our core/plugin JS bundles. This reduces the amount of data sent over the network.
This commit also introduces sourcemaps for theme JS. Browser developer tools will now be able show each source file separately when browsing, and also in backtraces.
For theme test JS, the sourcemap is inlined for simplicity. Network load is not a concern for tests.
Previously, compiling theme 'extra_js' was done with a number of steps. Each theme_field would be compiled into its own value_baked column, and then the JavascriptCache content would be built by concatenating all of those compiled values.
This commit streamlines things by removing the value_baked step. The raw value of all extra_js theme_fields are passed directly to the ThemeJavascriptCompiler, and then the result is stored in the JavascriptCache.
In itself, this commit should not cause any behavior change. It is designed to open the door to more advanced compilation features which have interdependencies between different source files (e.g. template colocation, sourcemaps).
This commit fixes an issue where we had a typo in the
UserAction.stream query which meant that action_code_path
was not loaded correctly. Once that was fixed, we were also
not actually using the action_code_path in the user-stream-item,
so that has been fixed here too.
The bug this caused was that, when the link for the action was
clicked within the user-stream-item, the user would be redirected
to a URL ending with `[missing%20%%7Bpath%7D%20value]` because
the I18n call did not have the path present.
* FIX: Reset related site settings on general category delete
If the new seeded General category is deleted we also need to delete the
corresponding site setting for it so that we don't try and reference it.
This fixes a bug in the category dropdown composer.
This change creates the `clear_related_site_settings` after destroy
hook that could also be used by other features in the future, like maybe
when we have a `default_category_id` site_setting.
Looks like if `nil` out a site setting it is set to `0`?
```
[9] pry(main)> SiteSetting.general_category_id = nil
SiteSetting Load (0.4ms) SELECT "site_settings".* FROM "site_settings" WHERE "site_settings"."name" = 'general_category_id' LIMIT 1
=> nil
[10] pry(main)> SiteSetting.general_category_id
=> 0
```
That is why the tests check if the value is `< 1` and not `nil`.
* Use -1 instead of nil because it is the default
This commit introduces a new framework for building user tutorials as
popups using the Tippy JS library. Currently, the new framework is used
to replace the old notification spotlight and tips and show a new one
related to the topic timeline.
All popups follow the same structure and have a title, a description and
two buttons for either dismissing just the current tip or all of them
at once.
The state of all seen popups is stored in a user option. Updating
skip_new_user_tips will automatically update the list of seen popups
accordingly.
The previous fix in e83d35d6 was incorrect, and the stub in the test was never actually hit. This commit moves the error handling to the right place and updates the specs to ensure the stub is always used.
This can no longer be used from the user interface and could be used to
generate useless topic invites notifications. This commit adds site
setting max_topic_invitations_per_minute to prevent invite spam.
Adds a new upload field for a second dark mode category logo.
This alternative will be used when the browser is in dark mode (similar to the global site setting for a dark logo).
These errors tend to indicate that the upload is missing on the remote store. This is bad, but we don't want it to block the dominant-color calculation process. This commit catches errors when there is an HTTP error, and fixes the `base_store.rb` implementation when `FileHelper.download` returns nil.
Previously, when categories were not muted by default, we were sending message about unmuted topics (topics which user explicitly set notification level to watching)
The same mechanism can be used to fix a bug. When the user was explicitly watching topic, but category was muted, then the user was not informed about new reply.
This commit makes pending reviewables show up in the main tab (a.k.a. "all notifications" tab). Pending reviewables along with unread notifications are always shown first and they're sorted based on their creation date (most recent comes first).
The dismiss button currently only shows up if there are unread notifications and it doesn't dismiss pending reviewables. We may follow up with another change soon that allows makes the dismiss button work with reviewables and remove them from the list without taking any action on them.
Follow-up to 079450c9e4.
This commit adds non-archived group messages and `group_message_summary` notifications in the messages tab in the user menu. With this change, the messages tab in the user menu now includes 3 types of items:
1. Unread `private_message` notifications (notifications when you receive a reply in a PM)
2. Unread and read `group_message_summary` notifications (notifications when there's a new message in a group inbox that you track)
3. Non-archived personal and group messages
Unread `private_message` notifications are always shown first, followed by unread `group_message_summary` notifications, and then everything else (messages and read `group_message_summary` notifications) sorted by recency (most recent first).
Internal topic: t/72976.
Currently, the reviewables tab in the user menu shows pending reviewables at the top of the menu and fills the remaining space in the menu with old/handled reviewables. This PR makes the revieables tab show only pending reviewables and hides the tab altogether from the menu if there are no pending reviewables. We're going to follow-up with another change soon that will show pending reviewables in the main tab of the user menu.
Internal topic: t/73220.
This commit renames all secure_media related settings to secure_uploads_* along with the associated functionality.
This is being done because "media" does not really cover it, we aren't just doing this for images and videos etc. but for all uploads in the site.
Additionally, in future we want to secure more types of uploads, and enable a kind of "mixed mode" where some uploads are secure and some are not, so keeping media in the name is just confusing.
This also keeps compatibility with the `secure-media-uploads` path, and changes new
secure URLs to be `secure-uploads`.
Deprecated settings:
* secure_media -> secure_uploads
* secure_media_allow_embed_images_in_emails -> secure_uploads_allow_embed_images_in_emails
* secure_media_max_email_embed_image_size_kb -> secure_uploads_max_email_embed_image_size_kb
In the past, CategoryFeaturedTopic.feature_topics raised an exception
sometimes because it tried to create multiple CategoryFeaturedTopic
records for the same topic.
This code should not raise any exceptions as long as the list of new
topic IDs is unique because all previous records are deleted first,
then recreated and everything happens inside a transaction. The previous
rescue block was dead code anyway because it tried to catch
PG::UniqueViolation instead of ActiveRecord::RecordNotUnique. This
commit includes a fix to ensure that the topic IDs are unique.
This will replace `enable_personal_messages` and
`min_trust_to_send_messages`, this commit introduces
the setting `personal_message_enabled_groups`
and uses it in all places that `enable_personal_messages`
and `min_trust_to_send_messages` currently apply.
A migration is included to set `personal_message_enabled_groups`
based on the following rules:
* If `enable_personal_messages` was false, then set
`personal_message_enabled_groups` to `3`, which is
the staff auto group
* If `min_trust_to_send_messages` is not default (1)
and the above condition is false, then set the
`personal_message_enabled_groups` setting to
the appropriate auto group based on the trust level
* Otherwise just set `personal_message_enabled_groups` to
11 which is the TL1 auto group
After follow-up PRs to plugins using these old settings, we will be
able to drop the old settings from core, in the meantime I've added
DEPRECATED notices to their descriptions and added them
to the deprecated site settings list.
This commit also introduces a `_map` shortcut method definition
for all `group_list` site settings, e.g. `SiteSetting.personal_message_enabled_groups`
also has `SiteSetting.personal_message_enabled_groups_map` available,
which automatically splits the setting by `|` and converts it into
an array of integers.
See https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499
for thorough reasoning.
This commit changes how we generate Message-IDs and do email
threading for emails sent from Discourse. The main changes are
as follows:
* Introduce an outbound_message_id column on Post that
is either a) filled with a Discourse-generated Message-ID
the first time that post is used for an outbound email
or b) filled with an original Message-ID from an external
mail client or service if the post was created from an
incoming email.
* Change Discourse-generated Message-IDs to be more consistent
and static, in the format `discourse/post/:post_id@:host`
* Do not send References or In-Reply-To headers for emails sent
for the OP of topics.
* Make sure that In-Reply-To is filled with either a) the OP's
Message-ID if the post is not a direct reply or b) the parent
post's Message-ID
* Make sure that In-Reply-To has all referenced post's Message-IDs
* Make sure that References is filled with a chain of Message-IDs
from the OP down to the parent post of the new post.
We also are keeping X-Discourse-Post-Id and X-Discourse-Topic-Id,
headers that we previously removed, for easier visual debugging
of outbound emails.
Finally, we backfill the `outbound_message_id` for posts that have
a linked `IncomingEmail` record, using the `message_id` of that record.
We do not need to do that for posts that don't have an incoming email
since they are backfilled at runtime if `outbound_message_id` is missing.
The `add_column` `limit` parameter has no effect on a postgres `text` column. Instead we can perform the check in ActiveRecord.
We never expect this condition to be hit - users cannot control this value. It's just a safety net.
Adds limits to location and website fields at model and DB level
to match the bio_raw field limits. A limit cannot be added at the
DB level for bio_raw because it is a postgres text field.
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
We previously had a system which would generate a 10x10px preview of images and add their URLs in a data-small-upload attribute. The client would then use that as the background-image of the `<img>` element. This works reasonably well on fast connections, but on slower connections it can take a few seconds for the placeholders to appear. The act of loading the placeholders can also break or delay the loading of the 'real' images.
This commit replaces the placeholder logic with a new approach. Instead of a 10x10px preview, we use imagemagick to calculate the average color of an image and store it in the database. The hex color value then added as a `data-dominant-color` attribute on the `<img>` element, and the client can use this as a `background-color` on the element while the real image is loading. That means no extra HTTP request is required, and so the placeholder color can appear instantly.
Dominant color will be calculated:
1. When a new upload is created
2. During a post rebake, if the dominant color is missing from an upload, it will be calculated and stored
3. Every 15 minutes, 25 old upload records are fetched and their dominant color calculated and stored. (part of the existing PeriodicalUpdates job)
Existing posts will continue to use the old 10x10px placeholder system until they are next rebaked
Right now the experimental user menu sorts notifications the same way that the old menu does: unread high-priority notifications are shown first in reverse-chronological order followed by everything else also in reverse-chronological order. However, since the experimental user menu has dedicated tabs for some notification types and each tab displays a badge with the count of unread notifications in the tab, we feel like it makes sense to change how notifications are sorted in the experimental user menu to this:
1. unread high-priority notifications
2. unread regular notifications
3. all read notifications (both high-priority and regular)
4. within each group, notifications are sorted in reverse-chronological order (i.e. newest is shown first).
This new sorting logic applies to all tabs in the experimental user menu, however it doesn't change anything in the old menu. With this change, if a tab in the experimental user menu shows an unread notification badge for a really old notification, it will be surfaced to the top and prevents confusing scenarios where a user sees an unread notification badge on a tab, but the tab doesn't show the unread notification because it's too old to make it to the list.
Internal topic: t72199.
Previously we were relying on a highly-customized version of the unmaintained Barber gem for theme template compilation. This commit switches us to use our own DiscourseJsProcessor, which makes use of more modern patterns and will be easier to maintain going forward.
In summary:
- Refactors DiscourseJsProcessor to move multiline JS heredocs into a companion `discourse-js-processor.js` file
- Use MiniRacer's `.call` method to avoid manually escaping JS strings
- Move Theme template AST transformers into DiscourseJsProcessor, and formalise interface for extending RawHandlebars AST transformations
- Update Ember template compilation to use a babel-based approach, just like Ember CLI. This gives each template its own ES6 module rather than directly assigning `Ember.TEMPLATES` values
- Improve testing of template compilation (and move some tests from `theme_javascript_compiler_spec.rb` to `discourse_js_processor_spec.rb`
It used to return the next URL anyway which lead to an additional
request. On the frontend, if the result set was empty, it kept retrying
until at least one result was returned. This bug is fixed in this commit
too.
* FIX: hide welcome topic banner as soon as the welcome topic is edited
This commit adds a message bus listener on client to hide the welcome
topic banner as soon as the welcome topic is edited.
* update test
* only subscribe when show_welcome_topic_banner is true
* Do not lookup for messageBus service if it's not required
* Remove unneeded code
* Cache result for Site.show_welcome_topic_banner
* Update tests per latest changes
* Changes per PR review
Each new user menu notifications should have their own count. Therefore, we need to include all types to serializer and not only `grouped_unread_high_priority_notifications`
Additional PR will be created for chat and assign plugin, as they will have to switch to `grouped_unread_notifications` as well.
The source-of-truth for our ember version is now the installed node_module. The `ember_source` gem carries an old version of Ember and so the constant is no longer useful. We'll be dropping the gem soon.
This commit makes a number of improvements to the DiscourseJsProcessor:
1. Remove dependence on the out-of-date Ember template compiler from the ember-rails gem; switch to modern template compiler
2. Refactor to make use of a proper module system with `define`/`require`
3. Introduce `babel-plugin-ember-template-compilation` to enable inline hbs compilation
The `mini-loader` is upgraded to support relative lookup and `require.has`, so that these new JS packages work correctly.
If a topic is converted to a private message, all posters were invited
to the new private message. This included users who only whispered or
posted small actions.
Topic allowed user records were created for small actions, which lead to
the system user being invited in many private topics when the user
removed themselves or if a group was invited but some members already
had access.
This commits skips creating topic allowed user. They are already skipped
for the whisper posts.
If a user was granted a trust level, joined a group that granted a trust
level and left the group, the trust level was reset. This commit tries
to restore the last known trust level before joining the group by
looking into staff logs.
This commit also migrates old :change_trust_level user history records
to use previous_value and new_value fields.
When preloading topic_list data we were giving it a 'preload key' which was loosely based on the parameters of the list. However, it did not include all parameters, and mismatches between client/server-side logic would cause the preloaded data to be ignored.
This commit simplifies things by using a single key for all topic_list preloading. This works on the assumption that "The first topic_list the JS app will load is the one which was preloaded". That assumption also existed to some extent in the old design, so we don't expect any regressions here.
Default sidebar tags for not authenticated users can be defined in admin panel. Otherwise, top 5 categories and tags are taken.
Optionally, if categories are set up in permanent order, then the first 5 categories are taken.
```
1) CurrentUserSerializer#sidebar_category_ids includes visible default sidebar categories
Failure/Error: expect(json[:sidebar_category_ids]).to eq([category.id, category_2.id])
expected: [378, 379]
got: [379, 378]
```
Note that in the Ruby doc it says "The order is preserved from the original array". In this case, we want to preserve the order of the site setting.
Handles edge-case when a user is an admin and has an associated reviewable. Hitting this exception should be rare since we clear the reviewable when
granting staff to the user.
A public key must be added to GitHub when installing private themes.
When the process happens asynchronously (for example if the admin does
not have admin permissions to the GitHub repository), installing
private themes becomes very difficult.
In this case, the Discourse admin can partially install the theme by
letting Discourse save the private key, create a placeholder theme and
give the admin a public key to be used as a deploy key. After the key
is installed, the admin can finish theme installation by pressing a
button on the theme page.
Hard deleting topics that contained soft deleted posts or small actions
used to create orphan posts because only the first post was hard
deleted. This commit adds an error message if there are still posts left
in the topic that must be hard deleted first or hard deletes all small
actions too immediately (there is no other way of hard deleting a small
action because there is no wrench menu).
Some of the changes in this PR are extracted from https://github.com/discourse/discourse/pull/17379.
Similar to the bookmarks tab in the new user menu, the messages tab also displays a mix of notifications and messages. When there are unread message notifications, the tab displays all of these notifications at the top and fills the remaining space in the menu with a list of the user's messages. The bubble/badge count on the messages tab indicates how many unread message notifications there are.
This is a much better description of its function. It performs idempotent normalization of a URL. If consumers truly need to `encode` a URL (including double-encoding of existing encoded entities), they can use the existing `.encode` method.
Some of the changes in this commit are extracted from https://github.com/discourse/discourse/pull/17379.
The bookmarks tab in the new user menu is different from the other tabs in that it can display a mixture of notifications and bookmarks. When there are unread bookmark reminder notifications, the tab displays all of these notifications at the top and fills the remaining space in the menu with the rest of the bookmarks. The bubble/badge count on the bookmarks tab indicates how many unread bookmark reminder notifications there are.
On the technical aspect, since this commit introduces a new `bookmark-item` component, we've done some refactoring so that all 3 "item" components (`notification-item`, `reviewable-item` and the new `bookmark-item`) inherit from a base component and get identical HTML structure so they all look consistent.
Internal tickets: t70584 and t65045.
Currently we can’t add a case-sensitive watched word if another one
exists with a different case. For example, the existing watched word
`Meta` has been created and is case-sensitive. Now an admin tries to add
`metA` while marking it as case-sensitive too, this won’t work and the
word won’t be added.
This patch changes this behavior by allowing to add same words that have
different cases, so the example above will now work as expected.
We still check for uniqueness but case-sensitivy is now taken
into account. It means that if the watched word `meta` already exists
and is not case-sensitive then it will not be possible to add `Meta`
(case-sensitive or not) as `meta` already matches every possible
variations of this word.
Follow-up to ce9eec8606.
I did a last-minute refactoring before merging the commit above where I extracted the Message Bus publish call into a new method, but forgot to delete the publish call after adding a call to the new method.
The content from the remote site and the footer get cached for 10 minutes, so Discourse should use the default locale instead of the user locale for the footer. Otherwise Discourse might cache the message in a different language.
* FEATURE: Add case-sensitivity flag to watched_words
Currently, all watched words are matched case-insensitively. This flag
allows a watched word to be flagged for case-sensitive matching.
To allow allow for backwards compatibility the flag is set to false by
default.
* FEATURE: Support case-sensitive creation of Watched Words via API
Extend admin creation and upload of Watched Words to support case
sensitive flag. This lays the ground work for supporting
case-insensitive matching of Watched Words.
Support for an extra column has also been introduced for the Watched
Words upload CSV file. The new column structure is as follows:
word,replacement,case_sentive
* FEATURE: Enable case-sensitive matching of Watched Words
WordWatcher's word_matcher_regexp now returns a list of regular
expressions instead of one case-insensitive regular expression.
With the ability to flag a Watched Word as case-sensitive, an action
can have words of both sensitivities.This makes the use of the global
Regexp::IGNORECASE flag added to all words problematic.
To get around platform limitations around the use of subexpression level
switches/flags, a list of regular expressions is returned instead, one for each
case sensitivity.
Word matching has also been updated to use this list of regular expressions
instead of one.
* FEATURE: Use case-sensitive regular expressions for Watched Words
Update Watched Words regular expressions matching and processing to handle
the extra metadata which comes along with the introduction of
case-sensitive Watched Words.
This allows case-sensitive Watched Words to matched as such.
* DEV: Simplify type casting of case-sensitive flag from uploads
Use builtin semantics instead of a custom method for converting
string case flags in uploaded Watched Words to boolean.
* UX: Add case-sensitivity details to Admin Watched Words UI
Update Watched Word form to include a toggle for case-sensitivity.
This also adds support for, case-sensitive testing and matching of Watched Word
in the admin UI.
* DEV: Code improvements from review feedback
- Extract watched word regex creation out to a utility function
- Make JS array presence check more explicit and readable
* DEV: Extract Watched Word regex creation to utility function
Clean-up work from review feedback. Reduce code duplication.
* DEV: Rename word_matcher_regexp to word_matcher_regexp_list
Since a list is returned now instead of a single regular expression,
change `word_matcher_regexp` to `word_matcher_regexp_list` to better communicate
this change.
* DEV: Incorporate WordWatcher updates from upstream
Resolve conflicts and ensure apply_to_text does not remove non-word characters in matches
that aren't at the beginning of the line.
Relative URLs will work just fine for Web Workers, which were the original reason for introducing the `theme_uploads_local` feature
Making them relative will mean that `loadScript()` automatically uses the CDN (when enabled), which is doubly important because our CSP doesn't allow loading theme-javascripts from the host domain when the CDN is enabled.
When viewing a topic, we execute two queries to fetch the topic's
public topic timer and slow mode timer. The former query happens to be
able to use a unique index but the latter has to do a seq scan which is
slow. The query itself is not expensive but since viewing a topic is a
hot path, the little cuts add up overtime and the query itself
contributes significantly to the load of the database.
This commit removes the ability to enable/disable the Sidebar on a per
user basis and introduces a site wide setting. For testing purposes, sidebar can be enabled/disabled via the `enable_sidebar=1` or `enable_sidebar=0` query param.
The previous method for reused the PrettyText logic which applied the
watched word logic, but had the unwanted effect of cooking the text too.
This meant that regular text values were converted to HTML.
Follow up to commit 5a4c35f627.
Our theme system is very complex and it can take a while to figure out how to invalidate the various types of caches that are used throughout the theme system. So, having a single helper method that invalidates everything can be useful in emergency situations where there is no time to read through the code and figure out how to clear the various caches.
Internal ticket: t64732.
The invites should be redeemed during the signup process. This was a
problem because when user tried to redeem an admin invite it tried to
authenticate the user using information from the session that was not
available.
This commit introduces a new plugin API to register
a group of stats that will be included in about.json
and also conditionally in the site about UI at /about.
The usage is like this:
```ruby
register_about_stat_group("chat_messages", show_in_ui: true) do
{
last_day: 1,
"7_days" => 10,
"30_days" => 100,
count: 1000,
previous_30_days: 120
}
end
```
In reality the stats will be generated any way the implementer
chooses within the plugin. The `last_day`, `7_days`, `30_days,` and `count`
keys must be present but apart from that additional stats may be added.
Only those core 4 stat keys will be shown in the UI, but everything will be shown
in about.json.
The stat group name is used to prefix the stats in about.json like so:
```json
"chat_messages_last_day": 2322,
"chat_messages_7_days": 2322,
"chat_messages_30_days": 2322,
"chat_messages_count": 2322,
```
The `show_in_ui` option (default false) is used to determine whether the
group of stats is shown on the site About page in the Site Statistics
table. Some stats may be needed purely for reporting purposes and thus
do not need to be shown in the UI to admins/users. An extension to the Site
serializer, `displayed_about_plugin_stat_groups`, has been added so this
can be inspected on the client-side.
* FIX: properly validate multiselect user fields on user creation
* Add test cases
* FIX: don't check multiselect user fields for watched words
* Clarifiy/simplify tests
* Roll back apply_watched_words changes
Since this method no longer needs to deal with arrays for now. If/when
we add new user fields which uses them, we can deal with it then.
All `DistributedCache` instances in Discourse are automatically keyed on the `Discourse.git_version`. Normally the theme compiler version is updated via a commit, and everything is fine. However, in some situations, it's possible for the BASE_COMPILER_VERSION to change without a change to the git_version (e.g. when applying patches directly to the codebase).
This commit adds the `BASE_COMPILER_VERSION` to the DistributedCache key to ensure that content from different compiler versions does not leak into other processes.
The `unread_not_too_old` attribute is a little odd because there should never be a case where
the user's first_unread_at column is less than the `Topic#updated_at`
column of an unread topic. The `unread_not_too_old` attribute is causing
a bug where topic states synced into `TopicTrackingState` do not appear
as unread because the attribute does not exsist on a normal `Topic`
object and hence never set.
It makes more sense to use user_ids for the UserCommScreener
introduced in fa5f3e228c since
in most cases the ID will be available, not the username. This
was discovered while starting work on a plugin that will
use this. In the cases where only usernames are available
the extra query is negligble.
The idea behind this refactor is to centralise all of the user ignoring / muting / disallow PM checks in a single place, so they can be used consistently in core as well as for plugins like chat, while improving the main bulk of the checks to run in a single fast non-AR query.
Also fixed up the invite error when someone is muting/ignoring the user that is trying to invite them to the topic.
Currently we only apply watched words of the `Block` type to custom user
fields and user profile fields.
This patch enables all rules to be applied such as `Censor` or
`Replace`.
Before, whispers were only available for staff members.
Config has been changed to allow to configure privileged groups with access to whispers. Post migration was added to move from the old setting into the new one.
I considered having a boolean column `whisperer` on user model similar to `admin/moderator` for performance reason. Finally, I decided to keep looking for groups as queries are only done for current user and didn't notice any N+1 queries.
Updates automatically data on the stats section of the topic.
It will update automatically the following information: likes, replies and last reply (timestamp and user)
We have a `cleanup!` class method on bookmarks that deletes
bookmarks X days after their related record (post/topic) are
deleted. This commit changes this method to use the
registered_bookmarkables for this instead, and each bookmarkable
type can delete related bookmarks in their own way.
In certain situations, a logged in user can redeem an invite with an email that
either doesn't match the invite's email or does not adhere to the email domain
restriction of an invite link. The impact of this flaw is aggrevated
when the invite has been configured to add the user that accepts the
invite into restricted groups.
At some point in the past we decided to rename the 'regular' notification state of topics/categories to 'normal'. However, some UI copy was missed when the initial renaming was done so this commit changes the spots that were missed to the new name.
This is pre-request work to introduce a splash screen while site assets load.
The only change this commit introduces is that it ensures we add the defer attribute to core/plugin/theme .JS files. This will allow us to insert markup before the browser starts evaluating those scripts later on. It has no visual or functional impact on core.
This will not have any impact on how themes and plugins work. The only exception is themes loading external scripts in the </head> theme field directly via script tags. Everything will work the same but those would need to add the defer attribute if they want to keep the benefits introduced in this PR.
This reverts commit 94c3bbc2d1.
At this current point in time, we do not have enough data on whether
this centralisation is the trade-offs of coupling features into a single
channel.
When sending emails with delivery_method_options -> return_response
set to true, the SMTP sending code inside Mail will return the SMTP
response when calling deliver! for mail within the app. This commit
ensures that Email::Sender captures this response if it is returned
and stores it against the EmailLog created for the sent email.
A follow up PR will make this visible within the admin email UI.
The query is very inefficient without any constraints on large sites and
the average of all time to first response since the beginning of time is
not useful as well.
We do not zero-pad our base62 short URLs, so there is no guarantee that the length is 27. Instead, let's greedily match all consecutive base62 characters and look for a matching upload.
This reverts bd32656157 and 36f5d5eada.
This table holds associations between uploads and other models. This can be used to prevent removing uploads that are still in use.
* DEV: Create upload_references
* DEV: Use UploadReference instead of PostUpload
* DEV: Use UploadReference for SiteSetting
* DEV: Use UploadReference for Badge
* DEV: Use UploadReference for Category
* DEV: Use UploadReference for CustomEmoji
* DEV: Use UploadReference for Group
* DEV: Use UploadReference for ThemeField
* DEV: Use UploadReference for ThemeSetting
* DEV: Use UploadReference for User
* DEV: Use UploadReference for UserAvatar
* DEV: Use UploadReference for UserExport
* DEV: Use UploadReference for UserProfile
* DEV: Add method to extract uploads from raw text
* DEV: Use UploadReference for Draft
* DEV: Use UploadReference for ReviewableQueuedPost
* DEV: Use UploadReference for UserProfile's bio_raw
* DEV: Do not copy user uploads to upload references
* DEV: Copy post uploads again after deploy
* DEV: Use created_at and updated_at from uploads table
* FIX: Check if upload site setting is empty
* DEV: Copy user uploads to upload references
* DEV: Make upload extraction less strict
This reverts one of the changes introduced just now in:
27d7b0c6de
I don't think we need this `activated_not_suspended_not_staged` scope
because we can just compose it ourselves via method chaining like
`User.activated.not_suspended.not_staged`.
This commit introduces a new site setting: `block_hotlinked_media`. When enabled, all attempts to hotlink media (images, videos, and audio) will fail, and be replaced with a linked placeholder. Exceptions to the rule can be added via `block_hotlinked_media_exceptions`.
`download_remote_image_to_local` can be used alongside this feature. In that case, hotlinked images will be blocked immediately when the post is created, but will then be replaced with the downloaded version a few seconds later.
This implementation is purely server-side, and does not impact the composer preview.
Technically, there are two stages to this feature:
1. `PrettyText.sanitize_hotlinked_media` is called during `PrettyText.cook`, and whenever new images are introduced by Onebox. It will iterate over all src/srcset attributes in the post HTML and check if they're allowed. If not, the attributes will be removed and replaced with a `data-blocked-hotlinked-src(set)` attribute
2. In the `CookedPostProcessor`, we iterate over all `data-blocked-hotlinked-src(set)` attributes and check whether we have a downloaded version of the media. If yes, we update the src to use the downloaded version. If not, the entire media element is replaced with a placeholder. The placeholder is labelled 'external media', and is a link to the offsite media.
* FIX: Email Send post has already been taken error
Adding a failing test first before coming up with a good solution.
Related: 357011eb3b
The above commit changed
```
PostReplyKey.find_or_create_by_safe!
```
to
```
PostReplyKey.create_or_find_by!
```
But I don't think it is working as a 1-1 replacement because of the
`Validation failed: Post has already been taken` error we are receiving
with this change. Also we need to make sure we don't re-introduce any
concurrency issues.
Reported: https://meta.discourse.org/t/224706/13
* Remove rails unique constraint and rely on db index
I believe this is what is causing `create_or_find_by!` to fail. Because
we have a unique constraint in the db I think we can remove this rails
unique constraint?
* clean up spec wording
This commit resolves a bug where users are not auto approved based on
`SiteSetting.auto_approve_email_domains` when
`SiteSetting.must_approve_users` has been enabled.
When a site has `SiteSetting.invite_only` enabled, we create a
`ReviewableUser`record when activating a user if the user is not
approved. Therefore, we need to approve the user when redeeming an
invite.
There are some uncertainties surrounding why a `ReviewableRecord` is
created for a user in an invites only site but this commit does not seek
to address that.
Follow-up to 7c4e2d33fa
This security fix affects sites which have `SiteSetting.must_approve_users`
enabled. There are intentional and unintentional cases where invited
users can be auto approved and are deemed to have skipped the staff approval process.
Instead of trying to reason about when auto-approval should happen, we have decided that
enabling the `must_approve_users` setting going forward will just mean that all new users
must be explicitly approved by a staff user in the review queue. The only case where users are auto
approved is when the `auto_approve_email_domains` site setting is used.
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
Previously this mapping of **cooked** images was only being run for oneboxes. Now it runs for all images, so we can transform hotlinked images without needing to immediately update `raw`
This commit migrates all bookmarks to be polymorphic (using the
bookmarkable_id and bookmarkable_type) columns. It also deletes
all the old code guarded behind the use_polymorphic_bookmarks setting
and changes that setting to true for all sites and by default for
the sake of plugins.
No data is deleted in the migrations, the old post_id and for_topic
columns for bookmarks will be dropped later on.
- Only validate if custom_fields are loaded, so that we don't trigger a db query
- Only validate public user fields, not all custom_fields
This commit also reverts the unrelated spec changes in ba148e08, which were required to work around these issues
Currently we don’t apply watched words to custom user fields nor user
profile fields.
This led to users being able to use blocked words in their bio, location
or some custom user fields.
This patch addresses this issue by adding some validations so it’s not
possible anymore to save the User model or the UserProfile model if they
contain blocked words.
This allows the category_id filter for the bookmark
report to work with polymorphic bookmarks. Honestly this
is a little hardcode-y at the moment but until we go and
make this report a lot more flexible with more filters
I don't think it's worth the work to add extra interfaces
to RegisteredBookmarkable and BaseBookmarkable to make
this more flexible. This is enough for now.
We have not used anything related to bookmarks for PostAction
or UserAction records since 2020, bookmarks are their own thing
now. Deleting all this is just cleaning up old cruft.
Latest redis interoduces a block form of multi / pipelined, this was incorrectly
passed through and not namespaced.
Fix also updates logster, we held off on upgrading it due to missing functions
A bit of a mixed bag, this addresses several edge areas of bookmarks and makes them compatible with polymorphic bookmarks (hidden behind the `use_polymorphic_bookmarks` site setting). The main ones are:
* ExportUserArchive compatibility
* SyncTopicUserBookmarked job compatibility
* Sending different notifications for the bookmark reminders based on the bookmarkable type
* Import scripts compatibility
* BookmarkReminderNotificationHandler compatibility
This PR also refactors the `register_bookmarkable` API so it accepts a class descended from a `BaseBookmarkable` class instead. This was done because we kept having to add more and more lambdas/properties inline and it was very messy, so a factory pattern is cleaner. The classes can be tested independently as well.
Some later PRs will address some other areas like the discourse narrative bot, advanced search, reports, and the .ics endpoint for bookmarks.
We're adding this column now in preparation for a future commit(s) that will
redesign the avatar/notifications menu. The reason the column is added in a
separate commit is because the redesign changes are going to be complex with a
high risk of getting (temporarily) reverted and if they included a database
migration, they wouldn't revert cleanly/easily.
Internal ticket: t65045.
When a user views a topic that contains a topic timer to publish to a
restricted category, an error occurs on the client side because the user
does not have access to information about the category.
This commit fixes it such that the topic timer is not shown to the user
if the user does not have access to the category.
On some installations, this would fail with 'index row size exceeds btree version 4 maximum'. This commit replaces the (post_id, url)` index with a `(post_id, md5(url))` index, which is much more space efficient.
This will make future changes to the 'pull hotlinked images' system easier. This commit should not introduce any functional change.
For now, the old post_custom_field data is kept in the database. This will be dropped in a future commit.
This commit introduces a new site setting: `use_name_for_username_suggestions` (default true)
Admins can disable it if they want to stop using Name values when generating usernames for users. This can be useful if you want to keep real names private-by-default or, when used in conjunction with the `use_email_for_username_and_name_suggestions` setting, you would prefer to use email-based username suggestions.
* hidden siteSetting to enable experimental sidebar
* user preference to enable experimental sidebar
* `experimental_sidebar_enabled` attribute for current user
* Empty glimmer component for Sidebar
We were calling `dup` on the hash and using that to check for changes. However, we were not duplicating the values, so changes to arrays or nested hashes would not be detected.
This pull request follows on from https://github.com/discourse/discourse/pull/16308. This one does the following:
* Changes `BookmarkQuery` to allow for querying more than just Post and Topic bookmarkables
* Introduces a `Bookmark.register_bookmarkable` method which requires a model, serializer, fields and preload includes for searching. These registered `Bookmarkable` types are then used when validating new bookmarks, and also when determining which serializer to use for the bookmark list. The `Post` and `Topic` bookmarkables are registered by default.
* Adds new specific types for Post and Topic bookmark serializers along with preloading of associations in `UserBookmarkList`
* Changes to the user bookmark list template to allow for more generic bookmarkable types alongside the Post and Topic ones which need to display in a particular way
All of these changes are gated behind the `use_polymorphic_bookmarks` site setting, apart from the .hbs changes where I have updated the original `UserBookmarkSerializer` with some stub methods.
Following this PR will be several plugin PRs (for assign, chat, encrypt) that will register their own bookmarkable types or otherwise alter the bookmark serializers in their own way, also gated behind `use_polymorphic_bookmarks`.
This commit also removes `BookmarkQuery.preloaded_custom_fields` and the functionality surrounding it. It was added in 0cd502a558 but only used by one plugin (discourse-assign) where it has since been removed, and is now used by no plugins. We don't need it anymore.
Previously, 'crop' would resize the image to have the requested width, then crop the height to the requested value. This works when cropping images vertically, but not when cropping them horizontally.
For example, trying to crop a 500x500 image to 200x500 was actually resulting in a 200x200 image. Having an OptimizedImage with width/height columns mismatching the actual OptimizedImage width/height causes some unusual issues.
This commit ensures that a call to `OptimizedImage.crop(from, to, width, height)` will always return an image of the requested width/height. The `w x h^` syntax defines minimum width/height, while maintaining aspect ratio.
This commit fixes two issues at play. The first was introduced
in f6c852b (or maybe not introduced
but rather revealed). When a user posted a new message in a topic,
they received the unread topic tracking state MessageBus message,
and the Unread (X) indicator was incremented by one, because with the
aforementioned perf commit we "guess" the correct last read post
for the user, because we no longer calculate individual users' read
status there. This meant that every time a user posted in a topic
they tracked, the unread indicator was incremented. To get around
this, we can just exclude the user who created the post from the
target users of the unread state message.
The second issue was related to the private message topic tracking
state, and was somewhat similar. Whenever a user created a new private
message, the New (X) indicator was incremented, and could not be
cleared until the page was refreshed. To solve this, we just don't
update the topic state for the user when the new_topic tracking state
message comes through if the user who created the topic is the
same as the current user.
cf. https://meta.discourse.org/t/bottom-of-topic-shows-there-is-1-unread-remaining-when-there-are-actually-0-unread-topics-remaining/220817
Discourse has the Discourse Connect Provider protocol that makes it possible to
use a Discourse instance as an identity provider for external sites. As a
natural extension to this protocol, this PR adds a new feature that makes it
possible to use Discourse as a 2FA provider as well as an identity provider.
The rationale for this change is that it's very difficult to implement 2FA
support in a website and if you have multiple websites that need to have 2FA,
it's unrealistic to build and maintain a separate 2FA implementation for each
one. But with this change, you can piggyback on Discourse to take care of all
the 2FA details for you for as many sites as you wish.
To use Discourse as a 2FA provider, you'll need to follow this guide:
https://meta.discourse.org/t/-/32974. It walks you through what you need to
implement on your end/site and how to configure your Discourse instance. Once
you're done, there is only one additional thing you need to do which is to
include `require_2fa=true` in the payload that you send to Discourse.
When Discourse sees `require_2fa=true`, it'll prompt the user to confirm their
2FA using whatever methods they've enabled (TOTP or security keys), and once
they confirm they'll be redirected back to the return URL you've configured and
the payload will contain `confirmed_2fa=true`. If the user has no 2FA methods
enabled however, the payload will not contain `confirmed_2fa`, but it will
contain `no_2fa_methods=true`.
You'll need to be careful to re-run all the security checks and ensure the user
can still access the resource on your site after they return from Discourse.
This is very important because there's nothing that guarantees the user that
will come back from Discourse after they confirm 2FA is the same user that
you've redirected to Discourse.
Internal ticket: t62183.
This commit improves the logic for rolling up IPv4 screened IP
addresses and extending it for IPv6. IPv4 addresses will roll up only
up to /24. IPv6 can rollup to /48 at most. The log message that is
generated contains the list of original IPs and new subnet.
* FEATURE: Let sites add a sitemap.xml file.
This PR adds the same features discourse-sitemap provides to core. Sitemaps are only added to the robots.txt file if the `enable_sitemap` setting is enabled and `login_required` disabled.
After merging discourse/discourse-sitemap#34, this change will take priority over the sitemap plugin because it will disable itself. We're also using the same sitemaps table, so our migration won't try to create it
again using `if_not_exists: true`.
It's possible in Rails to map a single route to multiple controller
actions with different constraints. We do this in at least 1 place in
our application for the root route (/) to make it possible to change the
page that root route displays.
This means that if you get the list of routes of your application,
you'll get the same route for each time the route is defined. And if
there's an API scope for 2 (or more) controller actions that map to the
same route, the route will be listed twice in the Allowed URLs list of
the scope.
To prevent this, this PR adds the allowed URLs in a set so that
duplicate routes are automatically removed.
The Allowed URLs list of an API scope only includes routes that
constraint the format for the route to JSON. However, some routes define
no format constraints, but that doesn't mean they can't be used by an
API key.
This commit amends the logic for the Allowed URLs list so that it
includes routes that have no format constraints or the format
constraints include JSON.
Due to default CSP web workers instantiated from CDN based assets are still
treated as "same-origin" meaning that we had no way of safely instansiating
a web worker from a theme.
This limits the theme system and adds the arbitrary restriction that WASM
based components can not be safely used.
To resolve this limitation all js assets in about.json are also cached on
local domain.
{
"name": "Header Icons",
"assets" : {
"worker" : "assets/worker.js"
}
}
This can then be referenced in JS via:
settings.theme_uploads_local.worker
local_js_assets are unconditionally served from the site directly and
bypass the entire CDN, using the pre-existing JavascriptCache
Previous to this change this code was completely dormant on sites which
used s3 based uploads, this reuses the very well tested and cached asset
system on s3 based sites.
Note, when creating local_js_assets it is highly recommended to keep the
assets lean and keep all the heavy working in CDN based assets. For example
wasm files can still live on the CDN but the lean worker that loads it can
live on local.
This change unlocks wasm in theme components, so wasm is now also allowed
in `theme_authorized_extensions`
* more usages of upload.content
* add a specific test for upload.content
* Adjust logic to ensure that after upgrades we still get a cached local js
on save
Previously we only supported a single 'required tag group' for a category. This commit allows admins to specify multiple required tag groups, each with their own minimum tag count.
A new category_required_tag_groups database table replaces the existing columns on the categories table. Data is automatically migrated.
This patch removes some of our freedom patches that have been deprecated
for some time now.
Some of them have been updated so we’re not shipping code based on an
old version of Rails.
Via the API it is possible to create a user with an integer username. So
123 instead of "123". This causes the following 500 error:
```
NoMethodError (undefined method `unicode_normalize' for 1:Integer)
app/models/user.rb:276:in `normalize_username'
```
See: https://meta.discourse.org/t/222281
Previous to this change if any of the assets were not allowed extensions
they would simply be silently ignored, this could lead to broken themes
that are very hard to debug
Our group fabrication creates groups with name "my_group_#{n}" where n
is the sequence number of the group being created. However, this can
cause the test to be flaky if and when a group with name `my_group_10`
is created as it will be ordered before
`my_group_9`. This commits makes the group names determinstic to
eliminate any flakiness.
This reverts commit 558bc6b746.
This commit introduces a new use_polymorphic_bookmarks site setting
that is default false and hidden, that will be used to help continuous
development of polymorphic bookmarks. This setting **should not** be
enabled anywhere in production yet, it is purely for local development.
This commit uses the setting to enable create/update/delete actions
for polymorphic bookmarks on the server and client side. The bookmark
interactions on topics/posts are all usable. Listing, searching,
sending bookmark reminders, and other edge cases will be handled
in subsequent PRs.
Comprehensive UI tests will be added in the final PR -- we already
have them for regular bookmarks, so it will just be a matter of
changing them to be for polymorphic bookmarks.
Tags (and tag groups) can be configured so that they can only be used in specific categories and (optionally) restrict topics in these categories to be able to add/use only these tags. These restrictions work as expected when a topic is created without going through the review queue; however, if the topic has to be reviewed by a moderator then these restrictions currently aren't checked before the topic is sent to the review queue, but they're checked later when a moderator tries to approve the topic. This is because if a user manages to submit a topic that doesn't meet the restrictions, moderators won't be able to approve and it'll be stuck in the review queue.
This PR prevents topics that don't meet the tags requirements from being sent to the review queue and shows the poster an error message that indicates which tags that cannot be used.
Internal ticket: t60562.
As we are gradually moving to having a polymorphic
bookmarkable relationship on the Bookmark table,
we need to make the post_id column nullable to be
able to develop and test the new columns, and
for cutover/migration purposes later as well.
PostAnalyzer and CookedPostProcessor both replace URLs with oneboxes.
PostAnalyzer did not use the max_oneboxes_per_post site and setting and
CookedPostProcessor replaced at most max_oneboxes_per_post URLs ignoring
the oneboxes that were replaced already by PostAnalyzer.
This commit is a redo of2f1ddadff7dd47f824070c8a3f633f00a27aacde
which we reverted because it blew up an internal CI check. I looked
into it, and it happened because the old migration to add the bookmark
columns still existed, and those columns were dropped in a post migrate,
so the two migrations to add the columns were conflicting before
the post migrate was run.
------
This commit only includes the creation of the new columns and index,
and does not add any triggers, backfilling, or new data.
A backfill will be done in the final PR when we switch this over.
Intermediate PRs will look something like this:
Add an experimental site setting for using polymorphic bookmarks,
and make sure in the places where bookmarks are created or updated
we fill in the columns. This setting will be used in subsequent
PRs as well.
Listing and searching bookmarks based on polymorphic associations
Creating post and topic bookmarks using polymorphic associations,
and changing special for_topic logic to just rely on the Topic
bookmarkable_type
Querying bookmark reminders based on polymorphic associations
Make sure various other areas like importers, bookmark guardian,
and others all rely on the associations
Prepare plugins that rely on the Bookmark model to use polymorphic
associations
The final core PR will remove all the setting gates and switch over
to using the polymorphic associations, backfill the bookmarks
table columns, and ignore the old post_id and for_topic colummns.
Then it will just be a matter of dropping the old columns down the
line.
This commit is a redo of e21c640a3c
which we reverted to not include half-done work in a release.
This commit is slightly different though, in that it only includes
the creation of the new columns and index, and does not add any
triggers, backfilling, or new data.
A backfill will be done in the final PR when we switch this over.
Intermediate PRs will look something like this:
1. Add an experimental site setting for using polymorphic bookmarks,
and make sure in the places where bookmarks are created or updated
we fill in the columns. This setting will be used in subsequent
PRs as well.
2. Listing and searching bookmarks based on polymorphic associations
3. Creating post and topic bookmarks using polymorphic associations,
and changing special for_topic logic to just rely on the Topic
bookmarkable_type
4. Querying bookmark reminders based on polymorphic associations
5. Make sure various other areas like importers, bookmark guardian,
and others all rely on the associations
6. Prepare plugins that rely on the Bookmark model to use polymorphic
associations
The final core PR will remove all the setting gates and switch over
to using the polymorphic associations, backfill the bookmarks
table columns, and ignore the old post_id and for_topic colummns.
Then it will just be a matter of dropping the old columns down the
line.
Plugins like chat add custom score type to override the title in the UI, but that should be reserved for situations when you need to manage the flag priority separately, which is configurable in the queue settings page.
Currently, if a plugin creates a custom score type, it won't be able to associate a priority, so there's no real gain from doing so. Priorities are tightly related to post-action types, which is something we might want to revise. For now, this change lets plugins move away from custom score types without compromises.
The meaning of reminder_at and reminder_last_sent_at changed after
commit 6d422a8033. A bookmark reminder
will fire only if reminder_last_sent_at is null, but before that it
fired everytime reminder_at was set. This is no longer true because
sometimes reminder_at continues to exist even after a reminder fired.
* FEATURE: use canonical links in posts.rss feed
Previously we used non canonical links in posts.rss
These links get crawled frequently by crawlers when discovering new
content forcing crawlers to hop to non canonical pages just to end up
visiting canonical pages
This uses up expensive crawl time and adds load on Discourse sites
Old links were of the form:
`https://DOMAIN/t/SLUG/43/21`
New links are of the form
`https://DOMAIN/t/SLUG/43?page=2#post_21`
This also adds a post_id identified element to crawler view that was
missing.
Note, to avoid very expensive N+1 queries required to figure out the
page a post is on during rss generation, we cache that information.
There is a smart "cache breaker" which ensures worst case scenario is
a "page drift" - meaning we would publicize a post is on page 11 when
it is actually on page 10 due to post deletions. Cache holds for up to
12 hours.
Change only impacts public post RSS feeds (`/posts.rss`)
In the API keys page where admins can create API keys with restricted scopes, each scope shows a list of URLs that it allows. But currently, this list of allowed URLs shows incomplete URLs for scopes that are added by plugins. For example, the allowed URL for the "run queries" scope of the data-explorer plugin is shown as `/queries/:id/run` when the correct URL for this scope is `/admin/plugins/explorer/queries/:id/run`. The first 3 segments of the path are the mount path of the plugin's engine and it's missing because the routes set of the engine doesn't include the mount path. To fix this, this commit gets the mount path and prepends it to the URL so the complete URL is shown to the user.
It's not possible to write tests for this change because plugins are not loaded in the test environment by default when core's tests suite is running.
The user can select what happens with a bookamrk after it expires. New
option allow bookmark's reminder to be kept even after it has expired.
After a bookmark's reminder notification is created, the reminder date
will be highlighted in red until the user resets the reminder date.
User can do that using the new Clear Reminder button from the dropdown.
This change adds support for the categories endpoint to have an api
scope. Only adds GET scope for listing categories and for fetching a
single category.
See: https://meta.discourse.org/t/218080/4
Our @mention user search prioritized users based on prefix matches.
So if searching for `sa` we will display `sam`, `asam` in that order
Previously, we did not prioritize group matches based on prefix. This change ensures better parity.
Implementation notes:
1. User search only prioritizes based on username prefix, not name prefix. TBD if we want to change that.
2. @mention on client side will show 0 group matches if we fill up all the spots with user matches. TBD if we want to unconditionally show the first / second group match.
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
* FEATURE: upload an avatar option for uploading avatars with selectable avatars
Allow staff or users at or above a trust level to upload avatars even when the site
has selectable avatars enabled.
Everyone can still pick from the list of avatars. The option to upload is shown
below the selectable avatar list.
refactored boolean site setting into an enum with the following values:
disabled: No selectable avatars enabled (default)
everyone: Show selectable avatars, and allow everyone to upload custom avatars
tl1: Show selectable avatars, but require tl1+ and staff to upload custom avatars
tl2: Show selectable avatars, but require tl2+ and staff to upload custom avatars
tl3: Show selectable avatars, but require tl3+ and staff to upload custom avatars
tl4: Show selectable avatars, but require tl4 and staff to upload custom avatars
staff: Show selectable avatars, but only allow staff to upload custom avatars
no_one: Show selectable avatars. No users can upload custom avatars
Co-authored-by: Régis Hanol <regis@hanol.fr>
Previously cached counting made redis calls in main thread and performed
the flush in main thread.
This could lead to pathological states in extreme heavy load.
This refactor reduces load and cleans up the interface
Previously we were publishing one messagebus message per user which was 'tracking' a topic. On large sites, this can easily be 1000+ messages. The important information in the message is common between all users, so we can manage with a single message on a shared channel, which will be much more efficient.
For user-specific values (notification_level and last_read_post_number), the JS app can infer values which are 'good enough'. Correct values will be loaded as soon as a topic-list containing the topic is visited.
aa1442fdc3 split theme stylesheets so that every component gets its own stylesheet. Therefore, there is now no need for parent themes to collate the settings/variables of its children during scss compilation.
Technically this is a breaking change for any themes which depend on the settings/variables of their child components. That was never a supported/recommended arrangement, so we don't expect this to cause issues.
If a theme is updated to introduce a new setting AND immediately make use of it in a stylesheet, then an error was being shown. This is because the stylesheet compilation was using the theme's cached settings, and the cache is only cleared **after** the theme has finished compiling.
This commit updates the SCSS compilation to use uncached values for settings. A similar fix was applied to other parts of theme compilation back in 2020: (a51b8d9c66)
The previous Discourse.cache usage was different to how other theme-related caching is handled, and also requires reaching out to redis every time. The common theme cache is held in memory (as a DistributedCache)
This commit makes sure that the email log's bounce_error_code
conforms to the SMTP error code RFC on save, so that
it is always in the format X.X.X or XXX without any
additional string details. Also included is a migration
to fix this issue for past records.
Similar site settings exist for likes and edits and the new ones work
in a similar way.
By default, users below TL2 have a limit of 20, the limit is increased
by 1.5 for TL2 users up to 30, by 2 for TL3 users up to 40 and by 3 for
TL4 users up to 60.
We validate the *format* of email addresses in many places with a match against
a regex, often with very slightly different syntax.
Adding a separate EmailAddressValidator simplifies the code in a few spots and
feels cleaner.
Deprecated the old location in case someone is using it in a plugin.
No functionality change is in this commit.
Note: the regex used at the moment does not support using address literals, e.g.:
* localpart@[192.168.0.1]
* localpart@[2001:db8::1]
* FIX: Don't accept accents in slug if generation_method == 'ascii'
Fixes bug reported in:
- https://meta.discourse.org/t/404-when-trying-to-edit-category-with-accent-in-slug/214762
- https://meta.discourse.org/t/formatting-and-accents-in-urls/215734/5
Assuming `SiteSetting.slug_generation_method == 'ascii'.
If the user provides a slug containing non-ascii characters while
creating the category, the user will receive a 404 error just
after saving the category since the slug will be escaped anyway but
Category.find_by_slug_path won't escape the category slug
causing the Edit Page of the category to be inaccessible.
This commit checks the provided slug and raises an error if the
provided slugcontains non-ascii characters ensuring that the
provided value is consistent with the site settings.
It also changes Category.find_by_slug_path to always escape the slug,
since if present, it is escaped anyway in Category.ensure_slug to
prevent the 404 in the Edit Category Page in case the user already
have some category with a non-ascii slug.
* Removed trailing whitespace
When parent category or grandparent category is muted, then category should be muted as well.
Still, it can be overridden by setting individual subcategory notification level.
CategoryUser record is not created, mute for subcategories is purely virtual.
This can happen if the topic to which a user is invited is in a private
category and the user was not invited to one of the groups that can see
that specific category.
This used to be a warning and this commit makes it an error.
This commit introduces two new APIs for handling unused uploads, one
can be used to exclude uploads in bulk when the data model allow and
the other one excludes uploads one by one.
Previously calls such as `Emoji["smile"]` would force a full dehydration of
objects from Redis.
This introduces a version safe site and global emoji cache so lookups are
cheap. It eliminates iterating through the list of emojis and pulling from
redis.
Distributed cache uses a normalized name as the key and stores an Array tuple
with version and Emoji. Successful hits always confirm version matches.
Interface to Emoji object remains unchanged.
We opted for 2 caches to improve reuse on multisites. misses though will be
stored in both caches. If there is a hit on the global cache we can avoid
looking up in site local cache and storing a miss there.
Previously we were calling `EXPIRE` every time we incremented a given key. Instead, we can call EXPIRE once when the key is first populated. A LUA script is used to make this as efficient as possible.
Consumers of this Concern use daily keys. Since we're now calling EXPIRE only at the beginning of the day, rather than throughout the day, the expire time has been increased from 3 to 4 days.
Whenever we got a bounced email in the Email::Receiver we
previously would just set bounced: true on the EmailLog and
discard the status/diagnostic code. This commit changes this
flow to store the bounce error code (defined in the RFC at
https://www.iana.org/assignments/smtp-enhanced-status-codes/smtp-enhanced-status-codes.xhtml)
not just in the Email::Receiver, but also via webhook events
from other mail services and from SNS.
This commit does not surface the bounce error in the UI,
we can do that later if necessary.
There is a couple of layers of caching for theme JavaScript in Discourse:
The first layer is the `javascript_caches` table in the database. When a theme
with JavaScript files is installed, Discourse stores each one of the JavaScript
files in the `theme_fields` table, and then concatenates the files, compiles
them, computes a SHA1 digest of the compiled JavaScript and store the results
along with the SHA1 digest in the `javascript_caches` table.
Now when a request comes in, we need to render `<script>` tags for the
activated theme(s) of the site. To do this, we retrieve the `javascript_caches`
records of the activated themes and generate a `<script>` tag for each record.
The `src` attribute of these tags is a path to the `/theme-javascripts/:digest`
route which simply responds with the compiled JavaScript that has the requested
digest.
The second layer is a distributed cache whose purpose is to make rendering
`<script>` a lot more efficient. Without this cache, we'd have to query the
`javascript_caches` table to retrieve the SHA1 digests for every single
request. So we use this cache to store the `<script>` tags themselves so that
we only have to retrieve the `javascript_caches` records of the activated
themes for the first request and future requests simply get the cached
`<script>` tags.
What this commit does it ensures that the SHA1 digest in the
`javascript_caches` table stay the same across compilations by adding an order
by id clause to the query that loads the `theme_fields` records. Currently, we
specify no order when retrieving the `theme_fields` records so the order in
which they're retrieved can change across compilations and therefore cause the
SHA1 to change even though the individual records have not changed at all.
An inconsistent SHA1 digest across compilations can cause the database cache
and the distributed cache to have different digests and that causes the
JavaScript to fail to load (and if the theme heavily customizes the site, it
gives the impression that the site is broken) until the cache is cleared.
This can happen in busy sites when 2 concurrent requests recompile the
JavaScript files of a theme at the same time (this can happen when deploying a
new Discourse version) and request A updates the database cache after request B
did, and request B updates the distributed cache after request A did.
Internal ticket: t60783.
Co-authored-by: David Taylor <david@taylorhq.com>
- Update UI to improve contrast
- Make it clear that the message is only shown to administrators
- Add theme name and id to the console output
- Parse the error backtrace to identify the theme-id for post-decoration errors
- Improve console output to include the theme name / URL
- Add `?safe_mode=no_custom` to the admin panel link, so that it will work even if the theme is causing the site to break
In 8e5b945b0f, we reverted the commit but
at the same time resulted in Theme::BASE_COMPILER_VERSION going
backwards which caused problems with themes caching.
This commit bumps the version to clear all the caches.
Follow-up to 8e5b945b0f
Breakdown of fixes in this commit:
* `UserStat#topic_count` was not updated when visibility of
the topic changed.
* `UserStat#post_count` was not updated when post was hidden or
unhidden.
* `TopicConverter` was only incrementing or decrementing the counts by 1
even if a user has multiple posts in the topic.
* The commit turns off the verbose logging by default as it is just
noise to normal users who are not debugging this problem.
- Update UI to improve contrast
- Make it clear that the message is only shown to administrators
- Add theme name and id to the console output
- Parse the error backtrace to identify the theme-id for post-decoration errors
- Improve console output to include the theme name / URL
- Add `?safe_mode=no_custom` to the admin panel link, so that it will work even if the theme is causing the site to break
This commits adds a new advance_draft to PostCreator that controls if
the draft sequence will be advanced or not. If the draft sequence is
advanced then the old drafts will be cleared. This used to happen for
posts created by plugins or through the API and cleared user drafts
by mistake.
* FEATURE: Add external_id to topics
This commit allows for topics to be created and fetched by an
external_id. These changes are API only for now as there aren't any
front changes.
* add annotations
* add external_id to this spec
* Several PR feedback changes
- Add guardian to find topic
- 403 is returned for not found as well now
- add `include_external_id?`
- external_id is now case insensitive
- added test for posts_controller
- added test for topic creator
- created constant for max length
- check that it redirects to the correct path
- restrain external id in routes file
* remove puts
* fix tests
* only check for external_id in webhook if exists
* Update index to exclude external_id if null
* annotate
* Update app/controllers/topics_controller.rb
We need to check whether the topic is present first before passing it to the guardian.
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
* Apply suggestions from code review
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
This commit allows group SMTP emails to be sent with a
different from email address that has been set up as an
alias in the email provider. Emails from the alias will
be grouped correctly using Message-IDs in the mail client,
and replies to the alias go into the correct group inbox.
Ensures that `UserStat#post_count` and `UserStat#topic_count` does not
go below 0. When it does like it did now, we tend to have bugs in our
code since we're usually coding with the assumption that the count isn't
negative.
In order to support the constraints, our post and topic fabricators in
tests will now automatically increment the count for the respective
user's `UserStat` as well. We have to do this because our fabricators
bypasss `PostCreator` which holds the responsibility of updating `UserStat#post_count` and
`UserStat#topic_count`.
Job arguments go via JSON, and so DateTime objects will appear as strings in the Job's `#execute` method. The latest version of Sidekiq has started warning about this to reduce developer confusion.
Non-staff users are not allowed to see whisper so this change prevents
non-staff user from seeing a like count that does not make sense to
them. In the future, we might consider adding another like count column
for staff user.
Follow-up to 4492718864
When the record is not saved, we should display a proper message.
One potential reason can be plugins for example discourse-calendar is specifying that only first post can contain event
Sometimes plugins need to have additional data or options available
when rendering custom markdown features/rules that are not available
on the default opts.discourse object. These additional options should
be namespaced to the plugin adding them.
```
Site.markdown_additional_options["chat"] = { limited_pretty_text_markdown_rules: [] }
```
These are passed down to markdown rules on opts.discourse.additionalOptions.
The main motivation for adding this is the chat plugin, which currently stores
chat_pretty_text_features and chat_pretty_text_markdown_rules on
the Site object via additions to the serializer, and the Site object is
not accessible to import via markdown rules (either through
Site.current() or through container.lookup). So, to have this working
for both front + backend code, we need to attach these additional options
from the Site object onto the markdown options object.
We initialize models as part of the warmup process in production, so this was being logged on every boot. We only want to log if a plugin is actually using the model, so after_save is a safer bet.
Follow up to: 12f041de5d
Probably best to lookup the "everyone" group_id instead of hard-coding
it to `0`. Also now its more clear what this `0` means.
Running `update_from_remote` and `save!` cause a number of side-effects, including instructing all clients to reload CSS files. If there are no changes, then this is wasteful, and can even cause a 'flicker' effect on clients as they reload CSS.
This commit checks if any updates are available before triggering `update_from_remote` / `save!`. This should be much faster, and stop the 'flickering' UX from happening on every themes:update run.
It also improves the output of the command to include the from/to commit hashes, which may be useful for debugging issues. For example:
```
Checking 'Alien Night | A Dark Discourse Theme' for 'default'... already up to date
Checking 'Star Wars' for 'default'... updating from d8a170dd to 66b9756f
Checking 'Media Overlay' for 'default'... already up to date
```
If a model class calls preload_custom_fields twice then
we have to clear this otherwise the fields are cached inside the
already existing proxy and no new ones are added, so when we check
for custom_fields[KEY] an error is likely to occur
This commit makes a more specific N1NotPreLoadedError from
StandardError to raise when a custom field is loaded before
being preloaded, so it is easier to test that this does
not happen from plugins. Also adds the name of the class
trying to load the custom field to the error message.
In the unlikely, but possible, scenario where a user has no email_tokens, and has an invite record for their email address, login would fail. This commit fixes the `Invite` `user_doesnt_already_exist` validation so that it only applies to new invites, or when changing the email address.
This regressed in d8fe0f4199 (based on `git bisect`)
* FIX: Tag watching for everyone tag groups
Tags in tag groups that have permissions set to everyone were not able
to be saved correctly. A user on their preferences page would mark the
tags that they wanted to save, but the watched_tags in the response
would be empty. This did not apply to admins, just regular users. Even
though the watched tags were being saved in the db, the user serializer
response was filtering them out. When a user refreshed their preferences
pages it would show zero watched tags.
This appears to be a regression introduced by:
0f598ca51e
The issue that needed to be fixed is that we don't track the "everyone"
group (which has an id of 0) in the group_users table. This is because
everyone has access to it, so why fill a row for every single user, that
would be a lot. The fix was to update the query to include tag groups
that had permissions set to the "everyone" group (group_id 0).
I also added another check to the existing spec for updating
watched tags for tags that aren't in a tag group so that it checks the
response body. I then added a new spec which updates watched tags for
tags in a tag group which has permissions set to everyone.
* Resolve failing tests
Improve SQL query syntax for including the "everyone" group with the id
of 0.
This commit also fixes a few failing tests that were introduced. It
turns out that the Fabrication of the Tag Group Permissions was faulty.
What happens when creating the tag groups without any permissions is
that it sets the permission to "everyone". If we then follow up with
fabricating a tag group permission on the tag group instead of having a
single permission it will have 2 (everyone + the group specified)! We
don't want this. To fix it I removed the fabrication of tag group
permissions and just set the permissions directly when creating the tag
group.
* Use response.parsed_body instead of JSON.parse
* FIX: Mark invites flash messages as HTML safe.
This change should be safe as all user inputs included in the errors are sanitized before sending it back to the client.
Context: https://meta.discourse.org/t/html-tags-are-explicit-after-latest-update/214220
* If somebody adds a new error message that includes user input and doesn't sanitize it, using html-safe suddenly becomes unsafe again. As an extra layer of protection, we make the client sanitize the error message received from the backend.
* Escape user input instead of sanitizing
The `plugin:pull_compatible_all` task is intended to take incompatible plugins and downgrade them to an earlier version. Problem is, when running the rake task in development/production environments, the plugins have already been activated. If an incompatible plugin raises an error in `plugin.rb` then the rake task will be unable to start.
This commit centralises our LOAD_PLUGINS detection, adds support for LOAD_PLUGINS=0 in dev/prod, and adds a warning to `plugin:pull_compatible_all` if it's run with plugins enabled.
An admin could search for all screened ip addresses in a block by
using wildcards. 192.168.* returned all IPs in range 192.168.0.0/16.
This feature allows admins to search for a single IP address in all
screened IP blocks. 192.168.0.1 returns all IP blocks that match it,
for example 192.168.0.0/16.
* FEATURE: Remove roll up button for screened IPs
* FIX: Match more specific screened IP address first
Having to load `ip_addr` is confusing especially when that file exists
to monkey patch Ruby's `IpAddr` class. Moving it to our freedom patches
folder which is automatically loaded on initialization.
This is a partial revert of 099b679fc5.
`Bookmark#topic_id` and `Bookmark#reminder_type` was dropped in
b22450c7a8 so we need to continue ignoring
the dropped columns so as to ensure a seamless deploy. Otherwise,
ActiveRecord's schema cache will still contain references to
`Bookmark#topic_id` when the column is dropped in a post migration.
It is too close to release of 2.8 for incomplete
feature shenanigans. Ignores and drops the columns and drops
the trigger/function introduced in
e21c640a3c.
Will pick this feature back up post-release.
We are planning on attaching bookmarks to more and
more other models, so it makes sense to make a polymorphic
relationship to handle this. This commit adds the new
columns and backfills them in the bookmark table, and
makes sure that any new bookmark changes fill in the columns
via DB triggers.
This way we can gradually change the frontend and backend
to use these new columns, and eventually delete the
old post_id and for_topic columns in `bookmarks`.
* File.exists? is deprecated and removed in Ruby 3.2 in favor of
File.exist?
* Dir.exists? is deprecated and removed in Ruby 3.2 in favor of
Dir.exist?
The rake task deleted here was added back in Feb 2020
when bookmarks were first converted from PostAction
records, it is no longer needed. The ignored columns
were removed in ed83d7573e.
This commit adds a check that runs regularly as per
2d68e5d942 which tests the
credentials of groups with SMTP or IMAP enabled. If any issues
are found with those credentials a high priority problem is added to the
admin dashboard.
This commit also formats the admin dashboard differently if
there are high priority problems, bringing them to the top of
the list and highlighting them.
The problem will be cleared if the issue is fixed before the next
problem check, or if the group's settings are updated with a valid
credential.
The `fancy_title` column in the `topics` table currently has a constraint that limits the column to 400 characters. We need to remove that constraint because it causes some automatic topics/PMs from the system to fail when using Discourse in locales that need more than 400 characters to the translate the content of those automatic messages.
Internal ticket: t58030.
This commit introduces scheduled problem checks for the admin dashboard, which are long running or otherwise cumbersome problem checks that will be run every 10 minutes rather than every time the dashboard is loaded. If these scheduled checks add a problem, the problem will remain until it is cleared or until the scheduled job runs again.
An example of a check that should be scheduled is validating credentials against an external provider.
This commit also introduces the concept of a `priority` to the problems generated by `AdminDashboardData` and the scheduled checks. This is `low` by default, and can be set to `high`, but this commit does not change any part of the UI with this information, only adds a CSS class.
I will be making a follow up PR to check group SMTP credentials.
Some time ago, we made this fix to external authentication – https://github.com/discourse/discourse/pull/13706. We didn't address Discourse Connect (https://meta.discourse.org/t/discourseconnect-official-single-sign-on-for-discourse-sso/13045) at that moment, so I wanted to fix it for Discourse Connect as well.
Turned out though that Discourse Connect doesn't contain this problem and already handles staged users correctly. This PR adds tests that confirm it. Also, I've extracted two functions in Discourse Connect implementation along the way and decided to merge this refactoring too (the refactoring is supported with tests).
This commit introduces a new site setting "google_oauth2_hd_groups". If enabled, group information will be fetched from Google during authentication, and stored in the Discourse database. These 'associated groups' can be connected to a Discourse group via the "Membership" tab of the group preferences UI.
The majority of the implementation is generic, so we will be able to add support to more authentication methods in the near future.
https://meta.discourse.org/t/managing-group-membership-via-authentication/175950
The `ReviewableScore` model was defining class methods on `self.class`
from a singleton context so instead of defining methods on
`ReviewableScore` it was defining them on `Class`, so basically on every
existing class.
This patch resolves this issue. Using `enum` from `ActiveRecord` in the
future will avoid this kind of problems.
Related to: 20f736aa11.
`auto_update` is true by default at the database level, but it doesn't make sense for `auto_update` to be true on themes that are not imported from a Git repository.
* REFACTOR: Improve support for consolidating notifications.
Before this commit, we didn't have a single way of consolidating notifications. For notifications like group summaries, we manually removed old ones before creating a new one. On the other hand, we used an after_create callback for likes and group membership requests, which caused unnecessary work, as we need to delete the record we created to replace it with a consolidated one.
We now have all the consolidation rules centralized in a single place: the consolidation planner class. Other parts of the app looking to create a consolidable notification can do so by calling Notification#consolidate_or_save!, instead of the default Notification#create! method.
Finally, we added two more rules: one for re-using existing group summaries and another for deleting duplicated dashboard problems PMs notifications when the user is tracking the moderator's inbox. Setting the threshold to one forces the planner to apply this rule every time.
I plan to add plugin support for adding custom rules in another PR to keep this one relatively small.
* DEV: Introduces a plugin API for consolidating notifications.
This commit removes the `Notification#filter_by_consolidation_data` scope since plugins could have to define their criteria. The Plan class now receives two blocks, one to query for an already consolidated notification, which we'll try to update, and another to query for existing ones to consolidate.
It also receives a consolidation window, which accepts an ActiveSupport::Duration object, and filter notifications created since that value.
Currently when a user creates posts that are moderated (for whatever
reason), a popup is displayed saying the post needs approval and the
total number of the user’s pending posts. But then this piece of
information is kind of lost and there is nowhere for the user to know
what are their pending posts or how many there are.
This patch solves this issue by adding a new “Pending” section to the
user’s activity page when there are some pending posts to display. When
there are none, then the “Pending” section isn’t displayed at all.
Skipping methods we don't use gives us mem/perf gains (minuscule but still), but more importantly fixes warnings about `Poll#open` (created by `enum :status`) conflicting with some internal AR method. 😃
`pending`, `approved`, `rejected`, `ignored`, and `deleted` scope method were accessible on all model classes… 😂
Fixes `Creating scope :pending. Overwriting existing method DiscoursePostEvent::EventDate.pending.` warnings.
This commit adds token_hash and scopes columns to email_tokens table.
token_hash is a replacement for the token column to avoid storing email
tokens in plaintext as it can pose a security risk. The new scope column
ensures that email tokens cannot be used to perform a different action
than the one intended.
To sum up, this commit:
* Adds token_hash and scope to email_tokens
* Reuses code that schedules critical_user_email
* Refactors EmailToken.confirm and EmailToken.atomic_confirm methods
* Periodically cleans old, unconfirmed or expired email tokens
When this setting is turned on, it will check that normalized emails
are unique. Normalized emails are emails without any dots or plus
aliases.
This setting can be used to block use of aliases of the same email
address.
Use @here to mention all users that were allowed to topic directly or
through group, who liked topics or read the topic. Only first 10 users
will be notified.
In b8c8909a9d, we introduced a regression
where users may have had their `UserStat.first_unread_pm_at` set
incorrectly. This commit introduces a migration to reset `UserStat.first_unread_pm_at` back to
`User#created_at`.
Follow-up to b8c8909a9d.
The code that checked this permission was duplicated everytime a new
settings of this type was added. This commit changes the behavior of
some functionality because some feature checks were bypassed for staff
members.
Similar to site settings, adds support for `refresh` option to theme settings.
```yaml
super_feature_enabled:
type: bool
default: false
refresh: true
```
We are pushing /notification-alert/#{user_id} and /notification/#{user_id}
messages to MessageBus from both PostAlerter and User#publish_notification_state.
This can cause memory issues on large sites with many users. This commit
stems the bleeding by only sending these alert messages if the user
in question has been seen in the last 30 days, which eliminates a large
chunk of users on some sites.
The inefficiency here is that we were previously fetching all the
records from `TopicAllowedUser` before filtering against a limited subset of
users based on `User#last_seen_at`.
Previously, incorrect reply counts are displayed in the "top categories" section of the user summary page since we included the `moderator_action` and `small_action` post types.
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
Currently, Discourse rate limits all incoming requests by the IP address they
originate from regardless of the user making the request. This can be
frustrating if there are multiple users using Discourse simultaneously while
sharing the same IP address (e.g. employees in an office).
This commit implements a new feature to make Discourse apply rate limits by
user id rather than IP address for users at or higher than the configured trust
level (1 is the default).
For example, let's say a Discourse instance is configured to allow 200 requests
per minute per IP address, and we have 10 users at trust level 4 using
Discourse simultaneously from the same IP address. Before this feature, the 10
users could only make a total of 200 requests per minute before they got rate
limited. But with the new feature, each user is allowed to make 200 requests
per minute because the rate limits are applied on user id rather than the IP
address.
The minimum trust level for applying user-id-based rate limits can be
configured by the `skip_per_ip_rate_limit_trust_level` global setting. The
default is 1, but it can be changed by either adding the
`DISCOURSE_SKIP_PER_IP_RATE_LIMIT_TRUST_LEVEL` environment variable with the
desired value to your `app.yml`, or changing the setting's value in the
`discourse.conf` file.
Requests made with API keys are still rate limited by IP address and the
relevant global settings that control API keys rate limits.
Before this commit, Discourse's auth cookie (`_t`) was simply a 32 characters
string that Discourse used to lookup the current user from the database and the
cookie contained no additional information about the user. However, we had to
change the cookie content in this commit so we could identify the user from the
cookie without making a database query before the rate limits logic and avoid
introducing a bottleneck on busy sites.
Besides the 32 characters auth token, the cookie now includes the user id,
trust level and the cookie's generation date, and we encrypt/sign the cookie to
prevent tampering.
Internal ticket number: t54739.
When there are multiple groups on a topic, we were selecting
the first from the topic allowed groups to act as the sender
email address when sending group SMTP replies via PostAlerter.
However, this was not ordered, and since there is no created_at
column on TopicAllowedGroup we cannot order this nicely, which
caused just a random group to be used (based on whatever postgres
decided it felt like that morning).
This commit changes the group used for SMTP sending to be the
group using the email_username of the to address of the first
incoming email for the topic, if there are more than one allowed
groups on the topic. Otherwise it just uses the only SMTP enabled
group.
Previously, suppressed category topics are included in the digest emails if the user visited that topic before and the `TopicUser` record is created with any notification level except 'muted'.
* FIX: allowed_theme_ids should not be persisted in GlobalSettings
It was observed that the memoized value of `GlobalSetting.allowed_theme_ids` would be persisted across requests, which could lead to unpredictable/undesired behaviours in a multisite environment.
This change moves that logic out of GlobalSettings so that the returned theme IDs are correct for the current site.
Uses get_set_cache, which ultimately uses DistributedCache, which will take care of multisite issues for us.
* DEV: Sanitize HTML admin inputs
This PR adds on-save HTML sanitization for:
Client site settings
translation overrides
badges descriptions
user fields descriptions
I used Rails's SafeListSanitizer, which [accepts the following HTML tags and attributes](018cf54073/lib/rails/html/sanitizer.rb (L108))
* Make sure that the sanitization logic doesn't corrupt settings with special characters
This PR doesn't change any behavior, but just removes code that wasn't in use. This is a pretty dangerous place to change, since it gets called during user's registration. At the same time the refactoring is very straightforward, it's clear that this code wasn't doing any work (it still needs to be double-checked during review though). Also, the test coverage of UserNameSuggester is good.
* PERF: Remove JOIN on categories for PM search
JOIN on categories is not needed when searchin in private messages as
PMs are not categorized.
* DEV: Use == for string comparison
* PERF: Optimize query for allowed topic groups
There was a query that checked for all topics a user or their groups
were allowed to see. This used UNION between topic_allowed_users and
topic_allowed_groups which was very inefficient. That was replaced with
a OR condition that checks in either tables more efficiently.
When inviting a group to a topic, there may be members of
the group already in the topic as topic allowed users. These
can be safely removed from the topic, because they are implicitly
allowed in the topic based on their group membership.
Also, this prevents issues with group SMTP emails, which rely
on the topic_allowed_users of the topic to send to and cc's
for emails, and if there are members of the group as topic_allowed_users
then that complicates things and causes odd behaviour.
We also ensure that the OP of the topic is not removed from
the topic_allowed_users when a group they belong to is added,
as it will make it harder to add them back later.
The `generate`, `rotate` and `suspicious` auth token logs are now always logged regardless of the `verbose_auth_token_logging` setting because we rely no these to detect suspicious logins.
This is a follow-up to https://github.com/discourse/discourse/pull/14541. This adds a hidden setting for restoring the old behavior for those users who rely on it. We'll likely deprecate this setting at some point in the future.
Sometimes administrators want to permanently delete posts and topics
from the database. To make sure that this is done for a good reasons,
administrators can do this only after one minute has passed since the
post was deleted or immediately if another administrator does it.
We don't want to be using emails as source for username and name suggestions in cases when it's possible that a user have no chance to intervene and correct a suggested username. It risks exposing email addresses.
Previosuly, quotes from original topics are rendered incorrectly since the moved posts are not rebaked.
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
Ruby 2.7 or earlier `+contents` returns self.dup
when `frozen_string_literal: true`. However, Ruby 3.0 returns self
because this string is interpolated one, which is not frozen anymore.
This commit uses self.dup to return duplicated string regardless Ruby
versions.
https://bugs.ruby-lang.org/issues/17104
It allows saving local date to calendar.
Modal is giving option to pick between ics and google. User choice can be remembered as a default for the next actions.
* FEATURE: Return subcategories on categories endpoint
When using the API subcategories will now be returned nested inside of
each category response under the `subcategory_list` param. We already
return all the subcategory ids under the `subcategory_ids` param, but
you then would have to make multiple separate API calls to fetch each of
those subcategories. This way you can get **ALL** of the categories
along with their subcategories in a single API response.
The UI will not be affected by this change because you need to pass in
the `include_subcategories=true` param in order for subcategories to be
returned.
In a follow up PR I'll add the API scoping for fetching categories so
that a readonly API key can be used for the `/categories.json` endpoint. This
endpoint should be used instead of the `/site.json` endpoint for
fetching a sites categories and subcategories.
* Update PR based on feedback
- Have spec check for specific subcategory
- Move comparison check out of loop
- Only populate subcategory list if option present
- Remove empty array initialization
- Update api spec to allow null response
* More PR updates based on feedback
- Use a category serializer for the subcategory_list
- Don't include the subcategory_list param if empty
- For the spec check for the subcategory by id
- Fix spec to account for param not present when empty
FinalDestination now supports the `follow_canonical` option, which will perform an initial GET request, parse the canonical link if present, and perform a HEAD request to it.
We use this mode during embeds to avoid treating URLs with different query parameters as different topics.
It was possible to see notifications of other users using routes:
- notifications/responses
- notifications/likes-received
- notifications/mentions
- notifications/edits
We weren't showing anything private (like notifications about private messages), only things that're publicly available in other places. But anyway, it feels strange that it's possible to look at notifications of someone else. Additionally, there is a risk that we can unintentionally leak something on these pages in the future.
This commit restricts these routes.
* PERF: Improve database query perf when loading topics for a category.
Instead of left joining the `topics` table against `categories` by filtering with `categories.id`,
we can improve the query plan by filtering against `topics.category_id`
first before joining which helps to reduce the number of rows in the
topics table that has to be joined against the other tables and also
make better use of our existing index.
The following is a before and after of the query plan for a category
with many subcategories.
Before:
```
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------
Limit (cost=1.28..747.09 rows=30 width=12) (actual time=85.502..2453.727 rows=30 loops=1)
-> Nested Loop Left Join (cost=1.28..566518.36 rows=22788 width=12) (actual time=85.501..2453.722 rows=30 loops=1)
Join Filter: (category_users.category_id = topics.category_id)
Filter: ((topics.category_id = 11) OR (COALESCE(category_users.notification_level, 1) <> 0) OR (tu.notification_level > 1))
-> Nested Loop Left Join (cost=1.00..566001.58 rows=22866 width=20) (actual time=85.494..2453.702 rows=30 loops=1)
Filter: ((COALESCE(tu.notification_level, 1) > 0) AND ((topics.category_id <> 11) OR (topics.pinned_at IS NULL) OR ((t
opics.pinned_at <= tu.cleared_pinned_at) AND (tu.cleared_pinned_at IS NOT NULL))))
Rows Removed by Filter: 1
-> Nested Loop (cost=0.57..528561.75 rows=68606 width=24) (actual time=85.472..2453.562 rows=31 loops=1)
Join Filter: ((topics.category_id = categories.id) AND ((categories.topic_id <> topics.id) OR (categories.id = 1
1)))
Rows Removed by Join Filter: 13938306
-> Index Scan using index_topics_on_bumped_at on topics (cost=0.42..100480.05 rows=715549 width=24) (actual ti
me=0.010..633.015 rows=464623 loops=1)
Filter: ((deleted_at IS NULL) AND ((archetype)::text <> 'private_message'::text))
Rows Removed by Filter: 105321
-> Materialize (cost=0.14..36.04 rows=30 width=8) (actual time=0.000..0.002 rows=30 loops=464623)
-> Index Scan using categories_pkey on categories (cost=0.14..35.89 rows=30 width=8) (actual time=0.006.
.0.040 rows=30 loops=1)
Index Cond: (id = ANY ('{11,53,57,55,54,56,112,94,107,115,116,117,97,95,102,103,101,105,99,114,106,1
13,104,98,100,96,108,109,110,111}'::integer[]))
-> Index Scan using index_topic_users_on_topic_id_and_user_id on topic_users tu (cost=0.43..0.53 rows=1 width=16) (a
ctual time=0.004..0.004 rows=0 loops=31)
Index Cond: ((topic_id = topics.id) AND (user_id = 1103877))
-> Materialize (cost=0.28..2.30 rows=1 width=8) (actual time=0.000..0.000 rows=0 loops=30)
-> Index Scan using index_category_users_on_user_id_and_last_seen_at on category_users (cost=0.28..2.29 rows=1 width
=8) (actual time=0.004..0.004 rows=0 loops=1)
Index Cond: (user_id = 1103877)
Planning Time: 1.359 ms
Execution Time: 2453.765 ms
(23 rows)
```
After:
```
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=1.28..438.55 rows=30 width=12) (actual time=38.297..657.215 rows=30 loops=1)
-> Nested Loop Left Join (cost=1.28..195944.68 rows=13443 width=12) (actual time=38.296..657.211 rows=30 loops=1)
Filter: ((categories.topic_id <> topics.id) OR (topics.category_id = 11))
Rows Removed by Filter: 29
-> Nested Loop Left Join (cost=1.13..193462.59 rows=13443 width=16) (actual time=38.289..657.092 rows=59 loops=1)
Join Filter: (category_users.category_id = topics.category_id)
Filter: ((topics.category_id = 11) OR (COALESCE(category_users.notification_level, 1) <> 0) OR (tu.notification_level > 1))
-> Nested Loop Left Join (cost=0.85..193156.79 rows=13489 width=20) (actual time=38.282..657.059 rows=59 loops=1)
Filter: ((COALESCE(tu.notification_level, 1) > 0) AND ((topics.category_id <> 11) OR (topics.pinned_at IS NULL) OR ((topics.pinned_at <= tu.cleared_pinned_at) AND (tu.cleared_pinned_at IS NOT NULL))))
Rows Removed by Filter: 1
-> Index Scan using index_topics_on_bumped_at on topics (cost=0.42..134521.06 rows=40470 width=24) (actual time=38.267..656.850 rows=60 loops=1)
Filter: ((deleted_at IS NULL) AND ((archetype)::text <> 'private_message'::text) AND (category_id = ANY ('{11,53,57,55,54,56,112,94,107,115,116,117,97,95,102,103,101,105,99,114,106,113,104,98,100,96,108,109,110,111}'::integer[])))
Rows Removed by Filter: 569895
-> Index Scan using index_topic_users_on_topic_id_and_user_id on topic_users tu (cost=0.43..1.43 rows=1 width=16) (actual time=0.003..0.003 rows=0 loops=60)
Index Cond: ((topic_id = topics.id) AND (user_id = 1103877))
-> Materialize (cost=0.28..2.30 rows=1 width=8) (actual time=0.000..0.000 rows=0 loops=59)
-> Index Scan using index_category_users_on_user_id_and_last_seen_at on category_users (cost=0.28..2.29 rows=1 width=8) (actual time=0.004..0.004 rows=0 loops=1)
Index Cond: (user_id = 1103877)
-> Index Scan using categories_pkey on categories (cost=0.14..0.17 rows=1 width=8) (actual time=0.001..0.001 rows=1 loops=59)
Index Cond: (id = topics.category_id)
Planning Time: 1.633 ms
Execution Time: 657.255 ms
(22 rows)
```
* PERF: Optimize index on topics bumped_at.
Replace `index_topics_on_bumped_at` index with a partial index on `Topic#bumped_at` filtered by archetype since there is already another index that covers private topics.
* DEV: use active record `save!` instead of mini sql.
The "save" method will trigger the before_save callback "match_primary_group_changes" for User model. Else `flair_group_id` won't be removed from the user.
* check whether the method `match_primary_group_changes` called or not.
Allows creating a bookmark with the `for_topic` flag introduced in d1d2298a4c set to true. This happens when clicking on the Bookmark button in the topic footer when no other posts are bookmarked. In a later PR, when clicking on these topic-level bookmarks the user will be taken to the last unread post in the topic, not the OP. Only the OP can have a topic level bookmark, and users can also make a post-level bookmark on the OP of the topic.
I had to do some pretty heavy refactors because most of the bookmark code in the JS topics controller was centred around instances of Post JS models, but the topic level bookmark is not centred around a post. Some refactors were just for readability as well.
Also removes some missed reminderType code from the purge in 41e19adb0d
After deleting a category, we should soft-delete the category definition topic instead of hard deleting it. Else it causes issues while doing the user merge action if the source user has an orphan post that belongs to the deleted topic.
We don't actually use the reminder_type for bookmarks anywhere;
we are just storing it. It has no bearing on the UI. It used
to be relevant with the at_desktop bookmark reminders (see
fa572d3a7a)
This commit marks the column as readonly, ignores it, and removes
the index, and it will be dropped in a later PR. Some plugins
are relying on reminder_type partially so some stubs have been
left in place to avoid errors.
This new column will be used to indicate that a bookmark
is at the topic level. The first post of a topic can be
bookmarked twice after this change -- with for_topic set
to true and with for_topic set to false.
A later PR will use this column for logic to bookmark the
topic, and then topic-level bookmark links will take you
to the last unread post in the topic.
See also 22208836c5
We don't need no stinkin' denormalization! This commit ignores
the topic_id column on bookmarks, to be deleted at a later date.
We don't really need this column and it's better to rely on the
post.topic_id as the canonical topic_id for bookmarks, then we
don't need to remember to update both columns if the bookmarked
post moves to another topic.
The previous excerpt was a simple truncated raw message. Starting with
this commit, the raw content of the draft is cooked and an excerpt is
extracted from it. The logic for extracting the excerpt mimics the the
`ExcerptParser` class, but does not implement all functionality, being
a much simpler implementation.
The two draft controllers have been merged into one and the /draft.json
route has been changed to /drafts.json to be consistent with the other
route names.
This is necessary to allow for large file uploads via
the direct S3 upload mechanism, as we convert the external
file to an Upload record via ExternalUploadManager once
it is complete.
This will allow for files larger than 2,147,483,647 bytes (2.14GB)
to be referenced in the uploads table.
This is a table locking migration, but since it is not as highly
trafficked as posts, topics, or users, the disruption should be minimal.
When a user archives a personal message, they are redirected back to the
inbox and will refresh the list of the topics for the given filter.
Publishing an event to the user results in an incorrect incoming message
because the list of topics has already been refreshed.
This does mean that if a user has two tabs opened, the non-active tab
will not receive the incoming message but at this point we do not think
the technical trade-offs are worth it to support this feature. We
basically have to somehow exclude a client from an incoming message
which is not easy to do.
Follow-up to fc1fd1b416
DEV: Allow passing cook_method to TopicEmbed.import to override default
This will be used in the rss-polling plugin when we want to have
oneboxes on feed content, like youtube for example.
This can be used to change the list of topic posters. For example,
discourse-solved can use this to move the user who posted the solution
after the original poster.
There are certain design decisions that were made in this commit.
Private messages implements its own version of topic tracking state because there are significant differences between regular and private_message topics. Regular topics have to track categories and tags while private messages do not. It is much easier to design the new topic tracking state if we maintain two different classes, instead of trying to mash this two worlds together.
One MessageBus channel per user and one MessageBus channel per group. This allows each user and each group to have their own channel backlog instead of having one global channel which requires the client to filter away unrelated messages.
This pull request introduces the endpoints required, and the JavaScript functionality in the `ComposerUppyUpload` mixin, for direct S3 multipart uploads. There are four new endpoints in the uploads controller:
* `create-multipart.json` - Creates the multipart upload in S3 along with an `ExternalUploadStub` record, storing information about the file in the same way as `generate-presigned-put.json` does for regular direct S3 uploads
* `batch-presign-multipart-parts.json` - Takes a list of part numbers and the unique identifier for an `ExternalUploadStub` record, and generates the presigned URLs for those parts if the multipart upload still exists and if the user has permission to access that upload
* `complete-multipart.json` - Completes the multipart upload in S3. Needs the full list of part numbers and their associated ETags which are returned when the part is uploaded to the presigned URL above. Only works if the user has permission to access the associated `ExternalUploadStub` record and the multipart upload still exists.
After we confirm the upload is complete in S3, we go through the regular `UploadCreator` flow, the same as `complete-external-upload.json`, and promote the temporary upload S3 into a full `Upload` record, moving it to its final destination.
* `abort-multipart.json` - Aborts the multipart upload on S3 and destroys the `ExternalUploadStub` record if the user has permission to access that upload.
Also added are a few new columns to `ExternalUploadStub`:
* multipart - Whether or not this is a multipart upload
* external_upload_identifier - The "upload ID" for an S3 multipart upload
* filesize - The size of the file when the `create-multipart.json` or `generate-presigned-put.json` is called. This is used for validation.
When the user completes a direct S3 upload, either regular or multipart, we take the `filesize` that was captured when the `ExternalUploadStub` was first created and compare it with the final `Content-Length` size of the file where it is stored in S3. Then, if the two do not match, we throw an error, delete the file on S3, and ban the user from uploading files for N (default 5) minutes. This would only happen if the user uploads a different file than what they first specified, or in the case of multipart uploads uploaded larger chunks than needed. This is done to prevent abuse of S3 storage by bad actors.
Also included in this PR is an update to vendor/uppy.js. This has been built locally from the latest uppy source at d613b849a6. This must be done so that I can get my multipart upload changes into Discourse. When the Uppy team cuts a proper release, we can bump the package.json versions instead.
When a staff member clicks on a user's number of flagged posts, we redirect them to the review queue, so it makes sense to count the number of items there to calculate the count.
We used to look at post action items to calculate this number, which doesn't match the number of items in the queue if old flags exist.
This reverts a part of changes introduced by https://github.com/discourse/discourse/pull/13947
In that PR I:
1. Disallowed topic feature links for TL-0 users
2. Additionally, disallowed just putting any URL in topic titles for TL-0 users
Actually, we don't need the second part. It introduced unnecessary complexity for no good reason. In fact, it tries to do the job that anti-spam plugins (like Akismet plugin) should be doing.
This PR reverts this second change.
This disallows putting URLs in topic titles for TL0 users, which means that:
If a TL-0 user puts a link into the title, a topic featured link won't be generated (as if it was disabled in the site settings)
Server methods for creating and updating topics will be refusing featured links when they are called by TL-0 users
TL-0 users won't be able to put any link into the topic title. For example, the title "Hey, take a look at https://my-site.com" will be rejected.
Also, it improves a bit server behavior when creating or updating feature links on topics in the categories with disabled featured links. Before the server just silently ignored a featured link field that was passed to him, now it will be returning 422 response.
* FIX: Update draft count when sequence is increased
Sometimes users ended up having a draft count higher than the actual
number of drafts.
* FIX: Do not update draft count twice
The call to DraftSequence.next! above already does it.
Group flair is not removed while removing a user from the group since the `before_save` callback methods are not triggered while using the `update_columns` method.
When a post is created, the draft sequence is increased and then older
drafts are automatically executing a raw SQL query. This skipped the
Draft model callbacks and did not update user's draft count.
I fixed another problem related to a raw SQL query from Draft.cleanup!
method.
This adds a few different things to allow for direct S3 uploads using uppy. **These changes are still not the default.** There are hidden `enable_experimental_image_uploader` and `enable_direct_s3_uploads` settings that must be turned on for any of this code to be used, and even if they are turned on only the User Card Background for the user profile actually uses uppy-image-uploader.
A new `ExternalUploadStub` model and database table is introduced in this pull request. This is used to keep track of uploads that are uploaded to a temporary location in S3 with the direct to S3 code, and they are eventually deleted a) when the direct upload is completed and b) after a certain time period of not being used.
### Starting a direct S3 upload
When an S3 direct upload is initiated with uppy, we first request a presigned PUT URL from the new `generate-presigned-put` endpoint in `UploadsController`. This generates an S3 key in the `temp` folder inside the correct bucket path, along with any metadata from the clientside (e.g. the SHA1 checksum described below). This will also create an `ExternalUploadStub` and store the details of the temp object key and the file being uploaded.
Once the clientside has this URL, uppy will upload the file direct to S3 using the presigned URL. Once the upload is complete we go to the next stage.
### Completing a direct S3 upload
Once the upload to S3 is done we call the new `complete-external-upload` route with the unique identifier of the `ExternalUploadStub` created earlier. Only the user who made the stub can complete the external upload. One of two paths is followed via the `ExternalUploadManager`.
1. If the object in S3 is too large (currently 100mb defined by `ExternalUploadManager::DOWNLOAD_LIMIT`) we do not download and generate the SHA1 for that file. Instead we create the `Upload` record via `UploadCreator` and simply copy it to its final destination on S3 then delete the initial temp file. Several modifications to `UploadCreator` have been made to accommodate this.
2. If the object in S3 is small enough, we download it. When the temporary S3 file is downloaded, we compare the SHA1 checksum generated by the browser with the actual SHA1 checksum of the file generated by ruby. The browser SHA1 checksum is stored on the object in S3 with metadata, and is generated via the `UppyChecksum` plugin. Keep in mind that some browsers will not generate this due to compatibility or other issues.
We then follow the normal `UploadCreator` path with one exception. To cut down on having to re-upload the file again, if there are no changes (such as resizing etc) to the file in `UploadCreator` we follow the same copy + delete temp path that we do for files that are too large.
3. Finally we return the serialized upload record back to the client
There are several errors that could happen that are handled by `UploadsController` as well.
Also in this PR is some refactoring of `displayErrorForUpload` to handle both uppy and jquery file uploader errors.
This commit adds the number of drafts a user has next to the "Draft"
label in the user preferences menu and activity tab. The count is
updated via MessageBus when a draft is created or destroyed.
Will show the last 6 seen users as filtering suggestions when typing @ in quick search. (Previously the user suggestion required a character after the @.)
This also adds a default limit of 6 to the user search query, previously the backend was returning 20 results but a maximum of 6 results was being shown anyway.
When configured, all topics in the category inherits the slow mode
duration from the category's default.
Note that currently there is no way to remove the slow mode from the
topics once it has been set.
When the Forever option is selected for suspending a user, the user is suspended for 1000 years. Without customizing the site’s text, this time period is displayed to the user in the suspension email that is sent to the user, and if the user attempts to log back into the site. Telling someone that they have been suspended for 1000 years seems likely to come across as a bad attempt at humour.
This PR special case messages when a user suspended or silenced forever.
Flips content_security_policy_frame_ancestors default to enabled, and
removes HTTP_REFERER checks on embed requests, as the new referer
privacy options made the check fragile.
* FIX: Clear stale status of reloaded reviewables
Navigating away from and back to the reviewables reloaded Reviewable
records, but did not clear the "stale" attribute.
* FEATURE: Show user who last acted on reviewable
When a user acts on a reviewable, all other clients are notified and a
generic "reviewable was resolved by someone" notice was shown instead of
the buttons. There is no need to keep secret the username of the acting
user.
Fixes two issues:
- ignores invalid XML in custom icon sprite SVG file (and outputs an error if sprite was uploaded via admin UI)
- clears SVG sprite cache when deleting an `icons-sprite` upload in a theme
Previously we would re-calculate topic_user.liked for all users who have ever viewed the source or destination topic. This can be very expensive on large sites. Instead, we can use the array of moved post ids to find which users are actually affected by the move, and restrict the update query to only check/update their records.
On an example site this reduced the `update_post_action_cache` time from ~27s to 300ms
Scanning for all possible invalid post_timing records in the destination topics can be a very expensive operation. The main aim is to avoid the data clashing with soon-to-be-moved posts, so we can reduce the scope of the query by targeting only rows which would actually cause a clash. post_timings has an index on (topic_id, post_number), so this is very fast.
On an example site, this reduced the query from ~6s to <10ms
When a staged user tried to redeem an invite, a different username was
suggested and manually typing the staged username failed because the
username was not available.
When secure media is enabled or when upload secure status
is updated, we also try and update the upload ACL. However
if the object storage provider does not implement this we
get an Aws::S3::Errors::NotImplemented error. This PR handles
this error so the update_secure_status method does not error
out and still returns whether the secure status changed.
User flair was given by user's primary group. This PR separates the
two, adds a new field to the user model for flair group ID and users
can select their flair from user preferences now.
And also move all the "top topics by period" routes to query string param.
/top/monthly => /top?period=monthly
/c/:slug/:id/l/top/monthly => /c/:slug/:id/l/top?period=monthly
/tag/:slug/l/top/daily => /tag/:slug/l/top?period=daily (new)
Users can invite people to topics from secured category, but they will
not be redirected to the topic after signing up unless they have the
permissions to view the topic. This commit shows a warning when invite
is saved if the topic is in a secured category and none of the invite
groups are allowed to see it.
Skip group SMTP email (and add log) if:
* topic is deleted
* post is deleted
* smtp has been disabled for the group
Skip without log if:
* enable_smtp site setting is false
* disable_emails site setting is yes
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
We want to put the name of the trust level in to generated URLs, not the human-readable form.
i.e.:
`/admin/users/list/newuser`
rather than:
`/admin/users/list/new user`
When a topic is fully merged into another topic we close it and schedule it for deleting. But last time I changed this place I added a bug – when merging all posts in topic except the first one the topic was closing too.
If the OP is not merged into another topic, the original topic shouldn't be closed and marked for deletion. This PR fixes this.
The duration column has been ignored since the commit
4af77f1e38
for topic_timers, we use duration_minutes instead.
Also removing the duration key from Topic.set_or_create_timer. The only
plugin to use this was discourse-solved, which doesn't use it any
longer
since
c722b94a97
This PR makes several changes to the group SMTP email contents to make it look more like a support inbox message.
* Remove the context posts, they only add clutter to the email and replies
* Display email addresses of staged users instead of odd generated usernames
* Add a "please reply above this line" message to sent emails
This PR backtracks a fair bit on this one https://github.com/discourse/discourse/pull/13220/files.
Instead of sending the group SMTP email for each user via `UserNotifications`, we are changing to send only one email with the existing `Jobs::GroupSmtpEmail` job and `GroupSmtpMailer`. We are changing this job and mailer along with `PostAlerter` to make the first topic allowed user the `to_address` for the email and any other `topic_allowed_users` to be the CC address on the email. This is to cut down on emails sent via SMTP, which is subject to daily limits from providers such as Gmail. We log these details in the `EmailLog` table now.
In addition to this, we have changed `PostAlerter` to no longer rely on incoming email email addresses for sending the `GroupSmtpEmail` job. This was unreliable as a user's email could have changed in the meantime. Also it was a little overcomplicated to use the incoming email records -- it is far simpler to reason about to just use topic allowed users.
This also adds a fix to include cc_addresses in the EmailLog.addressed_to_user scope.
ATM it only implements server side of it, as my need is for automation purposes. However it should probably be added in the UI too as it's unexpected to have pinned_until and no bannered_until.
This adds the following columns to EmailLog:
* cc_addresses
* cc_user_ids
* topic_id
* raw
This is to bring the EmailLog table closer in parity to
IncomingEmail so it can be better utilized for Group SMTP
and IMAP mailing.
The raw column contains the full content of the outbound email,
but _only_ if the new hidden site setting
enable_raw_outbound_email_logging is enabled. Most sites do not
need it, and it's mostly required for IMAP and SMTP sending.
In the next pull request, there will be a migration to backfill
topic_id on the EmailLog table, at which point we can remove the
topic fallback method on EmailLog.
Before this change, calling `StyleSheet::Manager.stylesheet_details`
for the first time resulted in multiple queries to the database. This is
because the code was modelled in a way where each `Theme` was loaded
from the database one at a time.
This PR restructures the code such that it allows us to load all the
theme records in a single query. It also allows us to eager load the
required associations upfront. In order to achieve this, I removed the
support of loading multiple themes per request. It was initially added
to support user selectable theme components but the feature was never
completed and abandoned because it wasn't a feature that we thought was
worth building.
The first thing we needed here was an enum rather than a boolean to determine how a directory_column was created. Now we have `automatic`, `user_field` and `plugin` directory columns.
This plugin API is assuming that the plugin has added a migration to a column to the `directory_items` table.
This was created to be initially used by discourse-solved. PR with API usage - https://github.com/discourse/discourse-solved/pull/137/
Profiling showed that we were roughly 10% of a request time creating all
the ActiveRecord objects for categories in the `Site` model on a site with 61 categories.
Instead of querying for the categories each time based on which categories the user can see,
we can just preload all of the categories upfront and filter out the
categories that the user can not see.
Adds the last updated at and by SMTP/IMAP fields to the UI, we were already storing them in the DB. Also makes sure that `imap_mailbox_name` being changed makes the last_updated_at/by field update for IMAP.
When we call Bookmark.cleanup! we want to make sure that
topic_user.bookmarked is updated for topics linked to the
bookmarks that were deleted. Also when PostDestroyer calls
destroy and recover. We have a job for this already --
SyncTopicUserBookmarked -- so we just utilize that.
Subclasses must call #delete_user_actions inside build_actions to support user deletion. The method adds a delete user bundle, which has a delete and a delete + block option. Every subclass is responsible for implementing these actions.
Adds a new `smtp_group_id` column to `EmailLog` which is filled in if the mail `from_address` matches a group's `email_username`. This is for easier debugging, so we know which emails have been sent via group SMTP.
When replying to a user_private_message email originating from
a group PM that does _not_ have a reply key (e.g. when replying
directly to the group's SMTP address), we were mistakenly linking
the new post created from the reply to the OP and the user who
created the topic, based on the first IncomingEmail message ID in
the topic, rather than using the correct reply to user and post number
that the user actually replied to.
We now use the In-Reply-To header to look up the corresponding EmailLog
record when the user who replied was sent a user_private_message email,
and use the post from that as the reply_to_user/post.
This also removes superfluous filtering of incoming_email records. After
already filtering by message_id and then addressed_to_user (which only
returns incoming emails where the to, from, or cc address includes any
of the user's emails), we were filtering again but in the ruby code for
the exact same conditions. After removing this all existing tests still
pass.
