This commit implements 2 new metrics/stats in the /about page for the _estimated_ numbers of unique visitors from the EU and the rest of the world. This new feature is currently off by default, but it can be enabled by turning on the hidden `display_eu_visitor_stats` site settings via the rails console.
There are a number of assumptions that we're making here in order to estimate the number of unique visitors, specifically:
1. we're assuming that the average of page views per anonymous visitor is similar to the average number of page views that a logged-in visitor makes, and
2. we're assuming that the ratio of logged in visitors from the EU is similar to the ratio of anonymous visitors from the EU
Discourse keeps track of the number of both logged-in and anonymous page views, and also the number of unique logged-in visitors and where they're from. So with those numbers and the assumptions above, we can estimate the number of unique anonymous visitors from the EU and the rest of the world.
Internal topic: t/128480.
We need a way to disable certain checks programatically, e.g. on Discourse hosting. This PR adds a configuration option for this, and makes it so that disabled checks aren't run as part of #run_all.
This commit fixes a bug where the silence button is incorrectly displayed on the admin page of a staff user. It's not actually possible to silence a staff user because the backend correctly prevents it, but the frontend isn't checking if the button should be displayed.
Another small bug that this commit fixes is the similar users list not showing up inside the silence/suspend modals due to also a bug in the frontend.
I've also changed the way similar users are loaded so that they're not returned by the `admin/users#show` endpoint anymore and moved them into a new endpoint that the penalize modals (suspend and silence) can call directly to retrieve the list of users. This is done because the similar users list is never shown on the admin user page (`/admin/users/:user_id/:username`); they're only needed when the suspend or silence modals are opened.
Internal topic: t/130014.
### Why?
Before, all flags were static. Therefore, they were stored in class variables and serialized by SiteSerializer. Recently, we added an option for admins to add their own flags or disable existing flags. Therefore, the class variable had to be dropped because it was unsafe for a multisite environment. However, it started causing performance problems.
### Solution
When a new Flag system is used, instead of using PostActionType, we can serialize Flags and use fragment cache for performance reasons.
At the same time, we are still supporting deprecated `replace_flags` API call. When it is used, we fall back to the old solution and the admin cannot add custom flags. In a couple of months, we will be able to drop that API function and clean that code properly. However, because it may still be used, redis cache was introduced to improve performance.
To test backward compatibility you can add this code to any plugin
```ruby
replace_flags do |flag_settings|
flag_settings.add(
4,
:inappropriate,
topic_type: true,
notify_type: true,
auto_action_type: true,
)
flag_settings.add(1001, :trolling, topic_type: true, notify_type: true, auto_action_type: true)
end
```
### Why?
Before, all flags were static. Therefore, they were stored in class variables and serialized by SiteSerializer. Recently, we added an option for admins to add their own flags or disable existing flags. Therefore, the class variable had to be dropped because it was unsafe for a multisite environment. However, it started causing performance problems.
### Solution
When a new Flag system is used, instead of using PostActionType, we can serialize Flags and use fragment cache for performance reasons.
At the same time, we are still supporting deprecated `replace_flags` API call. When it is used, we fall back to the old solution and the admin cannot add custom flags. In a couple of months, we will be able to drop that API function and clean that code properly. However, because it may still be used, redis cache was introduced to improve performance.
To test backward compatibility you can add this code to any plugin
```ruby
replace_flags do |flag_settings|
flag_settings.add(
4,
:inappropriate,
topic_type: true,
notify_type: true,
auto_action_type: true,
)
flag_settings.add(1001, :trolling, topic_type: true, notify_type: true, auto_action_type: true)
end
```
Adds a new statistics (hidden from the UI, but available via the API) that tracks daily participating users.
A user is considered as "participating" if they have
- Reacted to a post
- Replied to a topic
- Created a new topic
- Created a new PM
- Sent a chat message
- Reacted to a chat message
Internal ref - t/131013
Currently to handle stub topics after merging, there are only options to (1) never delete a stub topic and (2) delete a stub topic after X amount of days. This adds the option to immediately delete a stub topic upon merge.
---------
Co-authored-by: Mark VanLandingham <markvanlan@gmail.com>
Co-authored-by: Renato Atilio <renato@discourse.org>
This commit continues on work laid out by 6039b513fe to redesign the /about page. In this commit, we add the site age and a section on the right hand side to show site activities/statistics such as topics, posts, sign-ups, likes etc.
Following a recent refactor, some methods from `FlagSettings` have been
renamed (`custom_types` -> `additional_message_types`). The
`PostActionType` model was using `custom_types` but when the renaming
was done, it was renamed to `with_additional_message` instead of
`additional_message_types`, which under the right circumstances will
raise an error.
Admin can create up to 50 custom flags. It is limited for performance reasons.
When the limit is reached "Add button" is disabled and backend is protected by guardian.
We support a low-level construct called "inline checks", which you can use to register a problem ad-hoc from within application code.
Problems registered by inline checks never show up in the admin dashboard, this is because when loading the dashboard, we run all realtime checks and look for problems. Because of an oversight, we considered inline checks to be "realtime", causing them to be run and clear their problem status.
To fix this, we don't consider inline checks to be realtime, to prevent them from running when loading the admin dashboard.
When creating a shared draft, we're recording topic view stats on the draft and then pass those on when the draft is published, conflating the actual view count.
This fixes that by not registering topic views if the topic is a shared draft.
When `SiteSetting.review_every_post` is true and the category `require_topic_approval` system creates two reviewable items.
1. Firstly, because the category needs approval, the `ReviewableQueuePost` record` is created - at this stage, no topic is created.
2. Admin is approving the review. The topic and first post are created.
3. Because `review_every_post` is true `queue_for_review_if_possible` callback is evaluated and `ReviewablePost` is created.
4. Then `ReviewableQueuePost` is linked to the newly generated topic and post.
At the beginning, we were thinking about hooking to those guards:
```
def self.queue_for_review_if_possible(post, created_or_edited_by)
return unless SiteSetting.review_every_post
return if post.post_type != Post.types[:regular] || post.topic.private_message?
return if Reviewable.pending.where(target: post).exists?
...
```
And add something like
```
return if Reviewable.approved.where(target: post).exists?
```
However, because the callback happens in point 3. before the `ReviewableQueuePost` is linked to the `Topic`, it was not possible.
Therefore, when `ReviewableQueuePost` is creating a `Topic`, a new option called `:reviewed_queued_post` is passed to `PostCreator` to avoid creating a second `Reviewable`.
* SECURITY: Update default allowed iframes list
Change the default iframe url list to all include 3 slashes.
* SECURITY: limit group tag's name length
Limit the size of a group tag's name to 100 characters.
Internal ref - t/130059
* SECURITY: Improve sanitization of SVGs in Onebox
---------
Co-authored-by: Blake Erickson <o.blakeerickson@gmail.com>
Co-authored-by: Régis Hanol <regis@hanol.fr>
Co-authored-by: David Taylor <david@taylorhq.com>
* FIX: Ensure JsLocaleHelper to obly outputs up-to-date translations
The old implementation forgot to filter out deprecated
translations, causing these translations to incorrectly override the new
locale in the frontend.
This commit fills in the forgotten where clause, filtering only the
up-to-date part.
Related meta topic: https://meta.discourse.org/t/outdated-translation-replacement-causing-missing-translation/314352
Currently, when adding translation overrides, values aren’t validated
for MF strings. This results in being able to add invalid plural keys or
even strings containing invalid syntax.
This patch addresses this issue by compiling the string when saving an
override if the key is detected as an MF one.
If there’s an error from the compiler, it’s added to the model errors,
which in turn is displayed to the user in the admin UI, helping them to
understand what went wrong.
* FIX: Add post id to the anchor to prevent two identical anchors
We generate anchors for headings in posts. This works fine if there is
only one post in a topic with anchors. The problem comes when you have
two or more posts with the same heading. PrettyText generates anchors
based on the heading text using the raw context of each post, so it is
entirely possible to generate the same anchor for two posts in the same
topic, especially for topics with template replies
Post1:
# heading
context
Post2:
# heading
context
When both posts are on the page at the same time, the anchor will only
work for the first post, according to the [HTML specification](https://html.spec.whatwg.org/multipage/browsing-the-web.html#scroll-to-the-fragment-identifier).
> If there is an a element in the document tree whose root is document
> that has a name attribute whose value is equal to fragment, then
> return the *first* such element in tree order.
This bug is particularly serious in forums with non-Latin languages,
such as Chinese. We do not generate slugs for Chinese, which results in
the heading anchors being completely dependent on their order.
```ruby
[2] pry(main)> PrettyText.cook("# 中文")
=> "<h1><a name=\"h-1\" class=\"anchor\" href=\"#h-1\"></a>中文</h1>"
```
Therefore, the anchors in the two posts must be in exactly the same by
order, causing almost all of the anchors in the second post to be
invalid.
This commit solves this problem by adding the `post_id` to the anchor.
The new anchor generation method will add `p-{post_id}` as a prefix when
post_id is available:
```ruby
[3] pry(main)> PrettyText.cook("# 中文", post_id: 1234)
=> "<h1><a name=\"p-1234-h-1\" class=\"anchor\" href=\"#p-1234-h-1\"></a>中文</h1>"
```
This way we can ensure that each anchor name only appears once on the
same topic. Using post id also prevents the potential possibility of the
same anchor name when splitting/merging topics.
Previously in these 2 PRs, we introduced a new site setting `SiteSetting.enforce_second_factor_on_external_auth`.
https://github.com/discourse/discourse/pull/27547https://github.com/discourse/discourse/pull/27674
When disabled, it should enforce 2FA for local login with username and password and skip the requirement when authenticating with oauth2.
We stored information about the login method in a secure session but it is not reliable. Therefore, information about the login method is moved to the database.
* FEATURE: Clean up previously logged information after permanently deleting posts
When soft deleteing a topic or post, we will log some details in the
staff log, including the raw content of the post. Before this commit, we
will not clear the information in these records. Therefore, after
permanently deleting the post, `UserHistory` still retains copy of the
permanently deleted post. This is an unexpected behaviour and may raise
some potential legal issues.
This commit adds a behavior that when a post is permanently deleted, the
details column of the `UserHistory` associated with the post will be
overwritten to "(permanently deleted)". At the same time, for permanent
deletion, a new `action_id` is introduced to distinguish it from soft
deletion.
Related meta topic: https://meta.discourse.org/t/introduce-a-way-to-also-permanently-delete-the-sensitive-info-from-the-staff-logs/292546
This commit introduces the foundation for a new design for the /about page that we're currently working on. The current version will remain available and still be the default until we finish the new version and are ready to roll out. To opt into the new version right now, add one or more group to the `experimental_redesigned_about_page_groups` site setting and members in those groups will get the new version.
Internal topic: t/128545.
* FEATURE: Add logging for CustomEmoji
We didn't provide any logs for CustomEmoji before, nor did we record the
person who added any emoji in the database. As a result, the staff had
no way to trace back who added a certain emoji.
This commit adds a new column `user_id` to `custom_emojis` to record the
creator of an emoji. At the same time, a log is added for staff logs to
record who added or deleted a custom emoji.
Before migration is run flags code is evaluated. It is causing error:
```
NoMethodError: undefined method `require_message' for an instance of Flag (NoMethodError)
Did you mean? require_dependency
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activemodel-7.1.3.4/lib/active_model/attribute_methods.rb:489:in `method_missing'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.1.3.4/lib/active_record/relation/delegation.rb💯in `each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.1.3.4/lib/active_record/relation/delegation.rb💯in `each'
/var/www/discourse/app/models/post_action_type.rb:64:in `reject'
```
The solution is to temporarily fall back to old column name - custom_type
Our old group SMTP SSL option was a checkbox,
but this was not ideal because there are actually
3 different ways SSL can be used when sending
SMTP:
* None
* SSL/TLS
* STARTTLS
We got around this before with specific overrides
for Gmail, but it's not flexible enough and now people
want to use other providers. It's best to be clear,
though it is a technical detail. We provide a way
to test the SMTP settings before saving them so there
should be little chance of messing this up.
This commit also converts GroupEmailSettings to a glimmer
component.
Allow admin to create custom flag which requires an additional message.
I decided to rename the old `custom_flag` into `require_message` as it is more descriptive.
Previously, we did not log any topic slow mode changes. This allowed
some malicious (or just careless) TL4 users to delete slow modes created
by moderators at will. Administrators could not see who changed the slow
mode unless they had SQL knowledge and used Data Explorer.
This commit enables logging who turns slow mode on, off, or changes it.
Related meta topic: https://meta.discourse.org/t/why-is-there-no-record-of-who-added-or-removed-slow-mode/316354
The watch words controller creation function, create_or_update_word(), doesn’t validate the size of the replacement parameter, unlike the word parameter, when creating a replace watched word. So anyone with moderator privileges can create watched words with almost unlimited characters.
`after_commit` should be used before refreshing processes to be sure that the database is already updated.
Also, MessageBus is used instead of events as MessageBus works correctly with many processes;
The OutOfDateThemes problem check is using an old method of setting the message, by overriding #message. It should instead use #translation_keys. (By chance I noticed the same thing applies to UnreachableThemes.
Allow admin to create custom flag which requires an additional message.
I decided to rename the old `custom_flag` into `require_message` as it is more descriptive.
