This commit introduces the following changes which allows a site
administrator to mark `Upload` records with the `s3_file_missing`
verification status which will result in the `Upload` record being ignored when
`Discourse.store.list_missing_uploads` is ran on a site where S3 uploads
are enabled and `SiteSetting.enable_s3_inventory` is set to `true`.
1. Introduce `s3_file_missing` to `Upload.verification_statuses`
2. Introduce `Upload.mark_invalid_s3_uploads_as_missing` which updates
`Upload#verification_status` of all `Upload` records from `invalid_etag` to `s3_file_missing`.
3. Introduce `rake uploads:mark_invalid_s3_uploads_as_missing` Rake task
which allows a site administrator to change `Upload` records with
`invalid_etag` verification status to the `s3_file_missing`
verificaton_status.
4. Update `S3Inventory` to ignore `Upload` records with the
`s3_file_missing` verification status.
Followup 94fe31e5b3,
change the color of the "Known Crawler" bar on the
new "Consolidated Pageviews with Browser Detection (Experimental)"
report to be purple, like it was on the original
"Consolidated Pageviews" report to allow for easier
visual comparison.
Also removes the report colors to named keys in a hash
for easier reference than having to look up the
index of the array all the time.
This commit splits out the updating of `TopicUser#last_read_post_number` in
`TopicUser.ensure_consistency!` to a new
`TopicUser.update_last_read_post_number` method` which
`PostTiming.pretend_read` will now call instead. Previously,
`PostTiming.pretend_read` calls `TopicUser.ensure_consistency!` which in
turn calls `TopicUser.update_post_action_cache` but that is
unnecessary for `PostTiming.pretend_read` since `PostTiming.pretend_read` does not
affect the `TopicUser#liked` or `TopicUser.bookmarked` columns which
`TopicUser.update_post_action_cache` updates. As the query in
`TopicUser.update_post_action_cache` can be expensive, we should avoid
calling it when it isn't necessary.
One such scenario where it is unnecessary is when we are closing a
topic.
This gives us daily fidelity of topic view stats
New table stores a row per topic viewed per day tracking
anonymous and logged on views
We also have a new endpoint `/t/ID/views-stats.json` to get the statistics for the topic.
* FEATURE: add agree and edit
adds agree and edit - an alias for agree and keep -- but with a client action to
edit the post in the composer before the flag is agreed with
---------
Co-authored-by: Juan David Martinez <juan@discourse.org>
We're planning to implement a feature that allows adding required fields for existing users. This PR does some preparatory refactoring to make that possible. There should be no changes to existing behaviour. Just a small update to the admin UI.
This commit updates `Post#each_upload_url` to reject URLs that do not
have a host which matches `Discourse.current_hostname` but follows the
`/uploads/short-url` uploads URL format. This situation most commonly
happen when users copy upload URL link between different Discourse
sites.
This PR introduces a basic AdminNotice model to store these notices. Admin notices are categorized by their source/type (currently only notices from problem check.) They also have a priority.
In 07ecbb5a3b we ensure the mentions in a group's activity page worked properly but we missed adding proper support for infinite loading.
The client is using the `before` parameter instead of the `before_post_id` to do the pagination.
This adds support for `before` as well as some tests to ensure it doesn't regress.
I also added tests to the group's activity posts as well since those were missing.
Finally I deleted some unused code (`group.messages_for`) which is not used anymore.
Context - https://meta.discourse.org/t/-/308044/9
Whenever one creates, updates, or deletes a post, we should keep the `topic.word_count` counter in sync.
Context - https://meta.discourse.org/t/-/308062
When converting a PM to a public topic (and vice versa), if there was a validation error (like a topic already used, or a tag required or not allowed) the error message wasn't bubbled up nor shown to the user.
This fix ensures we properly stop the conversion whenever a validation error happens and bubble up the errors back to the user so they can be informed.
Internal ref - t/128795
Using the CategoryDrop on the categories page redirected the user to the
"latest topics" page with topics only from that category. With these
changes, selecting a category will take the user to a "subcategories
page" where only the subcategories of the selected property will be
displayed.
The users directory is updated on a daily cadence. However, when a site is new and doesn't have many users, it can be confusing that a user who has just joined doesn't show up in the users until a day after they join. To eliminate this confusion, this commit triggers a refresh for the users directory as soon as as a user joins, if the site is in bootstrap mode. The reason for the conditional trigger is that refreshing the users directory is an expensive operation and doing it often on a large site with many users could lead to performance problems.
Internal topic: t/126076.
It used to embed the objects which could lead to duplicated objects
when the same user or category was used multiple times (user was admin,
moderator and category or category was parent for multiple categories).
The automation plugin has 4 custom field types that are array typed. However, array typed custom fields are deprecated and should be migrated to JSON type.
This commit does a couple of things:
1. Migrate all four custom fields to JSON
2. Fix a couple of small bugs that have been discovered while migrating the custom fields to JSON (see the comments on this commit's PR for details https://github.com/discourse/discourse/pull/26939)
This reverts commit 0f4520867b.
This has led to two problems:
1. An incompatibility with Cloudflare's "auto minify" feature. They've deprecated this feature because of incompatibility with modern JS syntax. But unfortunately it will remain enabled on existing properties until 2024-08-05.
2. Discourse fails to boot in Safari 15. This is strange, because Safari does support all the required features in our production JS bundles. Even more strangely, things start working as soon as you open the developer tools. That suggests the cause could be a Safari bug rather than a simple incompatibility.
Reverting while we work out a path forward on both those issues.
decorator-transforms (https://github.com/ef4/decorator-transforms) is a modern replacement for babel's plugin-proposal-decorators. It provides a decorator implementation using modern browser features, without needing to enable babel's full suite of class feature transformations. This improves the developer experience and performance.
In local testing with Google's 'tachometer' tool, this reduces Discourse's 'init-to-render' time by around 3-4% (230ms -> 222ms).
It reduces our initial gzip'd JS payloads by 3.2% (2.43MB -> 2.35MB), or 7.5% (14.5MB -> 13.4MB) uncompressed.
Ignored columns can only be dropped when its associated post-deploy
migration has been promoted to a regular migration. This is so because
Discourse doesn't rely on a schema file system to setup a brand new
database and thus the column information will be loaded by the
application first before the post-deploy migration runs.
If there's ever a circular reference in categories, don't go into an infinite loop when generating the category slug.
Instead, keep track of parent ids, and bail out as soon as we're encountering one more than once.
In #22851 we added a dependent strategy for deleting upload references when a draft is destroyed. This, however, didn't catch all cases, because we still have some code that issues DELETE drafts queries directly to the database. Specifically in the weekly cleanup job handled by Draft#cleanup!.
This PR fixes that by turning the raw query into an ActiveRecord #destroy_all, which will invoke the dependent strategy that ultimately deletes the upload references. It also includes a post migration to clear orphaned upload references that are already in the database.
This commit introduces the `run_theme_migration` spec helper to allow
theme developers to write RSpec tests for theme migrations. For example,
this allows the following RSpec test to be written in themes:
```
RSpec.describe "0003-migrate-small-links-setting migration" do
let!(:theme) { upload_theme_component }
it "should set target property to `_blank` if previous target component is not valid or empty" do
theme.theme_settings.create!(
name: "small_links",
theme: theme,
data_type: ThemeSetting.types[:string],
value: "some text, #|some text 2, #, invalid target",
)
run_theme_migration(theme, "0003-migrate-small-links-setting")
expect(theme.settings[:small_links].value).to eq(
[
{ "text" => "some text", "url" => "#", "target" => "_blank" },
{ "text" => "some text 2", "url" => "#", "target" => "_blank" },
],
)
end
end
```
This change is being introduced because we realised that writting just
javascript tests for the migrations is insufficient since javascript
tests do not ensure that the migrated theme settings can actually be
successfully saved into the database. Hence, we are introduce this
helper as a way for theme developers to write "end-to-end" migrations
tests.
... wasn't working because it wasn't storing the proper "action" value.
Issue was that we were using the "action" parameter which is being used by Rails to determine which controller action to call.
We need to use the "action_key" parameter instead.
At the moment, there is no way to create a group of related watched words together. If a user needed a set of words to be created together, they'll have to create them individually one at a time.
This change attempts to allow related watched words to be created as a group. The idea here is to have a list of words be tied together via a common `WatchedWordGroup` record. Given a list of words, a `WatchedWordGroup` record is created and assigned to each `WatchedWord` record. The existing WatchedWord creation behaviour remains largely unchanged.
Co-authored-by: Selase Krakani <skrakani@gmail.com>
Co-authored-by: Martin Brennan <martin@discourse.org>
This change encourages users to title their threads to make it easier for other users to join in on conversations that matter to them.
The creator of the chat thread will receive a toast notification prompting them to add a thread title when on mobile and the thread has at least 5 sent replies.
In the "Consolidated Pageviews with Browser Detection (Experimental)"
report we used the same color for "Known Crawler" and "Other pageviews"
which makes the report confusing to look at, this commit makes them
different.
This commit introduces a few changes as a result of
customer issues with finding why a topic was relisted.
In one case, if a user edited the OP of a topic that was
unlisted and hidden because of too many flags, the topic
would get relisted by directly changing topic.visible,
instead of going via TopicStatusUpdater.
To improve tracking we:
* Introduce a visibility_reason_id to topic which functions
in a similar way to hidden_reason_id on post, this column is
set from the various places we change topic visibility
* Fix Post#unhide! which was directly modifying topic.visible,
instead we use TopicStatusUpdater which sets visibility_reason_id
and also makes a small action post
* Show the reason topic visibility changed when hovering the
unlisted icon in topic status on topic titles
Selecting the +subcategories option does not work sometimes when "lazy
load categories" is enabled because the subcategories may not be
fetched. This ensures that subcategories are loaded by requesting them
before being used.
In a large forum with millions of users and millions of user_fields
updating the list of dropdown user field options will result in a
502 now due to the large number of fields.
This commit moves the indexing into a job.
Our 'page_view_crawler' / 'page_view_anon' metrics are based purely on the User Agent sent by clients. This means that 'badly behaved' bots which are imitating real user agents are counted towards 'anon' page views.
This commit introduces a new method of tracking visitors. When an initial HTML request is made, we assume it is a 'non-browser' request (i.e. a bot). Then, once the JS application has booted, we notify the server to count it as a 'browser' request. This reliance on a JavaScript-capable browser matches up more closely to dedicated analytics systems like Google Analytics.
Existing data collection and graphs are unchanged. Data collected via the new technique is available in a new 'experimental' report.
This ensures we only ever store correct post and topic timing when the client
notifies.
Previous to this change we would blindly trust the client.
Additionally this has error correction code that will correct the last seen
post number when you visit a topic with incorrect timings.
This commit addresses an issue for sites where secure_uploads
is turned on after the site has been operating without it for
some time.
When uploads are linked when they are used inside a post,
we were setting the access_control_post_id unconditionally
if it was NULL to that post ID and secure_uploads was true.
However this causes issues if an upload has been used in a
few different places, especially if a post was previously
used in a PM and marked secure, so we end up with a case of
the upload using a public post for its access control, which
causes URLs to not use the /secure-uploads/ path in the post,
breaking things like image uploads.
We should only set the access_control_post_id if the post is the first time the
upload is referenced so it cannot hijack uploads from other places.
When lazy load categories is enabled, categories should be loaded with
user activity items and drafts because the categories may not be
preloaded on the client side.
This will automatically enable the glimmer header when all installed themes/plugins are ready. This replaces the old group-based site setting.
In 'auto' mode, we check for calls to deprecated APIs (e.g. decorateWidget) which affect the old header. If any are present, we stick to the old header implementation and print a message to the console alongside the normal deprecation messages.
To override this automatic behavior, a new `glimmer_header_mode` site setting can be set to 'disabled' or 'enabled'.
This change also means that our test suite is running with the glimmer header. This unveiled a couple of small issues (e.g. some incorrect `aria-*` and `alt` text) which are now fixed. A number of selectors had to be updated to ensure the tests were clicking the actual `<button>` elements rather than the surrounding `<li>` elements.
This modifier allows plugins to alter the outcome of
`should_secure_uploads?` on a Post record, for cases when
plugins need post-attached uploads to always be secure (or
not secure) in specific scenarios.
This method name is a bit confusing; with_secure_uploads implies
it may return a block or something with the uploads of the post,
and has_secure_uploads implies that it's checking whether the post
is linked to any secure uploads.
should_secure_uploads? communicates the true intent of this method --
which is to say whether uploads attached to this post should be
secure or not.
* DEV: Add `topic_embed_import_create_args` plugin modifier
This modifier allows a plugin to change the arguments used when creating
a new topic for an imported article.
For example: let's say you want to prepend "Imported: " to the title of
every imported topic. You could use this modifier like so:
```ruby
# In your plugin's code
plugin.register_modifier(:topic_embed_import_create_args) do |args|
args[:title] = "Imported: #{args[:title]}"
args
end
```
In this example, the modifier is prepending "Imported: " to the `title` in the `create_args` hash. This modified title would then be used when the new topic is created.
This PR improves the performance of the `most_replied_to_users` method on the `UserSummary` model.
### Old Query
```ruby
post_query
.joins(
"JOIN posts replies ON posts.topic_id = replies.topic_id AND posts.reply_to_post_number = replies.post_number",
)
# We are removing replies by @user, but we can simplify this by getting the using the user_id on the posts.
.where("replies.user_id <> ?", @user.id)
.group("replies.user_id")
.order("COUNT(*) DESC")
.limit(MAX_SUMMARY_RESULTS)
.pluck("replies.user_id, COUNT(*)")
.each { |r| replied_users[r[0]] = r[1] }
```
### Old Query with corrections
```ruby
post_query
.joins(
"JOIN posts replies ON posts.topic_id = replies.topic_id AND replies.reply_to_post_number = posts.post_number",
)
# Remove replies by @user but instead look on loaded posts (we do this so we don't count self replies)
.where("replies.user_id <> posts.user_id")
.group("replies.user_id")
.order("COUNT(*) DESC")
.limit(MAX_SUMMARY_RESULTS)
.pluck("replies.user_id, COUNT(*)")
.each { |r| replied_users[r[0]] = r[1] }
```
### New Query
```ruby
post_query
.joins(
"JOIN posts replies ON posts.topic_id = replies.topic_id AND posts.reply_to_post_number = replies.post_number",
)
# Only include regular posts in our joins, this makes sure we don't have the bloat of loading private messages
.joins(
"JOIN topics ON replies.topic_id = topics.id AND topics.archetype <> 'private_message'",
)
# Only include visible post types, so exclude posts like whispers, etc
.joins(
"AND replies.post_type IN (#{Topic.visible_post_types(@user, include_moderator_actions: false).join(",")})",
)
.where("replies.user_id <> posts.user_id")
.group("replies.user_id")
.order("COUNT(*) DESC")
.limit(MAX_SUMMARY_RESULTS)
.pluck("replies.user_id, COUNT(*)")
.each { |r| replied_users[r[0]] = r[1] }
```
# Conclusion
`most_replied_to_users` was untested, so I introduced a test for the logic, and have confirmed that it passes on both the new query **AND** the old query.
Thank you @danielwaterworth for the debugging assistance.
We will be collecting the logo URL and the site's default locale values along with existing basic details to display the site on the Discourse Discover listing page. It will be included only if the site is opted-in by enabling the "`include_in_discourse_discover`" site setting.
Also, we no longer going to use `about.json` and `site/statistics.json` endpoints retrieve these data. We will be using only the `site/basic-info.json` endpoint.
When a user is manually deactivated, they should not be deleted by our background job that purges inactive users.
In addition, site settings keywords should accept an array of keywords.
Previously the problem check registry simply looked at the subclasses of ProblemCheck. This was causing some confusion in environments where eager loading is not enabled, as the registry would appear empty as a result of the classes never being referenced (and thus never loaded.)
This PR changes the approach to a more explicit one. I followed other implementations (bookmarkable and hashtag autocomplete.) As a bonus, this now has a neat plugin entry point as well.
Why this change?
This is a follow up to 897be75941.
When updating `net-smtp` from `0.4.x` to `0.5.x`, our test suite passed
but the error `ArgumentError: SMTP-AUTH requested but missing user name`
was being thrown in production leading to emails being failed to send
out via SMTP.
This commit adds a test to ensure that our production SMTP settings will
at least attemp to connect to an SMTP server.
## Why this change?
The previous implementation of the method generated the query to find the relevant topics and iterated over the results, processing them.
This behavior made difficult reusing or changing the query logic in classes extending `CategoryList`.
This commit extracts the query logic into another method called `relevant_topics_query ` which can be reused or overwritten in descendant classes.
This was originally introduced in #26071, but that PR was closed, because the requirements changed. This PR lifts only the relevant parts, since they are a prerequisite for the new admin notice system.
This enables the following in Discourse AI
```
plugin.register_modifier(:chat_allowed_bot_user_ids) do |user_ids, guardian|
if guardian.user
mentionables = AiPersona.mentionables(user: guardian.user)
allowed_bot_ids = mentionables.map { |mentionable| mentionable[:user_id] }
user_ids.concat(allowed_bot_ids)
end
user_ids
end
```
some bots that are id < 0 need to be discoverable in search otherwise people can not talk to them.
---------
Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
When "lazy load categories" is enabled and parent_category_id was set,
the query fetching categories contained a contradiction filtering both
by parent_category_id and parent_category_id = NULL.
When categories are loaded by the frontend, the parent category is
looked up by ID and the `parentCategory` is set with the result. If the
categories returned are not in order, the parent category may miss.
We were using `OFFSET`/`LIMIT` to query topics without an 'ORDER'. Without an explicit order, postgres makes no guarantees about which rows will be returned for each query. This commit adds `ORDER BY id ASC` so that our sitemaps behave consistently.
All our link validation, and conversion from url -> route/model/query is expensive and prone to bugs. Instead, if people enter a link, we can just use it as-is.
Originally all this extra logic was added to handle unusual situations like `/safe-mode`, `/my/...`, etc. However, all of these are now handled correctly by our Ember router, so there is no need for it.
Now, we just pass the user-supplied `href` directly to the SectionLink component, and let Ember handle routing to it when clicked.
The only functional change here is that we no longer validate internal links by parsing them with the Ember router. But I'd argue this is fine, because the previous logic would cause both false positives (e.g. `/t/123` would be valid, even if topic 123 doesn't exist), and false negatives (for routes which are server-side only, like the new AI share pages).
We were incorrectly using `return` in a block which was causing exceptions at runtime. These exceptions were not causing much issues as they are in defer block.
While working on writing a test for this specific case, I noticed that our `upsert_custom_fields` function was using rails `update_all` which is not updating the `updated_at` timestamp. This commit also fixes it and adds a test for it.
In #26122 we promoted all problem checks defined as class methods on AdminDashboardData to their own first-class ProblemCheck instances.
This PR continues that by promoting problem checks that are implemented as blocks as well. This includes updating a couple plugins that have problem checks.
This is a follow up to e2da72b76c.
Why this change?
According to https://web.dev/articles/preload-critical-assets,
> By preloading a certain resource, you are telling the browser that you would like to fetch it sooner than the browser would otherwise discover it because you are certain that it is important for the current page.
The preload resource hint is meant to tell the browser to fetch
resources that it would not discover upfront or early. However, we are
not using it the right way because we are literally adding the resource
hint right before a `<script>` tag which means the browser would have
discovered the resource even without the resource hint.
What does this change do?
This commit removes the preload resource hint which are added right
before script tags since the optimization here is highly questionable at the expense of making
our initial DOM larger.
We never use that information and this also fixes an issue with the BCC plugin which ends up triggering a rate-limit because we were publishing a "NEW_PRIVATE_MESSAGE" to the user sending the BCC for every recipients 💥
Internal - t/118283
