We currently query the posts table without an order when notifying users of moved posts. Generally the query will return the lowest post number post (b/c ID correlates with post_number in most cases) but not always. This adds an order to the post query in notify_moved_posts job.
Also I removed some if statement nesting with early returns / guard clauses.
Admins and moderators can see a user's deleted posts via the `/u/:username/deleted-posts` route. Admins can always see any post on the site, but that's not always the case for moderators, e.g., they can't see all PMs. So, this route accounts for that and excludes posts that a moderator wouldn't be allowed to see if they were not deleted.
However, there's currently a problem with that logic where admins who also have moderation privileges, are treated the same way as moderators and prevented from seeing posts that pure moderators can't see. This commit fixes that problem and only applies the permission checks to moderators who don't have admin privileges.
Internal topic: t/143107.
This commit adds the `add_request_rate_limiter` plugin API which allows plugins to add custom rate limiters on top of the default rate limiters which requests by a user's id or the request's IP address.
Example to add a rate limiter that rate limits all requests from Googlebot under the same rate limit bucket:
```
add_request_rate_limiter(
identifier: :country,
key: ->(request) { "country/#{DiscourseIpInfo.get(request.ip)[:country]}" },
activate_when: ->(request) { DiscourseIpInfo.get(request.ip)[:country].present? },
)
```
When lurking on a Discourse as anonymous, if the sidebar is enabled, and a section contains only secondary links that are not visible to anonymous users, we should not display the "more..." button.
Otherwise it feels broken because clicking on it does nothing, since there are no "visible" links to be shown.
Internal ref t/144716
* wip: return full name in /notifications.json
* DEV: test for full name
* DEV: add test for enable_names=true
* DEV: add notification6, cleanup
* DEV: fix tests
This PR involves cleaning up the codebase from my (@keegangeorge's) todos.
In particular:
- Remove Form Template related todos (these are no longer in the roadmap)
- Remove old left-over AI summarization related code after moving to AI (https://github.com/discourse/discourse-ai/pull/658)
- Update one form template related spec
* Split `shutdown` into two separate methods for better control:
- `shutdown` - signals threads to stop accepting new work
- `wait_for_termination` - waits for threads to finish (with optional timeout)
* Add tracking of busy threads via `@busy_threads` Set
* Make idle_time parameter optional with 30-second default
* Improve thread spawning logic:
- Spawn initial thread immediately when work is posted
- Spawn additional threads when all threads are busy and work is queued
* Fix race condition in work distribution
* Add busy thread count to stats output
* Add test coverage for zero min_threads configuration
This commit makes the ThreadPool more reliable, easier to use, and adds
better visibility into its internal state.
---------
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
This commit introduces a new ThreadPool class that provides efficient worker
thread management for background tasks. Key features include:
- Dynamic scaling from min to max threads based on workload
- Proper database connection management in multisite setup
- Graceful shutdown with task completion
- Robust error handling and logging
- FIFO task processing with a managed queue
- Configurable idle timeout for worker threads
The implementation is thoroughly tested, including stress tests, error
scenarios, and multisite compatibility.
This update adds a ✨ _new_ `<PostList />` component, along with it's child components (`<PostListItem/>` and `<PostListItemDetails />`). This new generic component can be used to show a list of posts.
It can be used like so:
```js
/**
* A component that renders a list of posts
*
* @component PostList
*
* @args {Array<Object>} posts - The array of post objects to display
* @args {Function} fetchMorePosts - A function that fetches more posts. Must return a Promise that resolves to an array of new posts.
* @args {String} emptyText (optional) - Custom text to display when there are no posts
* @args {String|Array} additionalItemClasses (optional) - Additional classes to add to each post list item
* @args {String} titleAriaLabel (optional) - Custom Aria label for the post title
*
*/
```
```hbs
<PostList
@posts={{this.posts}}
@fetchMorePosts={{this.loadMorePosts}}
@emptyText={{i18n "custom_identifier.empty"}}
@additionalItemClasses="custom-class"
/>
```
Setting tab should be added to permalinks so admins do not need to have left `/permalinks`.
A new component called `AreaSetting` was added to avoid duplications and
simplify adding settings to other sections.
This allows plugins to skip the "posted" notifications for watching users, when posts get moved. The specs are kind of wild looking, as this unit tests a private method. This is difficult to isolate otherwise, with lots of trickery needed to make sure that this actually works.
I opted to unit test just this method instead.
There’s currently a bug when using a dedicated class as a policy in
services: if that class delegates its `#call` method (to an underlying
strategy object for example), then an error will be raised saying steps
aren’t allowed to provide default parameters.
This should not happen, and this patch fixes that issue.
This commit reimplements how we monitor Sidekiq processes that are
forked from the Unicorn master process. Prior to this change, we rely on
`Jobs::Heartbeat` to enqueue a `Jobs::RunHeartbeat` job every 3 minutes.
The `Jobs::RunHeartbeat` job then sets a Redis key with a timestamp. In
the Unicorn master process, we then fetch the timestamp that has been set
by the job from Redis every 30 minutes. If the timestamp has not been
updated for more than 30 minutes, we restart the Sidekiq process. The
fundamental flaw with this approach is that it fails to consider
deployments with multiple hosts and multiple Sidekiq processes. A
sidekiq process on a host may be in a bad state but the heartbeat check
will not restart the process because the `Jobs::RunHeartbeat` job is
still being executed by the working Sidekiq processes on other hosts.
In order to properly ensure that stuck Sidekiq processs are restarted,
we now rely on the [Sidekiq::ProcessSet](https://github.com/sidekiq/sidekiq/wiki/API#processes)
API that is supported by Sidekiq. The API provides us with "near real-time (updated every 5 sec)
info about the current set of Sidekiq processes running". The API
provides useful information like the hostname, pid and also when Sidekiq
last did its own heartbeat check. With that information, we can easily
determine if a Sidekiq process needs to be restarted from the Unicorn
master process.
This converts the `<AdminPageHeader />` component and the
`<AdminPageSubheader />` components into new components
that can be used outside of admin, and updates the CSS classes.
Also introduces a `<DPageActionButton />` component and child
components for the header action buttons.
I have to keep the old admin-only components around for
now until plugins are updated, then we can remove it,
and remove the re-exports that are done within
admin-page-action-button.gjs
When freeze_original option is passed to PostMover, and we are moving all posts there is an issue. We attempt to put the small_action right after the last moved post. The issue is when there is an existing small action after the last moved "real" post. We then try to put the moderator post at the same location of the existing small action, which causes an index conflict and the move fails.
This makes sure that we place the moderator post at the verrrrrry end of the topic :)
These controller tests are passing locally and in CI, but are failing the build when run in parallel.
I managed to recreate the failures by running the entire suite with turbo_spec and the right seed locally. After these changes, the parallel suite passes locally as well. 🤞
`new_in_category` was using `first` instead of `limit`
This meant it gets an array and that means that you can not operate on it easily in a modifier.
This ensures we always give the modifier a relation, with the notable exception of suggested topics.
We're changing the default of hide_email_address_taken to true. This is a trade-off we want to make, as it prevents account enumeration with minimal impact on legitimate users. If you forget you have an account and try to sign up again with the same e-mail you'll receive an e-mail letting you know.
Add flag reason filter and improve handling of deleted content in review queue
This commit enhances the review queue with several key improvements:
1. Adds a new "Reason" filter to allow filtering flags by their score type
2. Improves UI for deleted content by:
- Adding visual indication for deleted posts (red background)
- Properly handling deleted content visibility for staff (category mods can not see deleted content)
3. Refactors reviewable score type handling for better code organization
4. Adds tests for trashed topics/posts visibility
This change will help moderators more efficiently manage the review queue by
being able to focus on specific types of flags and better identify deleted
content.
This commit introduces <NotificationsTracking /> which is a wrapper component around <DMenu /> which replaces the select-kit component <TopicNotificationsButton />.
Each tracking case has its dedicated component:
- topic -> `<TopicNotificationsTracking />`
- group -> `<GroupNotificationsTracking />`
- tag -> `<TagNotificationsTracking />`
- category -> `<CategoryNotificationsTracking />`
- chat thread -> `<ThreadNotificationsTracking />`
Previously, theme hbr files were compiled to an IIFE, which would be executed before the app is booted. That is causing silenced deprecations to be printed, because the deprecation-workflow isn't set up when the IIFE is run.
This commit updates the theme compiler so that it matches the ember-cli-based raw-hbs compiler. Templates are output to normal modules, which will then be loaded by the existing `eager-load-raw-templates` initializer. This runs after the app has started booting.
* DEV: add db consistency check for UserEmail
* DEV: add db consistency check for UserAvatar
* DEV: ignore inconsistent data related to user avatars when deciding whether to rebake old posts
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
---------
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
Currently, there are two ways (kind of) for accessing `params` inside a
service:
- when there is no contract or it hasn’t been reached yet, `params` is
just the hash that was provided to the service. To access a key, you
have to use the bracket notation `params[:my_key]`.
- when there is a contract and it has been executed successfully,
`params` now references the contract and the attributes are accessible
using methods (`params.my_key`).
This patch unifies how `params` exposes its attributes. Now, even if
there is no contract at all in a service, `params` will expose its
attributes through methods, that way things are more consistent.
This patch also makes sure there is always a `params` object available
even when no `params` key is provided to the service (this allows a
contract to fail because its attributes are blank instead of having the
service raising an error because it doesn’t find `params` in its context).
This commit adds a new column full_move to the moved_posts table. This is useful to look back at history and determine if a whole topic was moved or partial.
This commit also adds an apply_modifier to skip the creation of the moved posts small action.
This patch aims to improve the steps inspector output:
- The service class name is displayed at the top.
- Next to each step is displayed the time it took to run said step.
- Steps that didn’t run are hidden.
- `#inspect` automatically outputs the error when it is present.
It doesn't make much sense to have the content of a `<details>` in an excerpt so I replaced them with "▶ summary" instead.
That way, they can't be (ab)used in user cards for example.
Reference - https://meta.discourse.org/t/335094
The new name may be too long for the bookmarks.name column and raise an
exception. This changes allows the remapper to truncate the new value to
fit (truncates to 100 characters).
The test was flaky and failing with the following errors:
```
Failure/Error:
klass
.connection
.select_raw(relation.arel) do |result, _|
result.type_map = DB.type_map
result.nfields == 1 ? result.column_values(0) : result.values
end
NoMethodError:
undefined method `select_raw' for nil
./lib/freedom_patches/fast_pluck.rb:60:in `pluck'
./vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/relation/calculations.rb:354:in `pick'
./app/models/web_crawler_request.rb:27:in `request_id'
./app/models/web_crawler_request.rb:31:in `rescue in request_id'
./app/models/web_crawler_request.rb:26:in `request_id'
./app/models/web_crawler_request.rb:19:in `write_cache!'
./app/models/concerns/cached_counting.rb:135:in `block (3 levels) in flush_to_db'
./vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
./vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management.rb:21:in `with_connection'
./app/models/concerns/cached_counting.rb:134:in `block (2 levels) in flush_to_db'
./app/models/concerns/cached_counting.rb:124:in `each'
./app/models/concerns/cached_counting.rb:124:in `block in flush_to_db'
./lib/distributed_mutex.rb:53:in `block in synchronize'
./lib/distributed_mutex.rb:49:in `synchronize'
./lib/distributed_mutex.rb:49:in `synchronize'
./lib/distributed_mutex.rb:34:in `synchronize'
./app/models/concerns/cached_counting.rb:120:in `flush_to_db'
./app/models/concerns/cached_counting.rb:187:in `perform_increment!'
./app/models/web_crawler_request.rb:15:in `increment!'
./lib/middleware/request_tracker.rb:74:in `log_request'
./lib/middleware/request_tracker.rb:409:in `block in log_later'
./lib/scheduler/defer.rb:125:in `block in do_work'
./vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
./vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management.rb:21:in `with_connection'
./lib/scheduler/defer.rb:119:in `do_work'
./lib/scheduler/defer.rb:105:in `block (2 levels) in start_thread'
```
This was due to running the defer thread in an async manner which is
actually no representative of the production environment. It also
revealed a spot in our code base where writes are happening in a GET
request which can cause requests to fail if ActiveRecord is in readonly
mode.
The directory items controller specs that have a search param were not
matching how things worked in production. In a non-test environment the
UserSearch class depends on the `user_search_data` table being
populated, so the tests I corrected now use this table as well to match
reality.
Also added a new test to match the 20 user limit for search results that
currently exists. This 20 user limit is on the line between a bug and a
feature but it is how it is currently working so we should document
that. We have plans to increase this limit and it has been documented
here: https://meta.discourse.org/t/296485
This PR is a no-op and only changes the tests.
Co-authored-by: brrusselburg <25828824+brrusselburg@users.noreply.github.com>
When a post containing an apostrophe (') is being cooked, the apostrophe is being converted to the "typographic" version (’) (because we enable markdown-it's **typographer** mode by default in Discourse)
When you select text that contains such apostrophe and then try to save your fast edit, it fails miserably without any error.
That's because when you select text from the DOM, it uses the cooked version which has the typographic apostrophe.
When you save your fast edit, we fetch the raw version of the post, which has the "regular" apostrophe. Thus doing `raw.replace(selectedText, editedText)` doesn't work because `raw` has the regular apostrophe but `selectedText` has the typographic apostrophe.
Since it's somewhat complicated to handle all typographic characters, we would basically have to reverse the process done in `custom-typographer-replacements.js`, we instead bail out and show the composer when we detect such character in the selection.
Internal ref - t/143836
This reverts commit 766ff723f8.
Ensure that we create the sidekiq log file first before opening it for
logging. This avoids any issue of the log file not being present when we
initialize an instance of the `Logger`.
Some pages like new/edit item should not display admin header. New attribute called `@shouldDisplay` was added.
As a proof of concept, the flags page was updated.
In 806e37aaec, I improved the conflict handling when editing a post to account for title and tags.
This fixes an edge cases when a topic has a hidden tag the current editor can't see. When they submit their edit, we automatically add the hidden tags before checking with the tags stored in the database.
Reported in https://meta.discourse.org/t/341375
We've seen in some communities abuse of user profile where bios and other fields are used in malicious ways, such as malware distribution. A common pattern between all the abuse cases we've seen is that the malicious actors tend to have 0 posts and have a low trust level.
To eliminate this abuse vector, or at least make it much less effective, we're making the following changes to user profiles:
1. Anonymous, TL0 and TL1 users cannot see any user profiles for users with 0 posts except for staff users
2. Anonymous and TL0 users can only see profiles of TL1 users and above
Users can always see their own profile, and they can still hide their profiles via the "Hide my public profile" preference. Staff can always see any user's profile.
Internal topic: t/142853.
Currently when copy an OP to another topic, the link is to the topic that wasn't moved. The notification should instead be to the new topic the OP was moved to -- we have duplicate logic already for this but first post creation get special treatment, and this applies the same treatment.
Follow-up from this commit - 9b8af0ea9f
Adds helpful data into MovedPost records for later lookup. ALSO fixes notifications for freeze_original to point to the newly created post, not the moved post.
PostMover has a new option called freeze_original implemented in this commit. It was previously unexposed in the controller. This PR permits the param in the controller, and passes it into PostMover.
Also, this applies a value transformer for move/merge payload options. In addition a plugin outlet in the move post modal. This allows plugins to add content to the modal, which can modify the payload (and use the freeze_original argument for example)
Followup c7e471d35a
It is currently possible to add a bundle (which is a collection
of actions used for a dropdown on the client) for a reviewable
via actions.add_bundle and then never add any actions to it.
This causes the client to explode, as seen in the referenced
commit, because of the way our store expects to resolve objects
referenced by ID that are passed down by the serializer, which
then causes Ember to have an unrecoverable render error.
Fixing this on the serializer level is not really possible because
of all the ActiveModel::Serializer magic that serializes
objects by ID reference when doing things like has_many.
`Reviewable#actions_for` is a better place to do this anyway,
because this is the main location where the bundles and actions
are built for every action via the serializer.
Followup aca6c462a6
Remove the warning message if DISCOURSE_INCLUDE_GEMS_IN_RSPEC_BACKTRACE
is not set for now while we decide whether we want to include
this or not, it's a little in-your-face.
Currently only system flags are translated. When we send message to the user that their post was deleted because of custom flag, we should default to custom flag name.
Previously when attempting to edit a globally shadowed setting, the
error message was not very helpful, it said "You are not allowed to
change hidden settings". This commit changes the error message to
reflect the actual problem, which is that the setting is shadowed by
a global setting via ENV var.
Followup:
* https://github.com/discourse/discourse/pull/28160
* https://github.com/discourse/discourse/pull/25921
In the previous PRs we added 2 environent variables
to control backtrace output for errors in rspec,
`RSPEC_EXCLUDE_NOISE_IN_BACKTRACE`, and
`RSPEC_EXCLUDE_GEMS_IN_BACKTRACE`
These largely do the same thing, and we want to enable
that behaviour by default.
This commit consolidates them into one env var,
`DISCOURSE_INCLUDE_GEMS_IN_RSPEC_BACKTRACE`, which is
disabled by default, meaning gem backtraces will not
be shown in rspec backtraces by default.
Also for the request spec use case with `RspecErrorTracker`,
we now show an indicator of how many lines were hidden from
the backtrace e.g. "...(21 framework line(s) excluded)",
and for this and the normal rspec backtrace exclusion we
show a warning if `DISCOURSE_INCLUDE_GEMS_IN_RSPEC_BACKTRACE`
is not enabled.
before this commit, when moving posts with freeze option, the rate limit was being applied leading to errors. This commit fixes that.
and also adds tests for the scenarios of moving posts with freeze option.
BEFORE: if you click the "reply" button on a post and then decided that you want to "edit" the same post, clicking the "edit" button would do nothing. Clicking "edit" on another post works, but editing the same post would appear broken.
AFTER: if you click the "edit" button, it will properly load the content of the post you're trying to edit. No matter which one it is.
This was somewhat tricky to track down as the system specs seemed to contradict the qunit tests until I realized that the qunit tests were only testing the edit on the 1st post and the system specs were testing on replies.
I improved the qunit tests to test both editing OP and a reply and (hopefully) made the system specs a little bit clearer.
This is a follow up to bbe62d88d2.
In order to limit issues with duplicate inline CSS definitions, this will now deduplicate inline CSS styles with the "last-to-be-defined-wins" strategy.
Also removes unecessary whitespaces in inline styles.
Context - https://meta.discourse.org/t/resolve-final-styles-in-email-notifications/310219
Co-authored-by: Thomas Kalka <thomas.kalka@gmail.com>
The hierarchical search for categories is composed of several complex
nested queries. This change ensures that the secured categories are
filtered out as soon as possible to ensure that the default limit of 5
categories is reached.
Without this fix, the search can return less than 5 categories if any
of the first 5 categories cannot be displayed due to permissions.
Non-admin/moderator users can bulk select items in new/unread, but not in
latest/top/hot. This commit ensures that when the user can no longer
bulk select items in a list, the bulk select checkboxes in the topic list
rows are hidden.
Spec was flaky cause work could still be in pipeline after the defer
length is 0. Our length denotes the backlog, not the in progress
count.
This adds a mechanism for gracefully stopping the queue and avoids
wait_for callse
We already add the "delete user" and "delete and block user" options to the drop-down for potential spam, but we should do this for potentially illegal posts as well.
This is entirely based on the implementation for the potential spam one, including caching the status on the reviewable record.
Also note that just as for potential spam, the user must be "deletable" for the option to appear.
I also took the liberty to move the options in the drop-down to what I think is a more intuitive place. (Between delete post and suspend/silence user.)
Sometimes changes to "What's new?" feed items are made or the feed items are
removed altogether, and the polling interval to check for new features is 1 day.
This is quite long, so this commit introduces a "Check for updates"
button for admins to click on the "What's new?" page which will bust
the cache for the feed and check again at the new features endpoint.
This is limited to 5 times per minute to avoid rapid sending of
requests.
* FEATURE: Add `freeze_original` option to `PostMover`
This option will allow the api user to specify if the original topic should be `frozen`(locked and posts not deleted neither moved)
With this option when moving topic posts your posts will be `copied` to the new topic and the original topic will be kept there.
* DEV: update tests to check raw instead of ids
* DEV: Implement `freeze_original` option for `PostMover`
update specs to use `*array` matcher
* DEV: add tests to `MovedPost` model in post mover
* DEV: Update `MovedPost` model rspec
* DEV: add back empty line to `post_mover.rb`
* FIX: Solve flaky tests in `PostMover`
