Likely we want the group smtp email job to retry. Also added a check to
see if we already have an email log entry for the message to avoid
possible duplicates on retry.
Related previous commit: ed47b550266e1ab669c756b0ecb48d1685b08fee
Until now, we were excluding uploads with a not null
access_control_post_id from cleanup, because we were still
considering them "secure" even though they might have been
`secure: false`.
It's not necessary to keep these around, these are no more
important than regular uploads if they are not referenced
by anything.
The optimization is not required because `find_each` fetches records in
batches of 1000 and no upload record will have more than 1000 optimized
images.
This commit converts the `AdminReport` component, which is quite
high complexity, to gjs. After this initial round, ideally this
component would be broken up into smaller components because it is
getting quite big now.
Also in this commit:
* Add an option to display the report description in a tooltip, which
was
the main way the description was shown until recently. We want to use
this on the dashboard view mostly.
* Move admin report "mode" definitions to the server-side Report model,
inside a `Report::MODES` constant, collecting the modes defined in
various
places in the UI into one place
* Refactor report code to refer to mode definitions
* Add a `REPORT_MODES` constant in JS via javascript.rake and refactor
JS to refer to the modes
* Delete old admin report components that are no longer used
(trust-level-counts, counts, per-day-counts) which were replaced
by admin-report-counters a while ago
* Add a new `registerReportModeComponent` plugin API, some plugins
introduce their own modes (like AI's `emotion`) and components and
we need a way to render them
This commit fixes an SQL syntax error in
`UserBadge.update_featured_ranks!` when
the `user_ids` param is an empty array `[]`.
This was causing the `Jobs::BackfillBadge` job to raise the following
exceptions:
```
Job exception: ERROR: syntax error at or near ")"
LINE 6: AND user_id IN ()
```
This commit fixes the same error in
`UserState.update_distinct_badge_count` as well
Follow-up to 3e4eac0fed05daedcdea50d6275e143469d55eda
This commit updates the `Jobs::BadgeGrant` scheduled job to enqueue on
`Job::BackfillBadge` regular job for each enabled badge on the site.
The rationale for this change is that we started seeing the
`Jobs::BadgeGrant` job taking hours on sites with lots of enabled badges
as well as users because the job was backfilling all enabled badges
serially within the job. This is bad as it means that a `mini_scheduler`
thread is tied up
by this job thus reducing the overall capacity of `mini_scheduler` for
hours.
The GDPR requires all users to be able to export their data, or request an export of their data. This is fine for active users as we have a data export button on user profiles, but suspended users have no way of accessing the data export function, and the workaround for admins to export data for suspended users involves temporarily unsuspending them, then impersonating the user to export the data as them.
Since suspended users no longer have access to their account, we can safely assume that the export request will be coming via a medium outside of Discourse (eg, email). This change is built with this workflow in mind.
This change adds a new "User exports" section to the admin user page, allowing admins to start a new export, and to download the latest export file.
This Type column is a special ":post" column on the
Flag Status report, so it did not show by default in
the CSV export of that report. This adds it so the
type of flag e.g. illegal, off topic, innapropriate
is shown in the CSV output.
(…and remove extraneous `stringify_keys` - enqueue already does
`deep_stringify_keys`)
The warning in question:
```
Deprecation notice: Jobs::RunProblemCheck was enqueued with argument values which do not cleanly serialize to/from JSON. This means that the job will be run with slightly different values than the ones supplied to `enqueue`. Argument values should be strings, booleans, numbers, or nil (or arrays/hashes of those value types). (deprecated since Discourse 2.9) (removal in Discourse 3.0)
```
It is not possible for an admin to generate a suspended user's archive now, disallowing SAR (subject access requests) under the GDPR.
This commit expands the export_user_archive job to allow specifying a requesting_user_id which will send the archive to an admin. When not specified, this defaults to the user itself.
I think the log messages were used for initial production trials but the
log messages are actually not very useful in the grand scheme of things.
If the timing of the job is important information, one can get it by
enabling Sidekiq logging and obtain the information from there. There is
no need for us to flood the logs with these messages.
Also the messages will contain at most 100 ids so it is just alot of
noise in general.
We currently query the posts table without an order when notifying users of moved posts. Generally the query will return the lowest post number post (b/c ID correlates with post_number in most cases) but not always. This adds an order to the post query in notify_moved_posts job.
Also I removed some if statement nesting with early returns / guard clauses.
This commit reimplements how we monitor Sidekiq processes that are
forked from the Unicorn master process. Prior to this change, we rely on
`Jobs::Heartbeat` to enqueue a `Jobs::RunHeartbeat` job every 3 minutes.
The `Jobs::RunHeartbeat` job then sets a Redis key with a timestamp. In
the Unicorn master process, we then fetch the timestamp that has been set
by the job from Redis every 30 minutes. If the timestamp has not been
updated for more than 30 minutes, we restart the Sidekiq process. The
fundamental flaw with this approach is that it fails to consider
deployments with multiple hosts and multiple Sidekiq processes. A
sidekiq process on a host may be in a bad state but the heartbeat check
will not restart the process because the `Jobs::RunHeartbeat` job is
still being executed by the working Sidekiq processes on other hosts.
In order to properly ensure that stuck Sidekiq processs are restarted,
we now rely on the [Sidekiq::ProcessSet](https://github.com/sidekiq/sidekiq/wiki/API#processes)
API that is supported by Sidekiq. The API provides us with "near real-time (updated every 5 sec)
info about the current set of Sidekiq processes running". The API
provides useful information like the hostname, pid and also when Sidekiq
last did its own heartbeat check. With that information, we can easily
determine if a Sidekiq process needs to be restarted from the Unicorn
master process.
* DEV: add db consistency check for UserEmail
* DEV: add db consistency check for UserAvatar
* DEV: ignore inconsistent data related to user avatars when deciding whether to rebake old posts
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
---------
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
This reverts commit 766ff723f8c216b2bf59053dba302b82e1d34d4b.
Ensure that we create the sidekiq log file first before opening it for
logging. This avoids any issue of the log file not being present when we
initialize an instance of the `Logger`.
In `Jobs::Base::JobInstrumenter.raw_log`, we were creating an instance
of `Queue` and then pushing messages to the queue before popping it off
the queue in a thread. However, this complexity is not necessary when
we can just write directly to the logger without much overhead. This is
how all logging is done in other parts of the app as well.
Sometimes `Jobs::PushNotification` gets stuck, probably because of the
network call. This commit replaces `Excon` with `FinalDestination::HTTP`
which is safer.
Constants should always be only assigned once. The logical OR assignment
of a constant is a relic of the past before we used zeitwerk for
autoloading and had bugs where a file could be loaded twice resulting in
constant redefinition warnings.
In our production environment, we have been seeing Sidekiq processes
getting stuck randomly when a USR1 signal is sent to the Unicorn master
process. We have not been able to identify the root cause of why the
Sidekiq process gets stuck. We however noticed that when the Unicorn
master process receives a USR1 signal, it will reopen the logs for the
Unicorn master process first before sending a USR1 signal for the
Unicorn worker processes to reopen the logs. We figured that we should
do the same for the Sidekiq process as well when a USR1 signal.
In this commit, we introduce an arbitrary delay of 1 second before we
the Sidekiq process reopens its log files so as to allow enough time for the Unicorn
master to finish reopening it logs first.
We also do not send reopen logs for the Sidekiq process if the `DISCOURSE_LOG_SIDEKIQ`
env is not present because there is no need to reopen any logs.
When running checks, we look to the existing problem check trackers and try to grab their ProblemCheck classes.
In some cases this is no longer in the problem check repository, e.g. when the check was part of a plugin that has been uninstalled.
In the case where the check was scheduled, this would lead to an error in one of the jobs
This patch removes the `with_service` helper from the code base.
Instead, we can pass a block with actions directly to the `.call` method
of a service.
This simplifies how to use services:
- use `.call` without a block to run the service and get its result
object.
- use `.call` with a block of actions to run the service and execute
arbitrary code depending on the service outcome.
It also means a service is now “self-contained” and can be used anywhere
without having to include a helper or whatever.
Currently, categories support designating only 1 group as a moderation group on the category. This commit removes the one group limitation and makes it possible to designate multiple groups as mods on a category.
Internal topic: t/124648.
Introduce a new log line for Sidekiq jobs that consume more than
`DISCOURSE_LIVE_SLOTS_SIDEKIQ_LIMIT` live slots. This is useful to
track down jobs that may leak memory.
This is enabled only when Sidekiq's job instrumenter is enabled (set
`DISCOURSE_LOG_SIDEKIQ` to `1`).
This patch allows using an AR relation as a model in services without
fetching associated records. It will just check if the relation is empty
or not. In the former case, the execution will stop at that point, as
expected.
### Why?
Before, all flags were static. Therefore, they were stored in class variables and serialized by SiteSerializer. Recently, we added an option for admins to add their own flags or disable existing flags. Therefore, the class variable had to be dropped because it was unsafe for a multisite environment. However, it started causing performance problems.
### Solution
When a new Flag system is used, instead of using PostActionType, we can serialize Flags and use fragment cache for performance reasons.
At the same time, we are still supporting deprecated `replace_flags` API call. When it is used, we fall back to the old solution and the admin cannot add custom flags. In a couple of months, we will be able to drop that API function and clean that code properly. However, because it may still be used, redis cache was introduced to improve performance.
To test backward compatibility you can add this code to any plugin
```ruby
replace_flags do |flag_settings|
flag_settings.add(
4,
:inappropriate,
topic_type: true,
notify_type: true,
auto_action_type: true,
)
flag_settings.add(1001, :trolling, topic_type: true, notify_type: true, auto_action_type: true)
end
```
This change ensures native push notifications respect the site setting for push_notification_time_window_mins. Previously only web push notifications would account for the delay, now we can bring more consistency between Discourse in browser vs Hub, by applying the same delay strategy to both forms of push notifications.
### Why?
Before, all flags were static. Therefore, they were stored in class variables and serialized by SiteSerializer. Recently, we added an option for admins to add their own flags or disable existing flags. Therefore, the class variable had to be dropped because it was unsafe for a multisite environment. However, it started causing performance problems.
### Solution
When a new Flag system is used, instead of using PostActionType, we can serialize Flags and use fragment cache for performance reasons.
At the same time, we are still supporting deprecated `replace_flags` API call. When it is used, we fall back to the old solution and the admin cannot add custom flags. In a couple of months, we will be able to drop that API function and clean that code properly. However, because it may still be used, redis cache was introduced to improve performance.
To test backward compatibility you can add this code to any plugin
```ruby
replace_flags do |flag_settings|
flag_settings.add(
4,
:inappropriate,
topic_type: true,
notify_type: true,
auto_action_type: true,
)
flag_settings.add(1001, :trolling, topic_type: true, notify_type: true, auto_action_type: true)
end
```
This change is mainly a refactor of the desktop notifications service to improve readability and have standardised values for tracking state for current user in regards to the Notification API and Push API.
Also improves readability when handling push notification jobs, especially in scenarios where the push_notification_time_window_mins site setting is set to 0, which will allow sending push notifications instantly.
* FIX: Ensure JsLocaleHelper to obly outputs up-to-date translations
The old implementation forgot to filter out deprecated
translations, causing these translations to incorrectly override the new
locale in the frontend.
This commit fills in the forgotten where clause, filtering only the
up-to-date part.
Related meta topic: https://meta.discourse.org/t/outdated-translation-replacement-causing-missing-translation/314352
In development, classes are lazy loaded so `Jobs::Onceoff.onceoff_job_klasses`
may not have been set. This is not a problem in production cause stuff
is eager loaded.
Follow-up to f4d06f195d583794313943e73b1b31d28151b85c
We are investigating a memory leak in Sidekiq and saw the following line
when comparing heap dumps over time.
`Allocated IMEMO 14775 objects of size 591000/7389528 (in bytes) at:
/var/www/discourse/app/jobs/onceoff/onceoff.rb:36`
That line in question was doing a `.select { |klass| klass < self }` on
`ObjectSpace.each_object(Class)`. This for some reason is allocating a
whole bunch of `IMEMO` objects which are instruction sequence objects.
Instead of diving deeper into why this might be leaking, we can just
save our time by switching to an implementation that is more efficient
and does not require looping through a ton of objects.
