discourse/app/jobs/scheduled
Alan Guo Xiang Tan 9812407f76
FIX: Redo Sidekiq monitoring to restart stuck sidekiq processes (#30198)
This commit reimplements how we monitor Sidekiq processes that are
forked from the Unicorn master process. Prior to this change, we rely on
`Jobs::Heartbeat` to enqueue a `Jobs::RunHeartbeat` job every 3 minutes.
The `Jobs::RunHeartbeat` job then sets a Redis key with a timestamp. In
the Unicorn master process, we then fetch the timestamp that has been set
by the job from Redis every 30 minutes. If the timestamp has not been
updated for more than 30 minutes, we restart the Sidekiq process. The
fundamental flaw with this approach is that it fails to consider
deployments with multiple hosts and multiple Sidekiq processes. A
sidekiq process on a host may be in a bad state but the heartbeat check
will not restart the process because the `Jobs::RunHeartbeat` job is
still being executed by the working Sidekiq processes on other hosts.

In order to properly ensure that stuck Sidekiq processs are restarted,
we now rely on the [Sidekiq::ProcessSet](https://github.com/sidekiq/sidekiq/wiki/API#processes)
API that is supported by Sidekiq. The API provides us with "near real-time (updated every 5 sec)
info about the current set of Sidekiq processes running". The API
provides useful information like the hostname, pid and also when Sidekiq
last did its own heartbeat check. With that information, we can easily
determine if a Sidekiq process needs to be restarted from the Unicorn
master process.
2024-12-18 12:48:50 +08:00
..
about_stats.rb
activation_reminder_emails.rb
admin_problems.rb
aggregate_web_hooks_events.rb FEATURE: Add WebHookEventsDailyAggregate (#27542) 2024-06-25 13:56:47 -03:00
auto_queue_handler.rb DEV: replace .each with .find_each for paginated queries (#27159) 2024-05-23 15:42:21 -07:00
backfill_dominant_colors.rb
badge_grant.rb
bookmark_reminder_notifications.rb
call_discourse_hub.rb
category_stats.rb
check_new_features.rb REVERT: Check for features sooner (#29746) 2024-11-13 11:30:34 -08:00
check_out_of_date_themes.rb
check_translation_overrides.rb FIX: Ensure JsLocaleHelper to not output deprecated translations (#28037) 2024-07-29 15:21:25 +08:00
clean_dismissed_topic_users.rb
clean_up_api_keys_max_life.rb
clean_up_associated_accounts.rb
clean_up_associated_groups.rb
clean_up_crawler_stats.rb
clean_up_email_change_requests.rb
clean_up_email_logs.rb
clean_up_email_tokens.rb
clean_up_exports.rb
clean_up_inactive_users.rb
clean_up_post_reply_keys.rb
clean_up_search_logs.rb
clean_up_stylesheet_cache.rb
clean_up_tags.rb
clean_up_unmatched_emails.rb
clean_up_unmatched_ips.rb
clean_up_unsubscribe_keys.rb
clean_up_unused_api_keys.rb
clean_up_unused_registered_user_api_key_clients.rb Require permitted scopes when registering a client (#29718) 2024-11-19 15:28:04 -05:00
clean_up_unused_staged_users.rb
clean_up_unused_user_api_keys.rb
clean_up_uploads.rb
clean_up_user_api_keys_max_life.rb
cleanup_imap_sync_log.rb
cleanup_redelivering_web_hook_events.rb FEATURE: Add Mechanism to redeliver all failed webhook events (#27609) 2024-07-08 15:43:16 -05:00
create_missing_avatars.rb
create_recent_post_search_indexes.rb
destroy_old_deletion_stubs.rb
destroy_old_hidden_posts.rb
directory_refresh_daily.rb
directory_refresh_older.rb
disable_bootstrap_mode.rb
drop_backup_schema.rb
enqueue_digest_emails.rb FIX: send activity summaries based on "last seen" (#27035) 2024-05-22 10:23:03 +02:00
enqueue_onceoffs.rb
enqueue_suspect_users.rb
ensure_db_consistency.rb DEV: ensure rebaking works even when some users have inconsistent data (#30261) 2024-12-16 19:48:25 +08:00
ensure_s3_uploads_existence.rb FIX: Jobs::EnsureS3UploadsExistence broken for multisite (#27401) 2024-06-10 16:26:39 +08:00
fix_user_usernames_and_groups_names_clash.rb
grant_anniversary_badges.rb
grant_new_user_of_the_month_badges.rb
heartbeat.rb FIX: Redo Sidekiq monitoring to restart stuck sidekiq processes (#30198) 2024-12-18 12:48:50 +08:00
ignored_users_summary.rb
invalidate_inactive_admins.rb
migrate_upload_scheme.rb
old_keys_reminder.rb
pending_queued_posts_reminder.rb
pending_reviewables_reminder.rb
pending_users_reminder.rb
periodical_updates.rb DEV: ensure rebaking works even when some users have inconsistent data (#30261) 2024-12-16 19:48:25 +08:00
poll_mailbox.rb DEV: Database backed admin notices (#26192) 2024-05-23 09:29:08 +08:00
presence_channel_auto_leave.rb
process_badge_backlog.rb
process_shelved_notifications.rb
process_user_notification_schedules.rb
purge_deleted_uploads.rb
purge_expired_ignored_users.rb
purge_old_web_hook_events_daily_aggregate.rb FEATURE: Add WebHookEventsDailyAggregate (#27542) 2024-06-25 13:56:47 -03:00
purge_old_web_hook_events.rb
purge_unactivated.rb
redeliver_web_hook_events.rb FEATURE: Add Mechanism to redeliver all failed webhook events (#27609) 2024-07-08 15:43:16 -05:00
regenerate_sitemaps.rb
reindex_search.rb
reviewable_priorities.rb
run_problem_checks.rb FIX: Appropriately handle uninstalled problem checks (#28771) 2024-09-18 10:11:52 +08:00
schedule_backup.rb
tl3_promotions.rb DEV: Add tl3_custom_promotions plugin modifier to tl3_promotions.rb (#29834) 2024-11-22 15:28:43 -05:00
top_refresh_older.rb
top_refresh_today.rb
topic_timer_enqueuer.rb
unsilence_users.rb
update_animated_uploads.rb DEV: Remove logical OR assignment of constants (#29201) 2024-10-16 10:09:07 +08:00
update_heat_settings.rb
update_topic_hot_scores.rb
weekly.rb