discourse/lib/demon/sidekiq.rb
Alan Guo Xiang Tan c1f25cdf5b
FIX: Unicorn master and Sidekiq reopening logs at the same time ()
In our production environment, we have been seeing Sidekiq processes
getting stuck randomly when a USR1 signal is sent to the Unicorn master
process. We have not been able to identify the root cause of why the
Sidekiq process gets stuck. We however noticed that when the Unicorn
master process receives a USR1 signal, it will reopen the logs for the
Unicorn master process first before sending a USR1 signal for the
Unicorn worker processes to reopen the logs. We figured that we should
do the same for the Sidekiq process as well when a USR1 signal.

In this commit, we introduce an arbitrary delay of 1 second before we
the Sidekiq process reopens its log files so as to allow enough time for the Unicorn
master to finish reopening it logs first.

We also do not send reopen logs for the Sidekiq process if the `DISCOURSE_LOG_SIDEKIQ`
env is not present because there is no need to reopen any logs.
2024-10-10 08:01:40 +08:00

88 lines
2.0 KiB
Ruby

# frozen_string_literal: true
require "demon/base"
class Demon::Sidekiq < ::Demon::Base
def self.prefix
"sidekiq"
end
def self.after_fork(&blk)
blk ? (@blk = blk) : @blk
end
private
def suppress_stdout
false
end
def suppress_stderr
false
end
def log_in_trap(message, level: :info)
SignalTrapLogger.instance.log(@logger, message, level: level)
end
def after_fork
Demon::Sidekiq.after_fork&.call
SignalTrapLogger.instance.after_fork
log("Loading Sidekiq in process id #{Process.pid}")
require "sidekiq/cli"
cli = Sidekiq::CLI.instance
# Unicorn uses USR1 to indicate that log files have been rotated
Signal.trap("USR1") { reopen_logs }
Signal.trap("USR2") do
sleep 1
reopen_logs
end
options = ["-c", GlobalSetting.sidekiq_workers.to_s]
[["critical", 8], ["default", 4], ["low", 2], ["ultra_low", 1]].each do |queue_name, weight|
custom_queue_hostname = ENV["UNICORN_SIDEKIQ_#{queue_name.upcase}_QUEUE_HOSTNAME"]
if !custom_queue_hostname || custom_queue_hostname.split(",").include?(Discourse.os_hostname)
options << "-q"
options << "#{queue_name},#{weight}"
end
end
# Sidekiq not as high priority as web, in this environment it is forked so a web is very
# likely running
Discourse::Utils.execute_command("renice", "-n", "5", "-p", Process.pid.to_s)
cli.parse(options)
load Rails.root + "config/initializers/100-sidekiq.rb"
cli.run
rescue => error
log(
"Error encountered while starting Sidekiq: [#{error.class}] #{error.message}\n#{error.backtrace.join("\n")}",
level: :error,
)
exit 1
end
private
def reopen_logs
begin
log_in_trap("Sidekiq reopening logs...")
Unicorn::Util.reopen_logs
log_in_trap("Sidekiq done reopening logs...")
rescue => error
log_in_trap(
"Error encountered while reopening logs: [#{error.class}] #{error.message}\n#{error.backtrace.join("\n")}",
level: :error,
)
exit 1
end
end
end