Commit Graph

28 Commits

Author SHA1 Message Date
Sam
384135845b FEATURE: introduce ultra_low priority queue
This commit introduces an ultra low priority queue for post rebakes. This
way rebakes can never interfere with regular sidekiq processing for cases
where we perform a large scale rebake.

Additionally it allows Post.rebake_old to be run with rate_limiter: false
to avoid triggering the limiter when rebaking. This is handy for cases
where you want to just force the full rebake and not wait for it to trickle
2019-01-17 14:53:19 +11:00
Sam
df460b4abd PERF: run sidekiq with nice 5
This ensures that unicorn master forks of sidekiq run with a lower priority
than the webs. It means that a busy sidekiq is less likely to impact web
performance
2019-01-09 09:29:14 +11:00
Sam
cc3fc87dd7 DEV: handle termination cleanly in autospec 2018-06-19 16:13:36 +10:00
Sam
fc05164667 demo script for demonizing using fork exec
minor refinements to demon
2018-01-11 13:51:52 +11:00
Guo Xiang Tan
470b1a5bc1 Don't print Sidekiq starting message to STDERR. 2017-11-03 21:02:31 +08:00
Guo Xiang Tan
5012d46cbd Add rubocop to our build. (#5004) 2017-07-28 10:20:09 +09:00
Guo Xiang Tan
7ea288140d Allow multiple host when restricting Sidekiq queues. 2017-06-19 14:45:51 +09:00
Guo Xiang Tan
84490c4558 Allow a sidekiq queue to be configured to only run on a certain hostname. 2017-04-27 15:32:16 +08:00
Sam
0b3aec9c94 FEATURE: set UNICORN_STATS_SOCKET_DIR for status socket
eg:

sam@ubuntu stats_sockets % socat - UNIX-CONNECT:9622.sock
gc_stat
{"count":46,"heap_allocated_pages":2459,"heap_sorted_length":2460,"heap_allocatable_pages":0,"heap_available_slots":1002267,"heap_live_slots":647293,"heap_free_slots":354974,"heap_final_slots":0,"heap_marked_slots":503494,"heap_swept_slots":498773,"heap_eden_pages":2459,"heap_tomb_pages":0,"total_allocated_pages":2459,"total_freed_pages":0,"total_allocated_objects":4337014,"total_freed_objects":3689721,"malloc_increase_bytes":6448248,"malloc_increase_bytes_limit":29188387,"minor_gc_count":36,"major_gc_count":10,"remembered_wb_unprotected_objects":19958,"remembered_wb_unprotected_objects_limit":39842,"old_objects":462019,"old_objects_limit":895782,"oldmalloc_increase_bytes":6448696,"oldmalloc_increase_bytes_limit":19350882}
2017-04-21 11:37:03 -04:00
Sam
8ec7fd84fd FEATURE: prioritize sidekiq jobs
This commit introduces 3 queues for sidekiq

"critical" for urgent jobs (weighted at 4x weight)
"default" for standard jobs(weighted at 2x weight)
"low" for less important jobs


"critical jobs"

Reset Password emails has been seperated to its own job
Heartbeat which is required to keep sidekiq running
Test email which needs to return real quick


"low priority jobs"

Notify mailing list
Pull hotlinked images
Update gravatar

"default"

All the rest

Note: for people running sidekiq from command line use

bin/sidekiq -q critical,4 -q default,2 -q low
2016-04-07 12:56:43 +10:00
Sam
65edbb609c Revert "Revert message bus upgrade"
This reverts commit 47e718f5b2.
2015-12-09 11:48:41 +11:00
Sam
47e718f5b2 Revert message bus upgrade 2015-12-09 11:45:11 +11:00
Sam
2cc95af69b Revert "REVERT: message bus changes"
This reverts commit 4820d5c7b0.
2015-12-09 07:36:36 +11:00
Robin Ward
4820d5c7b0 REVERT: message bus changes 2015-12-08 15:32:31 -05:00
Sam
a3ba564b03 missing spot where initializer was renamed 2015-12-08 07:13:29 +11:00
Sam
0e4883a8ae correct regression where monitoring thread crashed out
add logging
2015-06-16 11:16:33 +10:00
Sam
861cd5d9b0 FIX: ensure child demon is correctly terminated from parent on stop 2015-06-15 12:36:47 +10:00
Sam
1721872084 cleanup out-of-memory detection and correction code 2015-03-27 15:44:52 +11:00
Sam
58f3fcbc1a BUGFIX: not terminating self correctly on hangups from parent 2014-06-13 11:15:40 +10:00
Sam
95159fb82a BUGFIX: Sidekiq could be initialized incorrectly in some cases
Symptom, no jobs run
2014-06-03 17:17:10 +10:00
Sam
dc06401479 PERF: reduce sidekiq worker count to 5 2014-05-14 10:21:11 +10:00
Sam
7c57d74e85 FEATURE: unicorn sidekiq will restart sidekiq on complete failure.
(checks every 30 minutes for complete failure)
2014-04-23 13:13:18 +10:00
Sam
ead7c52a06 Refactor demonizer in prep for unicorn forking
Upgrade sidekiq
2014-04-17 15:58:00 +10:00
Sam
f3cc7360e0 BUGFIX: Correct after_fork semantics
After fork SiteSettings was not getting a new process id,
causing site settings not to refresh properly in unicorn

This code also centralizes the logic
2014-03-31 12:34:13 +11:00
Régis Hanol
b56b11d96a add qunit to autospec 2013-11-01 23:57:50 +01:00
Sam
89f801ac04 fix no sidetiq when using demon 2013-10-24 15:58:28 +11:00
Sam
28a0cb494a rails 4 upgrade
rack lock is trouble, nuke it out of orbit
more aggressive suicide for forked sidekiq
2013-10-10 14:23:24 +11:00
Sam
c4bab8915c fix initialization issues with unicorn
amend unicorn script to demonize sidekiq
create a sidekiq demon that unicorn consumes
correct bug in exec_sql with empty params
2013-10-10 14:23:24 +11:00