discourse/spec
Martin Brennan faf5b4d3e9
PERF: Speed up secure media and ACL sync rake tasks (#16849)
Incorporates learnings from /t/64227:

* Changes the code to set access control posts in the rake
  task to be an efficient UPDATE SQL query.
  The original version was timing out with 312017 post uploads,
  the new query took ~3s to run.
* Changes the code to mark uploads as secure/not secure in
  the rake task to be an efficient UPDATE SQL query rather than
  using UploadSecurity. This took a very long time previously,
  and now takes only a few seconds.
* Spread out ACL syncing for uploads into jobs with batches of
  100 uploads at a time, so they can be parallelized instead
  of having to wait ~1.25 seconds for each ACL to be changed
  in S3 serially.

One issue that still remains is post rebaking. Doing this serially
is painfully slow. We have a way to do this in sidekiq via PeriodicalUpdates
but this is limited by max_old_rebakes_per_15_minutes. It would
be better to fan this rebaking out into jobs like we did for the
ACL sync, but that should be done in another PR.
2022-05-23 13:14:11 +10:00
..
fabricators FEATURE: Promote polymorphic bookmarks to default and migrate (#16729) 2022-05-23 10:07:15 +10:00
fixtures FIX: Missing translation when translation override contained a %{key} (#16625) 2022-05-04 17:35:22 +02:00
helpers DEV: Allow Ember CLI assets to be used by development Rails app (#16511) 2022-04-21 16:26:34 +01:00
import_export DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
initializers DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
integration FIX: Use our header value instead of custom header on duplicates (#16711) 2022-05-11 13:47:12 +10:00
integrity DEV: Upgrade to Rails 7 2022-04-28 11:51:03 +02:00
jobs PERF: Speed up secure media and ACL sync rake tasks (#16849) 2022-05-23 13:14:11 +10:00
lib FEATURE: Promote polymorphic bookmarks to default and migrate (#16729) 2022-05-23 10:07:15 +10:00
mailers FEATURE: Site setting to cap the recipient list in notification emails 2022-04-21 10:43:13 +02:00
models FEATURE: Promote polymorphic bookmarks to default and migrate (#16729) 2022-05-23 10:07:15 +10:00
multisite DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
requests FEATURE: Promote polymorphic bookmarks to default and migrate (#16729) 2022-05-23 10:07:15 +10:00
script/import_scripts FEATURE: Promote polymorphic bookmarks to default and migrate (#16729) 2022-05-23 10:07:15 +10:00
serializers FEATURE: Promote polymorphic bookmarks to default and migrate (#16729) 2022-05-23 10:07:15 +10:00
services FEATURE: Promote polymorphic bookmarks to default and migrate (#16729) 2022-05-23 10:07:15 +10:00
support FIX: Allow .ics for polymorphic bookmarks (#16694) 2022-05-11 09:29:24 +10:00
tasks PERF: Speed up secure media and ACL sync rake tasks (#16849) 2022-05-23 13:14:11 +10:00
views DEV: Upgrade to Rails 7 2022-04-28 11:51:03 +02:00
rails_helper.rb DEV: Don't cache watched words in test env (#16731) 2022-05-12 14:45:05 +08:00
regenerate_swagger_docs DEV: Add API docs for uploads and API doc watcher (#15387) 2021-12-23 08:40:15 +10:00
swagger_helper.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00