github-mirror/discourse
2
0
Fork 0
mirror of https://github.com/discourse/discourse.git synced 2024-11-23 12:35:25 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
faf5b4d3e9
discourse/spec/jobs
History
Martin Brennan faf5b4d3e9
PERF: Speed up secure media and ACL sync rake tasks (#16849) 
Incorporates learnings from /t/64227:

* Changes the code to set access control posts in the rake
  task to be an efficient UPDATE SQL query.
  The original version was timing out with 312017 post uploads,
  the new query took ~3s to run.
* Changes the code to mark uploads as secure/not secure in
  the rake task to be an efficient UPDATE SQL query rather than
  using UploadSecurity. This took a very long time previously,
  and now takes only a few seconds.
* Spread out ACL syncing for uploads into jobs with batches of
  100 uploads at a time, so they can be parallelized instead
  of having to wait ~1.25 seconds for each ACL to be changed
  in S3 serially.

One issue that still remains is post rebaking. Doing this serially
is painfully slow. We have a way to do this in sidekiq via PeriodicalUpdates
but this is limited by max_old_rebakes_per_15_minutes. It would
be better to fan this rebaking out into jobs like we did for the
ACL sync, but that should be done in another PR.
2022-05-23 13:14:11 +10:00
..
regular FIX: Skip upload extension validation when changing security (#16498) 2022-04-20 14:11:39 +10:00
about_stats_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
activation_reminder_emails_spec.rb DEV: Make user field validation more specific (#16746) 2022-05-16 14:21:33 +01:00
auto_expire_user_api_keys_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
auto_queue_handler_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
automatic_group_membership_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
bookmark_reminder_notifications_spec.rb FEATURE: Promote polymorphic bookmarks to default and migrate (#16729) 2022-05-23 10:07:15 +10:00
bulk_grant_trust_level_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
bulk_invite_spec.rb DEV: Make user field validation more specific (#16746) 2022-05-16 14:21:33 +01:00
bump_topic_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
clean_dismissed_topic_users_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
clean_up_associated_accounts_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
clean_up_crawler_stats_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
clean_up_email_change_requests_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
clean_up_email_logs_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
clean_up_inactive_users_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
clean_up_post_reply_keys_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
clean_up_unused_staged_users_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
clean_up_uploads_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
clean_up_user_export_topics_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
cleanup_imap_sync_log_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
close_topic_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
correct_missing_dualstack_urls_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
crawl_topic_link_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
create_linked_topic_spec.rb DEV: Upgrade to Rails 7 2022-04-28 11:51:03 +02:00
create_missing_avatars_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
create_recent_post_search_indexes_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
create_user_reviewable_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
dashboard_stats_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
delete_replies_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
delete_topic_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
disable_bootstrap_mode_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
download_avatar_from_url_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
download_backup_email_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
download_profile_background_from_url_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
emit_web_hook_event_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
enable_bootstrap_mode_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
enqueue_digest_emails_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
enqueue_suspect_users_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
ensure_s3_uploads_existence_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
export_csv_file_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
export_user_archive_spec.rb FEATURE: Promote polymorphic bookmarks to default and migrate (#16729) 2022-05-23 10:07:15 +10:00
feature_topic_users_spec.rb DEV: Upgrade to Rails 7 2022-04-28 11:51:03 +02:00
fix_out_of_sync_user_uploaded_avatar_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
fix_primary_emails_for_staged_users_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
fix_s3_etags_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
fix_user_usernames_and_groups_names_clash_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
grant_anniversary_badges_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
grant_new_user_of_the_month_badges_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
heartbeat_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
ignored_users_summary_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
invalidate_inactive_admins_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
invite_email_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
jobs_base_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
jobs_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
mass_award_badge_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
migrate_badge_image_to_uploads_spec.rb DEV: Use FakeLogger in RequestTracker specs (#16640) 2022-05-05 09:53:54 +08:00
migrate_group_flair_images_spec.rb DEV: Use FakeLogger in RequestTracker specs (#16640) 2022-05-05 09:53:54 +08:00
notify_mailing_list_subscribers_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
notify_moved_posts_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
notify_reviewable_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
notify_tag_change_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
old_keys_reminder_spec.rb DEV: Upgrade to Rails 7 2022-04-28 11:51:03 +02:00
open_topic_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
pending_queued_posts_reminder_spec.rb DEV: Convert notify_about_queued_posts_after to accept a float (#16637) 2022-05-04 11:33:43 -05:00
pending_reviewables_reminder_spec.rb DEV: Convert notify_about_flags_after to float (#16633) 2022-05-04 11:19:43 -05:00
pending_users_reminder_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
periodical_updates_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
poll_mailbox_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
post_update_topic_tracking_state_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
post_uploads_recovery_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
problem_checks_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
process_bulk_invite_emails_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
process_email_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
process_post_spec.rb DEV: Upgrade to Rails 7 2022-04-28 11:51:03 +02:00
process_shelved_notifications_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
publish_topic_to_category_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
pull_hotlinked_images_spec.rb DEV: Upgrade to Rails 7 2022-04-28 11:51:03 +02:00
pull_user_profile_hotlinked_images_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
purge_expired_ignored_users_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
rebake_custom_emoji_posts_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
reindex_search_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
remove_banner_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
reviewable_priorities_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
send_system_message_spec.rb DEV: Upgrade to Rails 7 2022-04-28 11:51:03 +02:00
suspicious_login_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
sync_acls_for_uploads_spec.rb PERF: Speed up secure media and ACL sync rake tasks (#16849) 2022-05-23 13:14:11 +10:00
sync_topic_user_bookmarked_spec.rb FEATURE: Promote polymorphic bookmarks to default and migrate (#16729) 2022-05-23 10:07:15 +10:00
tl3_promotions_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
toggle_topic_closed_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
topic_timer_enqueuer_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
truncate_user_flag_stats_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
unsilence_users_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
update_animated_uploads_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
update_gravatar_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
update_s3_inventory_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
update_username_spec.rb DEV: Automatically require 'rails_helper' in all specs (#16077) 2022-03-01 17:50:50 +00:00
user_email_spec.rb DEV: Accept force_respect_seen_recently argument in UserEmail job (#16460) 2022-04-18 13:32:11 -05:00