discourse/lib
Martin Brennan faf5b4d3e9
PERF: Speed up secure media and ACL sync rake tasks (#16849)
Incorporates learnings from /t/64227:

* Changes the code to set access control posts in the rake
  task to be an efficient UPDATE SQL query.
  The original version was timing out with 312017 post uploads,
  the new query took ~3s to run.
* Changes the code to mark uploads as secure/not secure in
  the rake task to be an efficient UPDATE SQL query rather than
  using UploadSecurity. This took a very long time previously,
  and now takes only a few seconds.
* Spread out ACL syncing for uploads into jobs with batches of
  100 uploads at a time, so they can be parallelized instead
  of having to wait ~1.25 seconds for each ACL to be changed
  in S3 serially.

One issue that still remains is post rebaking. Doing this serially
is painfully slow. We have a way to do this in sidekiq via PeriodicalUpdates
but this is limited by max_old_rebakes_per_15_minutes. It would
be better to fan this rebaking out into jobs like we did for the
ACL sync, but that should be done in another PR.
2022-05-23 13:14:11 +10:00
..
action_dispatch/session DEV: Upgrade to Rails 7 2022-04-28 11:51:03 +02:00
auth FEATURE: Add fallback to suggested value when auth_overrides_username (#16612) 2022-05-04 11:22:09 +01:00
autospec DEV: Support for running theme test with Ember CLI (third attempt) 2022-01-13 16:02:07 -05:00
backup_restore DEV: Upgrade to Rails 7 2022-04-28 11:51:03 +02:00
common_passwords DEV: Upgrade to Rails 7 2022-04-28 11:51:03 +02:00
compression
content_security_policy DEV: Upgrade to Rails 7 2022-04-28 11:51:03 +02:00
demon DEV: Route Sidekiq logs to Rails logger (#15817) 2022-02-04 16:28:20 +00:00
discourse_dev DEV: Add ability to populate private messages (#16705) 2022-05-12 14:29:49 -04:00
email FIX: Use our header value instead of custom header on duplicates (#16711) 2022-05-11 13:47:12 +10:00
emoji FEATURE: Add missing emojis (#15582) 2022-01-14 17:51:13 -03:00
faker DEV: Fix methods removed in Ruby 3.2 (#15459) 2022-01-05 18:45:08 +01:00
file_store DEV: Upgrade to Rails 7 2022-04-28 11:51:03 +02:00
freedom_patches DEV: Upgrade to Rails 7.0.3 2022-05-10 12:01:00 +02:00
generators/rails DEV: removes plugin generator (#14101) 2021-08-20 11:29:06 +02:00
guardian FEATURE: Promote polymorphic bookmarks to default and migrate (#16729) 2022-05-23 10:07:15 +10:00
i18n DEV: Upgrade to Rails 7 2022-04-28 11:51:03 +02:00
imap FEATURE: Introduce site setting to allow for non staff pm tagging (#16671) 2022-05-10 10:02:28 -05:00
import
import_export FEATURE: include user custom fields in base exporter (#14690) 2021-10-22 10:02:56 -07:00
javascripts DEV: Disallow Ember global usage (#16147) 2022-03-09 17:54:07 +01:00
middleware DEV: Upgrade to Rails 7 2022-04-28 11:51:03 +02:00
migration DEV: Upgrade to Rails 7 2022-04-28 11:51:03 +02:00
onebox DEV: Don’t patch Sanitize::Config 2022-04-06 17:10:51 +02:00
plugin DEV: Add discourse-gamification to the official plugins (#16639) 2022-05-04 21:06:06 +02:00
pretty_text DEV: replaces huge generated emoji list by a simpler regex (#11053) 2021-04-22 08:43:06 +02:00
rate_limiter FEATURE: Apply rate limits per user instead of IP for trusted users (#14706) 2021-11-17 23:27:30 +03:00
reviewable DEV: Upgrade to Rails 7 2022-04-28 11:51:03 +02:00
scheduler
search DEV: Remove monkey patch that is no longer required (#16648) 2022-05-06 10:33:42 +08:00
second_factor FEATURE: Add 2FA support to the Discourse Connect Provider protocol (#16386) 2022-04-13 15:04:09 +03:00
seed_data FIX: Support Ruby 3 keyword arguments 2021-10-05 11:25:00 -04:00
sidekiq
site_settings FEATURE: Validate setting combination between exif strip and img opt (#16662) 2022-05-05 15:13:17 -03:00
stylesheet DEV: Fix stylesheet watcher (#16635) 2022-05-04 11:58:52 -04:00
tasks PERF: Speed up secure media and ACL sync rake tasks (#16849) 2022-05-23 13:14:11 +10:00
theme_store DEV: Upgrade to Rails 7 2022-04-28 11:51:03 +02:00
topic_query FIX: exclude topics from muted tag in category featured list. (#14925) 2021-11-16 12:10:50 +05:30
turbo_tests FIX: Make thumbnail tests start with a clean slate (#15216) 2021-12-07 13:07:45 -06:00
validators FEATURE: Allow category moderators to post consecutively (#16706) 2022-05-10 22:18:17 +03:00
webauthn
wizard FEATURE: Enable auto dark mode on new instances (#14208) 2021-09-02 14:55:38 -04:00
admin_confirmation.rb FEATURE: Centralized 2FA page (#15377) 2022-02-17 12:12:59 +03:00
admin_constraint.rb Revert "DEV: Add context in AdminConstraint (#15838)" (#15845) 2022-02-07 21:05:19 +03:00
admin_user_index_query.rb FIX: Add email to admin user list when show_emails is enabled (#16636) 2022-05-04 14:07:22 -03:00
age_words.rb
archetype.rb
auth.rb FEATURE: Experimental support for group membership via google auth (#14835) 2021-12-09 12:30:27 +00:00
backup_restore.rb FIX: Handle enum types during database restore (#16624) 2022-05-04 08:40:34 +10:00
badge_posts_view_manager.rb DEV: stop freezing frozen strings 2020-04-30 16:48:53 +10:00
badge_queries.rb FIX: Don't grant sharing badges to users who don't exist (#13851) 2021-07-27 16:32:59 +10:00
base62.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
bookmark_manager.rb FEATURE: Promote polymorphic bookmarks to default and migrate (#16729) 2022-05-23 10:07:15 +10:00
bookmark_query.rb FEATURE: Promote polymorphic bookmarks to default and migrate (#16729) 2022-05-23 10:07:15 +10:00
bookmark_reminder_notification_handler.rb FEATURE: Promote polymorphic bookmarks to default and migrate (#16729) 2022-05-23 10:07:15 +10:00
browser_detection.rb
cache.rb DEV: Fix rubocop issues (#14715) 2021-10-27 11:39:28 +03:00
canonical_url.rb FEATURE: Send a 'noindex' header in non-canonical responses (#15026) 2021-11-25 16:58:39 -03:00
category_badge.rb
chrome_installed_checker.rb DEV: Add chromium to ChromeInstalledChecker (#16224) 2022-03-19 11:00:06 +01:00
comment_migration.rb
common_passwords.rb DEV: Upgrade to Rails 7 2022-04-28 11:51:03 +02:00
composer_messages_finder.rb FEATURE: Make allow_uploaded_avatars accept TL (#14091) 2021-08-24 10:46:28 +03:00
configurable_urls.rb Replace base_uri with base_path (#10879) 2020-10-09 12:51:24 +01:00
content_buffer.rb
content_security_policy.rb PERF: Eager load Theme associations in Stylesheet Manager. 2021-06-21 11:06:58 +08:00
cooked_post_processor.rb FIX: handle quote rendering for external Discourse instance (#16722) 2022-05-12 10:07:43 -05:00
cooked_processor_mixin.rb FIX: Make sure max_oneboxes_per_post is enforced (#16215) 2022-03-23 17:36:08 +02:00
crawler_detection.rb FEATURE: Implement browser update in crawler view (#12448) 2021-03-22 19:41:42 +02:00
csrf_token_verifier.rb
current_user.rb
custom_renderer.rb
db_helper.rb DEV: Upgrade to Rails 7 2022-04-28 11:51:03 +02:00
directory_helper.rb
discourse_connect_base.rb FEATURE: Add 2FA support to the Discourse Connect Provider protocol (#16386) 2022-04-13 15:04:09 +03:00
discourse_connect_provider.rb FEATURE: Add 2FA support to the Discourse Connect Provider protocol (#16386) 2022-04-13 15:04:09 +03:00
discourse_dev.rb DEV: move discourse_dev gem to the core. (#13360) 2021-06-14 20:34:44 +05:30
discourse_diff.rb Escape values of HTML attributes 2021-08-10 10:25:15 -04:00
discourse_event.rb DEV: Remove site_setting_saved event (#15164) 2021-12-02 09:33:03 -06:00
discourse_hub.rb
discourse_ip_info.rb DEV: Typo. (#16092) 2022-03-03 09:24:58 +08:00
discourse_js_processor.rb DEV: Allow Ember CLI assets to be used by development Rails app (#16511) 2022-04-21 16:26:34 +01:00
discourse_logstash_logger.rb FIX: Use 'hostname' when Discourse.os_hostname is not available 2020-02-18 13:37:39 +02:00
discourse_plugin_registry.rb REFACTOR: Improve support for consolidating notifications. (#14904) 2021-11-30 13:36:14 -03:00
discourse_redis.rb DEV: Readonly Redis support for DiscourseRedis#multi/pipelined (#16744) 2022-05-13 16:18:13 +08:00
discourse_sourcemapping_url_processor.rb DEV: Fix production sourcemaps with Ember CLI (#16707) 2022-05-11 10:23:32 +01:00
discourse_tagging.rb FEATURE: Show prompt for required tag groups (#16458) 2022-04-21 13:13:52 +01:00
discourse_updates.rb FIX: Regression introduced in #14715 (#14842) 2021-11-09 17:20:09 +11:00
discourse.rb DEV: New readonly mode. Only applies to non-staff (#16243) 2022-05-17 13:06:08 -05:00
disk_space.rb FIX: correct upload statistics report for external storage 2020-02-20 15:15:53 +11:00
distributed_cache.rb FIX: Handle nil values in DistributedCache#defer_get_set (#15978) 2022-02-17 14:52:14 +00:00
distributed_memoizer.rb DEV: Make DistributedMemoizer use DistributedMutex (#16229) 2022-04-05 19:29:58 +02:00
distributed_mutex.rb FIX: add support for pipelined and multi redis commands (#16682) 2022-05-10 08:19:02 +10:00
edit_rate_limiter.rb FEATURE: Increase daily edit limits proportionally to trust level (#13090) 2021-05-19 13:57:21 +04:00
email_backup_token.rb
email_cook.rb PERF: Avoid lookbehinds when replacing links in imported emails (#11931) 2021-02-02 17:34:00 +01:00
email_updater.rb DEV: Update :critical_user_email calls to use strings (#15827) 2022-02-04 23:43:53 +00:00
email.rb DEV: pull email address validation out to a new EmailAddressValidator 2022-02-17 21:49:22 -05:00
ember_cli.rb DEV: Fix production sourcemaps with Ember CLI (#16707) 2022-05-11 10:23:32 +01:00
encodings.rb
enum_site_setting.rb
enum.rb
excerpt_parser.rb DEV: Remove dead code 2021-05-31 10:22:50 +08:00
external_upload_helpers.rb FIX: Use hidden site setting for batch presign rate limit (#16692) 2022-05-10 11:14:26 +10:00
feed_element_installer.rb
feed_item_accessor.rb FIX: Select best link from Atom feed (#15663) 2022-01-21 17:54:18 +02:00
file_helper.rb DEV: Expand UploadMarkdown generation capabilities (#15930) 2022-02-14 15:48:27 +10:00
filter_best_posts.rb
final_destination.rb FIX: Do not raise if title cannot be crawled (#16247) 2022-03-22 20:13:27 +02:00
flag_query.rb DEV: Remove deprecated methods (#14885) 2021-11-11 12:21:25 -06:00
flag_settings.rb DEV: Compatibility with TruffleRuby (#16864) 2022-05-19 21:56:55 +02:00
gaps.rb
global_path.rb FEATURE: Centralized 2FA page (#15377) 2022-02-17 12:12:59 +03:00
group_email_credentials_check.rb FEATURE: Scheduled group email credential problem check (#15396) 2022-01-04 10:14:33 +10:00
guardian.rb FIX: staff should not be able to PM groups that "Nobody" can message (#16163) 2022-03-22 10:23:14 +10:00
has_errors.rb
highlight_js.rb DEV: Upgrade to Rails 7 2022-04-28 11:51:03 +02:00
hijack.rb DEV: Add more debugging context to onebox generation 2020-10-22 12:50:22 +08:00
homepage_constraint.rb FEATURE: Apply rate limits per user instead of IP for trusted users (#14706) 2021-11-17 23:27:30 +03:00
html_prettify.rb DEV: stop freezing frozen strings 2020-04-30 16:48:53 +10:00
html_to_markdown.rb DEV: Upgrade to Rails 7 2022-04-28 11:51:03 +02:00
http_language_parser.rb FIX: Include resolved locale in anonymous cache key (#10289) 2020-07-22 18:00:07 +01:00
image_sizer.rb
import_export.rb FEATURE: Rake task to export groups (#9450) 2020-04-17 14:59:54 -07:00
inline_oneboxer.rb Only block domains at the final destination (#15689) 2022-01-31 15:35:12 +08:00
introduction_updater.rb FIX: replace default welcome topic post with new value from wizard 2020-04-01 15:42:45 -04:00
js_locale_helper.rb FIX: Overridden MessageFormat fallbacks (#15855) 2022-02-08 12:31:08 +11:00
json_error.rb
letter_avatar.rb DEV: Fix methods removed in Ruby 3.2 (#15459) 2022-01-05 18:45:08 +01:00
markdown_linker.rb
mem_info.rb
message_bus_diags.rb PERF: avoid shelling to get hostname aggressively 2020-02-18 15:13:19 +11:00
method_profiler.rb DEV: Add output_sql_to_stderr! to MethodProfiler (#12445) 2021-03-19 17:48:30 +10:00
mini_sql_multisite_connection.rb DEV: Use MiniSql ActiveRecordPostgres adapter (#15767) 2022-02-03 10:00:28 +00:00
mobile_detection.rb FIX: include crawler content on old mobile browsers (#16387) 2022-04-06 11:09:12 +01:00
new_post_manager.rb SECURITY: Escape watched word in error message (#14434) 2021-09-24 11:55:15 +03:00
new_post_result.rb DEV: Let's always give a drop_from param to deprecate (#14901) 2021-11-12 08:52:59 -06:00
notification_levels.rb
onebox.rb DEV: Don’t patch Sanitize::Config 2022-04-06 17:10:51 +02:00
oneboxer.rb DEV: Don’t patch Sanitize::Config 2022-04-06 17:10:51 +02:00
onpdiff.rb
pbkdf2.rb Use Xorcist.xor! instead of refinements since Ruby 3.2+ removes Refinment-include (#15694) 2022-02-03 16:19:30 +11:00
permalink_constraint.rb
pinned_check.rb
plain_text_to_markdown.rb DEV: stop freezing frozen strings 2020-04-30 16:48:53 +10:00
plugin_gem.rb DEV: Don't load bundler when installing plugin gem. (#16117) 2022-03-07 13:20:43 +08:00
plugin.rb DEV: Upgrade to Rails 7 2022-04-28 11:51:03 +02:00
post_action_creator.rb DEV: Remove PostAction/UserAction bookmark refs (#16681) 2022-05-10 10:42:18 +10:00
post_action_destroyer.rb FIX: Background like count update didn't account for own user actions (#16688) 2022-05-09 17:23:39 -03:00
post_action_result.rb DEV: Upgrade to Rails 7 2022-04-28 11:51:03 +02:00
post_creator.rb FIX: Update user stat counts when post/topic visibility changes. (#15883) 2022-02-11 09:00:58 +08:00
post_destroyer.rb FIX: avoid validations when destroying posts (#16049) 2022-02-25 11:20:54 +11:00
post_jobs_enqueuer.rb FIX: Do not send emails to mailing_list_mode subscribers for PMs (#14159) 2021-08-26 15:16:35 +10:00
post_locker.rb
post_merger.rb FEATURE: TL4 & category moderators can merge posts (#12843) 2021-04-27 18:24:27 +02:00
post_revisor.rb FIX: Don't advance draft sequence when editing topic title (#16002) 2022-02-23 10:39:54 +03:00
presence_channel.rb DEV: Fix random typos (#16066) 2022-02-28 10:20:58 +08:00
pretty_text.rb DEV: Drop lodash (#16110) 2022-03-06 18:15:25 +01:00
promotion.rb FIX: check if BasicBadge is enabled for TL1 welcome message (#13983) 2021-08-11 08:39:25 +10:00
quote_comparer.rb FIX: handle quote rendering for external Discourse instance (#16722) 2022-05-12 10:07:43 -05:00
rake_helpers.rb Try fix upload_spec flakys and remove logging from tasks/uploads_spec 2020-02-18 15:08:58 +10:00
rate_limiter.rb DEV: Consolidate Redis evalsha logic into DiscourseRedis::EvalHelper (#15957) 2022-02-15 16:06:12 +00:00
read_only_mixin.rb DEV: New readonly mode. Only applies to non-staff (#16243) 2022-05-17 13:06:08 -05:00
redis_snapshot.rb FIX: add support for pipelined and multi redis commands (#16682) 2022-05-10 08:19:02 +10:00
require_dependency_backward_compatibility.rb DEV: Upgrade to Rails 7 2022-04-28 11:51:03 +02:00
retrieve_title.rb FIX: Do not raise if title cannot be crawled (#16247) 2022-03-22 20:13:27 +02:00
route_format.rb
route_matcher.rb REFACTOR: Introduce RouteMatcher class 2020-10-19 10:40:55 +01:00
rtl.rb
s3_cors_rulesets.rb DEV: Upgrade to Rails 7 2022-04-28 11:51:03 +02:00
s3_helper.rb FEATURE: Direct S3 multipart uploads for backups (#14736) 2021-11-11 08:25:31 +10:00
s3_inventory.rb DEV: Fix methods removed in Ruby 3.2 (#15459) 2022-01-05 18:45:08 +01:00
score_calculator.rb
screening_model.rb
search.rb FEATURE: Promote polymorphic bookmarks to default and migrate (#16729) 2022-05-23 10:07:15 +10:00
secure_session.rb
shrink_uploaded_image.rb DEV: Improve script/downsize_uploads.rb (#13508) 2021-06-24 00:09:40 +02:00
sidekiq_logster_reporter.rb DEV: Move SidekiqLogsterReporter out of initializer (#16687) 2022-05-09 13:01:31 -05:00
site_icon_manager.rb PERF: Defer setting of distributed cache in more spots. 2021-06-04 09:13:18 +08:00
site_setting_extension.rb FIX: Resetting selectable avatars was failing (#16302) 2022-03-28 14:15:28 -04:00
slug.rb FIX: Make category slugs lowercase (#11277) 2021-01-12 17:28:33 +02:00
socket_server.rb
spam_handler.rb FIX: use allowlist and blocklist terminology (#10209) 2020-07-27 10:23:54 +10:00
staff_constraint.rb FEATURE: Apply rate limits per user instead of IP for trusted users (#14706) 2021-11-17 23:27:30 +03:00
staff_message_format.rb
suggested_topics_builder.rb
svg_sprite.rb FIX: uses tippy for popover (#15409) 2022-05-02 17:10:26 +02:00
system_message.rb REFACTOR: Add full_url and display_name to User 2022-04-14 11:53:57 +02:00
temporary_db.rb DEV: Fix methods removed in Ruby 3.2 (#15459) 2022-01-05 18:45:08 +01:00
temporary_redis.rb DEV: Introduce TemporaryRedis and unset DISCOURSE_* env vars in the themes:isolated_test rake task (#13401) 2021-06-23 07:38:43 +03:00
text_cleaner.rb FEATURE: Correctly convert topic title to uppercase and lowercase for Turkish default locale (#13115) 2021-05-24 18:13:30 +10:00
text_sentinel.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
theme_javascript_compiler.rb FEATURE: Allow theme tests to be run in production (take 2) (#12845) 2021-04-28 23:12:08 +03:00
theme_modifier_helper.rb Code review comments. 2021-06-21 11:06:58 +08:00
theme_settings_manager.rb FEATURE: Allow theme settings to request refresh (#15037) 2021-11-22 13:16:56 +01:00
theme_settings_parser.rb FEATURE: Allow theme settings to request refresh (#15037) 2021-11-22 13:16:56 +01:00
theme_translation_manager.rb
theme_translation_parser.rb
timeline_lookup.rb FIX: ensures timeline_lookup includes last tuple (#11829) 2021-01-25 11:30:59 +01:00
tiny_japanese_segmenter.rb FEATURE: Split up text segmentation for Chinese and Japanese. 2022-02-07 09:21:14 +08:00
topic_creator.rb FIX: Validate category tag restrictions before sending new topics to review (#16292) 2022-03-28 21:25:26 +03:00
topic_list_responder.rb DEV: Refactor draft attributes for CategoryList and TopicList. 2020-07-24 10:11:30 +08:00
topic_publisher.rb FIX: Use destroy_all instead of delete_all for shared drafts 2020-03-05 11:13:43 -08:00
topic_query_params.rb FIX: Build correct topic list filter (#11473) 2020-12-11 14:20:48 +02:00
topic_query.rb DEV: Remove PostAction/UserAction bookmark refs (#16681) 2022-05-10 10:42:18 +10:00
topic_retriever.rb FEATURE: Fallback to system users when creating new TopicEmbed (#12386) 2021-03-15 11:58:53 -03:00
topic_subtype.rb
topic_upload_security_manager.rb DEV: Add security_last_changed_at and security_last_changed_reason to uploads (#11860) 2021-01-29 09:03:44 +10:00
topic_view.rb FEATURE: Promote polymorphic bookmarks to default and migrate (#16729) 2022-05-23 10:07:15 +10:00
topics_bulk_action.rb FEATURE Add reset bump dates bulk action (#16885) 2022-05-22 12:32:55 -04:00
trust_level.rb FIX: Don't store translated trust level names in anonymous cache (#13224) 2021-06-01 22:11:48 +02:00
turbo_tests.rb FIX: Make thumbnail tests start with a clean slate (#15216) 2021-12-07 13:07:45 -06:00
twitter_api.rb FIX: Replace Twitter handles one at a time (#15870) 2022-02-09 13:54:02 +02:00
unicorn_logstash_patch.rb DEV: Fix lint. 2020-07-21 15:55:03 +08:00
unread.rb FEATURE: Add last visit indication to topic view page. (#13471) 2021-07-05 14:17:31 +08:00
upload_creator.rb FIX: Blurry onebox favicon images (#15258) 2021-12-10 12:25:50 -07:00
upload_fixer.rb
upload_markdown.rb DEV: Expand UploadMarkdown generation capabilities (#15930) 2022-02-14 15:48:27 +10:00
upload_recovery.rb FIX: Support Ruby 3 keyword arguments 2021-10-05 11:25:00 -04:00
upload_security.rb PERF: Speed up secure media and ACL sync rake tasks (#16849) 2022-05-23 13:14:11 +10:00
url_helper.rb FEATURE: revert disallowing putting URLs in titles for TL0 users (#13970) 2021-08-06 20:07:42 +04:00
user_lookup.rb REVERT "FIX: do not show private group flair on user avatars" (#13991) 2021-08-10 17:25:11 +05:30
user_name_suggester.rb FEATURE: when suggesting usernames skip input that consist entirely of disallowed characters (#15368) 2021-12-21 21:13:05 +04:00
vary_header.rb FIX: Include the Vary:Accept header on all Accept-based responses (#14647) 2021-10-25 12:53:50 +01:00
version.rb Version bump to v2.9.0.beta4 (#16477) 2022-04-14 10:10:23 -04:00
webauthn.rb FEATURE: RS512, RS384 and RS256 COSE algorithms (#15804) 2022-02-08 14:07:47 +02:00
wizard.rb DEV: Allow plugins to add wizard steps after specific steps (#9315) 2020-04-01 08:36:50 -05:00