mirror of
https://github.com/discourse/discourse.git
synced 2024-11-26 12:53:42 +08:00
cfd507822f
This commit fixes the follow quality issue with `PostSearchData#raw_data`: 1. URLs are being tokenized and links with similar href and characters are being duplicated in the raw data. `Post#cooked`: ``` <p><a href=\"https://meta.discourse.org/some.png\" class=\"onebox\" target=\"_blank\" rel=\"nofollow noopener\">https://meta.discourse.org/some.png</a></p> ``` `PostSearchData#raw_data` Before: ``` This is a test topic 0 Uncategorized https://meta.discourse.org/some.png discourse org/some png https://meta.discourse.org/some.png discourse org/some png ``` `PostSearchData#raw_data` After: ``` This is a test topic 0 Uncategorized https://meta.discourse.org/some.png meta discourse org ``` 2. Ligthbox being included in search pollutes the `PostSearchData#raw_data` unncessarily. From 28 March 2018 to 28 March 2019, searches for the term `image` on `meta.discourse.org` had a click through rate of 2.1%. Non-lightboxed images are not included in indexing for search yet we were indexing content within a lightbox. Also, search for terms like `image` was affected we were using `Pasted image` as the filename for uploads that were pasted. `Post#cooked` ``` <p>Let me see how I can fix this image<br>\n<div class=\"lightbox-wrapper\"><a class=\"lightbox\" href=\"https://meta.discourse.org/some.png\" title=\"some.png\" rel=\"nofollow noopener\"><img src=\"https://meta.discourse.org/some.png\" width=\"275\" height=\"299\"><div class=\"meta\">\n<svg class=\"fa d-icon d-icon-far-image svg-icon\" aria-hidden=\"true\"><use xlink:href=\"#far-image\"></use></svg><span class=\"filename\">some.png</span><span class=\"informations\">1750×2000</span><svg class=\"fa d-icon d-icon-discourse-expand svg-icon\" aria-hidden=\"true\"><use xlink:href=\"#discourse-expand\"></use></svg>\n</div></a></div></p> ``` `PostSearchData#raw_data` Before: ``` This is a test topic 0 Uncategorized Let me see how I can fix this image some.png png https://meta.discourse.org/some.png discourse org/some png some.png png 1750×2000 ``` `PostSearchData#raw_data` After: ``` This is a test topic 0 Uncategorized Let me see how I can fix this image ``` In terms of indexing performance, we now have to parse the given HTML through nokogiri twice. However performance is not a huge worry here since a string length of 194170 takes only 30ms to scrub plus the indexing takes place in a background job. |
||
|---|---|---|
| .. | ||
| anonymous_shadow_creator_spec.rb | ||
| auto_silence_spec.rb | ||
| badge_granter_spec.rb | ||
| color_scheme_revisor_spec.rb | ||
| destroy_task_spec.rb | ||
| flag_sockpuppets_spec.rb | ||
| group_action_logger_spec.rb | ||
| group_mentions_updater_spec.rb | ||
| group_message_spec.rb | ||
| i18n_interpolation_keys_finder_spec.rb | ||
| notification_emailer_spec.rb | ||
| post_action_notifier_spec.rb | ||
| post_alerter_spec.rb | ||
| post_owner_changer_spec.rb | ||
| push_notification_pusher_spec.rb | ||
| random_topic_selector_spec.rb | ||
| search_indexer_spec.rb | ||
| site_settings_spec.rb | ||
| staff_action_logger_spec.rb | ||
| topic_status_updater_spec.rb | ||
| topic_timestamp_changer_spec.rb | ||
| trust_level_granter_spec.rb | ||
| user_activator_spec.rb | ||
| user_anonymizer_spec.rb | ||
| user_authenticator_spec.rb | ||
| user_destroyer_spec.rb | ||
| user_merger_spec.rb | ||
| user_silencer_spec.rb | ||
| user_updater_spec.rb | ||
| username_changer_spec.rb | ||
| username_checker_service_spec.rb | ||
| wildcard_domain_checker_spec.rb | ||
| word_watcher_spec.rb | ||
