discourse/spec
David Taylor 3329484e2d
FEATURE: Simplify crawler content for non-canonical post URLs (#26324)
When crawlers visit a post-specific URL like `/t/-/{topic-id}/{post-number}`, we use the canonical to direct them to the appropriate crawler-optimised paginated view (e.g. `?page=3`).

However, analysis of google results shows that the post-specific URLs are still being included in the index. Google doesn't tell us exactly why this is happening. However, as a general rule, 'A large portion of the duplicate page's content should be present on the canonical version'.

In our previous implementation, this wasn't 100% true all the time. That's because a request for a post-specific URL would include posts 'surrounding' that post, and won't exactly conform to the page boundaries which are used in the canonical version of the page. Essentially: in some cases, the content of the post-specific pages would include many posts which were not present on the canonical paginated version.

This commit aims to resolve that problem by simplifying the implementation. Instead of rendering posts surrounding the target post_number, we will only render the target post, and include a link to 'show post in topic'. With this new implementation, 100% of the post-specific page content will be present on the canonical paginated version, which will hopefully mean google reduces their  indexing of the non-canonical post-specific pages.
2024-03-26 15:18:46 +00:00
..
fabricators DEV: Make all admins TL4 in tests (#25435) 2024-03-26 11:41:12 +08:00
fixtures DEV: Support description for properties in objects schema (#26172) 2024-03-15 07:47:42 +08:00
generator DEV: Silence the output of migration specs (#26365) 2024-03-26 11:32:44 +01:00
helpers PERF: Remove unnecessary <link rel="preload"> (#26219) 2024-03-18 20:07:29 +08:00
import_export DEV: Allow fab! without block (#24314) 2023-11-09 16:47:59 -06:00
initializers DEV: Allow fab! without block (#24314) 2023-11-09 16:47:59 -06:00
integration FEATURE: Enable strict-dynamic Content-Security-Policy by default (#26051) 2024-03-07 15:20:31 +00:00
integrity Enable Embroider/Webpack code spliting for Wizard (#24919) 2023-12-20 13:15:06 +00:00
jobs DEV: Remove unnecessary rails_helper requiring (#26364) 2024-03-26 11:32:01 +01:00
lib DEV: Remove unnecessary rails_helper requiring (#26364) 2024-03-26 11:32:01 +01:00
mailers DEV: Update rubocop-discourse to latest version 2024-03-04 15:08:35 +01:00
migrations DEV: Silence the output of migration specs (#26365) 2024-03-26 11:32:44 +01:00
models DEV: Remove unnecessary rails_helper requiring (#26364) 2024-03-26 11:32:01 +01:00
multisite DEV: Add S3 upload system specs using minio (#22975) 2023-08-23 11:18:33 +10:00
requests FEATURE: Simplify crawler content for non-canonical post URLs (#26324) 2024-03-26 15:18:46 +00:00
script/import_scripts DEV: Allow fab! without block (#24314) 2023-11-09 16:47:59 -06:00
serializers DEV: Load theme objects typed setting metadata when routing to editor (#26354) 2024-03-26 14:02:05 +08:00
services DEV: Remove unnecessary rails_helper requiring (#26364) 2024-03-26 11:32:01 +01:00
support DEV: Move non scheduled problem checks to classes (#26122) 2024-03-14 10:55:01 +08:00
system DEV: Load theme objects typed setting metadata when routing to editor (#26354) 2024-03-26 14:02:05 +08:00
tasks DEV: Introduce rake task to validate discourse-compatibility file (#26158) 2024-03-13 13:57:41 +00:00
views FEATURE: Simplify crawler content for non-canonical post URLs (#26324) 2024-03-26 15:18:46 +00:00
rails_helper.rb DEV: Add plugin_file_from_fixtures helper (#26359) 2024-03-26 16:17:51 +10:00
regenerate_swagger_docs DEV: Add API docs for uploads and API doc watcher (#15387) 2021-12-23 08:40:15 +10:00
swagger_helper.rb DEV: Bump rswag-specs from 2.11.0 to 2.13.0 (#24654) 2023-12-07 08:16:47 +08:00