mirror of
https://github.com/discourse/discourse.git
synced 2025-02-21 03:57:10 +08:00
bb4e8899c4
Google insists on indexing pages so it can figure out if they can be removed from the index. see: https://support.google.com/webmasters/answer/6332384?hl=en This change ensures the we have special behavior for Googlebot where we allow indexing, but block the actual indexing via X-Robots-Tag