mirror of
https://github.com/discourse/discourse.git
synced 2025-02-23 09:15:25 +08:00

Google insists on indexing pages so it can figure out if they can be removed from the index. see: https://support.google.com/webmasters/answer/6332384?hl=en This change ensures the we have special behavior for Googlebot where we allow indexing, but block the actual indexing via X-Robots-Tag