discourse/app/views/robots_txt/no_index.erb

# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
#

# Googlebot must be allowed to index so it can remove items from the index
# we return the X-Robots-Tag with noindex, nofollow which will ensure
# indexing is minimized and nothing shows up in Google search results
User-agent: googlebot
Allow: <%= Discourse.base_uri + "/" %>
Disallow: <%= Discourse.base_uri + "/uploads/*" %>

User-agent: *
Disallow: <%= Discourse.base_uri + "/" %>
Initial release of Discourse 2013-02-06 03:16:51 +08:00			`# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file`
			`#`
FEATURE: let Google index pages so it can remove them Google insists on indexing pages so it can figure out if they can be removed from the index. see: https://support.google.com/webmasters/answer/6332384?hl=en This change ensures the we have special behavior for Googlebot where we allow indexing, but block the actual indexing via X-Robots-Tag 2020-05-11 10:14:21 +08:00
			`# Googlebot must be allowed to index so it can remove items from the index`
			`# we return the X-Robots-Tag with noindex, nofollow which will ensure`
			`# indexing is minimized and nothing shows up in Google search results`
			`User-agent: googlebot`
			`Allow: <%= Discourse.base_uri + "/" %>`
			`Disallow: <%= Discourse.base_uri + "/uploads/*" %>`

Fix all the errors to get our tests green on Rails 5.1. 2017-08-31 12:06:56 +08:00			`User-agent: *`
prefix the robots.txt rules with the directory when using subfolder 2018-04-12 04:05:02 +08:00			`Disallow: <%= Discourse.base_uri + "/" %>`
FEATURE: let Google index pages so it can remove them Google insists on indexing pages so it can figure out if they can be removed from the index. see: https://support.google.com/webmasters/answer/6332384?hl=en This change ensures the we have special behavior for Googlebot where we allow indexing, but block the actual indexing via X-Robots-Tag 2020-05-11 10:14:21 +08:00