Commit Graph

6 Commits

Author SHA1 Message Date
Tobias Eigen
709bed5d1c
removed broken link and comments from no_index.erb (#25648)
We intend to move guidance about how to configure robots.txt to the admin interface and a documentation topic on meta.
2024-02-14 12:09:24 +08:00
Daniel Waterworth
721ee36425
Replace base_uri with base_path (#10879)
DEV: Replace instances of Discourse.base_uri with Discourse.base_path

This is clearer because the base_uri is actually just a path prefix. This continues the work started in 555f467.
2020-10-09 12:51:24 +01:00
Sam Saffron
bb4e8899c4
FEATURE: let Google index pages so it can remove them
Google insists on indexing pages so it can figure out if they
can be removed from the index.

see: https://support.google.com/webmasters/answer/6332384?hl=en

This change ensures the we have special behavior for Googlebot
where we allow indexing, but block the actual indexing via
X-Robots-Tag
2020-05-11 12:15:18 +10:00
Régis Hanol
df7970a6f6 prefix the robots.txt rules with the directory when using subfolder 2018-04-11 22:05:02 +02:00
Guo Xiang Tan
77d4c4d8dc Fix all the errors to get our tests green on Rails 5.1. 2017-09-25 13:48:58 +08:00
Sam Saffron
c50a9e4d01 added support for disabling indexing by google using SiteSetting.allow_index_in_robots_txt = false 2013-02-11 11:02:57 +11:00