discourse/app/views/robots_txt/index.erb

<%= @robots_info[:header] %>
<% if Discourse.base_path.present? %>
# This robots.txt file is not used. Please append the content below in the robots.txt file located at the root
<% end %>
#
<% @robots_info[:agents].each do |agent| %>
User-agent: <%= agent[:name] %>
<% agent[:disallow].each do |path| %>
Disallow: <%= path %>
<% end %>


<% end %>

<%- if SiteSetting.enable_sitemap? && !SiteSetting.login_required? %>
Sitemap: <%= request.protocol %><%= request.host_with_port %>/sitemap.xml
<% end %>

<%= server_plugin_outlet "robots_txt_index" %>
FEATURE: An API to help sites build robots.txt files programatically This is mainly useful for subfolder sites, who need to expose their robots.txt contents to a parent site. 2018-04-17 03:43:20 +08:00			`<%= @robots_info[:header] %>`
Replace `base_uri` with `base_path` (#10879) DEV: Replace instances of Discourse.base_uri with Discourse.base_path This is clearer because the base_uri is actually just a path prefix. This continues the work started in 555f467. 2020-10-09 19:51:24 +08:00			`<% if Discourse.base_path.present? %>`
add a warning in robots.txt when using subfolder 2018-04-12 06:00:15 +08:00			`# This robots.txt file is not used. Please append the content below in the robots.txt file located at the root`
			`<% end %>`
Initial release of Discourse 2013-02-06 03:16:51 +08:00			`#`
FEATURE: An API to help sites build robots.txt files programatically This is mainly useful for subfolder sites, who need to expose their robots.txt contents to a parent site. 2018-04-17 03:43:20 +08:00			`<% @robots_info[:agents].each do \|agent\| %>`
			`User-agent: <%= agent[:name] %>`
			`<% agent[:disallow].each do \|path\| %>`
			`Disallow: <%= path %>`
prefix the robots.txt rules with the directory when using subfolder 2018-04-12 04:05:02 +08:00			`<% end %>`
new: server plugin outlet for indexable robots.txt 2017-02-13 20:01:10 +08:00
FEATURE: control which web crawlers can access using a whitelist or blacklist 2018-03-16 05:10:45 +08:00
			`<% end %>`

FEATURE: Let sites add a sitemap.xml file. (#16357) * FEATURE: Let sites add a sitemap.xml file. This PR adds the same features discourse-sitemap provides to core. Sitemaps are only added to the robots.txt file if the `enable_sitemap` setting is enabled and `login_required` disabled. After merging discourse/discourse-sitemap#34, this change will take priority over the sitemap plugin because it will disable itself. We're also using the same sitemaps table, so our migration won't try to create it again using `if_not_exists: true`. 2022-04-12 21:33:59 +08:00			`<%- if SiteSetting.enable_sitemap? && !SiteSetting.login_required? %>`
			`Sitemap: <%= request.protocol %><%= request.host_with_port %>/sitemap.xml`
			`<% end %>`

new: server plugin outlet for indexable robots.txt 2017-02-13 20:01:10 +08:00			`<%= server_plugin_outlet "robots_txt_index" %>`