Robin Ward
3d7dbdedc0
FEATURE: An API to help sites build robots.txt files programatically
...
This is mainly useful for subfolder sites, who need to expose their
robots.txt contents to a parent site.
2018-04-16 15:43:20 -04:00
Régis Hanol
df7970a6f6
prefix the robots.txt rules with the directory when using subfolder
2018-04-11 22:05:02 +02:00
Sam
3a7b696703
FEATURE: allow for setting crawl delay per user agent
...
Also moved to default crawl delay bing so no more than a req every 5 seconds is allowed
New site settings:
"slow_down_crawler_user_agents" - list of crawlers that will be slowed down
"slow_down_crawler_rate" - how many seconds to wait between requests
Not enforced server side yet
2018-04-06 10:15:23 +10:00
Neil Lalonde
ced7e9a691
FEATURE: control which web crawlers can access using a whitelist or blacklist
2018-03-22 15:41:02 -04:00
Guo Xiang Tan
77d4c4d8dc
Fix all the errors to get our tests green on Rails 5.1.
2017-09-25 13:48:58 +08:00
Régis Hanol
d75cc67d86
FIX: robots.txt should be accessible even when login is required
2015-10-15 11:42:41 +02:00
Sam
e5888cf090
PERF: avoid preloading json in cases where it is not needed
...
(uploads / avatars / non GET requests)
2015-05-20 17:12:16 +10:00
Neil Lalonde
a86b35c873
Remove the access_password site setting
2013-06-25 15:05:25 -04:00
Robin Ward
d2596c3c4c
Remove unusued site_settings, show checkbox in UI for boolean values, remove restrict_access
...
boolean to avoid locking yourself out by setting access_password to empty string. Minor
UI tweaks.
2013-03-01 14:27:41 -05:00
Gosha Arinich
cafc75b238
remove trailing whitespaces ❤️
2013-02-26 07:31:35 +03:00
Sam Saffron
1c12c91d0c
forgot to skip a filter
2013-02-11 17:14:36 +11:00
Sam Saffron
c50a9e4d01
added support for disabling indexing by google using SiteSetting.allow_index_in_robots_txt = false
2013-02-11 11:02:57 +11:00
