discourse/app
Sam 3a7b696703 FEATURE: allow for setting crawl delay per user agent
Also moved to default crawl delay bing so no more than a req every 5 seconds is allowed

New site settings:

"slow_down_crawler_user_agents" - list of crawlers that will be slowed down
"slow_down_crawler_rate" - how many seconds to wait between requests

Not enforced server side yet
2018-04-06 10:15:23 +10:00
..
assets FEATURE: implements minimum selection for select-kit 2018-04-05 16:45:19 +02:00
controllers FEATURE: allow for setting crawl delay per user agent 2018-04-06 10:15:23 +10:00
helpers FIX: update Google Tag Manager javascript 2018-04-03 14:22:06 -04:00
jobs FEATURE: Webhook for tag events 2018-04-04 17:49:20 +05:30
mailers Make rubocop happy. 2018-03-12 09:48:41 +08:00
models FEATURE: Send a different PM when a post has been hidden more than once 2018-04-05 14:03:21 +02:00
serializers Remove use of rescue nil. 2018-04-02 13:52:51 +08:00
services Remove use of rescue nil. 2018-04-02 13:52:51 +08:00
views FEATURE: allow for setting crawl delay per user agent 2018-04-06 10:15:23 +10:00