Sam 3a7b696703 FEATURE: allow for setting crawl delay per user agent
Also moved to default crawl delay bing so no more than a req every 5 seconds is allowed

New site settings:

"slow_down_crawler_user_agents" - list of crawlers that will be slowed down
"slow_down_crawler_rate" - how many seconds to wait between requests

Not enforced server side yet
2018-04-06 10:15:23 +10:00
..
2018-03-26 10:58:01 -04:00
2018-03-06 20:51:41 +01:00
2018-03-26 10:58:01 -04:00
2018-03-26 10:58:01 -04:00
2018-03-26 10:58:01 -04:00
2018-03-26 10:58:01 -04:00
2018-03-26 10:58:01 -04:00
2018-03-26 10:58:01 -04:00
2018-03-26 10:58:01 -04:00
2018-03-26 10:58:01 -04:00
2018-03-26 10:58:01 -04:00
2018-03-26 10:58:01 -04:00
2018-03-26 10:58:01 -04:00
2018-02-15 16:36:03 -05:00
2018-03-26 10:58:01 -04:00
2018-03-26 10:58:01 -04:00
2018-03-26 10:58:01 -04:00
2018-03-26 10:58:01 -04:00
2018-03-26 10:58:01 -04:00
2017-08-16 11:38:45 -04:00
2018-03-26 10:58:01 -04:00
2018-03-26 10:58:01 -04:00
2018-03-26 10:58:01 -04:00
2018-03-26 10:58:01 -04:00
2018-03-26 10:58:01 -04:00
2018-03-26 10:58:01 -04:00
2018-03-26 10:58:01 -04:00
2018-03-26 10:58:01 -04:00
2018-03-26 10:58:01 -04:00
2018-03-26 10:58:01 -04:00
2018-03-26 10:58:01 -04:00
2018-03-26 10:58:01 -04:00
2018-03-26 10:58:01 -04:00
2018-03-26 10:58:01 -04:00
2018-03-26 10:58:01 -04:00
2017-02-24 11:35:33 +01:00