3 Commits

Author	SHA1	Message	Date
Neil Lalonde	2bd44bbf13	WebCrawlerRequest.clear_cache needs to clear user agent list too	2018-03-27 15:11:48 -04:00
Neil Lalonde	4d12ff2e8a	when writing cache, remove elements from the user agents list. also return a message and content type when blocking a crawler.	2018-03-27 13:44:14 -04:00
Neil Lalonde	ced7e9a691	FEATURE: control which web crawlers can access using a whitelist or blacklist	2018-03-22 15:41:02 -04:00