56 Commits

Author	SHA1	Message	Date
Robin Ward	b81e7be9a1	FEATURE: Rate limit how often we'll crawl a destination IP	2017-05-23 15:03:04 -04:00
Robin Ward	36e477750c	FIX: Use same code path for downloading images	2017-05-23 14:51:30 -04:00
Robin Ward	e5e7a15a85	SECURITY: Never crawl by IP	2017-05-23 13:07:18 -04:00
Robin Ward	93a5fc62bf	FEATURE: A site setting to prevent crawling on private IP blocks	2017-05-23 11:56:06 -04:00
Robin Ward	b8d78b33c6	FIX: Other content types like images are fine	2017-05-22 16:51:37 -04:00
Robin Ward	b23fc2bf84	Helper to find the final destination for a URL	2017-05-22 15:52:41 -04:00