10 Commits

Author	SHA1	Message	Date
Robin Ward	5d04cb4b47	FIX: Always allow the host the forum is hosted on	2017-06-13 10:55:15 -04:00
Robin Ward	502bca2c0d	FIX: If HEAD is not supported, try GET. Also set cookies	2017-06-13 10:54:27 -04:00
Robin Ward	0a08c18a14	FIX: Don't rate limit gravatar downloads	2017-05-24 13:54:26 -04:00
Robin Ward	3b0cbf7013	FIX: Always allow downloads from CDN	2017-05-23 16:32:54 -04:00
Robin Ward	b81e7be9a1	FEATURE: Rate limit how often we'll crawl a destination IP	2017-05-23 15:03:04 -04:00
Robin Ward	36e477750c	FIX: Use same code path for downloading images	2017-05-23 14:51:30 -04:00
Robin Ward	e5e7a15a85	SECURITY: Never crawl by IP	2017-05-23 13:07:18 -04:00
Robin Ward	93a5fc62bf	FEATURE: A site setting to prevent crawling on private IP blocks	2017-05-23 11:56:06 -04:00
Robin Ward	b8d78b33c6	FIX: Other content types like images are fine	2017-05-22 16:51:37 -04:00
Robin Ward	b23fc2bf84	Helper to find the final destination for a URL	2017-05-22 15:52:41 -04:00