11 Commits

Author	SHA1	Message	Date
Guo Xiang Tan	56f98de7b2	Use webmock to stub external web requests.	2017-05-26 15:19:09 +08:00
Guo Xiang Tan	f8f1548fd4	Revert "FIX: Use Excon to do its own stubbing" ... This reverts commit 80af54460a.	2017-05-26 13:04:25 +08:00
Robin Ward	3b0cbf7013	FIX: Always allow downloads from CDN	2017-05-23 16:32:54 -04:00
Robin Ward	b81e7be9a1	FEATURE: Rate limit how often we'll crawl a destination IP	2017-05-23 15:03:04 -04:00
Robin Ward	36e477750c	FIX: Use same code path for downloading images	2017-05-23 14:51:30 -04:00
Robin Ward	e5e7a15a85	SECURITY: Never crawl by IP	2017-05-23 13:07:18 -04:00
Robin Ward	93a5fc62bf	FEATURE: A site setting to prevent crawling on private IP blocks	2017-05-23 11:56:06 -04:00
Robin Ward	80af54460a	FIX: Use Excon to do its own stubbing	2017-05-22 18:19:20 -04:00
Robin Ward	b51126dd5e	FIX: Reset the WebMock after before every test	2017-05-22 17:52:31 -04:00
Robin Ward	4c690f7089	Use FinalDestination to ensure public redirects for onebox	2017-05-22 16:42:49 -04:00
Robin Ward	b23fc2bf84	Helper to find the final destination for a URL	2017-05-22 15:52:41 -04:00