Commit Graph

54 Commits

Author SHA1 Message Date
Robin Ward
e5e7a15a85 SECURITY: Never crawl by IP 2017-05-23 13:07:18 -04:00
Robin Ward
93a5fc62bf FEATURE: A site setting to prevent crawling on private IP blocks 2017-05-23 11:56:06 -04:00
Robin Ward
b8d78b33c6 FIX: Other content types like images are fine 2017-05-22 16:51:37 -04:00
Robin Ward
b23fc2bf84 Helper to find the final destination for a URL 2017-05-22 15:52:41 -04:00