9 Commits

Author	SHA1	Message	Date
Camille Roux	f14c71b9d4	Fix the Amazon links regex	2014-05-06 19:19:32 +02:00
Camille Roux	e77e7f23ca	Update the Amazon links regexp ... Added all the countries displayed in the Amazon footer	2014-05-06 18:36:07 +02:00
Robin Ward	a57f802048	If there's a TopicEmbed record for a url, we don't have to crawl it. ... This should help sites like Boing Boing where sometimes links are crawled before saved in WordPress.	2014-04-17 14:00:22 -04:00
Robin Ward	e80851b0fa	Special case: When crawling a link to an image, just put the filename as ... the title.	2014-04-10 13:45:13 -04:00
Robin Ward	99e2bab62d	Use update_all to prevent after_commit from executing again.	2014-04-10 13:19:57 -04:00
Robin Ward	aa63868d5e	FIX: Problem crawling amazon titles	2014-04-08 16:39:47 -04:00
Robin Ward	1e3faddfe4	FIX: Change crawl size to 10k. Youtube for example doesn't work with the ... first 1k	2014-04-07 16:03:47 -04:00
Robin Ward	7e0028ba50	FIX: Don't crawl in test mode, raise correct exception when parameters ... are missing	2014-04-07 14:38:18 -04:00
Robin Ward	7e3ea5d644	Support for crawling topic links	2014-04-07 14:08:34 -04:00