github-mirror/discourse
2
0
Fork 0
mirror of https://github.com/discourse/discourse.git synced 2024-11-23 23:54:16 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
14,146 Commits 190 Branches 500 Tags 938 MiB
edae9d7ad9
Commit Graph

6 Commits

Author SHA1 Message Date
Jeff Atwood
bbb1348118 add Swiftbot to crawler regex 2015-05-02 03:18:58 -07:00
Erick Guan
026cdd8fc3 FEATURE: add 360Spider UA to allow 360 crawl Discourse sites 2015-03-16 22:58:33 +08:00
Jeff Atwood
ceef06e771 add support for "Save Page Now" archive.org/web 2015-01-06 01:05:45 -08:00
riking
37dbc4b5e6 Add archive.org to crawler list to serve no-js to 2014-11-02 16:51:23 -08:00
Vikhyat Korrapati
e3702ecb30 Improved crawler detection: add Twitterbot, Facebook, curl, Bing, Baidu. 2014-03-16 19:30:20 +05:30
Robin Ward
c4b5455c21 REFACTOR: Rename GooglebotDetection to CrawlerDetection because we 
will likely whitelist more crawlers in the future.
 2014-02-20 16:07:02 -05:00