discourse/lib/crawler_detection.rb

module CrawlerDetection
  # added 'ia_archiver' based on https://meta.discourse.org/t/unable-to-archive-discourse-pages-with-the-internet-archive/21232
  # added 'Wayback Save Page' based on https://meta.discourse.org/t/unable-to-archive-discourse-with-the-internet-archive-save-page-now-button/22875
  # added 'Swiftbot' based on https://meta.discourse.org/t/how-to-add-html-markup-or-meta-tags-for-external-search-engine/28220

  def self.crawler?(user_agent)
    !/Googlebot|Mediapartners|AdsBot|curl|HTTrack|Twitterbot|facebookexternalhit|bingbot|Baiduspider|ia_archiver|Wayback Save Page|360Spider|Swiftbot|YandexBot/.match(user_agent).nil?
  end
end
REFACTOR: Rename `GooglebotDetection` to `CrawlerDetection` because we will likely whitelist more crawlers in the future. 2014-02-21 05:07:02 +08:00			`module CrawlerDetection`
add support for "Save Page Now" archive.org/web 2015-01-06 17:05:45 +08:00			`# added 'ia_archiver' based on https://meta.discourse.org/t/unable-to-archive-discourse-pages-with-the-internet-archive/21232`
			`# added 'Wayback Save Page' based on https://meta.discourse.org/t/unable-to-archive-discourse-with-the-internet-archive-save-page-now-button/22875`
add Swiftbot to crawler regex 2015-05-02 18:18:58 +08:00			`# added 'Swiftbot' based on https://meta.discourse.org/t/how-to-add-html-markup-or-meta-tags-for-external-search-engine/28220`
FEATURE: add 360Spider UA to allow 360 crawl Discourse sites 2015-02-14 22:24:51 +08:00
REFACTOR: Rename `GooglebotDetection` to `CrawlerDetection` because we will likely whitelist more crawlers in the future. 2014-02-21 05:07:02 +08:00			`def self.crawler?(user_agent)`
Update crawler_detection.rb Add HTTrack to the list of detected crawlers so that Discourse will serve vanilla HTML per https://meta.discourse.org/t/a-basic-discourse-archival-tool/62614/25 2017-05-16 23:17:05 +08:00			`!/Googlebot\|Mediapartners\|AdsBot\|curl\|HTTrack\|Twitterbot\|facebookexternalhit\|bingbot\|Baiduspider\|ia_archiver\|Wayback Save Page\|360Spider\|Swiftbot\|YandexBot/.match(user_agent).nil?`
Detect Googlebot from user agent and use a different layout that doesn't load javascript 2014-02-15 06:10:08 +08:00			`end`
			`end`