github-mirror/discourse
2
0
Fork 0
mirror of https://github.com/discourse/discourse.git synced 2024-11-23 02:50:00 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

FEATURE: block screaming frog SEO spider by default

Browse Source 
This crawler is not respectful and often can make 1000s of reqs a minute.

Block by default, site operators can amend this default if they want to.
This commit is contained in:
Sam Saffron 2019-11-18 16:47:47 +11:00
parent a9704da34c
commit d28de23ac8
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
config/site_settings.yml
View File

@ -1371,7 +1371,7 @@ security:
list_type: compact
blacklisted_crawler_user_agents:
type: list
default: "mauibot|semrushbot|ahrefsbot|blexbot"
default: "mauibot|semrushbot|ahrefsbot|blexbot|seo spider"
list_type: compact
slow_down_crawler_user_agents:
type: list