Joshua Rosenfeld 417bdcb53a FIX: Remove paths from robots.txt in favor of noindex header ... Google no longer supports the use of robots.txt to block indexing. See https://support.google.com/webmasters/answer/6062608 and https://support.google.com/webmasters/answer/93710 Previous commits have added the `noindex` header to appropriate pages, now we need to remove the paths from robots.txt so the pages can be crawled. Follow up to: 13f229808a b6765aac4b 676be3a853 07b728c5e5 c94e6a9a66		2020-07-02 15:09:40 -04:00
..
assets	FIX: Uploads cannot be mapped due to the cook-text's element attr being null (#10136)	2020-06-30 12:07:50 -03:00
controllers	FIX: Remove paths from robots.txt in favor of noindex header	2020-07-02 15:09:40 -04:00
helpers	New bootstrap.json endpoint for starting up Discourse	2020-06-03 14:45:23 -04:00
jobs	DEV: improve verbose mode for reindexer	2020-06-24 17:29:45 +10:00
mailers	FIX: Use correct URL for unsubscribe (#10077)	2020-06-24 09:31:20 +02:00
models	Merge diffs from master	2020-06-24 13:48:37 -04:00
serializers	FIX: skip category notification_level unless scoped	2020-06-24 17:08:12 +10:00
services	FEATURE: allow disabling of extra term injection in search	2020-07-02 15:08:53 -04:00
views	FIX: published-page-header should be a sibling to published-page-body not a parent (#10126)	2020-06-25 14:59:33 -07:00