Natalie Tay 038a391cfe FIX: Remove /u/ from robots (#30782) ... Follow up from https://github.com/discourse/discourse/pull/27712. Currently, we already add `noindex` to /u routes. However, due to robots.txt blocking this, search engines are not able to see the header. This commit removes /u from our robots.txt to allow search engines to see the header. This commit also includes a migration to remove the /u from admins who have overridden the file. I had contemplated not including this migration, but seeing there are existing site admins who are trying to remove errors from their dashboard, they would probably welcome this change. The migration replaces overridden text at this area, and will not modify if additional routes have been added in-between: <img width="500" alt="Screenshot 2025-01-15 at 11 28 43 AM copy" src="https://github.com/user-attachments/assets/60db64c9-ed33-48a5-a917-a10545282a5c" /> Side effect note: This might potentially result in more pageviews* from GoogleBot (for example) for a period of time as Google starts visiting the user routes they have been denied before.		2025-01-15 13:42:08 +08:00
..
fixtures	DEV: update more deprecated font awesome icon names (#30123)	2024-12-05 15:20:02 +08:00
migrate	FIX: Remove /u/ from robots (#30782)	2025-01-15 13:42:08 +08:00
post_migrate
api_test_seeds.rb