discourse/app/controllers
Natalie Tay 038a391cfe
FIX: Remove /u/ from robots (#30782)
Follow up from https://github.com/discourse/discourse/pull/27712.

Currently, we already add `noindex` to /u routes. However, due to
robots.txt blocking this, search engines are not able to see the header.

This commit removes /u from our robots.txt to allow search engines to
see the header. This commit also includes a migration to remove the /u
from admins who have overridden the file. I had contemplated not
including this migration, but seeing there are existing site admins who
are trying to remove errors from their dashboard, they would probably
welcome this change.

The migration replaces overridden text at this area, and will not modify
if additional routes have been added in-between:

<img width="500" alt="Screenshot 2025-01-15 at 11 28 43 AM copy"
src="https://github.com/user-attachments/assets/60db64c9-ed33-48a5-a917-a10545282a5c"
/>


Side effect note: This might potentially result in more pageviews* from
GoogleBot (for example) for a period of time as Google starts visiting
the user routes they have been denied before.
2025-01-15 13:42:08 +08:00
..
admin DEV: Convert admin API keys to conform to UI guidelines (#30660) 2025-01-09 10:57:40 +08:00
users SECURITY: When enabled only allow Discourse Connect logins 2024-12-19 13:13:23 -03:00
about_controller.rb DEV: Revert guardian changes (#24742) 2023-12-06 16:37:32 +10:00
application_controller.rb DEV: Include controller namespace in X-Discourse-Route (#29783) 2024-11-29 17:11:17 +11:00
associated_groups_controller.rb
badges_controller.rb
bookmarks_controller.rb FEATURE: Add bulk action to bookmark (#26856) 2024-05-22 12:50:21 -03:00
bootstrap_controller.rb DEV: Simplify ember-cli proxy strategy (#24242) 2023-11-10 11:16:06 +00:00
categories_controller.rb FEATURE: add new setting to select view for category page on mobile (#30519) 2025-01-06 09:25:25 +05:30
clicks_controller.rb
composer_controller.rb UX: hide warning if all users mentioned via group are already invited. (#23557) 2023-09-13 19:21:44 +05:30
composer_messages_controller.rb DEV: Move distance_of_time_in_words/time_ago_in_words (#21745) 2023-05-25 14:53:59 +02:00
csp_reports_controller.rb
custom_homepage_controller.rb DEV: allow themes to render their own custom homepage (#26291) 2024-04-02 11:05:08 -04:00
directory_columns_controller.rb
directory_items_controller.rb FEATURE: Add links to searchable user fields in users directory and user profile (#29338) 2024-11-06 13:35:30 -04:00
do_not_disturb_controller.rb
drafts_controller.rb FIX: corrently handle hidden tags when checking for edit conflicts 2024-12-09 19:17:16 +01:00
edit_directory_columns_controller.rb DEV: Implement staff logs for user columns edits (#21774) 2023-06-07 17:19:58 -05:00
email_controller.rb DEV: refactor email_controller.rb 2024-12-31 15:28:59 +01:00
embed_controller.rb DEV: Also noindex embedded comments (#27221) 2024-05-28 12:59:24 +08:00
emojis_controller.rb DEV: unifies emoji picker (#28277) 2025-01-08 11:41:36 +01:00
exceptions_controller.rb
export_csv_controller.rb SECURITY: Prevent large staff actions causing DoS 2024-03-15 14:24:04 +08:00
extra_locales_controller.rb DEV: Remove logical OR assignment of constants (#29201) 2024-10-16 10:09:07 +08:00
finish_installation_controller.rb FEATURE: User fields required for existing users - Part 2 (#27172) 2024-06-25 19:32:18 +08:00
form_templates_controller.rb DEV: Show form templates in the composer (#21190) 2023-05-29 14:47:18 -07:00
forums_controller.rb
groups_controller.rb FEATURE: Allow add group member endpoint to skip invite emails (#29962) 2024-11-27 11:33:09 -06:00
hashtags_controller.rb FEATURE: Async load of category and chat hashtags (#25526) 2024-02-12 12:07:14 +02:00
highlight_js_controller.rb FEATURE: User fields required for existing users - Part 2 (#27172) 2024-06-25 19:32:18 +08:00
inline_onebox_controller.rb
invites_controller.rb SECURITY: When enabled only allow Discourse Connect logins 2024-12-19 13:13:23 -03:00
list_controller.rb DEV: Add per_page as public param for TopicQuery (#30716) 2025-01-14 09:19:19 -06:00
metadata_controller.rb FEATURE: User fields required for existing users - Part 2 (#27172) 2024-06-25 19:32:18 +08:00
new_invite_controller.rb FEATURE: Add invite link to the sidebar (#29448) 2024-10-30 05:31:14 +03:00
new_topic_controller.rb
notifications_controller.rb DEV: Dedicated route for current user notification counts (#26106) 2024-03-15 12:08:37 -04:00
offline_controller.rb FEATURE: User fields required for existing users - Part 2 (#27172) 2024-06-25 19:32:18 +08:00
onebox_controller.rb
pageview_controller.rb FEATURE: User fields required for existing users - Part 2 (#27172) 2024-06-25 19:32:18 +08:00
permalinks_controller.rb FIX: Don’t raise an error on permalinks with external URL 2024-06-28 10:09:37 +02:00
post_action_users_controller.rb DEV: Add post_action_users_list modifier for PostActionUsersController (#25740) 2024-02-20 09:48:09 +10:00
post_actions_controller.rb
post_readers_controller.rb
posts_controller.rb FIX: All admins should be allowed to see deleted PM posts regardless of their mod status (#30206) 2024-12-23 12:48:03 +03:00
presence_controller.rb DEV: Remove logical OR assignment of constants (#29201) 2024-10-16 10:09:07 +08:00
published_pages_controller.rb FEATURE: User fields required for existing users - Part 2 (#27172) 2024-06-25 19:32:18 +08:00
push_notification_controller.rb
qunit_controller.rb FEATURE: User fields required for existing users - Part 2 (#27172) 2024-06-25 19:32:18 +08:00
reviewable_claimed_topics_controller.rb FEATURE: Support designating multiple groups as mods on category (#28655) 2024-09-04 04:38:46 +03:00
reviewables_controller.rb FEATURE: Reason and deleted content support in the review queue (#30295) 2024-12-17 11:44:46 +11:00
robots_txt_controller.rb FIX: Remove /u/ from robots (#30782) 2025-01-15 13:42:08 +08:00
safe_mode_controller.rb DEV: Add safe_mode=deprecation_errors mode (#24870) 2023-12-13 14:06:59 +00:00
search_controller.rb DEV: Add user_agent column to search_logs (#27742) 2024-07-05 14:05:00 -05:00
session_controller.rb FIX: staff only mode blocks admin password resets (#29289) 2024-10-21 09:29:37 +02:00
sidebar_sections_controller.rb DEV: Use has_many and ArraySerializer for SidebarSectionsSerializer (#26716) 2024-05-06 11:32:18 -05:00
similar_topics_controller.rb
site_controller.rb FEATURE: User fields required for existing users - Part 2 (#27172) 2024-06-25 19:32:18 +08:00
sitemap_controller.rb
slugs_controller.rb
static_controller.rb FIX: Do not ignore redirects containing "/login" in the path (#29960) 2024-11-27 11:22:45 -05:00
steps_controller.rb
stylesheets_controller.rb FIX: Write stylesheet cache atomically (#28457) 2024-08-21 12:44:17 +01:00
svg_sprite_controller.rb FIX: bump the number of svg icons we return to first 500 (#29286) 2024-10-18 19:22:13 +02:00
tag_groups_controller.rb FEATURE: Log tag group changes in staff action log (#28787) 2024-09-09 10:50:48 +08:00
tags_controller.rb DEV: Ignore invalid tag parameter in TagsController (#28557) 2024-08-27 12:06:54 -04:00
test_requests_controller.rb FIX: Set sane default for Net::HTTP when processing a request (#28141) 2024-08-06 07:12:42 +08:00
theme_javascripts_controller.rb FEATURE: User fields required for existing users - Part 2 (#27172) 2024-06-25 19:32:18 +08:00
topic_view_stats_controller.rb FEATURE: topic_view_stats table with daily fidelity (#27197) 2024-05-27 15:25:32 +10:00
topics_controller.rb DEV: Allow freeze_original argument in topics controller & JS transformer (#30120) 2024-12-05 08:31:05 -06:00
uploads_controller.rb FIX: Extension-less secure uploads (#29914) 2024-11-25 12:18:21 +00:00
user_actions_controller.rb FIX: Load categories with user activity and drafts (#26553) 2024-04-10 17:35:42 +03:00
user_api_key_clients_controller.rb Add user api key client rate limit settings (#30402) 2024-12-30 11:10:48 -05:00
user_api_keys_controller.rb Require permitted scopes when registering a client (#29718) 2024-11-19 15:28:04 -05:00
user_avatars_controller.rb FEATURE: User fields required for existing users - Part 2 (#27172) 2024-06-25 19:32:18 +08:00
user_badges_controller.rb DEV:refactor user badges create to get grant opts from method (#29372) 2024-10-30 18:03:20 -03:00
user_status_controller.rb
users_controller.rb FIX: Allow admins to use reserved usernames (#30262) 2025-01-13 17:30:36 +10:00
users_email_controller.rb FEATURE: User fields required for existing users - Part 2 (#27172) 2024-06-25 19:32:18 +08:00
webhooks_controller.rb FEATURE: Add Mailpace webhook (#21981) 2023-06-08 20:06:20 +03:00
wizard_controller.rb