Expand SiteSetting.allow_index_in_robots_txt so it also adds a noindex header if set to false. This makes sure that nothing is indexed even if it somehow reaches Google.
response.parsed_body