Add "test mode" to project to avoid Google Search traffic begin redirected to test environments and improving page rank

[monty241]

A test instance of Hypha is indexed and followed by Google Search and other crawlers when discovered. This is not hypothetical, but already affecting sites and withholding search traffic from production sites. For example, the following picture displays a search on site:test.destadsbr0n.nl (with intended typo) having Google Search display test data:

Suggestion is to add a boolean attribute "testMode" or a boolean "allowIndex" to control availability for indexing.

When set to try, the following HTTP header is set on each page:

X-Robots-Tag noindex,nofollow

Plus for instance a meta tag generated into the head:

<meta name="robots" content="noindex,nofollow">

Additional bonus is - given absence of canonical URLs - the remaining pages rank higher.

Workaround Caddy

As a workaround, on Caddy the following definition implements the same:

domainname
{
  import hsts

  header * X-Robots-Tag "noindex,nofollow"

  header {
    -Server
    -Via
  }

  reverse_proxy http://1.2.3.4:5 {
  }
}