check_restart: move feature to task level hook

[mismithhisler]

Description

Currently, if check_restart is specified in a group service block, it is not part of the task's lifecycle, so a task that restarts and relies on check_restart.grace will be restarted unnessecarily due to failing checks. These changes move the check_restart functionality to always be part of the tasks lifecycle, so when a task is restarted the check watch is restarted.

Testing & Reproduction steps

mysql.hcl

job "mysql-server" {
  type        = "service"

  group "mysql-server" {

    service {
      name = "mysql-server"
      port = "db"
      task = "mysql-server"

      check {
        type     = "tcp"
        interval = "10s"
        timeout  = "2s"

        check_restart {
          grace = "20s"
          limit = 1
        }
      }
    }
    network {
      port "db" {
        static = 3306
        to = 3306
      }
    }

    task "mysql-server" {
      driver = "docker"

      config {
        image = "mysql:9.6.0"
        ports = ["db"]
      }

      env {
        MYSQL_ROOT_PASSWORD = "password"
      }
    }
  }
}

Take the above job, run it, then restart the task. It will get restarted by the Nomad CheckWatcher due to failing healthchecks.

Links

Fixes #9431

Contributor Checklist

• Changelog Entry If this PR changes user-facing behavior, please generate and add a
  changelog entry using the make cl command.
• Testing Please add tests to cover any new functionality or to demonstrate bug fixes and
  ensure regressions will be caught.
• Documentation If the change impacts user-facing functionality such as the CLI, API, UI,
  and job configuration, please update the Nomad product documentation, which is stored in the
  web-unified-docs repo. Refer to the web-unified-docs contributor guide for docs guidelines.
  Please also consider whether the change requires notes within the upgrade
  guide. If you would like help with the docs, tag the nomad-docs team in this PR.

Reviewer Checklist

• Backport Labels Please add the correct backport labels as described by the internal
  backporting document.
• Commit Type Ensure the correct merge method is selected which should be "squash and merge"
  in the majority of situations. The main exceptions are long-lived feature branches or merges where
  history should be preserved.
• Enterprise PRs If this is an enterprise only PR, please add any required changelog entry
  within the public repository.

• If a change needs to be reverted, we will roll out an update to the code within 7 days.

Changes to Security Controls

Are there any changes to security controls (access controls, encryption, logging) in this pull request? If so, explain.

[mismithhisler]

c.TaskName == h.taskName means we would only register that for group level checks, only the task specified in the service/check will be restarted. Also if the task is not specified the check won't even be registered.

[mismithhisler]

In a subsequent PR, I want to remove the CheckWatcher from the serviceregistration.Wrapper. Currently we create a check watcher on the nomad client, and a check watcher for every consul API on the client and server (even though servers don't watch checks.. Arguably we could do a isNomadClient check and not run the watcher on the server, but the consul service client code is difficult enough, so I think pulling it out could help reduce some cognitive complexity).

I'd like to see this done as part of the normal client startup process. But it will take a little refactoring of how we create consul api clients.