Host volumes with per_alloc allow deletion with alloc running

[the-nando]

Nomad version

v1.10.8+ent

Issue

A dynamic host volume created to be used with per_alloc=true doesn't show any allocations using it and Nomad allows its deletion leaving the alloc running with a volume path on the host that doesn't exist anymore.

Reproduction steps

Create a dynamic host volumes:

❯ nomad volume status -type=host -verbose -namespace='my-test' 8148ec82-e916-109a-9e9f-80dc11b6b593
ID        = 8148ec82-e916-109a-9e9f-80dc11b6b593
Name      = namespace-my-test-job-testing[0]
Namespace = my-test
Plugin ID = mkdir
Node ID   = 7d399036-11a3-58fa-1101-b7da7e663a10
Node Pool = my-pool
Capacity  = 0 B
State     = ready
Host Path = /data/nomad/host_volumes/8148ec82-e916-109a-9e9f-80dc11b6b593

Capabilities
Access Mode         Attachment Mode
single-node-writer  file-system

Allocations
No allocations placed
❯

Start a job that uses the volume:

volume "data" {
  type      = "host"
  source    = "namespace-my-test-job-testing"
  read_only = false
  per_alloc = true
}

[...]

volume_mount {
  volume = "data"
  destination = "/data"
}

Confirm with a docker inspect the volume is actually in use by the Docker container:

{
  "Type": "bind",
  "Source": "/data/nomad/host_volumes/8148ec82-e916-109a-9e9f-80dc11b6b593",
  "Destination": "/data",
  "Mode": "",
  "RW": true,
  "Propagation": "rprivate"
}

Volume status still shows no allocation placed:

❯ nomad volume status -type=host -verbose -namespace='my-test' 8148ec82-e916-109a-9e9f-80dc11b6b593
ID        = 8148ec82-e916-109a-9e9f-80dc11b6b593
Name      = namespace-my-test-job-testing[0]
Namespace = my-test
Plugin ID = mkdir
Node ID   = 7d399036-11a3-58fa-1101-b7da7e663a10
Node Pool = my-pool
Capacity  = 0 B
State     = ready
Host Path = /data/nomad/host_volumes/8148ec82-e916-109a-9e9f-80dc11b6b593

Capabilities
Access Mode         Attachment Mode
single-node-writer  file-system

Allocations
No allocations placed

Delete the volume:

❯ nomad volume delete -type=host -namespace='my-test' 8148ec82-e916-109a-9e9f-80dc11b6b593
Successfully deleted volume "8148ec82-e916-109a-9e9f-80dc11b6b593"!

The alloc stays running but the binded mount is now broken.

Expected Result

Nomad volume status show the volume as in being use and the API prevents deletion of the volume.

Actual Result

The volume can be deleted while there is an alloc running.

[tgross]

I just took a quick look at this and it looks like there's two different bugs here with the same underlying cause: when we lookup a volume we're not always accounting for the notion that the name has [#] concatenated if per_alloc = true (ref state_store_host_volumes.go#L44-L47 state_store_host_volumes.go#L156-L160). This is probably because supporting this feature was tacked on after the fact.

Fixing those two spots seems like a quick fix and I've marked this issue for roadmapping. But we should take a pass over other spots in the code base where we might be making a similar query and ensure we're not missing it there too.