Allocation recovery does not re-derive Consul tokens via Workload Identity, falling back to anonymous token

[Diogo-Almeida3]

Nomad version

Nomad v1.11.1

Operating system and Environment details

Azure VM with ubuntu 24.04

Issue

When a Nomad client restarts and recovers allocations from its saved state, the Consul hook that exchanges Workload identity JWTs for Consul ACL tokens is not re-executed. The recovered Envoy sidecars start with an invalid/missing Consul token and fall back to the anonymous token, resulting in Permission denied: anonymous token lacks permission 'service:write'. Manually rescheduling allocations resolves the issue because new allocations go through the full hook lifecycle and obtain fresh Consul tokens. The recovery path should re-derive Consul WI tokens the same way the new allocation path does.

Reproduction steps

1. Enable Consul ACLs with a restrictive anonymous token (no service:write)
2. Configure a Consul auth method (e.g., nomad-workloads) for Nomad Workload Identity
3. Configure Nomad with service_auth_method and task_auth_method pointing to the auth method, and a Consul token in a separate .hcl for agent-level communication
4. Deploy jobs using Consul Connect service mesh. Allocations receive proper WI-derived Consul tokens and sidecars work correctly
5. Restart the VM
6. Nomad recovers allocations from client state. Envoy sidecars fail with Permission denied: anonymous token lacks permission 'service:write'
7. Running nomad job restart -reschedule for the affected jobs resolves the issue, confirming that fresh allocations correctly obtain Consul tokens or deleting the data in /opt/nomad/data/alloc and /opt/nomad/data/client/state

Expected Result

Recovered allocations re-derive Consul tokens via the WI auth method.

Actual Result

Recovered allocations skip the Consul WI token exchange and fall back to the anonymous token.

Sidecar log

DeltaAggregatedResources gRPC config stream to local_agent closed: 7, Permission denied: anonymous token lacks permission 'service:write' on "". The anonymous token is used implicitly when a request does not specify a token.{via_upstream}

[Diogo-Almeida3]

Additional: Consul ACL tokens derived via the WI auth method persist across restarts and remain valid (verified with consul acl token list).

[tehut]

Hi @Diogo-Almeida3 I added this to our board for prioritization last week. I wasn't able to get to the bottom of this so we'll get some more eyes on it.

[mismithhisler]

Hello @Diogo-Almeida3! We were looking to reproduce this but usually when you restart a Nomad client host machine, the Nomad client will restart, see it has allocations in it's state, attempt to reattach to those processes, and fail (because the processes were stopped when the host stopped). It then reports back the allocations have failed, and new allocations would get started which would re-derive the Consul tokens.

Can you provide a bit more about your reproduction steps that could help us reproduce this?

[Diogo-Almeida3]

Hello @tehut @mismithhisler. Yes give me a few hours i will try to provide a detailed reproduction step list.

[Diogo-Almeida3]

Sorry for the delay @tehut @mismithhisler . Let me know if you need anything else.

Setup

• Two Ubuntu 24.04 VMs (Azure)

  • One running nomad server+client
  • The other client-only.

• consul 1.22.3 with ACLs in default-deny mode and Connect enabled.

• nomad 1.11.1 with Workload Identity configured to derive per-service consul tokens via a JWT auth method.

The nomad.hcl config is the following:

# Copyright (c) HashiCorp, Inc.
# SPDX-License-Identifier: BUSL-1.1

data_dir  = "/opt/nomad/data"
bind_addr = "0.0.0.0"

server {
  enabled          = true
  bootstrap_expect = 1

  default_scheduler_config {
    scheduler_algorithm             = "spread"
    memory_oversubscription_enabled = true

    preemption_config {
      system_scheduler_enabled   = true
      service_scheduler_enabled  = true
      batch_scheduler_enabled    = true
      sysbatch_scheduler_enabled = true
    }
  }
}

client {
  host_volumes_dir = "<custom-volume-path>"
  enabled = true
  servers = ["<server-ip>"] # SERVER IPs
  meta {
    consul.grpc = "1"
  }

  cni_path = "/opt/cni/bin"
  cni_config_dir = "/opt/cni/config"

  reserved {
    cpu    = 940
    memory = 1600
  }
}

plugin "docker" {
  config {
    volumes {
      enabled = true
    }
  }
}

advertise {
  http = "<own-node-ip>:4646" # OWN NODE IP
  rpc  = "<own-node-ip>:4647" # OWN NODE IP
  serf = "<own-node-ip>:4648" # OWN NODE IP
}

acl {
  enabled = false
}

consul {
  address = "127.0.0.1:8500" # Address Consul Agent
  server_service_name = "nomad"
  client_service_name = "nomad-client"
  auto_advertise = true
  server_auto_join = true
  client_auto_join = true

  # Workload Identity
  service_auth_method = "nomad-workloads"
  task_auth_method    = "nomad-workloads"
  service_identity {
    aud = ["consul.io"]
    ttl = "1h"
  }
  task_identity {
    aud = ["consul.io"]
    ttl = "1h"
  }
}

ui {
  enabled = true
  label {
    text              = "<custom-text>"
    background_color  = "blue"
    text_color        = "white"
  }
}

telemetry {
  prometheus_metrics         = true
  publish_allocation_metrics = true
  publish_node_metrics       = true
  disable_hostname           = true
}

Then the consul-token.hcl stored in the same folder as nomad.hcl at /etc/nomad.d/ which is nomad config path (ExecStart=/usr/bin/nomad agent -config /etc/nomad.d). nomad loads all .hcl files in the directory and merges them:

$ sudo cat /etc/nomad.d/consul-token.hcl
consul {
  token = "<nomad-client-or-nomad-server-consul-token>"
}

The consul.hcl is the following:

# Copyright (c) HashiCorp, Inc.
# SPDX-License-Identifier: BUSL-1.1

datacenter = "dc1"
server = true
bootstrap_expect=1

rejoin_after_leave=true
server_rejoin_age_max = "8760h" # This setting needs to be revisited

connect {
  enabled = true
  ca_provider = "consul"
  ca_config {
    csr_max_concurrent = 1
    leaf_cert_ttl = "72h"
    intermediate_cert_ttl = "8760h"
    root_cert_ttl = "87600h"
    private_key_type = "ec"
    private_key_bits = 256
  }
}

ports {
  grpc = 8502
  grpc_tls = -1
}

data_dir = "/opt/consul"

client_addr = "0.0.0.0"
bind_addr = "0.0.0.0"

advertise_addr = "<own-node-ip>" # OWN NODE IP
retry_join=["<server-ip>"] # SERVER IPs

ui_config {
  enabled = true
}

telemetry {
  prometheus_retention_time = "30s"
  disable_hostname          = true
}

acl {
  enabled                  = true
  default_policy           = "deny"
  enable_token_persistence = true
}

The consul auth method so nomad can send a JWT that gets validated by consul, and the binding rule so consul knows what service identity to assign from the token:

resource "consul_acl_auth_method" "nomad_workloads" {
  name = "nomad-workloads"
  type = "jwt"
  config_json = jsonencode({
    JWKSURL          = "http://127.0.0.1:4646/.well-known/jwks.json"
    JWTSupportedAlgs = ["RS256"]
    BoundAudiences   = ["consul.io"]
    ClaimMappings = {
      nomad_namespace = "nomad_namespace"
      nomad_job_id    = "nomad_job_id"
      nomad_task      = "nomad_task"
      nomad_service   = "nomad_service"
    }
  })
}

resource "consul_acl_binding_rule" "nomad_services" {
  auth_method = consul_acl_auth_method.nomad_workloads.name
  bind_type   = "service"
  bind_name   = "${value.nomad_service}"
  selector    = "\"nomad_service\" in value"
}

consul has a global default-deny intention (* -> * = deny) with explicit allow intentions per service pair. The anonymous token has no permissions as can be seen in the following image:

Other then that i have these tokens:

1. dns-requests - read-only policy used as the fallback token for unauthenticated requests to the local consul agent, like DNS queries on port 8600. Without it, DNS returns empty in default-deny mode.
2. nomad-client - lets nomad client nodes read the catalog and register their node with consul
3. nomad-server - lets nomad register and deregister services and health checks in the consul catalog
4. consul-agent-server - lets the consul agent register itself as a node and sync its own state

nomad runs as a systemd service with ordering dependencies so it starts after consul and Docker.

# /etc/systemd/system/nomad.service.d/override.conf
[Unit]
Requires=network-online.target
After=network-online.target

Requires=docker.service
After=docker.service

Requires=consul.service
After=consul.service

Docker is using its default configuration.

Initial state (everything working)

19 jobs are deployed in a single namespace. All of but one them are type = "service" running Docker containers with consul Connect sidecars (connect { sidecar_service {} }). Only my web app runs without sidecars on a network_mode = "host". Each has a reschedule { unlimited = true } block.

Before the reboot, everything is healthy. Each allocation Envoy sidecar has a valid consul token derived through the WI auth method. I confirmed this by listing tokens:

consul acl token list -format=json -token=$BOOTSTRAP_TOKEN

consul UI shows the generated tokens:

And by checking a sidecar bootstrap config, used by Envoy to authenticate with the consul xDS API:

grep -A1 "x-consul-token" /opt/nomad/data/alloc/<alloc-id>/<connect-proxy-task>/secrets/envoy_bootstrap.json
# "key": "x-consul-token",
# "value": "acfbebe6-xxxx-xxxx-xxxx-xxxxxxxxxxxx"

Reproduction

I tried two things. Doing a shutdown of the Azure VM through the azure UI and starting it again, and also just doing a restart of the Azure VMs. After it comes back up, consul and Docker start, then nomad starts and recovers its allocations from the persisted client state in /opt/nomad/data/client/state.

So nomad reattaches to the existing containers through its allocation recovery. The allocations transition back to running without going through the full lifecycle. Although they show as healthy the allocation history shows a different story.

The Envoy sidecars are now running but their consul token is empty. I can see this in the sidecar logs:

DeltaAggregatedResources gRPC config stream to local_agent closed: 7, Permission denied: anonymous token lacks permission 'service:write' on "<my-service>". The anonymous token is used implicitly when a request does not specify a token.{via_upstream}
[2026-04-21 10:36:19.241][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:188] DeltaAggregatedResources gRPC config stream to local_agent closed: 7, Permission denied: anonymous token lacks permission 'service:write' on "<my-service>". The anonymous token is used implicitly when a request does not specify a token.{via_upstream}
[2026-04-21 10:36:19.975][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:188] DeltaAggregatedResources gRPC config stream to local_agent closed: 7, Permission denied: anonymous token lacks permission 'service:write' on "<my-service>". The anonymous token is used implicitly when a request does not specify a token.{via_upstream}
[2026-04-21 10:36:21.691][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:188] DeltaAggregatedResources gRPC config stream to local_agent closed: 7, Permission denied: anonymous token lacks permission 'service:write' on "<my-service>". The anonymous token is used implicitly when a request does not specify a token.{via_upstream}
[2026-04-21 10:36:23.120][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:188] DeltaAggregatedResources gRPC config stream to local_agent closed: 7, Permission denied: anonymous token lacks permission 'service:write' on "<my-service>". The anonymous token is used implicitly when a request does not specify a token.{via_upstream}

Checking the bootstrap config confirms the token is empty:

grep -A1 "x-consul-token" /opt/nomad/data/alloc/<alloc-id>/<connect-proxy-task>/secrets/envoy_bootstrap.json
# "key": "x-consul-token",
# "value": ""

The old WI tokens still exist in consul (due to enable_token_persistence = true), but they're orphaned.

I also tested with drain before shutdown

To address the question about containers surviving: I also tested with nomad node drain -enable -self -yes before shutting down (via a systemd ExecStop hook). But behaviour was the same.

Running nomad job restart -reschedule manually after each reboot fixes the issue and all tokens are derived but takes too much time.

for job in $(nomad job status -namespace=$NS | tail -n +2 | awk '{print $1}'); do
   nomad job restart -namespace=$NS -reschedule "$job"
 done

The other approach i found to fix this was creating a new systemd service that deletes the allocations and state directories data, which forces nomad to actually create new allocations.

[Unit]
Description=Clean Nomad client state for fresh start
Before=nomad.service

[Service]
Type=oneshot
ExecStart=/bin/bash -c 'rm -rf /opt/nomad/data/client/state && rm -rf /opt/nomad/data/alloc'
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Summary

Recovered allocations (reattached containers) never re-derive their consul tokens through consul login. The hook responsible for this is skipped during recovery, resulting in an empty x-consul-token in the Envoy bootstrap config.

Running nomad job restart -reschedule after the system is up resolves it, confirming the WI infrastructure itself is correct.