sagas hang after local disk deletion from expunged disk

[askfongjojo]

We've tested local disk deletion after expunging disks on racklets and it appears to be working just fine since the disk is logically deleted in the database and disk accounting shows reduced utilization. However, the disk-delete saga actually never entirely completes as it is unable to delete the local volume backend. The saga could stay running indefinitely until the nexus zone is gone and doesn't really affect anything else - till we need to quiesce it. And we have such a situation today on rack2. Unlike racklets, the system is not wiped clean and is mostly upgraded via nexus-driven update so we only realize this blocking situation for the first time.

I saw that final nexus handoff step in online update was not making progress:

root@oxz_switch0:~# omdb nexus background-tasks show blueprint_planner 2>/dev/null
task: "blueprint_planner"
  configured period: every 1m
  currently executing: no
  last completed activation: iter 1162, triggered by a dependent task completing
    started at 2026-04-04T01:18:19.362Z (36s ago) and ran for 687ms
    plan unchanged from parent 29fe25f9-db93-4c3e-9e34-dbe2564758a5
    note: 3255/5000 blueprints in database
planning report:
* 3 remaining out-of-date zones
* 3 zones waiting to be expunged:
  * zone 03ee5ea0-a003-4ff3-9125-bf54d41b1868: nexus image out-of-date, but nexus_generation 46 is still active
  * zone 84be6867-c3b1-4f54-92c8-1ba3390a9ff7: nexus image out-of-date, but nexus_generation 46 is still active
  * zone f0d6e08d-0cdb-4f83-8f50-6115f2ebfb84: nexus image out-of-date, but nexus_generation 46 is still active
* will ensure cockroachdb setting: "22.1"

and there were warnings about some nexus instances not drained

01:22:35.803Z WARN 03ee5ea0-a003-4ff3-9125-bf54d41b1868 (ServerContext): not yet quiesced
    error = at least one Nexus instance is not drained as of blueprint Some(29fe25f9-db93-4c3e-9e34-dbe2564758a5 (blueprint)): Nexus 84be6867-c3b1-4f54-92c8-1ba3390a9ff7
    file = nexus/src/app/quiesce.rs:193

@jgallagher found out that:

root@oxz_switch0:~# OMDB_NEXUS_URL=http://[fd00:1122:3344:108::6e]:12232 omdb nexus quiesce show
note: using Nexus URL http://[fd00:1122:3344:108::6e]:12232
quiescing since 2026-04-03T23:20:51.574Z (2h 7m 42s 219ms ago)
details: waiting for running sagas to finish
saga quiesce:
    new sagas: DisallowedQuiesce
    drained as of blueprint: none
    blueprint for last completed recovery pass: 29fe25f9-db93-4c3e-9e34-dbe2564758a5
    blueprint for last reassignment pass: 29fe25f9-db93-4c3e-9e34-dbe2564758a5
    reassignment generation: 1 (pass running: no)
    recovered generation: 1
    recovered at least once successfully: yes
    recovery pending: no
    sagas running: 1
        saga 16933790-0e7c-402a-bb4e-7f95b662023d pending since 2026-04-03T20:45:16.673Z (disk-delete)

This is the current state of the saga:

oot@oxz_switch0:~# omdb db saga show 16933790-0e7c-402a-bb4e-7f95b662023d 2>/dev/null
 id                                   | current_sec                          | time_created             | name        | state   
--------------------------------------+--------------------------------------+--------------------------+-------------+---------
 16933790-0e7c-402a-bb4e-7f95b662023d | 84be6867-c3b1-4f54-92c8-1ba3390a9ff7 | 2026-04-01T21:08:26.353Z | disk-delete | Running 

DAG: {"end_node":5,"graph":{"edge_property":"directed","edges":[[0,1,null],[1,2,null],[2,3,null],[4,0,null],[3,5,null]],"node_holes":[],"nodes":[{"Action":{"action_name":"disk_delete.delete_disk_record","label":"DeleteDiskRecord","name":"deleted_disk"}},{"Action":{"action_name":"disk_delete.space_account","label":"SpaceAccount","name":"no_result1"}},{"Action":{"action_name":"disk_delete.delete_local_storage","label":"DeleteLocalStorage","name":"delete_local_storage"}},{"Action":{"action_name":"disk_delete.deallocate_local_storage","label":"DeallocateLocalStorage","name":"deallocate_local_storage"}},{"Start":{"params":{"disk":{"LocalStorage":{"disk":{"attach_instance_id":null,"block_size":"AdvancedFormat","disk_state":"detached","disk_type":"LocalStorage","identity":{"description":"application ephemeral data disk","id":"ce87e9f1-20c3-4ba8-bcae-a97386854284","name":"app-data2-17","time_created":"2026-04-01T18:22:33.224012Z","time_deleted":null,"time_modified":"2026-04-01T18:22:33.224012Z"},"project_id":"fe0da422-5c48-4b52-8010-f2fc401f090f","rcgen":1,"size":53687091200,"slot":null,"state_generation":2,"time_state_updated":"2026-04-01T18:22:33.326067Z"},"disk_type_local_storage":{"disk_id":"ce87e9f1-20c3-4ba8-bcae-a97386854284","local_storage_dataset_allocation_id":null,"local_storage_unencrypted_dataset_allocation_id":"f6f9b6a1-3322-48af-b74a-e00b49d33eeb","required_dataset_overhead":3670016000},"local_storage_dataset_allocation":{"Unencrypted":{"dataset_size":57357107200,"id":"f6f9b6a1-3322-48af-b74a-e00b49d33eeb","local_storage_unencrypted_dataset_id":"d0f6c1a2-fbee-4a39-b861-3fdce10e475a","pool_id":"f522118c-5dcd-4116-8044-07f0cceec52e","sled_id":"87c2c4fc-b0c7-4fef-a305-78f0ed265bbc","time_created":"2026-04-01T18:23:07.637294Z","time_deleted":null}}}},"project_id":"fe0da422-5c48-4b52-8010-f2fc401f090f","serialized_authn":{"kind":{"Authenticated":[{"actor":{"SiloUser":{"silo_id":"7bd7623a-68ed-4636-8ecb-b59e3b068787","silo_user_id":"906e74cb-eab8-4a87-bda8-2cb0914bf853"}},"credential_id":"a84b123a-cad9-4c49-8d89-e67a0356d843","device_token_expiration":null},{"mapped_fleet_roles":{"admin":["admin"]}}]}}}}},"End"]},"saga_name":"disk-delete","start_node":4}

              event time | sub saga | node id                               | event type | data
------------------------ | -------- | ------------------------------------- | ---------- | ---
2026-04-01T21:08:26.357Z |          |                              4: start | started    | 
2026-04-01T21:08:26.362Z |          |                              4: start | succeeded  | 
2026-04-01T21:08:26.366Z |          |     0: disk_delete.delete_disk_record | started    | 
2026-04-01T21:08:26.374Z |          |     0: disk_delete.delete_disk_record | succeeded  | "deleted_disk" => {"attach_instance_id":null,"block_size":"AdvancedFormat","disk_state":"detached","disk_type":"LocalStorage","identity":{"description":"application ephemeral data disk","id":"ce87e9f1-20c3-4ba8-bcae-a97386854284","name":"app-data2-17","time_created":"2026-04-01T18:22:33.224012Z","time_deleted":null,"time_modified":"2026-04-01T18:22:33.224012Z"},"project_id":"fe0da422-5c48-4b52-8010-f2fc401f090f","rcgen":1,"size":53687091200,"slot":null,"state_generation":2,"time_state_updated":"2026-04-01T18:22:33.326067Z"}
2026-04-01T21:08:26.376Z |          |          1: disk_delete.space_account | started    | 
2026-04-01T21:08:26.396Z |          |          1: disk_delete.space_account | succeeded  | 
2026-04-01T21:08:26.400Z |          |   2: disk_delete.delete_local_storage | started    | 

[askfongjojo]

It turns out that there is a different saga on a different nexus node also stuck on using a local disk:

root@oxz_switch1:~# OMDB_NEXUS_URL=http://[fd00:1122:3344:101::7c]:12232 omdb nexus quiesce show
note: using Nexus URL http://[fd00:1122:3344:101::7c]:12232
quiescing since 2026-04-03T23:20:47.129Z (5h 45m 32s 666ms ago)
details: waiting for running sagas to finish
saga quiesce:
    new sagas: DisallowedQuiesce
    drained as of blueprint: none
    blueprint for last completed recovery pass: 29fe25f9-db93-4c3e-9e34-dbe2564758a5
    blueprint for last reassignment pass: 29fe25f9-db93-4c3e-9e34-dbe2564758a5
    reassignment generation: 1 (pass running: no)
    recovered generation: 1
    recovered at least once successfully: yes
    recovery pending: no
    sagas running: 1
        saga 81ac6e59-a575-4830-a2e5-82d6f82cd0d4 pending since 2026-04-03T21:13:35.893Z (instance-start)
database connections held: 2
    claim 1819703 held since 2026-04-04 05:06:14.872514991 UTC (4s 927ms ago)
    claim 1819760 held since 2026-04-04 05:06:19.725006077 UTC (74ms ago)

root@oxz_switch1:~# omdb db saga show 81ac6e59-a575-4830-a2e5-82d6f82cd0d4 2>/dev/null
 id                                   | current_sec                          | time_created             | name           | state   
--------------------------------------+--------------------------------------+--------------------------+----------------+---------
 81ac6e59-a575-4830-a2e5-82d6f82cd0d4 | f0d6e08d-0cdb-4f83-8f50-6115f2ebfb84 | 2026-04-03T20:10:50.855Z | instance-start | Running 

DAG: {"end_node":25,"graph":{"edge_property":"directed","edges":[[0,1,null],[1,2,null],[2,3,null],[3,4,null],[4,5,null],[5,6,null],[5,7,null],[5,8,null],[5,9,null],[5,10,null],[5,11,null],[5,12,null],[5,13,null],[5,14,null],[5,15,null],[5,16,null],[5,17,null],[6,18,null],[7,18,null],[8,18,null],[9,18,null],[10,18,null],[11,18,null],[12,18,null],[13,18,null],[14,18,null],[15,18,null],[16,18,null],[17,18,null],[18,19,null],[19,20,null],[20,21,null],[21,22,null],[22,23,null],[24,0,null],[23,25,null]],"node_holes":[],"nodes":[{"Action":{"action_name":"instance_start.generate_propolis_id","label":"GeneratePropolisId","name":"propolis_id"}},{"Action":{"action_name":"instance_start.alloc_server","label":"AllocServer","name":"sled_id"}},{"Action":{"action_name":"instance_start.alloc_propolis_ip","label":"AllocPropolisIp","name":"propolis_ip"}},{"Action":{"action_name":"instance_start.create_vmm_record","label":"CreateVmmRecord","name":"vmm_record"}},{"Action":{"action_name":"instance_start.mark_as_starting","label":"MarkAsStarting","name":"started_record"}},{"Action":{"action_name":"instance_start.list_local_storage","label":"ListLocalStorage","name":"local_storage_records"}},{"Action":{"action_name":"instance_start.ensure_local_storage_0","label":"EnsureLocalStorage_0","name":"ensure_local_storage_0"}},{"Action":{"action_name":"instance_start.ensure_local_storage_1","label":"EnsureLocalStorage_1","name":"ensure_local_storage_1"}},{"Action":{"action_name":"instance_start.ensure_local_storage_2","label":"EnsureLocalStorage_2","name":"ensure_local_storage_2"}},{"Action":{"action_name":"instance_start.ensure_local_storage_3","label":"EnsureLocalStorage_3","name":"ensure_local_storage_3"}},{"Action":{"action_name":"instance_start.ensure_local_storage_4","label":"EnsureLocalStorage_4","name":"ensure_local_storage_4"}},{"Action":{"action_name":"instance_start.ensure_local_storage_5","label":"EnsureLocalStorage_5","name":"ensure_local_storage_5"}},{"Action":{"action_name":"instance_start.ensure_local_storage_6","label":"EnsureLocalStorage_6","name":"ensure_local_storage_6"}},{"Action":{"action_name":"instance_start.ensure_local_storage_7","label":"EnsureLocalStorage_7","name":"ensure_local_storage_7"}},{"Action":{"action_name":"instance_start.ensure_local_storage_8","label":"EnsureLocalStorage_8","name":"ensure_local_storage_8"}},{"Action":{"action_name":"instance_start.ensure_local_storage_9","label":"EnsureLocalStorage_9","name":"ensure_local_storage_9"}},{"Action":{"action_name":"instance_start.ensure_local_storage_10","label":"EnsureLocalStorage_10","name":"ensure_local_storage_10"}},{"Action":{"action_name":"instance_start.ensure_local_storage_11","label":"EnsureLocalStorage_11","name":"ensure_local_storage_11"}},{"Action":{"action_name":"instance_start.dpd_ensure","label":"DpdEnsure","name":"dpd_ensure"}},{"Action":{"action_name":"instance_start.v2p_ensure","label":"V2PEnsure","name":"v2p_ensure"}},{"Action":{"action_name":"instance_start.ensure_registered","label":"EnsureRegistered","name":"ensure_registered"}},{"Action":{"action_name":"instance_start.update_multicast_sled_id","label":"UpdateMulticastSledId","name":"multicast_sled_id"}},{"Action":{"action_name":"instance_start.add_virtual_resources","label":"AddVirtualResources","name":"virtual_resources"}},{"Action":{"action_name":"instance_start.ensure_running","label":"EnsureRunning","name":"ensure_running"}},{"Start":{"params":{"db_instance":{"auto_restart":{"cooldown":null,"policy":null},"boot_disk_id":null,"cpu_platform":null,"dst_propolis_id":null,"hostname":"primary","identity":{"description":"mongodb cluster","id":"9b38ecba-5408-4538-b3ae-f87c13a3c982","name":"mongodb-xfs-bs4096-primary","time_created":"2026-02-03T16:40:03.370124Z","time_deleted":null,"time_modified":"2026-02-03T16:40:03.370124Z"},"intended_state":"Running","memory":8589934592,"migration_id":null,"ncpus":4,"nexus_state":"Failed","project_id":"5e49b6de-cb2d-438d-83af-95c415bbb901","propolis_id":null,"state_generation":70,"time_last_auto_restarted":"2026-04-03T16:02:49.386264Z","time_state_updated":"2026-04-03T20:09:46.836309Z","updater_gen":2881,"updater_id":null,"user_data":[35,99,108,111,117,100,45,99,111,110,102,105,103,10,109,97,110,97,103,101,95,101,116,99,95,104,111,115,116,115,58,32,116,114,117,101,10,115,115,104,95,32,112,119,97,117,116,104,58,32,49,10,99,104,112,97,115,115,119,100,58,10,32,32,108,105,115,116,58,10,32,32,32,32,45,32,117,98,117,110,116,117,58,111,120,105,100,101,10,32,32,101,120,112,105,114,101,58,32,102,97,108,115,101,10,114,117,110,99,109,100,58,10,45,32,39,115,101,100,32,45,105,32,34,115,47,45,32,117,112,100,97,116,101,95,101,116,99,95,104,111,115,116,115,47,35,32,38,47,34,32,47,101,116,99,47,99,108,111,117,100,47,99,108,111,117,100,46,99,102,103,39,10,45,32,39,99,117,114,108,32,104,116,116,112,115,58,47,47,103,105,115,116,46,103,105,116,104,117,98,117,115,101,114,99,111,110,116,101,110,116,46,99,111,109,47,97,115,107,102,111,110,103,106,111,106,111,47,49,56,49,102,99,49,100,49,102,52,97,98,56,100,100,48,55,100,49,99,51,53,55,54,97,49,101,54,51,102,49,54,47,114,97,119,47,50,99,50,50,56,56,101,52,102,98,50,53,50,99,99,50,101,48,48,100,53,102,100,53,53,97,50,52,99,56,101,51,54,99,98,98,49,55,49,49,47,109,111,110,103,111,100,98,45,105,110,115,116,97,108,108,45,99,111,110,102,105,103,32,124,32,47,98,105,110,47,98,97,115,104,32,45,115,32,120,102,115,39,10,45,32,91,32,115,104,44,32,45,99,44,32,34,115,108,101,101,112,32,49,50,48,34,32,93,10,45,32,39,99,117,114,108,32,104,116,116,112,115,58,47,47,103,105,115,116,46,103,105,116,104,117,98,117,115,101,114,99,111,110,116,101,110,116,46,99,111,109,47,97,115,107,102,111,110,103,106,111,106,111,47,49,56,49,102,99,49,100,49,102,52,97,98,56,100,100,48,55,100,49,99,51,53,55,54,97,49,101,54,51,102,49,54,47,114,97,119,47,50,99,50,50,56,56,101,52,102,98,50,53,50,99,99,50,101,48,48,100,53,102,100,53,53,97,50,52,99,56,101,51,54,99,98,98,49,55,49,49,47,109,111,110,103,111,100,98,45,99,114,101,97,116,101,45,99,108,117,115,116,101,114,32,124,32,47,98,105,110,47,98,97,115,104,32,45,115,32,104,116,116,112,115,58,47,47,111,120,105,100,101,46,115,121,115,46,114,97,99,107,50,46,101,110,103,46,111,120,105,100,101,46,99,111,109,112,117,116,101,114,32,111,120,105,100,101,45,116,111,107,101,110,45,100,98,100,52,102,56,98,57,102,52,52,56,50,98,52,99,98,97,51,51,56,101,51,53,48,52,55,55,99,101,100,102,51,51,53,53,99,98,57,99,32,120,102,115,32,52,48,57,54,32,116,114,121,39,10,45,32,101,99,104,111,32,39,42,42,42,42,42,42,42,42,42,42,32,99,117,115,116,111,109,32,115,101,116,117,112,32,99,111,109,112,108,101,116,101,100,32,42,42,42,42,42,42,42,42,42,42,39,10]},"reason":"AutoRestart","serialized_authn":{"kind":{"Authenticated":[{"actor":{"UserBuiltin":{"user_builtin_id":"001de000-05e4-4000-8000-000000000002"}},"credential_id":null,"device_token_expiration":null},null]}}}}},"End"]},"saga_name":"instance-start","start_node":24}

              event time | sub saga | node id                                     | event type | data
------------------------ | -------- | ------------------------------------------- | ---------- | ---
2026-04-03T20:10:50.917Z |          |                                   24: start | started    | 
2026-04-03T20:10:50.923Z |          |                                   24: start | succeeded  | 
2026-04-03T20:10:50.927Z |          |      0: instance_start.generate_propolis_id | started    | 
2026-04-03T20:10:50.931Z |          |      0: instance_start.generate_propolis_id | succeeded  | "propolis_id" => "37d25ab9-e18b-4e99-9c3d-eae4e170745c"
2026-04-03T20:10:50.934Z |          |              1: instance_start.alloc_server | started    | 
2026-04-03T20:10:50.979Z |          |              1: instance_start.alloc_server | succeeded  | "sled_id" => "87c2c4fc-b0c7-4fef-a305-78f0ed265bbc"
2026-04-03T20:10:50.989Z |          |         2: instance_start.alloc_propolis_ip | started    | 
2026-04-03T20:10:50.999Z |          |         2: instance_start.alloc_propolis_ip | succeeded  | "propolis_ip" => "fd00:1122:3344:108::1:13e2"
2026-04-03T20:10:51.002Z |          |         3: instance_start.create_vmm_record | started    | 
2026-04-03T20:10:51.070Z |          |         3: instance_start.create_vmm_record | succeeded  | "vmm_record" => {"cpu_platform":"AmdMilan","gen":1,"id":"37d25ab9-e18b-4e99-9c3d-eae4e170745c","instance_id":"9b38ecba-5408-4538-b3ae-f87c13a3c982","propolis_ip":"fd00:1122:3344:108::1:13e2/128","propolis_port":12400,"sled_id":"87c2c4fc-b0c7-4fef-a305-78f0ed265bbc","state":"Creating","time_created":"2026-04-03T20:10:51.015579Z","time_deleted":null,"time_state_updated":"2026-04-03T20:10:51.015579Z"}
2026-04-03T20:10:51.075Z |          |          4: instance_start.mark_as_starting | started    | 
2026-04-03T20:10:51.182Z |          |          4: instance_start.mark_as_starting | succeeded  | "started_record" => {"auto_restart":{"cooldown":null,"policy":null},"boot_disk_id":null,"cpu_platform":null,"dst_propolis_id":null,"hostname":"primary","identity":{"description":"mongodb cluster","id":"9b38ecba-5408-4538-b3ae-f87c13a3c982","name":"mongodb-xfs-bs4096-primary","time_created":"2026-02-03T16:40:03.370124Z","time_deleted":null,"time_modified":"2026-02-03T16:40:03.370124Z"},"intended_state":"Running","memory":8589934592,"migration_id":null,"ncpus":4,"nexus_state":"Vmm","project_id":"5e49b6de-cb2d-438d-83af-95c415bbb901","propolis_id":"37d25ab9-e18b-4e99-9c3d-eae4e170745c","state_generation":71,"time_last_auto_restarted":"2026-04-03T20:10:51.118017018Z","time_state_updated":"2026-04-03T20:09:46.836309Z","updater_gen":2881,"updater_id":null,"user_data":[35,99,108,111,117,100,45,99,111,110,102,105,103,10,109,97,110,97,103,101,95,101,116,99,95,104,111,115,116,115,58,32,116,114,117,101,10,115,115,104,95,32,112,119,97,117,116,104,58,32,49,10,99,104,112,97,115,115,119,100,58,10,32,32,108,105,115,116,58,10,32,32,32,32,45,32,117,98,117,110,116,117,58,111,120,105,100,101,10,32,32,101,120,112,105,114,101,58,32,102,97,108,115,101,10,114,117,110,99,109,100,58,10,45,32,39,115,101,100,32,45,105,32,34,115,47,45,32,117,112,100,97,116,101,95,101,116,99,95,104,111,115,116,115,47,35,32,38,47,34,32,47,101,116,99,47,99,108,111,117,100,47,99,108,111,117,100,46,99,102,103,39,10,45,32,39,99,117,114,108,32,104,116,116,112,115,58,47,47,103,105,115,116,46,103,105,116,104,117,98,117,115,101,114,99,111,110,116,101,110,116,46,99,111,109,47,97,115,107,102,111,110,103,106,111,106,111,47,49,56,49,102,99,49,100,49,102,52,97,98,56,100,100,48,55,100,49,99,51,53,55,54,97,49,101,54,51,102,49,54,47,114,97,119,47,50,99,50,50,56,56,101,52,102,98,50,53,50,99,99,50,101,48,48,100,53,102,100,53,53,97,50,52,99,56,101,51,54,99,98,98,49,55,49,49,47,109,111,110,103,111,100,98,45,105,110,115,116,97,108,108,45,99,111,110,102,105,103,32,124,32,47,98,105,110,47,98,97,115,104,32,45,115,32,120,102,115,39,10,45,32,91,32,115,104,44,32,45,99,44,32,34,115,108,101,101,112,32,49,50,48,34,32,93,10,45,32,39,99,117,114,108,32,104,116,116,112,115,58,47,47,103,105,115,116,46,103,105,116,104,117,98,117,115,101,114,99,111,110,116,101,110,116,46,99,111,109,47,97,115,107,102,111,110,103,106,111,106,111,47,49,56,49,102,99,49,100,49,102,52,97,98,56,100,100,48,55,100,49,99,51,53,55,54,97,49,101,54,51,102,49,54,47,114,97,119,47,50,99,50,50,56,56,101,52,102,98,50,53,50,99,99,50,101,48,48,100,53,102,100,53,53,97,50,52,99,56,101,51,54,99,98,98,49,55,49,49,47,109,111,110,103,111,100,98,45,99,114,101,97,116,101,45,99,108,117,115,116,101,114,32,124,32,47,98,105,110,47,98,97,115,104,32,45,115,32,104,116,116,112,115,58,47,47,111,120,105,100,101,46,115,121,115,46,114,97,99,107,50,46,101,110,103,46,111,120,105,100,101,46,99,111,109,112,117,116,101,114,32,111,120,105,100,101,45,116,111,107,101,110,45,100,98,100,52,102,56,98,57,102,52,52,56,50,98,52,99,98,97,51,51,56,101,51,53,48,52,55,55,99,101,100,102,51,51,53,53,99,98,57,99,32,120,102,115,32,52,48,57,54,32,116,114,121,39,10,45,32,101,99,104,111,32,39,42,42,42,42,42,42,42,42,42,42,32,99,117,115,116,111,109,32,115,101,116,117,112,32,99,111,109,112,108,101,116,101,100,32,42,42,42,42,42,42,42,42,42,42,39,10]}
2026-04-03T20:10:51.188Z |          |        5: instance_start.list_local_storage | started    | 
2026-04-03T20:10:51.234Z |          |        5: instance_start.list_local_storage | succeeded  | "local_storage_records" => [{"disk":{"attach_instance_id":"9b38ecba-5408-4538-b3ae-f87c13a3c982","block_size":"AdvancedFormat","disk_state":"attached","disk_type":"LocalStorage","identity":{"description":"empty volume for data","id":"7c96e660-dec7-4093-b240-fd61272d683a","name":"dbdata-xfs-bs4096-primary","time_created":"2026-02-03T16:39:36.038302Z","time_deleted":null,"time_modified":"2026-02-03T16:39:36.038302Z"},"project_id":"5e49b6de-cb2d-438d-83af-95c415bbb901","rcgen":1,"size":85899345920,"slot":1,"state_generation":2,"time_state_updated":"2026-02-03T16:39:36.201271Z"},"disk_type_local_storage":{"disk_id":"7c96e660-dec7-4093-b240-fd61272d683a","local_storage_dataset_allocation_id":null,"local_storage_unencrypted_dataset_allocation_id":"45c1619d-b0a0-47d4-a8f0-939786a4c2bc","required_dataset_overhead":5872025600},"local_storage_dataset_allocation":{"Unencrypted":{"dataset_size":91771371520,"id":"45c1619d-b0a0-47d4-a8f0-939786a4c2bc","local_storage_unencrypted_dataset_id":"d0f6c1a2-fbee-4a39-b861-3fdce10e475a","pool_id":"f522118c-5dcd-4116-8044-07f0cceec52e","sled_id":"87c2c4fc-b0c7-4fef-a305-78f0ed265bbc","time_created":"2026-02-03T16:40:07.415191Z","time_deleted":null}}}]
2026-04-03T20:10:51.238Z |          |   15: instance_start.ensure_local_storage_9 | started    | 
2026-04-03T20:10:51.242Z |          |  17: instance_start.ensure_local_storage_11 | started    | 
2026-04-03T20:10:51.246Z |          |   15: instance_start.ensure_local_storage_9 | succeeded  | 
2026-04-03T20:10:51.249Z |          |  17: instance_start.ensure_local_storage_11 | succeeded  | 
2026-04-03T20:10:51.253Z |          |   13: instance_start.ensure_local_storage_7 | started    | 
2026-04-03T20:10:51.257Z |          |  16: instance_start.ensure_local_storage_10 | started    | 
2026-04-03T20:10:51.260Z |          |   13: instance_start.ensure_local_storage_7 | succeeded  | 
2026-04-03T20:10:51.264Z |          |  16: instance_start.ensure_local_storage_10 | succeeded  | 
2026-04-03T20:10:51.267Z |          |   12: instance_start.ensure_local_storage_6 | started    | 
2026-04-03T20:10:51.271Z |          |   11: instance_start.ensure_local_storage_5 | started    | 
2026-04-03T20:10:51.274Z |          |   12: instance_start.ensure_local_storage_6 | succeeded  | 
2026-04-03T20:10:51.277Z |          |   11: instance_start.ensure_local_storage_5 | succeeded  | 
2026-04-03T20:10:51.281Z |          |   10: instance_start.ensure_local_storage_4 | started    | 
2026-04-03T20:10:51.285Z |          |    9: instance_start.ensure_local_storage_3 | started    | 
2026-04-03T20:10:51.288Z |          |   10: instance_start.ensure_local_storage_4 | succeeded  | 
2026-04-03T20:10:51.291Z |          |    9: instance_start.ensure_local_storage_3 | succeeded  | 
2026-04-03T20:10:51.295Z |          |    8: instance_start.ensure_local_storage_2 | started    | 
2026-04-03T20:10:51.299Z |          |    7: instance_start.ensure_local_storage_1 | started    | 
2026-04-03T20:10:51.301Z |          |    8: instance_start.ensure_local_storage_2 | succeeded  | 
2026-04-03T20:10:51.305Z |          |    7: instance_start.ensure_local_storage_1 | succeeded  | 
2026-04-03T20:10:51.308Z |          |   14: instance_start.ensure_local_storage_8 | started    | 
2026-04-03T20:10:51.312Z |          |    6: instance_start.ensure_local_storage_0 | started    | 
2026-04-03T20:10:51.314Z |          |   14: instance_start.ensure_local_storage_8 | succeeded  |

[askfongjojo]

Since these sagas will never complete and we have an omdb command built for the purpose of abandoning sagas that are hitting a dead end, I went ahead with abandoning the two sagas (I had to manually diable the nexus SMF service before abandoning the saga and re-enable it afterwards - as required by the command). Nexus handoff finished afterwards for the round of online update.

I wonder if there is something different that can be done to make this clean-up less manual. Alternatively, sagas dealing with local disks that cease to respond might need to be able to terminate based on some conditions. The iocal disk associated with the abandoned instance-start saga is still there on rack2. It will result in another unfinishable disk-delete as soon as I try to delete it (blocking the next online update).

[davepacheco]

In general, using omdb saga abandon can create state that requires manual cleanup. Has someone assessed whether some cleanup is required on dogfood here?

[davepacheco]

I've changed the synopsis here to reflect the saga hang. This seems like a blocker for using physical disk expungement on systems that are also using local storage, isn't it?

[jmpesp]

In general, using omdb saga abandon can create state that requires manual cleanup. Has someone assessed whether some cleanup is required on dogfood here?

at minimum, the instance lock is probably still taken for that instance, and will need to be manually unlocked. maybe there's an omdb command for this, or maybe there should be? paging @hawkw

[jmpesp]

I've changed the synopsis here to reflect the saga hang. This seems like a blocker for using physical disk expungement on systems that are also using local storage, isn't it?

#10257 should fix both these cases (hung disk delete, and hung instance start)

[askfongjojo]

In general, using omdb saga abandon can create state that requires manual cleanup. Has someone assessed whether some cleanup is required on dogfood here?

For the disk-delete saga, I've verified that against a successful one; the failed step is the last one in the saga to remove the raw volume that no longer exists.

The VM with the stuck instance-start has since been in failed state and configured to not auto-restart. I think the instance lock has been reset (we've done 2 rounds of online update since then so any related propolis zones will also be gone)?

root@oxz_switch0:~# omdb db instance info 9b38ecba-5408-4538-b3ae-f87c13a3c982

== INSTANCE ====================================================================
                        ID: 9b38ecba-5408-4538-b3ae-f87c13a3c982
                project ID: 5e49b6de-cb2d-438d-83af-95c415bbb901
                      name: mongodb-xfs-bs4096-primary
               description: mongodb cluster
                created at: 2026-02-03 16:40:03.370124 UTC
          last modified at: 2026-04-04 05:03:52.852822 UTC

== CONFIGURATION ===============================================================
                     vCPUs: 4
                    memory: 8 GiB
                  hostname: primary
                 boot disk: None
              auto-restart:
                  InstanceAutoRestart {
                      policy: Some(
                          Never,
                      ),
                      cooldown: None,
                  }

== RUNTIME STATE ===============================================================
               nexus state: Failed
/!\     external API state: Failed
            intended state: stopped
           last updated at: 2026-04-04T05:15:20.893265Z (generation 72)
       needs reincarnation: false
             karmic status: nirvāṇa (reincarnation disabled)
      last reincarnated at: Some(2026-04-03T20:10:51.118017Z)
             active VMM ID: None
             target VMM ID: None
              migration ID: None
              updater lock: UNLOCKED at generation: 2884

== ATTACHED DISKS ==============================================================
# ID                                   SIZE   TYPE     STATE    NAME                       
1 7c96e660-dec7-4093-b240-fd61272d683a 80 GiB local    attached dbdata-xfs-bs4096-primary  
0 361c3d84-1df6-4c14-a9e3-318f86934ed0 30 GiB crucible attached mongodb-xfs-bs4096-primary