Skip to content

storage-detaching hook fails on Juju 3.5 due to secrets creation for dying units #243

@BON4

Description

@BON4

Steps to reproduce

  1. Clone the source code for cassandra-operator.
    If this issue is reviewed after a temporary bug fix on the operator side, remove the following lines from src/core/state.py:
        if (self.model.juju_version <= JujuVersion("3.5.7")) and  os.getenv("JUJU_HOOK_NAME", "") == f"{DATA_STORAGE}-storage-detaching":
            unit_additional_secret_fields = None
  1. Build and deploy 2 units:
juju deploy ./cassandra_ubuntu@24.04-amd64.charm cassandra --config profile=testing --num-units=2
  1. Remove a secondary unit:
juju remove-unit <secondary-unit>
  1. Wait until the storage-detaching hook completes on the departing unit.

  2. Observe that the hook fails on any remaining unit with the error:

cannot apply changes: creating secrets: cannot create secret for owner "<secondary-unit>" which is not alive

Expected behavior

  • After a unit begins removal (life = dying), no new secrets should be created on its behalf, neither by the departing unit nor by surviving peers.
  • DataPeerUnitData should skip secret creation for units whose life is dying or defer until the unit is alive again.
  • The unit should be removed cleanly after the storage-detaching hook completes successfully.

Actual behavior

  • DataPeerUnitData automatically attempts to create Juju secrets for all fields in additional_secret_fields.
  • Juju rejects requests for units that are not alive, causing the hook to fail and retry indefinitely, which blocks the application.

Versions

Operating system: Ubuntu 24.04.1 LTS

Juju CLI: 3.5.7-genericlinux-amd64

Juju agent: 3.5.7

Log output

unit-cassandra-1: 17:10:48 ERROR juju.worker.uniter.context cannot apply changes: creating secrets: cannot create secret for owner "unit-cassandra-1" which is not alive
unit-cassandra-1: 17:10:48 ERROR juju.worker.uniter.operation hook "data-storage-detaching" (via hook dispatching script: dispatch) failed: creating secrets: cannot create secret for owner "unit-cassandra-1" which is not alive

Additional context

  • This issue occurs only on Juju v3.5; Juju v3.6 does not have this problem.
  • The issue can be worked around by removing additional_secret_fields from DataPeerUnitData during unit removal:
        if (self.model.juju_version <= JujuVersion("3.5.7")) and  os.getenv("JUJU_HOOK_NAME", "") == f"{DATA_STORAGE}-storage-detaching":
            unit_additional_secret_fields = None

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions