Skip to content

Re using storage from previous charmed-zookeeper installation fails #85

@zmraul

Description

@zmraul

Steps to reproduce

  1. Run zookeeper with 4 units
  2. Scale down one, keeping storage
  3. Scale back up, attaching the storage from previous unit.

Expected behavior

Zookeeper charm uses the storage

Actual behavior

New unit fails to install charmed-zookeeper

unit-zookeeper-4: 14:24:42 WARNING unit.zookeeper/4.install error: cannot perform the following tasks:                                                                                                                           
unit-zookeeper-4: 14:24:42 WARNING unit.zookeeper/4.install - Copy snap "charmed-zookeeper" data (unlinkat /var/snap/charmed-zookeeper/common/var/lib/zookeeper: device or resource busy)                                        
unit-zookeeper-4: 14:24:42 WARNING unit.zookeeper/4.install - Run install hook of "charmed-zookeeper" snap if present (run hook "install":                                                                                       
unit-zookeeper-4: 14:24:42 WARNING unit.zookeeper/4.install -----                                                                                                                                                                
unit-zookeeper-4: 14:24:42 WARNING unit.zookeeper/4.install + export COMPONENT=zookeeper                                                                                                                                         
unit-zookeeper-4: 14:24:42 WARNING unit.zookeeper/4.install + COMPONENT=zookeeper                                                                                                                                                
unit-zookeeper-4: 14:24:42 WARNING unit.zookeeper/4.install + export CONF=/var/snap/charmed-zookeeper/7/etc/zookeeper                                                                                                            
unit-zookeeper-4: 14:24:42 WARNING unit.zookeeper/4.install + CONF=/var/snap/charmed-zookeeper/7/etc/zookeeper                                                                                                                   
unit-zookeeper-4: 14:24:42 WARNING unit.zookeeper/4.install + export LOGS=/var/snap/charmed-zookeeper/common/var/log/zookeeper                                                                                                   
unit-zookeeper-4: 14:24:42 WARNING unit.zookeeper/4.install + LOGS=/var/snap/charmed-zookeeper/common/var/log/zookeeper                                                                                                          
unit-zookeeper-4: 14:24:42 WARNING unit.zookeeper/4.install + export DATA=/var/snap/charmed-zookeeper/common/var/lib/zookeeper                                                                                                   
unit-zookeeper-4: 14:24:42 WARNING unit.zookeeper/4.install + DATA=/var/snap/charmed-zookeeper/common/var/lib/zookeeper                                                                                                          
unit-zookeeper-4: 14:24:42 WARNING unit.zookeeper/4.install + mkdir -p /var/snap/charmed-zookeeper/7/etc/zookeeper                                                                                                               
unit-zookeeper-4: 14:24:42 WARNING unit.zookeeper/4.install + mkdir -p /var/snap/charmed-zookeeper/common/var/log/zookeeper                                                                                                      
unit-zookeeper-4: 14:24:42 WARNING unit.zookeeper/4.install + mkdir -p /var/snap/charmed-zookeeper/common/var/lib/zookeeper                                                                                                      
unit-zookeeper-4: 14:24:42 WARNING unit.zookeeper/4.install + cp /snap/charmed-zookeeper/7/etc/zookeeper/jmx_prometheus.yaml /snap/charmed-zookeeper/7/etc/zookeeper/log4j.properties /var/snap/charmed-zookeeper/7/etc/zookeeper
unit-zookeeper-4: 14:24:42 WARNING unit.zookeeper/4.install + chmod -R 770 /var/snap/charmed-zookeeper/7/etc                                                                                                                     
unit-zookeeper-4: 14:24:42 WARNING unit.zookeeper/4.install + chmod -R 770 /var/snap/charmed-zookeeper/common/var                                                                                                                
unit-zookeeper-4: 14:24:42 WARNING unit.zookeeper/4.install chmod: changing permissions of '/var/snap/charmed-zookeeper/common/var/lib/zookeeper': Operation not permitted                                                       
unit-zookeeper-4: 14:24:42 WARNING unit.zookeeper/4.install chmod: changing permissions of '/var/snap/charmed-zookeeper/common/var/lib/zookeeper/data': Operation not permitted                                                  
unit-zookeeper-4: 14:24:42 WARNING unit.zookeeper/4.install chmod: changing permissions of '/var/snap/charmed-zookeeper/common/var/lib/zookeeper/data/myid': Operation not permitted                                             
unit-zookeeper-4: 14:24:42 WARNING unit.zookeeper/4.install chmod: changing permissions of '/var/snap/charmed-zookeeper/common/var/lib/zookeeper/data/version-2': Operation not permitted                                        
unit-zookeeper-4: 14:24:42 WARNING unit.zookeeper/4.install chmod: changing permissions of '/var/snap/charmed-zookeeper/common/var/lib/zookeeper/data/version-2/snapshot.0': Operation not permitted                             
unit-zookeeper-4: 14:24:42 WARNING unit.zookeeper/4.install chmod: changing permissions of '/var/snap/charmed-zookeeper/common/var/lib/zookeeper/data/version-2/acceptedEpoch': Operation not permitted                          
unit-zookeeper-4: 14:24:42 WARNING unit.zookeeper/4.install chmod: changing permissions of '/var/snap/charmed-zookeeper/common/var/lib/zookeeper/data/version-2/currentEpoch': Operation not permitted 
unit-zookeeper-4: 14:24:42 WARNING unit.zookeeper/4.install chmod: changing permissions of '/var/snap/charmed-zookeeper/common/var/lib/zookeeper/data-log': Operation not permitted                    
unit-zookeeper-4: 14:24:42 WARNING unit.zookeeper/4.install chmod: changing permissions of '/var/snap/charmed-zookeeper/common/var/lib/zookeeper/data-log/version-2': Operation not permitted         
unit-zookeeper-4: 14:24:42 WARNING unit.zookeeper/4.install chmod: changing permissions of '/var/snap/charmed-zookeeper/common/var/lib/zookeeper/data-log/version-2/log.100000001': Operation not permitted                      
unit-zookeeper-4: 14:24:42 WARNING unit.zookeeper/4.install chmod: changing permissions of '/var/snap/charmed-zookeeper/common/var/lib/zookeeper/data-log/version-2/log.200000001': Operation not permitted
unit-zookeeper-4: 14:24:42 WARNING unit.zookeeper/4.install -----)                                                                                                                                         
unit-zookeeper-4: 14:24:42 ERROR unit.zookeeper/4.juju-log Snap: 'charmed-zookeeper'; command ['snap', 'install', 'charmed-zookeeper', '--revision="7"'] failed with output = ''                          
. . . # cut some logs

unit-zookeeper-4: 14:24:49 INFO unit.zookeeper/4.juju-log Running legacy hooks/start.                                                                                                                                
unit-zookeeper-4: 14:24:56 INFO unit.zookeeper/4.juju-log cluster:0: zookeeper/4 initializing...                                                                                                                     
unit-zookeeper-4: 14:24:56 ERROR unit.zookeeper/4.juju-log cluster:0: Could not ['snap', 'start', 'charmed-zookeeper.daemon'] for snap [charmed-zookeeper]: error: snap "charmed-zookeeper" has no service "daemon"  
Traceback (most recent call last):                                                                                
  File "/var/lib/juju/agents/unit-zookeeper-4/charm/lib/charms/operator_libs_linux/v1/snap.py", line 309, in _snap_daemons                                                                                           
    return subprocess.run(_cmd, universal_newlines=True, check=True, capture_output=True)                                                                                                                            
  File "/usr/lib/python3.10/subprocess.py", line 526, in run                                                      
    raise CalledProcessError(retcode, process.args,                                                               
subprocess.CalledProcessError: Command '['snap', 'start', 'charmed-zookeeper.daemon']' returned non-zero exit status 1.                                                                                              

During handling of the above exception, another exception occurred:                                                                                                                                                  

Traceback (most recent call last):                                                                                
  File "/var/lib/juju/agents/unit-zookeeper-4/charm/src/snap.py", line 60, in start_snap_service                                                                                                                     
    self.zookeeper.start(services=[self.SNAP_SERVICE])                                                            
  File "/var/lib/juju/agents/unit-zookeeper-4/charm/lib/charms/operator_libs_linux/v1/snap.py", line 347, in start
    self._snap_daemons(args, services)                                                                            
  File "/var/lib/juju/agents/unit-zookeeper-4/charm/lib/charms/operator_libs_linux/v1/snap.py", line 311, in _snap_daemons                                                                                           
    raise SnapError("Could not {} for snap [{}]: {}".format(_cmd, self._name, e.stderr))                                                                                                                             
charms.operator_libs_linux.v1.snap.SnapError: Could not ['snap', 'start', 'charmed-zookeeper.daemon'] for snap [charmed-zookeeper]: error: snap "charmed-zookeeper" has no service "daemon"                          

unit-zookeeper-4: 14:24:56 INFO unit.zookeeper/4.juju-log cluster:0: zookeeper/4 started                                                                                                                             
unit-zookeeper-4: 14:25:14 INFO unit.zookeeper/4.juju-log restart:1: zookeeper/4 restarting...                                                                                                                       
unit-zookeeper-4: 14:25:14 ERROR unit.zookeeper/4.juju-log restart:1: Could not ['snap', 'restart', 'charmed-zookeeper.daemon'] for snap [charmed-zookeeper]: error: snap "charmed-zookeeper" has no service "daemon"
Traceback (most recent call last):                                                                                
  File "/var/lib/juju/agents/unit-zookeeper-4/charm/lib/charms/operator_libs_linux/v1/snap.py", line 309, in _snap_daemons                                                                                           
    return subprocess.run(_cmd, universal_newlines=True, check=True, capture_output=True)                                                                                                                            
  File "/usr/lib/python3.10/subprocess.py", line 526, in run                                                      
    raise CalledProcessError(retcode, process.args,                                                               
subprocess.CalledProcessError: Command '['snap', 'restart', 'charmed-zookeeper.daemon']' returned non-zero exit status 1.                                                                                            

During handling of the above exception, another exception occurred:                                                                                                                                                  

Traceback (most recent call last):                                                                                
  File "/var/lib/juju/agents/unit-zookeeper-4/charm/src/snap.py", line 86, in restart_snap_service                                                                                                                   
    self.zookeeper.restart(services=[self.SNAP_SERVICE])                                                          
  File "/var/lib/juju/agents/unit-zookeeper-4/charm/lib/charms/operator_libs_linux/v1/snap.py", line 424, in restart                                                                                                 
    self._snap_daemons(args, services)                                                                            
  File "/var/lib/juju/agents/unit-zookeeper-4/charm/lib/charms/operator_libs_linux/v1/snap.py", line 311, in _snap_daemons                                                                                           
    raise SnapError("Could not {} for snap [{}]: {}".format(_cmd, self._name, e.stderr))                                                                                                                             
charms.operator_libs_linux.v1.snap.SnapError: Could not ['snap', 'restart', 'charmed-zookeeper.daemon'] for snap [charmed-zookeeper]: error: snap "charmed-zookeeper" has no service "daemon"                        


Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions