TagFuse/Sparse open failure

[cire831]

some kind of crash/hang prior to this occurring.

Tagdump was running against Tib5. Tagfuse had many timeouts (Tib5 stopped responding?), couldn't gain control. Tried to C-C no response, had to kill -9 tagfuse and tagdump. Tib5 was at Reset when reconnected via Jtag.

Last record seen from Tib5 is:

--- @59488832 136646 993418957 28 3 SYNC prev: @59484708 (0x38baa24)

we tried to read this when we restarted tagfuse after the kills above via:

tagdump -t 5 --tail -j 59488832 byte

exception in tagfuse is below:

TIB5: ~ cire831/mm@1505e8b
TagFuse: danome/TagNet@d0187a1

DEBUG:fuse.log-mixin:-> open /1fbcd99fd29f/tag/sd/0/dblk/byte (<fuse.fuse_file_info object at 0x75dac5d0>,)
DEBUG:fuse.log-mixin:<- open 0
DEBUG:fuse.log-mixin:-> read /1fbcd99fd29f/tag/sd/0/dblk/byte (20L, 59488832L, <fuse.fuse_file_info object at 0x75d355d0>)
DEBUG:fuse.log-mixin:<- read '[Unhandled Exception]'
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 495, in _wrapper
    return func(*args, **kwargs) or 0
  File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 583, in read
    offset, fh)
  File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 972, in __call__
    ret = getattr(self, op)(path, *args)
  File "tagfuse/tagfuse.py", line 188, in read
    return (str(handler.read(path_list, size, offset)))
  File "tagfuse/taghandlers.py", line 297, in read
    work_list = self._get_sparse(offset, size)
  File "tagfuse/taghandlers.py", line 252, in _get_sparse
    return self.sparse.get_bytes_and_holes(offset, size)
  File "tagfuse/sparsefile.py", line 193, in get_bytes_and_holes
    for b_s, block in self._overlay(a_s, a_e):
  File "tagfuse/sparsefile.py", line 228, in _overlay
    block = self[b_s]
  File "/usr/local/lib/python2.7/dist-packages/chest/core.py", line 185, in __getitem__
    self.get_from_disk(key)
  File "/usr/local/lib/python2.7/dist-packages/chest/core.py", line 170, in get_from_disk
    with self.open(fn, mode='r'+self.mode) as f:
IOError: [Errno 2] No such file or directory: u'/tmp/<node_id:1fbcd99fd29f>_tag_sd_0_dblk_byte/fb9f0b6620cd4e8175f7f907b05f2aeb'
DEBUG:fuse.log-mixin:-> flush /1fbcd99fd29f/tag/sd/0/dblk/byte (<fuse.fuse_file_info object at 0x75dac5d0>,)
*** sparse IO flush [u'<node_id:1fbcd99fd29f>', u'tag', u'sd', u'0', u'dblk', u'byte']
DEBUG:fuse.log-mixin:<- flush 0
DEBUG:fuse.log-mixin:-> release /1fbcd99fd29f/tag/sd/0/dblk/byte (<fuse.fuse_file_info object at 0x75dac5d0>,)
** tag release
DEBUG:fuse.log-mixin:<- release 0
DEBUG:fuse.log-mixin:-> getattr /1fbcd99fd29f/tag/sd/0/dblk/byte (None,)
** getattr: path /1fbcd99fd29f/tag/sd/0/dblk/byte
** getattr: pathlist: [u'<node_id:1fbcd99fd29f>', u'tag', u'sd', u'0', u'dblk', u'byte']
DEBUG:fuse.log-mixin:<- getattr SparseIOFileHandler([('st_ctime', 1525106067.275733), ('st_mtime', 1525107077.897952), ('st_nlink', 1), ('st_gid', 1000), ('st_blksize', 512), ('st_size', 60384104), ('st_atime', 1525107077.898392), ('st_uid', 1000), ('st_mode', 33060)])
DEBUG:fuse.log-mixin:-> open /1fbcd99fd29f/tag/sd/0/dblk/byte (<fuse.fuse_file_info object at 0x75dac5d0>,)
DEBUG:fuse.log-mixin:<- open 0
DEBUG:fuse.log-mixin:-> flush /1fbcd99fd29f/tag/sd/0/dblk/byte (<fuse.fuse_file_info object at 0x75dac5d0>,)
*** sparse IO flush [u'<node_id:1fbcd99fd29f>', u'tag', u'sd', u'0', u'dblk', u'byte']
DEBUG:fuse.log-mixin:<- flush 0
DEBUG:fuse.log-mixin:-> release /1fbcd99fd29f/tag/sd/0/dblk/byte (<fuse.fuse_file_info object at 0x75dac5d0>,)
** tag release
DEBUG:fuse.log-mixin:<- release 0
DEBUG:fuse.log-mixin:-> open /1fbcd99fd29f/tag/sd/0/dblk/byte (<fuse.fuse_file_info object at 0x75dac5d0>,)
DEBUG:fuse.log-mixin:<- open 0
DEBUG:fuse.log-mixin:-> read /1fbcd99fd29f/tag/sd/0/dblk/byte (20L, 59488832L, <fuse.fuse_file_info object at 0x75d355d0>)
DEBUG:fuse.log-mixin:<- read '[Unhandled Exception]'

[cire831]

we tried to clear the sparse via 'rm byte' and it failed with the Errno 2 on one of the hash files.

we need to handle this by blowing all the sparse away and restart the sparse.