FileNotFoundError when running datalad containers-run cmd in git worktree

[just-meng]

What is the problem?

originally reported in datalad/datalad#7759, now slightly different behavior, but I believe it is the same underlying bug.

A datalad containers-run command that works fine in the master worktree

meng@meng-MS-7D06 /m/D/M/d/L5b (master)> datalad containers-run -n fissa -m "Run fissa v1.0.0 for mouse I, LSD with Suite2p roiset." \
                                                                                         -i "01_suite2p/*I/*LSD/plane-0/reg_tif/*.tif" \
                                                                                         -i "01_suite2p/*I/*LSD/plane-0/allcell.npy" \
                                                                                         -i "01_suite2p/*I/*LSD/plane-0/ops.npy" \
                                                                                         -i "01_suite2p/*I/*LSD/plane-0/stat.npy" \
                                                                                         -o "03_fissa/*I/*LSD/plane-0/roiset_suite2p" \
                                                                                         "python code/src/process2p/run_fissa.py {inputs} {outputs}"
[INFO   ] Making sure inputs are available (this may take some time) 
[INFO   ] == Command start (output follows) ===== 
Extracting traces:   0%|                                                                                                               | 0/77 [00:00<?, ?it/s]
...

fails in the runs worktree with FileNotFoundError:

meng@meng-MS-7D06 /m/D/L5b-runs (runs)> datalad containers-run -n fissa -m "Run fissa v1.0.0 for mouse I, LSD with Suite2p roiset." \
                                            -i "01_suite2p/*I/*LSD/plane-0/reg_tif/*.tif" \
                                            -i "01_suite2p/*I/*LSD/plane-0/allcell.npy" \
                                            -i "01_suite2p/*I/*LSD/plane-0/ops.npy" \
                                            -i "01_suite2p/*I/*LSD/plane-0/stat.npy" \
                                            -o "03_fissa/*I/*LSD/plane-0/roiset_suite2p" \
                                            "python code/src/process2p/run_fissa.py {inputs} {outputs}"
[INFO   ] Making sure inputs are available (this may take some time) 
[INFO   ] == Command start (output follows) ===== 
Traceback (most recent call last):
  File "/mnt/Data/L5b-runs/code/src/process2p/run_fissa.py", line 252, in <module>
    main()
  File "/mnt/Data/L5b-runs/code/src/process2p/run_fissa.py", line 215, in main
    ops = np.load(path_ops, allow_pickle=True).item()
  File "/usr/local/lib/python3.9/site-packages/numpy/lib/npyio.py", line 405, in load
    fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] No such file or directory: '01_suite2p/sub-240226I/exp-LSD/plane-0/ops.npy'
[INFO   ] == Command exit (modification check follows) ===== 
run(error): /mnt/Data/L5b-runs (dataset) [singularity exec -B /mnt/Data/L5b-runs -...]
action summary:
  get (notneeded: 83)
  run (error: 1)

The file is present in the shared annex though. datalad get -s gin which worked in the example in datalad/datalad#7759 does not work here. git annex get --from also does not work. git annex fsck does not help.

meng@meng-MS-7D06 /m/D/L5b-runs (runs) [1]> git annex whereis 01_suite2p/sub-240226I/exp-LSD/plane-0/ops.npy
whereis 01_suite2p/sub-240226I/exp-LSD/plane-0/ops.npy 
  unable to query remote ria-backup-storage for urls: RIA store unavailable. -caused by- file:///media/meng/ProData/ria-store/ria-layout-version not found, self.ria_store_url: ria+file:///media/meng/ProData/ria-store, self.store_base_pass: /media/meng/ProData/ria-store, self.store_base_pass_push: None, path: <class 'pathlib._local.PosixPath'> /media/meng/ProData/ria-store/ria-layout-version -caused by- [Errno 2] No such file or directory: '/media/meng/ProData/ria-store/ria-layout-version'
(9 copies) 
  	01586b18-304b-42d2-b3ab-a3a688853808 -- [ria-backup-storage]
  	1fe65de2-a8f1-403f-92a4-28b9fcf980d2 -- create subds for the suit2p outputs for layer5b exp.
  	492fb0e6-7980-40b3-9fbb-97caebaaaced -- meng@meng-MS-7D06:/mnt/Data/Meng_Psychedelics/02_processed/L5b
  	4f2d53ba-7cce-4d96-ba8e-947b0d8dd8cf -- git@b7807f24a9b9:/data/repos/jwu/meng_psychedelics-01_data-02_preprocessed_data-02_psychedelics_l5b_suite2p.git [gin]
  	6a7efbaa-2842-4742-9412-ed7824682fab -- jiame@RAZOR:D:\Meng_Psychedelics\01_data\02_preprocessed_data\02_Psychedelics_L5b_suite2p
  	7f789fa4-25be-4b21-b4dd-6522864e9c7a -- meng@meng-MS-7D06:/mnt/Data/Meng_Psychedelics/derived/L5b [here]
  	97b30a7d-e532-4bd6-a91c-37b955e9f467 -- fabia@Fabian:~/Desktop\ROIs\ds_Psychedelics_L5b_suite2p
  	a4a3a0c9-4c58-453f-8041-25d11be84ca9 -- jiame@RAZOR:D:\Meng_Psychedelics\derived\L5b
  	f12fa2a7-6ad7-42f0-a811-0306f2bce0d7 -- git@be610756bf80:/data/repos/jwu/ds_processed_l5b.git
ok

meng@meng-MS-7D06 /m/D/L5b-runs (runs)> git annex fsck 01_suite2p/sub-240226I/exp-LSD/plane-0/ops.npy
fsck 01_suite2p/sub-240226I/exp-LSD/plane-0/ops.npy (checksum...) ok
(recording state in git...)

meng@meng-MS-7D06 /m/D/L5b-runs (runs) [1]> datalad get "01_suite2p/sub-240226I/exp-LSD/plane-0/ops.npy" -s gin
meng@meng-MS-7D06 /m/D/L5b-runs (runs)> git annex get "01_suite2p/sub-240226I/exp-LSD/plane-0/ops.npy"
meng@meng-MS-7D06 /m/D/L5b-runs (runs)> git annex get "01_suite2p/sub-240226I/exp-LSD/plane-0/ops.npy" --from gin

meng@meng-MS-7D06 /m/D/L5b-runs (runs)> ls 01_suite2p/sub-240226I/exp-LSD/plane-0
allcell.npy@  Fneu.npy@       iscell-02.npy@  iscell-05.npy@  iscell-08.npy@  iscell-11.npy@      notcell.npy@  spks.npy@
data.bin@     F.npy@          iscell-03.npy@  iscell-06.npy@  iscell-09.npy@  iscell.npy@         ops.npy@      stat.npy@
Fall.mat@     iscell-01.npy@  iscell-04.npy@  iscell-07.npy@  iscell-10.npy@  iscell-somata.npy@  reg_tif/      whichcell.npy@

meng@meng-MS-7D06 /m/D/L5b-runs (runs)> datalad containers-run -n fissa -m "Run fissa v1.0.0 for mouse I, LSD with Suite2p roiset." \
                                            -i "01_suite2p/*I/*LSD/plane-0/reg_tif/*.tif" \
                                            -i "01_suite2p/*I/*LSD/plane-0/allcell.npy" \
                                            -i "01_suite2p/*I/*LSD/plane-0/ops.npy" \
                                            -i "01_suite2p/*I/*LSD/plane-0/stat.npy" \
                                            -o "03_fissa/*I/*LSD/plane-0/roiset_suite2p" \
                                            "python code/src/process2p/run_fissa.py {inputs} {outputs}"
[INFO   ] Making sure inputs are available (this may take some time) 
[INFO   ] == Command start (output follows) ===== 
Traceback (most recent call last):
  File "/mnt/Data/L5b-runs/code/src/process2p/run_fissa.py", line 252, in <module>
    main()
  File "/mnt/Data/L5b-runs/code/src/process2p/run_fissa.py", line 215, in main
    ops = np.load(path_ops, allow_pickle=True).item()
  File "/usr/local/lib/python3.9/site-packages/numpy/lib/npyio.py", line 405, in load
    fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] No such file or directory: '01_suite2p/sub-240226I/exp-LSD/plane-0/ops.npy'
[INFO   ] == Command exit (modification check follows) ===== 
run(error): /mnt/Data/L5b-runs (dataset) [singularity exec -B /mnt/Data/L5b-runs -...]
action summary:
  get (notneeded: 83)
  run (error: 1)

I remember from the troubleshooting session with @yarikoptic that it wouldn't work if the worktree were created before getting the file content in the master worktree, but I don't remember the solution. I suppose, for now, I should just kill the worktree and create a new one, but it would be nice if I didn't have to go through the setup process for nested worktrees every time new content is available on master ... that's not fun!

What steps will reproduce the problem?

1. drop the content of a file
2. create a worktree
3. get content in the master worktree
4. try to get the file or use the file in the other worktree

DataLad information

DataLad 1.3.0 WTF (configuration, credentials, datalad, dataset, dependencies, environment, extensions, git-annex, location, metadata.extractors, metadata.filters, metadata.indexers, python, system)

WTF

configuration <SENSITIVE, report disabled by configuration>

credentials

• keyring:
  • active_backends:
    • SecretService Keyring
    • PlaintextKeyring with no encyption v.1.0 at /home/meng/.local/share/python_keyring/keyring_pass.cfg
  • config_file: /home/meng/.config/python_keyring/keyringrc.cfg
  • data_root: /home/meng/.local/share/python_keyring

datalad

• version: 1.3.0

dataset

• branches:
  • git-annex@f524a69
  • master@0a20c61
  • runs@0a20c61
• id: 869c8a4d-338b-4f34-97d2-3fc0d6bf73e2
• path: /mnt/Data/L5b-runs/containers/fissa-1.0.0
• repo: AnnexRepo

dependencies

• annexremote: 1.6.6
• boto3: 1.42.30
• cmd:7z: 16.02
• cmd:annex: 10.20251114-geeb21b831e7c45078bd9447ec2b0532a691fe471
• cmd:bundled-git: UNKNOWN
• cmd:git: 2.34.1
• cmd:ssh: 8.9p1
• cmd:system-git: 2.34.1
• cmd:system-ssh: 8.9p1
• humanize: 4.15.0
• iso8601: 2.1.0
• keyring: 25.7.0
• keyrings.alt: 5.0.2
• msgpack: 1.1.2
• platformdirs: 4.5.1
• requests: 2.32.5

environment

• LANG: en_US.UTF-8
• LANGUAGE: en_US
• LC_ADDRESS: de_DE.UTF-8
• LC_IDENTIFICATION: de_DE.UTF-8
• LC_MEASUREMENT: de_DE.UTF-8
• LC_MONETARY: de_DE.UTF-8
• LC_NAME: de_DE.UTF-8
• LC_NUMERIC: de_DE.UTF-8
• LC_PAPER: de_DE.UTF-8
• LC_TELEPHONE: de_DE.UTF-8
• LC_TIME: de_DE.UTF-8
• PATH: /home/meng/bin:/home/meng/go/bin:/home/meng/.cargo/bin:/home/meng/.local/bin:/home/meng/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/meng/.local/share/JetBrains/Toolbox/scripts:/usr/local/go/bin

extensions

• container:
  • description: Containerized environments
  • entrypoints:
    • datalad_container.containers_add.ContainersAdd:
      • class: ContainersAdd
      • module: datalad_container.containers_add
      • names:
        • containers-add
        • containers_add
    • datalad_container.containers_list.ContainersList:
      • class: ContainersList
      • module: datalad_container.containers_list
      • names:
        • containers-list
        • containers_list
    • datalad_container.containers_remove.ContainersRemove:
      • class: ContainersRemove
      • module: datalad_container.containers_remove
      • names:
        • containers-remove
        • containers_remove
    • datalad_container.containers_run.ContainersRun:
      • class: ContainersRun
      • module: datalad_container.containers_run
      • names:
        • containers-run
        • containers_run
  • module: datalad_container
  • version: 1.2.6
• deprecated:
  • description: Deprecated functionality
  • entrypoints:
    • datalad_deprecated.annotate_paths.AnnotatePaths:
      • class: AnnotatePaths
      • module: datalad_deprecated.annotate_paths
      • names:
    • datalad_deprecated.ls.Ls:
      • class: Ls
      • module: datalad_deprecated.ls
      • names:
    • datalad_deprecated.metadata.aggregate.AggregateMetaData:
      • class: AggregateMetaData
      • module: datalad_deprecated.metadata.aggregate
      • names:
        • aggregate-metadata
        • aggregate_metadata
    • datalad_deprecated.metadata.extract_metadata.ExtractMetadata:
      • class: ExtractMetadata
      • module: datalad_deprecated.metadata.extract_metadata
      • names:
        • extract-metadata
        • extract_metadata
    • datalad_deprecated.metadata.metadata.Metadata:
      • class: Metadata
      • module: datalad_deprecated.metadata.metadata
      • names:
    • datalad_deprecated.metadata.search.Search:
      • class: Search
      • module: datalad_deprecated.metadata.search
      • names:
    • datalad_deprecated.publish.Publish:
      • class: Publish
      • module: datalad_deprecated.publish
      • names:
  • module: datalad_deprecated
  • version: 1.0.0
• next:
  • description: What is next in DataLad
  • entrypoints:
    • datalad_next.commands.create_sibling_webdav.CreateSiblingWebDAV:
      • class: CreateSiblingWebDAV
      • module: datalad_next.commands.create_sibling_webdav
      • names:
        • create-sibling-webdav
    • datalad_next.commands.credentials.Credentials:
      • class: Credentials
      • module: datalad_next.commands.credentials
      • names:
    • datalad_next.commands.download.Download:
      • class: Download
      • module: datalad_next.commands.download
      • names:
        • download
    • datalad_next.commands.ls_file_collection.LsFileCollection:
      • class: LsFileCollection
      • module: datalad_next.commands.ls_file_collection
      • names:
        • ls-file-collection
    • datalad_next.commands.status.Status:
      • class: Status
      • module: datalad_next.commands.status
      • names:
        • next-status
        • next_status
    • datalad_next.commands.tree.TreeCommand:
      • class: TreeCommand
      • module: datalad_next.commands.tree
      • names:
        • tree
  • module: datalad_next
  • version: 1.5.0

git-annex

• build flags:
  • Assistant
  • Webapp
  • Pairing
  • Inotify
  • TorrentParser
  • MagicMime
  • Benchmark
  • Feeds
  • Testsuite
  • S3
  • WebDAV
  • Servant
  • OsPath
• dependency versions:
  • aws-0.24.4
  • bloomfilter-2.0.1.2
  • crypton-1.0.4
  • DAV-1.3.4
  • feed-1.3.2.1
  • ghc-9.10.2
  • http-client-0.7.19
  • persistent-sqlite-2.13.3.1
  • torrent-10000.1.3
  • uuid-1.3.16
  • yesod-1.6.2.1
• key/value backends:
  • SHA256E
  • SHA256
  • SHA512E
  • SHA512
  • SHA224E
  • SHA224
  • SHA384E
  • SHA384
  • SHA3_256E
  • SHA3_256
  • SHA3_512E
  • SHA3_512
  • SHA3_224E
  • SHA3_224
  • SHA3_384E
  • SHA3_384
  • SKEIN256E
  • SKEIN256
  • SKEIN512E
  • SKEIN512
  • BLAKE2B256E
  • BLAKE2B256
  • BLAKE2B512E
  • BLAKE2B512
  • BLAKE2B160E
  • BLAKE2B160
  • BLAKE2B224E
  • BLAKE2B224
  • BLAKE2B384E
  • BLAKE2B384
  • BLAKE2BP512E
  • BLAKE2BP512
  • BLAKE2S256E
  • BLAKE2S256
  • BLAKE2S160E
  • BLAKE2S160
  • BLAKE2S224E
  • BLAKE2S224
  • BLAKE2SP256E
  • BLAKE2SP256
  • BLAKE2SP224E
  • BLAKE2SP224
  • SHA1E
  • SHA1
  • MD5E
  • MD5
  • WORM
  • URL
  • GITBUNDLE
  • GITMANIFEST
  • VURL
  • X*
• local repository version: 10
• operating system: linux x86_64
• remote types:
  • git
  • gcrypt
  • p2p
  • S3
  • bup
  • directory
  • rsync
  • web
  • bittorrent
  • webdav
  • adb
  • tahoe
  • glacier
  • ddar
  • git-lfs
  • httpalso
  • borg
  • rclone
  • hook
  • external
  • compute
  • mask
• supported repository versions:
  • 8
  • 9
  • 10
• upgrade supported from repository versions:
  • 0
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
• version: 10.20251114-geeb21b831e7c45078bd9447ec2b0532a691fe471

location

• path: /mnt/Data/L5b-runs/containers/fissa-1.0.0
• type: dataset

metadata.extractors

• annex:
  • distribution: datalad_deprecated 1.0.0
  • module: datalad_deprecated.metadata.extractors.annex
• audio:
  • distribution: datalad_deprecated 1.0.0
  • module: datalad_deprecated.metadata.extractors.audio
• container_inspect:
  • distribution: datalad_container 1.2.6
  • load_error: ModuleNotFoundError(No module named 'datalad_metalad')
  • module: datalad_container.extractors.metalad_container
• datacite:
  • distribution: datalad_deprecated 1.0.0
  • module: datalad_deprecated.metadata.extractors.datacite
• datalad_core:
  • distribution: datalad_deprecated 1.0.0
  • module: datalad_deprecated.metadata.extractors.datalad_core
• datalad_rfc822:
  • distribution: datalad_deprecated 1.0.0
  • module: datalad_deprecated.metadata.extractors.datalad_rfc822
• exif:
  • distribution: datalad_deprecated 1.0.0
  • module: datalad_deprecated.metadata.extractors.exif
• frictionless_datapackage:
  • distribution: datalad_deprecated 1.0.0
  • module: datalad_deprecated.metadata.extractors.frictionless_datapackage
• image:
  • distribution: datalad_deprecated 1.0.0
  • module: datalad_deprecated.metadata.extractors.image
• xmp:
  • distribution: datalad_deprecated 1.0.0
  • module: datalad_deprecated.metadata.extractors.xmp

metadata.filters

metadata.indexers

python

• implementation: CPython
• version: 3.13.5

system

• distribution: linuxmint/21.3/virginia
• encoding:
  • default: utf-8
  • filesystem: utf-8
  • locale.prefered: UTF-8
• filesystem:
  • CWD:
    • path: /mnt/Data/L5b-runs/containers/fissa-1.0.0
  • HOME:
    • path: /home/meng
  • TMP:
    • path: /tmp
• max_path_length: 297
• name: Linux
• release: 5.15.0-168-generic
• type: posix
• version: Morpho & https://knb.ecoinformatics.org for ecological data datalad#178-Ubuntu SMP Fri Jan 9 19:05:03 UTC 2026

Hint: install psutil to get filesystem information

Additional context

No response

Have you had any success using DataLad before?

No response

[just-meng]

I created another nested worktree and still got the same FileNotFoundError error. Then I copied the venv and tried with datalad run instead of datalad containers-run and was surprised to see it work in the worktree! So it seems to be an issue with the combination of datalad containers-run + git worktree. Because I verified that:

• the specific datalad containers-run cmd with the specific container works in master worktree
• the equivalent datalad run cmd in a venv that is the same as the container (supposedly, I really have no clew, it was such a long fight to get this stupid package working in either condition...) works in run2 worktree
• both cmd work in master worktree
• but not datalad containers-run in runs worktree

What the hell is going on ...

# container does not work in worktree
meng@meng-MS-7D06 /m/D/L5b-runs2 (runs2)> datalad containers-run -n fissa -m "Run fissa v1.0.0 for mouse I, LSD with Suite2p roiset." \
                                              -i "01_suite2p/*I/*LSD/plane-0/reg_tif/*.tif" \
                                              -i "01_suite2p/*I/*LSD/plane-0/allcell.npy" \
                                              -i "01_suite2p/*I/*LSD/plane-0/ops.npy" \
                                              -i "01_suite2p/*I/*LSD/plane-0/stat.npy" \
                                              -o "03_fissa/*I/*LSD/plane-0/roiset_suite2p" \
                                              "python code/src/process2p/run_fissa.py {inputs} {outputs}"
[INFO   ] Making sure inputs are available (this may take some time) 
[INFO   ] == Command start (output follows) ===== 
Traceback (most recent call last):
  File "/mnt/Data/L5b-runs2/code/src/process2p/run_fissa.py", line 252, in <module>
    main()
  File "/mnt/Data/L5b-runs2/code/src/process2p/run_fissa.py", line 215, in main
    ops = np.load(path_ops, allow_pickle=True).item()
  File "/usr/local/lib/python3.9/site-packages/numpy/lib/npyio.py", line 405, in load
    fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] No such file or directory: '01_suite2p/sub-240226I/exp-LSD/plane-0/ops.npy'
[INFO   ] == Command exit (modification check follows) ===== 
run(error): /mnt/Data/L5b-runs2 (dataset) [singularity exec -B /mnt/Data/L5b-runs2 ...]
action summary:
  get (notneeded: 83)
  run (error: 1)

meng@meng-MS-7D06 /m/D/L5b-runs2 (runs2) [1]> cp /mnt/Data/Meng_Psychedelics/derived/L5b/environments . -r
meng@meng-MS-7D06 /m/D/L5b-runs2 (runs2)> source environments/fissa/bin/activate.fish

# datalad run works in worktree
(fissa) meng@meng-MS-7D06 /m/D/L5b-runs2 (runs2)> datalad run -m "Run fissa (1.0.0) for mouse I, LSD with Suite2p roiset." \
                                                      -i "01_suite2p/*I/*LSD/plane-0/reg_tif/*.tif" \
                                                      -i "01_suite2p/*I/*LSD/plane-0/allcell.npy" \
                                                      -i "01_suite2p/*I/*LSD/plane-0/ops.npy" \
                                                      -i "01_suite2p/*I/*LSD/plane-0/stat.npy" \
                                                      -o "03_fissa/*I/*LSD/plane-0/roiset_suite2p" \
                                                      "python code/src/process2p/run_fissa.py {inputs} {outputs}"
[INFO   ] Making sure inputs are available (this may take some time) 
[INFO   ] == Command start (output follows) ===== 
Loading data from cache 03_fissa/sub-240226I/exp-LSD/prepared.npz
...  

# container works in master
meng@meng-MS-7D06 /m/D/M/d/L5b (master)> datalad containers-run -n fissa --explicit -m "Run fissa (1.0.0) for mouse I, LSD with Suite2p roiset." \
                                                                                           -i "01_suite2p/*I/*LSD/plane-0/reg_tif/*.tif" \
                                                                                           -i "01_suite2p/*I/*LSD/plane-0/allcell.npy" \
                                                                                           -i "01_suite2p/*I/*LSD/plane-0/ops.npy" \
                                                                                           -i "01_suite2p/*I/*LSD/plane-0/stat.npy" \
                                                                                           -o "03_fissa/*I/*LSD/plane-0/roiset_suite2p" \
                                                                                           "python code/src/process2p/run_fissa.py {inputs} {outputs}"

[INFO   ] Making sure inputs are available (this may take some time) 
[INFO   ] == Command start (output follows) ===== 
Loading data from cache 03_fissa/sub-240226I/exp-LSD/prepared.npz

[yarikoptic]

hm, what is the difficulty you having of getting a single environment with "both commands" (run and containers-run) available? if you have containers-run you must have run already. With the switch between environments I wonder if it is anyhow related to differences in installation!? FWIW, containers-run and run should be identical in their treatment of -i inputs, hence I am alittle puzzled by your observation of difference.

[just-meng]

I might have a clew now: I did not install datalad and datalad-container inside the container. But if that was the issue, it should not have worked in the master worktree neither, no?

I rebuilt the container with datalad 1.3.0 and datalad-container 1.2.6, but still get the same FileNotFoundError.

[just-meng]

Here is the container: https://gin.g-node.org/jwu/container-fissa

[yarikoptic]

FWIW: datalad + datalad-container should be in your working environment - e.g. under some miniconda or uv . In https://gin.g-node.org/jwu/container-fissa/src/master/fissa.def I also see installation of datalad-next which can alter how datalad functionality works (e.g. fix via its patches or break) so it might be the decisive factor . I would recommend to

• not use container to run datalad*
• use miniconda or uv env with pip install datalad datalad-container and potentially datalad-next and see if that anyhow changes the behavior

[yarikoptic]

as for versions reporting, just use datalad wtf -S dependencies -S datalad -S extensions to get a more complete view (albeit more verbosely formatted as well)

[just-meng]

now I think about it, I can't remember that I ever tried datalad containers-run in a worktree. Maybe there is some decisive difference between run and containers-run when executed in a worktree?

[just-meng]

FWIW: datalad + datalad-container should be in your working environment - e.g. under some miniconda or uv . In https://gin.g-node.org/jwu/container-fissa/src/master/fissa.def I also see installation of datalad-next which can alter how datalad functionality works (e.g. fix via its patches or break) so it might be the decisive factor . I would recommend to

I tried without datalad-next here https://gin.g-node.org/jwu/container-fissa/commit/0a101523ba744e327d6598f0531e8b0365c65aad did not help

• not use container to run datalad*

I don't understand this one

• use miniconda or uv env with pip install datalad datalad-container and potentially datalad-next and see if that anyhow changes the behavior

Ok I'll try that.

[just-meng]

I created a dedicated venv and installed datalad, datalad-container and git-annex, and then ran datalad containers-run inside, it failed with the same error.

meng@meng-MS-7D06 /m/D/L/environments (runs)> uv venv datalad-container
Using CPython 3.13.5
Creating virtual environment at: datalad-container
Activate with: source datalad-container/bin/activate.fish
meng@meng-MS-7D06 /m/D/L/environments (runs)> source datalad-container/bin/activate.fish
(datalad-container) meng@meng-MS-7D06 /m/D/L/environments (runs)> uv pip install datalad datalad-container git-annex
Using Python 3.13.5 environment at: datalad-container
Resolved 39 packages in 459ms
Prepared 2 packages in 2.08s
░░░░░░░░░░░░░░░░░░░░ [0/39] Installing wheels...                                                                                                              warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance.
         If the cache and target directories are on different filesystems, hardlinking may not be supported.
         If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning.
Installed 39 packages in 678ms
 + annexremote==1.6.6
 + boto3==1.42.42
 + botocore==1.42.42
 + certifi==2026.1.4
 + cffi==2.0.0
 + chardet==5.2.0
 + charset-normalizer==3.4.4
 + cryptography==46.0.4
 + datalad==1.3.1
 + datalad-container==1.2.6
 + distro==1.9.0
 + fasteners==0.20
 + git-annex==10.20260115
 + humanize==4.15.0
 + idna==3.11
 + iso8601==2.1.0
 + jaraco-classes==3.4.0
 + jaraco-context==6.1.0
 + jaraco-functools==4.4.0
 + jeepney==0.9.0
 + jmespath==1.1.0
 + keyring==25.7.0
 + keyrings-alt==5.0.2
 + looseversion==1.3.0
 + more-itertools==10.8.0
 + msgpack==1.1.2
 + packaging==26.0
 + patool==4.0.4
 + platformdirs==4.5.1
 + pycparser==3.0
 + python-dateutil==2.9.0.post0
 + python-gitlab==8.0.0
 + requests==2.32.5
 + requests-toolbelt==1.0.0
 + s3transfer==0.16.0
 + secretstorage==3.5.0
 + six==1.17.0
 + tqdm==4.67.3
 + urllib3==2.6.3
(datalad-container) meng@meng-MS-7D06 /m/D/L/environments (runs)> cd ..
(datalad-container) meng@meng-MS-7D06 /m/D/L5b-runs (runs)> datalad rerun
[INFO   ] run commit 1356d4d; (Run fissa (1.0.0)...) 
[INFO   ] Making sure inputs are available (this may take some time) 
unlock(ok): 03_fissa/sub-240229K/exp-Saline2/plane-0/roiset-suite2p/DFF.npy (file)                                                                            
unlock(ok): 03_fissa/sub-240229K/exp-Saline2/plane-0/roiset-suite2p/F.npy (file)                                                                              
unlock(ok): 03_fissa/sub-240229K/exp-Saline2/plane-0/roiset-suite2p/prepared.npz (file)                                                                       
unlock(ok): 03_fissa/sub-240229K/exp-Saline2/plane-0/roiset-suite2p/rois.npz (file)                                                                           
unlock(ok): 03_fissa/sub-240229K/exp-Saline2/plane-0/roiset-suite2p/separated.npz (file)                                                                      
[INFO   ] == Command start (output follows) =====                                                                                                             
Traceback (most recent call last):
  File "/mnt/Data/L5b-runs/code/src/process2p/run_fissa.py", line 240, in <module>
    main()
  File "/mnt/Data/L5b-runs/code/src/process2p/run_fissa.py", line 208, in main
    ops = np.load(path_ops, allow_pickle=True).item()
  File "/usr/local/lib/python3.9/site-packages/numpy/lib/npyio.py", line 405, in load
    fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] No such file or directory: '01_suite2p/sub-240229K/exp-Saline2/plane-0/ops.npy'
[INFO   ] == Command exit (modification check follows) ===== 
run(error): /mnt/Data/L5b-runs (dataset) [singularity exec -B /mnt/Data/L5b-runs -...]
add(ok): 03_fissa/sub-240229K/exp-Saline2/plane-0/roiset-suite2p/DFF.npy (file)                                                                               
add(ok): 03_fissa/sub-240229K/exp-Saline2/plane-0/roiset-suite2p/F.npy (file)                                                                                 
add(ok): 03_fissa/sub-240229K/exp-Saline2/plane-0/roiset-suite2p/prepared.npz (file)                                                                          
add(ok): 03_fissa/sub-240229K/exp-Saline2/plane-0/roiset-suite2p/rois.npz (file)                                                                              
add(ok): 03_fissa/sub-240229K/exp-Saline2/plane-0/roiset-suite2p/separated.npz (file)                                                                         
action summary:                                                                                                                                               
  add (ok: 5)
  get (notneeded: 79)
  run (error: 1)
  save (notneeded: 8)
  unlock (ok: 5)

[just-meng]

The bug: When running containers with git worktrees, symlinks to git-annex objects resolve incorrectly because:

1. Git worktrees use .git files (not directories)

2. os.path.realpath() in containers inserts /worktrees/name/ path component

3. Git-annex objects live in main repo's .git/annex/, not worktree's (non-existent) .git/worktrees/name/annex/

Indeed, the path /mnt/Data/Meng_Psychedelics/derived/L5b/.git/worktrees/L5b-runs/annex/objects/... does not exist.

# 2. Create a test script that checks symlinks
cat > test_symlink_resolution.py << 'EOF'
#!/usr/bin/env python
import os
import sys

path = sys.argv[1]
print(f"Path: {path}")
print(f"Exists: {os.path.exists(path)}")
print(f"Is link: {os.path.islink(path)}")
if os.path.islink(path):
    print(f"Link target: {os.readlink(path)}")
    # Try to resolve it
    try:
        real_path = os.path.realpath(path)
        print(f"Real path: {real_path}")
        print(f"Real path exists: {os.path.exists(real_path)}")
    except Exception as e:
        print(f"Error resolving: {e}")
EOF

# 3. Run test on host
$ python test_symlink_resolution.py 01_suite2p/sub-240226I/exp-LSD/plane-0/ops.npy
Path: 01_suite2p/sub-240226I/exp-LSD/plane-0/ops.npy
Exists: True
Is link: True
Link target: ../../../../.git/annex/objects/9k/QW/MD5E-s99289729--60a4599fb738ac50f29b5a1b10d30031.npy/MD5E-s99289729--60a4599fb738ac50f29b5a1b10d30031.npy
Real path: /mnt/Data/Meng_Psychedelics/derived/L5b/.git/annex/objects/9k/QW/MD5E-s99289729--60a4599fb738ac50f29b5a1b10d30031.npy/MD5E-s99289729--60a4599fb738ac50f29b5a1b10d30031.npy  ✅ EXISTS
Real path exists: True

# 4. Run test in container
$ datalad containers-run -n fissa -m "Test" \
    -i "01_suite2p/*I/*LSD/plane-0/ops.npy" \
    "python test_symlink_resolution.py {inputs}"
Path: 01_suite2p/sub-240226I/exp-LSD/plane-0/ops.npy
Exists: False
Is link: True
Link target: ../../../../.git/annex/objects/9k/QW/MD5E-s99289729--60a4599fb738ac50f29b5a1b10d30031.npy/MD5E-s99289729--60a4599fb738ac50f29b5a1b10d30031.npy
Real path: /mnt/Data/Meng_Psychedelics/derived/L5b/.git/worktrees/L5b-runs/annex/objects/9k/QW/MD5E-s99289729--60a4599fb738ac50f29b5a1b10d30031.npy/MD5E-s99289729--60a4599fb738ac50f29b5a1b10d30031.npy  ❌ DOESN'T EXIST
Real path exists: False

[just-meng]

reproduce with this

#!/bin/bash
set -Eeuo pipefail

mkdir -p tmp-container-worktree
cd tmp-container-worktree

# Create venv
uv venv
source .venv/bin/activate
uv pip install datalad datalad-container git-annex

# Clone project
datalad clone git@gin.g-node.org:/jwu/ds_processed_L5b.git L5b
cd L5b
datalad get code containers/fissa -n

# Also get the critical files before creating the worktrees
# this one for the path resolution test
datalad get 01_suite2p/sub-240229K/exp-Saline/plane-0/ops.npy
# this one to prevent the unlock/remove issue
datalad get 03_fissa/sub-240229K/exp-Saline/plane-0/roiset-suite2p/prepared.npz

# Make sure you can rerun in master (if you insist or just skip to the bottom)
datalad rerun d22a6038802363baee5d547aee888bd6a626d0ad

# Remove prev. worktree and branch
#datalad foreach-dataset git worktree prune
#datalad foreach-dataset git branch -d runs

# Create worktree
name="runs"
git worktree add "../L5b-${name}" -b "${name}"

# Get the root directory for submodules
root="$(pwd)/.."

cd code
git worktree add "../../L5b-${name}/code" -b "${name}"
cd ..

cd containers/fissa
git worktree add "../../../L5b-${name}/containers/fissa" -b "${name}"
cd ../..

# Rerun in worktree and look out for the FileNotFoundError
cd "../L5b-${name}"
datalad rerun d22a6038802363baee5d547aee888bd6a626d0ad

# 2. Create a test script that checks symlinks
cd ..
cat > test_symlink_resolution.py << 'EOF'
#!/usr/bin/env python
import os
import sys

path = sys.argv[1]
print(f"Path: {path}")
print(f"Exists: {os.path.exists(path)}")
print(f"Is link: {os.path.islink(path)}")
if os.path.islink(path):
    print(f"Link target: {os.readlink(path)}")
    # Try to resolve it
    try:
        real_path = os.path.realpath(path)
        print(f"Real path: {real_path}")
        print(f"Real path exists: {os.path.exists(real_path)}")
    except Exception as e:
        print(f"Error resolving: {e}")
EOF

chmod +x test_symlink_resolution.py
cp test_symlink_resolution.py L5b/
cp test_symlink_resolution.py "../L5b-${name}"/

# Run test on host
cd L5b
#python test_symlink_resolution.py 01_suite2p/sub-240226I/exp-LSD/plane-0/ops.npy
datalad containers-run --explicit -n fissa -m "Test" \
    -i "01_suite2p/*I/*LSD/plane-0/ops.npy" \
    "python test_symlink_resolution.py {inputs}"

# Run test in container
cd "../L5b-${name}"
datalad containers-run --explicit -n fissa -m "Test" \
    -i "01_suite2p/*I/*LSD/plane-0/ops.npy" \
    "python test_symlink_resolution.py {inputs}"

[just-meng]

I confirm that the same error occurs also for a different container/project.

meng@meng-MS-7D06 /m/D/CFDHW_Peru (main)> datalad containers-run -n regridder -i sourcedata/Precipitation/RAIN4PE_daily_0.1d_1981_2015_v1.0.nc -i sourcedata/Temperature/tmin_daily_1981_2020_010.nc -o derivatives/regridded "python code/regrid.py {inputs} {outputs}"
[INFO   ] Making sure inputs are available (this may take some time) 
unlock(ok): derivatives/regridded/RAIN4PE_daily_0.1d_1981_2015_v1.0.nc (file)                                                                                 
Unlocking:  50%|███████████████████████████████████████████████████▌            Unlocking: 100%|█████████████████████████| 2.00/2.00 [00:40<00:00, 20.4s/ Files]unlock(ok): derivatives/regridded/tmax_daily_1981_2020_010.nc (file)            
[INFO   ] == Command start (output follows) =====                               
Selecting time period 1981-01-01 - 2015-12-31 ...
Creating notnull masks for regridding ...
Creating weight files...
Regridding ...
Saving to disk as netCDF...
[INFO   ] == Command exit (modification check follows) ===== 
run(ok): /mnt/Data/CFDHW_Peru (dataset) [singularity exec -B /mnt/Data/CFDHW_Peru...]
add(ok): derivatives/regridded/tmin_daily_1981_2020_010.nc (file)               
add(ok): derivatives/regridded/RAIN4PE_daily_0.1d_1981_2015_v1.0.nc (file)      
add(ok): derivatives/regridded/tmax_daily_1981_2020_010.nc (file)               
save(ok): . (dataset)                                                           
action summary:                                                                 
  add (ok: 3)
  get (notneeded: 6)
  run (ok: 1)
  save (notneeded: 3, ok: 1)
  unlock (ok: 2)
meng@meng-MS-7D06 /m/D/CFDHW_Peru (main)> cd ../Nina-runs/

meng@meng-MS-7D06 /m/D/Nina-runs (runs) [128]> git reset --hard main     (base) 
HEAD is now at aca715a [DATALAD RUNCMD] singularity exec -B /mnt/Data/CFDHW_Peru...

meng@meng-MS-7D06 /m/D/Nina-runs (runs)> datalad rerun                   (base) 
[INFO   ] run commit aca715a; (singularity exec ...) 
[INFO   ] Making sure inputs are available (this may take some time) 
unlock(ok): derivatives/regridded/RAIN4PE_daily_0.1d_1981_2015_v1.0.nc (file)   
unlock(ok): derivatives/regridded/tmax_daily_1981_2020_010.nc (file)            
unlock(ok): derivatives/regridded/tmin_daily_1981_2020_010.nc (file)            
[INFO   ] == Command start (output follows) =====                               
Traceback (most recent call last):
  File "/opt/conda/lib/python3.14/site-packages/xarray/backends/file_manager.py", line 219, in _acquire_with_cache_info
    file = self._cache[self._key]
           ~~~~~~~~~~~^^^^^^^^^^^
  File "/opt/conda/lib/python3.14/site-packages/xarray/backends/lru_cache.py", line 56, in __getitem__
    value = self._cache[key]
            ~~~~~~~~~~~^^^^^
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/mnt/Data/Nina-runs/sourcedata/Temperature/tmin_daily_1981_2020_010.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), '88665066-6d95-4aa5-b572-822ce0dd2d7f']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/Data/Nina-runs/code/regrid.py", line 86, in <module>
    main()
    ~~~~^^
  File "/mnt/Data/Nina-runs/code/regrid.py", line 29, in main
    ds_in = xr.open_dataset(path_ds_in[0])
  File "/opt/conda/lib/python3.14/site-packages/xarray/backends/api.py", line 606, in open_dataset
    backend_ds = backend.open_dataset(
        filename_or_obj,
    ...<2 lines>...
        **kwargs,
    )
  File "/opt/conda/lib/python3.14/site-packages/xarray/backends/netCDF4_.py", line 771, in open_dataset
    store = NetCDF4DataStore.open(
        filename_or_obj,
    ...<8 lines>...
        autoclose=autoclose,
    )
  File "/opt/conda/lib/python3.14/site-packages/xarray/backends/netCDF4_.py", line 529, in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
  File "/opt/conda/lib/python3.14/site-packages/xarray/backends/netCDF4_.py", line 429, in __init__
    self.format = self.ds.data_model
                  ^^^^^^^
  File "/opt/conda/lib/python3.14/site-packages/xarray/backends/netCDF4_.py", line 538, in ds
    return self._acquire()
           ~~~~~~~~~~~~~^^
  File "/opt/conda/lib/python3.14/site-packages/xarray/backends/netCDF4_.py", line 532, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "/opt/conda/lib/python3.14/contextlib.py", line 141, in __enter__
    return next(self.gen)
  File "/opt/conda/lib/python3.14/site-packages/xarray/backends/file_manager.py", line 207, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "/opt/conda/lib/python3.14/site-packages/xarray/backends/file_manager.py", line 225, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
  File "src/netCDF4/_netCDF4.pyx", line 2521, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2158, in netCDF4._netCDF4._ensure_nc_success
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/Data/Nina-runs/sourcedata/Temperature/tmin_daily_1981_2020_010.nc'
[INFO   ] == Command exit (modification check follows) ===== 
run(error): /mnt/Data/Nina-runs (dataset) [singularity exec -B /mnt/Data/Nina-runs ...]
add(ok): derivatives/regridded/RAIN4PE_daily_0.1d_1981_2015_v1.0.nc (file)      
add(ok): derivatives/regridded/tmax_daily_1981_2020_010.nc (file)               
add(ok): derivatives/regridded/tmin_daily_1981_2020_010.nc (file)               
action summary:                                                                 
  add (ok: 3)
  get (notneeded: 6)
  run (error: 1)
  save (notneeded: 4)
  unlock (ok: 3)