Enhance threaded_walk with exclusion of vcs and other subfolders

[yarikoptic]

We have https://github.com/dandi/dandi-cli/blob/master/dandi/support/threaded_walk.py which is used during zarr uploads. As now we do have Datalad datasets for (some/most) of zarr folders, we better add exclusion of .git, .datalad, .dotdirs etc folders there as we do in find_files. Ideally RF should then include RFing of find_files to use threaded_walk with those added exclusion features so we gain speed up in places where we use find_files

[jwodder]

@yarikoptic The threaded walk is not used during uploads; it's only used to verify downloads and by dandi digest. Also, the results in con/fscacher#67 indicate that the threaded walk is not an improvement when you're just listing files; its value in Zarr digesting comes from being able to calculate MD5 hashes of files concurrenctly.

[yarikoptic]

hm, interesting... may be I was under wrong impression that it does provide some benefits, will need to recall details better.

[yarikoptic]

Resolved. threaded_walk() in dandi/support/threaded_walk.py now accepts an exclude: Callable[[Path], Any] | None = None parameter (line 34). When provided, it skips matching paths with log.debug("Excluding %s from traversal", p) (line 61-62). Callers can pass any exclusion function to filter VCS or other subdirectories.