What happens?
_match_files_with_pattern in _glob.py uses glob.glob(), which silently excludes dotfiles and
dot-directories (e.g. .gtn/, .config/). The worker output sync in asset_sync.py uses
pathlib.Path.glob(), which includes them. This means files that exist on disk after a job runs can be
missing from output manifests.
Expected behavior
Both code paths should handle dotfiles consistently. Files present on disk should appear in the output
manifest.
Additionally, a customer reported that they spent time trying to figure out why dotfiles were not attached to jobs. We might consider always attaching them or following .gitignore files or similar.
Reproduction
- Submit a job whose output directory contains a dotfile or dot-directory (e.g.
.gtn/result.txt)
- After the job completes, observe that the worker output sync picks up the dotfile
- Generate a manifest via
_glob_paths — the dotfile is missing
Considerations
- A naive fix of switching to
pathlib.Path.glob() introduces Python-version-dependent behavior:
pathlib.Path.glob() handles dotfiles differently across versions (consistent dotfile matching only in
3.11+), and trailing ** only yields directories on <3.13.
- Including dotfiles by default could cause users to unintentionally upload large hidden trees (e.g.
.git/)
in output manifests. A default exclude pattern for common cases like .git may be worth considering.
- Whatever approach is chosen should produce identical results regardless of Python version.
Relevant code
src/deadline/job_attachments/_glob.py — _match_files_with_pattern (uses glob.glob())
src/deadline/job_attachments/asset_sync.py — worker output sync (uses pathlib.Path.glob())
See draft PR along with @leongdl 's feedback: #1039
What happens?
_match_files_with_patternin_glob.pyusesglob.glob(), which silently excludes dotfiles anddot-directories (e.g.
.gtn/,.config/). The worker output sync inasset_sync.pyusespathlib.Path.glob(), which includes them. This means files that exist on disk after a job runs can bemissing from output manifests.
Expected behavior
Both code paths should handle dotfiles consistently. Files present on disk should appear in the output
manifest.
Additionally, a customer reported that they spent time trying to figure out why dotfiles were not attached to jobs. We might consider always attaching them or following .gitignore files or similar.
Reproduction
.gtn/result.txt)_glob_paths— the dotfile is missingConsiderations
pathlib.Path.glob()introduces Python-version-dependent behavior:pathlib.Path.glob()handles dotfiles differently across versions (consistent dotfile matching only in3.11+), and trailing
**only yields directories on <3.13..git/)in output manifests. A default exclude pattern for common cases like
.gitmay be worth considering.Relevant code
src/deadline/job_attachments/_glob.py—_match_files_with_pattern(usesglob.glob())src/deadline/job_attachments/asset_sync.py— worker output sync (usespathlib.Path.glob())See draft PR along with @leongdl 's feedback: #1039