Skip to content

Bug: job attachments excludes dotfiles from output manifests, inconsistent with worker output sync #1101

@crowecawcaw

Description

@crowecawcaw

What happens?

_match_files_with_pattern in _glob.py uses glob.glob(), which silently excludes dotfiles and
dot-directories (e.g. .gtn/, .config/). The worker output sync in asset_sync.py uses
pathlib.Path.glob(), which includes them. This means files that exist on disk after a job runs can be
missing from output manifests.

Expected behavior

Both code paths should handle dotfiles consistently. Files present on disk should appear in the output
manifest.

Additionally, a customer reported that they spent time trying to figure out why dotfiles were not attached to jobs. We might consider always attaching them or following .gitignore files or similar.

Reproduction

  1. Submit a job whose output directory contains a dotfile or dot-directory (e.g. .gtn/result.txt)
  2. After the job completes, observe that the worker output sync picks up the dotfile
  3. Generate a manifest via _glob_paths — the dotfile is missing

Considerations

  • A naive fix of switching to pathlib.Path.glob() introduces Python-version-dependent behavior:
    pathlib.Path.glob() handles dotfiles differently across versions (consistent dotfile matching only in
    3.11+), and trailing ** only yields directories on <3.13.
  • Including dotfiles by default could cause users to unintentionally upload large hidden trees (e.g. .git/)
    in output manifests. A default exclude pattern for common cases like .git may be worth considering.
  • Whatever approach is chosen should produce identical results regardless of Python version.

Relevant code

  • src/deadline/job_attachments/_glob.py_match_files_with_pattern (uses glob.glob())
  • src/deadline/job_attachments/asset_sync.py — worker output sync (uses pathlib.Path.glob())

See draft PR along with @leongdl 's feedback: #1039

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingjob attachmentsFor an issue with job attachmentsneeds triageA new report that needs a first look

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions