Support for heterogeneous components.

[pvandyken]

Summary

The proposal addresses Snakebids' inability to index and process files with missing or variably ordered entities, a limitation caused by its reliance on fixed templates. It utilizes Snakemake's input functions and constrained wildcard templates to allow potentially missing entities, enabling flexible handling of heterogeneous filenames. Optional entities are paired with dummy wildcards that contain the tag. For users, the main effect is the appearance of these dummy wildcards in dry-run output, but workflows gain the ability to support datasets with inconsistent or partially specified entities without requiring significant migration.

Theory

The proposed solution replaces the coherent template BidsComponent.path with an input function. The initial BidsComponent would retain the zip_list table of wildcard values, but instead of a unified template (.path), it would save the paths for each file in raw form. Missing entities would be encoded in the zip_list as None. The initial snakemake rule would collect its input from this BidsComponent via a method. Its output would be a new template standardized by bids() that supports potentially missing entities.

Such a template is possible using snakemake's wildcard_constraints, as in the following example:

r"sub-{subject}/ses-{session}/sub-{subject}_ses-{session}{_acq_,(?:_acq\-)?}{acq,(?:(?<=_acq\-)[^_]+)?}{_dir_,(?:_dir\-)?}{dir,(?:(?<=_dir\-)[^_]+)?}_suffix.nii.gz"

This ghastly form simplified by ignoring the ,constraint portion of each {wildcard}:

r"sub-{subject}/ses-{session}/sub-{subject}_ses-{session}{_acq_}{acq}{_dir_}{dir}_suffix.nii.gz"

Clearly, both acq and dir are optional wildcards. {_acq_} is intended to match the tag portion of the path (_acq-) while {acq} matches its value. This intent is enforced in snakemake via the constraints:

_acq_: (?:_acq\-)? # regex optionally matching `_acq-`
acq: (?:(?<=_acq\-)[^_]+)? # regex matching the associated value when the negative look-behind detects the tag

This can be tested using the regex_from_filepattern utility function used in snakemake to process output templates:

>>> import re
>>> from snakemake.io import regex_from_filepattern
>>> PATTERN = regex_from_filepattern(
...    r"sub-{subject}/ses-{session}/sub-{subject}_ses-{session}{_acq_,(?:_acq\-)?}{acq,(?:(?<=_acq\-)[^_]+)?}{_dir_,(?:_dir\-)?}{dir,(?:(?<=_dir\-)[^_]+)?}_suffix.nii.gz"
... )
>>> re.match(
...     PATTERN,
...    "sub-001/ses-01/sub-001_ses-01_dir-AP_acq-MPRAGE_dir-AP_suffix.nii.gz"
... ).groupdict()
{'subject': '001', 'session': '01', '_acq_': '_acq-', 'acq': 'MPRAGE', '_dir_': '_dir-', 'dir': 'AP'}
>>> re.match(
...     PATTERN,
...    "sub-001/ses-01/sub-001_ses-01_dir-AP_acq-MPRAGE_suffix.nii.gz"
... ).groupdict()
{'subject': '001', 'session': '01', '_acq_': '_acq-', 'acq': 'MPRAGE', '_dir_': '', 'dir': ''}
>>> re.match(
...     PATTERN,
...    "sub-001/ses-01/sub-001_ses-01_dir-AP_suffix.nii.gz"
... ).groupdict()
{'subject': '001', 'session': '01', '_acq_': '', 'acq': '', '_dir_': '', 'dir': ''}
>>> re.match(
...     PATTERN,
...    "sub-001/ses-01/sub-001_ses-01_dir-AP_acq-MPRAGE_suffix.nii.gz"
... ).groupdict()
AttributeError

Importantly, these constrained patterns can be used in input, which ignores the constraints, as well as output. Furthermore, all outputs accessed later by rules.RULE_NAME.output.OUTPUT will have been converted into ordinary strings with no constraint syntax, so an ordinary Python .format() will work without issue.

Although snakemake's constraint syntax was not likely intended to allow such optional components, I do not believe this approach to be an unstable "hack". Wildcard restraints have been a long-standing feature and have used python regex expression since the beginning. The above constraints use perfectly valid regex.

In practice, this results in extra, dummy wildcards containing the tag portion of the path when present. By convention, these wildcards are named the same as the entity, wrapped in single underscores (e.g. _acq_). Within the workflow, I do not anticipate this causing any issues. The main consequence to users will be the appearance of these dummy wildcards in the dry run summary, but I think this is a tolerable side effect. However, snakebids will need to meet three new requirements:

Requirements

1. A method on `BidsComponent` for retrieval of input files. This method will receive a `dict` of wildcards from snakemake as its argument. It must anticipate optional wildcards in two manifestations:
   1. Absent wildcards will be presented as a blank string, not as `None` or a similar unambiguous marker.
   2. A dummy wildcard (e.g. `_acq_`) will be present and set either to a blank string or the tag value.
2. A method of producing snakemake templates with optional components
3. At the end of the workflow, a finalized list of paths is typically compiled using `.expand()`. This method will need support templates with dummy wildcards and insert the appropriate values.

Implementation

Template-free components

Currently, all BidsComponents are initialized with a template stored in BidsComponent.path, the format of which is not compatible with optional entities. Three different situations must be handled.

Heterogeneous components

These are components with entries of differing roots, entities in differing orders, or additional path elements unaccounted for by pybids (e.g. extra suffixes). In principle, with sufficiently expansive wildcard selection, an entire dataset could be fit within a single such component.

Potential handling methods are as follows:

1. Prohibit. Insist on a coherent template to prevent accidental overinclusion of files.
2. Opt-in. Heterogeneous inputs would be useful for disorganized datasets, and handling them is feasible within the scope of this feature.
3. Opt-out or force. Would require careful, explicit filtering to prevent accidental overinclusion.

In any case where a heterogeneous component is indexed (options 2 or 3), no coherent template would exist for BidsComponent.path, so on invocation, it must error.

Coherent template with optional entities

Coherent means all entries have the same root, the same directory structures, and a set of entities in the same order. Optional entities are absent from the path.

In this case, BidsComponent.path could:

1. Return a template with constrained wildcards as shown above.

   Advantages:

   1. Existing applications could potentially support optional entities without migration.

   Disadvantages:

   1. This would wed a core property closely with snakemake and, as such, hurt snakebid's generalizability. For instance, these templates would not be compatible with str.format().
   2. Applications relying on BidsComponent.path would fail after indexing heterogeneous inputs.

2. Raise an error. This would ensure applications can handle any potential dataset. Practically, this would deprecate BidsComponent.path.

Coherent templates without optional entities

These are the components currently supported by snakebids. This will obviously continue to be handled, however, if BidsComponent.path is essentially deprecated per above, it should eventually be removed entirely even for this standard case.

Retrieving paths

Template-free components require a means of retrieving paths. Replacing BidsComponent.path, a method (working name .get) shall be devised as follows:

class BidsComponent:
    def get(
        self, schema: dict[str, str | bool | None], /, **entities: str | bool | None
    ) -> str:
        """Look up an entry based on the provided entities and return its path.

        Entities may be provided as a dict or as keyword arguments.

        Errors if no matching entry is found, or if not all entities in the component
        are specified.
        """

This method could be used as an input function for snakemake rules in lieu of a template, seamlessly handling all the above cases. As specified in I, entities set to the blank string would be interpreted as absent, potentially only if accompanied by a corresponding dummy entity (e.g. _acq_) also set to the blank string.

Template generation

Requirement II will be satisfied by endowing the bids function with a convention to indicate optional wildcards. Note that, in its current state, bids() hard-codes all _tags-, whereas optional entities require a dummy wildcard for the tag. A few options are available:

1. A keyword-only argument (e.g. optional_entities) that takes a list of entities to be made optional. BidsComponent.wildcards must include this key with the associated list in lieu of "key": "{value}" pairings for each optional entity. For instance, if "subject" and "session" were mandatory and "acq" optional:

   component.wildcards == {
       "subject": "{subject}",
       "session": "{session}",
       "optional_entities": ["acq"],
   }

   bids() would format these optional entities into the path using the appropriately constrained wildcard entries.

   This breaks the pure meaning of BidsComponent.wildcards. A modification would move "optional_entities" to a new property: BidsComponent.optional_entities, but this would inconveniently need to be specified manually in every call to bids().

   It also adds an additional magic keyword argument to bids(), something that we have been trying to avoid.

2. A sentinel object indicating an optional entity (e.g. OPTIONAL_WILDCARD). When given to bids() as an entity value, instead of being formatted as a string, it would cause the entity to be inserted using constrained wildcard syntax. The wildcards dict would be as follows:

   component.wildcards == {
       "subject": "{subject}",
       "session": "{session}",
       "acq": OPTIONAL_WILDCARD,
   }

   This also breaks the dict[str, str] type of .wildcards, but is less invasive than the previous option and retains the nominal meaning of the keys.

   As a variation, OPTIONAL_WILDCARD could be replaced with an existing Python object, such as .... This would simplify the implementation but obscure the meaning.

Generic expansion

For requirement III, the BidsComponent.expand() implementation will be modified to correctly detect and handle both dummy wildcards and wildcards with snakemake constraints.

Templates would be sanitized of any constraints (wrapped in braces and preceded by a comma: {acq,restraint}.)

When formatting a path with an absent entity (e.g. "acq": None), .expand() will expect to find a corresponding dummy wildcard ({_acq_}), erroring if otherwise.

Additional entities may not be dummy-formatted (i.e. wrapped in single underscores, e.g. _acq_). For instance, component.expand(TEMPLATE, _acq_="foo") will raise an error, regardless whether "{_acq_}{acq}" appears in the template.

Allowance for dummy wildcards

It is not uncommon for workflows to call BidsComponent.filter() within an input function using the wildcards argument. As this argument will potentially include dummy wildcards, all BidsComponent methods must be able to handle them. As with .get(), the solution will generally be to ignore dummy wildcards and intepret wildcards set to the blank string as absent.

I do not necessarily propose we modify the filter_list function, as its usage should generally give way to the object-oriented .filter() method.

Migration

Amazingly, existing workflows using the most recent snakebids API could begin supporting generic datasets with minimal to no migration. Supporting truly heterogeneous datasets would require using BidsComponent.get instead of BidsComponent.path. Such a switch may be generally required if BidsComponent.path is deprecated. Otherwise, workflows correctly using BidsComponent.wildcards within bids() calls would immediately get access to generic templates. BidsComponent.expand() would begin handling these templates without any user change required.

Note that workflows using the expand() function from Snakemake would not get this benefit, as this function will not support generic templates.

Rejected Alternative

Previous issue #470 attempted to correct missing entities by adding a generic label. I have found this approach to be relatively unfeasible compared to the above.

It can essentially be conceptualized as entity remapping: paths with an entity initially set to None are remapped to a given value. This operation could be performed with an input function such as BidsComponent.get. The problem is that the function would need knowledge of the remapping. I would be very hesitant to further complexify the internal state of BidsComponent by saving knowledge of remapping operations. This precludes any automatic remapping within generate_inputs. Potentially a wrapper function could be devised that translates the remapped wildcards into the input values before calling BidsComponent.get. But this would boilerplate code in workflows to be supported.

Thus, this solution would require fairly significant migration before being useful. It is certainly possible, and it is not mutually exclusive with the generic template solution, but I do not think it should be a priority at this time.

[pvandyken]

@akhanf, @jordandekraker, @zihuaihuai.

I think this solution should be quite workable, please let me know what you think. Moving into Christmas and the new year, I'll have quite a bit more time, so I think finishing this is quite feasible.

[pvandyken]

@Dhananjhay, would appreciate any input you have as well!

[akhanf]

Looks interesting! Alot to digest and haven't given it the most thorough read yet, but seems like a promising avenue..

[Dhananjhay]

Looks exciting! Will give it a read over the break, and get back to you soon.

[pvandyken]

Thanks all. I'm planning to start implementation next week, so please get back with any major objections before then.