Skip to content

Pipeline does not verify that split key is a list #71

@ARMmaster17

Description

@ARMmaster17

Describe the bug
This was found while debugging #6 because there was another bug where the split mode would not be cleared after a context copy. If the pipeline is set to split after a step completes, and the split key is not a list, a subset of the remaining contexts to be run will be deleted. This is an issue with downstream implementation, but it would be better if the developer was warned that they are trying to split on an invalid data type.

To Reproduce
Steps to reproduce the behavior:

  1. Build a pipeline (SA or HA)
  2. Create a step that sets a context value to a primitive value (e.g. 1).
  3. Set the context to split on step completion.
  4. Note that subsequent steps will not run the correct number of times relative to previous splits/filters.

Expected behavior
Pipeline should throw an error when attempting to split on a key that is not a list, and this should halt the pipeline completely as this is most likely a code error and not a data error.

Watergrid specific
Watergrid version/commit: 77e5755
Python version: 3.10
Redis version (if using HAPipeline): N/A

Additional context
N/A

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions