WIP: Add node setup&teardown #16
Conversation
example/example.py
Outdated
| @pipeline | ||
| def infer_pipeline(translate_model: FilePath, | ||
| sentiment_model: FilePath, | ||
| def infer_pipeline(sentiment_model: FilePath, |
There was a problem hiding this comment.
Should not be changing the structure of the example DAG with this PR. Should maintain backward compatibility.
There was a problem hiding this comment.
so make a new example for new features?
There was a problem hiding this comment.
No I mean, this PR adds the setup and teardown methods. The only change needed in the example should be to add a setup and teardown step to one of the tasks. Not sure why we need to make these other changes like to the task arguments, removing a file input, etc.
There was a problem hiding this comment.
I removed the file input for translation model since now the translation model is initialized by task's setup method. If we need to keep it, then I guess it should mean something else? For example, model checkpoint.
Address issue #15