Future ideas and RFC

[bgruening]

While studying this project I had two ideas which I would like to discuss.

1: Galaxy is planning and has actually started the development of a V2 of the workflow definition - YAML based. It would be great if you can get in touch with upstream and discuss this format and optionally add missing pieces. Imho, we should include an optional test section into the workflow definition, pretty much like we do in tools.

2: In the planemo project we had some plans to extend this functionality to also support testing/running workflows. Which was blocked by (1) the definition of the new workflow spec. I encourage you to talk to the planemo devs and maybe consider to merge this project into planemo. This would have multiple advantages. For example you would gain the entire infrastructure from planemo to start/fetch/setup a Galaxy instance if you don't have one running. Testing tools and workflows belong somehow together and I think from a user-perspective it would make sense to have it in one place. This would also magically solve #31 as planemo is already using galaxy-lib. And last but not least if you plan to add CWL support, planemo already has this.

ping @jmchilton

[pcm32]

I think that this would be very good for the sustainability of wft4galaxy, as raised in one of our meetings, but then decision would come from @ilveroluca and @kikkomep of course.

[bgruening]

Thanks @pcm32! I'm open to help if this is wanted. ... and no, this decision has no impact on the publication ;)

[jmchilton]

In the planemo project we had some plans to extend this functionality to also support testing/running workflows. Which was blocked by (1) the definition of the new workflow spec.

Planemo does already support testing workflows and it isn't blocking on a new workflow definition at all IMO. You can today take original Galaxy workflows or workflows in the new beta format and run planemo test, planemo serve, planemo run, etc... on them. There are some small differences between old and new workflows that I need to address - like for new workflows tool shed tools will automatically be installed but not for Galaxy's traditional workflow format. As mentioned by @bgruening - it also supports CWL workflows the same way. It uses galaxy-lib for comparing outputs - so all the same operations available in Galaxy tools are available for workflow outputs. In order to scale up tool testing and make the workflow running production-grade and scalable - it can be configured to create on-demand postgres databases, talk to clusters, and use docker-galaxy-stable.

None of this planemo functionality is documented unfortunately - I presented it at last years GCC and there wasn't a lot of interest so I haven't put more work into the effort. I thought before the conference that what I was doing with planemo or what you are trying to do with this project would really be highly desirable - but I failed to generate any buzz I think and so I haven't put effort into documenting it and polishing it. Perhaps your paper will generate buzz for workflow testing - and that would be great.

Testing tools and workflows belong somehow together and I think from a user-perspective it would make sense to have it in one place.

Yeah I think I agree with this - I'm hoping that planemo can provide a consistent user experience, consistent deployment options, etc... across tools and workflows, across Galaxy and CWL and other future target formats and workflow engines.

I'm eager to have either help or competition. Planemo started as a passion project for me and I'm so thankful for every single contribution it gets and I want to see it blossom into a really helpful workflow SDK. Galaxy workflows are my day job though so every thing that encourages people to build more and build better workflows is awesome IMO.

[ilveroluca]

I'm happy to read about your openness to contributions @jmchilton ! We started developing wft4galaxy out of a necessity that arose in PhenoMeNal (http://phenomenal-h2020.eu/) to automatically test the workflows that we're distributing with the platform -- both as a test of the platform and of the workflow. We evidently missed the workflow testing functionality in Planemo. Now that it's done though, I'm sure our approaches each have their strengths and weaknesses, so we'd be happy to work with you on integrating this functionality and documentation into Planemo, taking the best of each tool. It would be good to see this work help bring a standard way to test Galaxy workflows to the community.