Add tutorial for running small demo workflow on local machine

[SJaffa]

For people who are new to MatFlow there are a lot of concepts to pick up in order to understand the documentation and effectively set up your own workflow. A good way to learn some of the concepts without extra cognitive load might be to follow a tutorial to run a minimal workflow on your own laptop. This would have very detailed instructions and no decisions for the user to make.

This can help people learn the general purpose of each step in the setup, some of the jargon, and make sure they can see the whole process to the end. This should make it much easier when trying to install on a HPC cluster.

This should be linked from the landing page of the documentation.

[SJaffa]

@Gbowker which workflows would be suitable for running on a laptop?

[Gbowker]

Hi @SJaffa so since I said this I have realised that only the tasks which use python in practace anyone could use, becuase MTEX requires a matlab licence and Damask requires a powerful computer. The best demo workflows in this case would be ones in which just use python. BUT the only one at the moment is read_tensile_test_CSV.yaml. I think there should be one which just generates microstructure seeds, and generates a volume element using these in python. Then users can visualise the output 3D .vti file using paraview, a free 3D data visualisation program.

Proposed easy to run demo-workflow:
generate_volume_element.yaml

  - schema: generate_microstructure_seeds_from_random
    inputs:
      VE_size: [1, 1, 1]
      num_grains: 4
      phase_label: demo

  - schema: generate_volume_element_from_voronoi
    inputs:
      homog_label: SX
      VE_grid_size: [8, 8, 8]

  - schema: visualise_volume_element

I think this would be the best workflow for a user to run first becuase it introduces them to the idea of making the models which our team create for our work.

@gcapes what do you think?

[aplowman]

The tension_DAMASK_Al demo workflow runs quite quickly locally, so that would be my preference. DAMASK can run via docker, so for a local tutorial, installing Docker (desktop) and pulling the DAMASK image could be part of the set up steps.

[gcapes]

@Gbowker I like your suggestion. For me, the ideal first demo would only need python and no other software, and run quickly. As such, I chose the existing demo workflow https://github.com/hpcflow/matflow-new/blob/develop/matflow/data/workflows/modify_volume_element_grid_size.yaml for my getting started section of the docs that I'm working on, on the docs/getting_started branch: hpcflow/matflow-new@ccf078c

It's very similar (the same even?) as what you proposed (I'm not sure if visualise_volume_element does the same thing as visualise_VE_VTK).

I don't like the idea of complicating a quick demo with having to install docker. My motivation for this is to get something running as quickly as easily as possible so that someone can verify their installation/config.

[SJaffa]

In Monday meeting discussed the scale of tutorials and agreed it would be best to have a progression:

• Very basic "dummy" workflow like hello-world, simple maths. This will run on a laptop with no extra software installed, and will be in the HPCFlow docs
• Intermediate material science workflow, creating a 3D set of points and visualising. Could run on laptop or cluster, in Matflow docs
• Advanced workflow with DAMASK, in Matflow docs, might run on laptops but targeted at cluster.

[SJaffa]

Me and @mikey247 worked through setting up and submitting some workflows on our laptops. We think we successfully installed MatFlow and dependencies, added the config and envs files to ~/.matflow-new, and submitted two workflows:

• demo workflow tension_DAMASK_Al
• Gerard's dummy workflow advanced_workflow.yaml

They both said "Submitted N jobs" and produced some output files and folders but running matflow show revealed that the tasks all had errors or skipped, so they didn't run successfully.

We also tried Gerard's dummy workflows hello.yaml and groups.yaml but both of these had errors on trying to run, they didn't get as far as submitting the jobs.

Follow up questions and next steps:

• Why did hello.yaml and groups.yaml fail to submit? To solve, investigate error messages and discuss with Gerard. (Update from Gerard: this issue has a proposed solution but not yet implemented, I can test locally. Update 2: that didn't fix hello, made a new issue). The groups.yaml does submit now, but I can't tell how far it's got because matflow show is now broken??
• Why did advanced_workflow.yaml and tension_DAMASK_Al fail in the tasks? To solve, find out where the log files from each task are and what the output should look like if they succeed.
• Since the installation & config were ok, write this up in the tutorial.

[SJaffa]

I got one to run successfully! Using MatFlow, version 0.3.0a149 and hpcFlow, version 0.2.0a209 I can run groups.yaml and matflow show --no-update --full reports success for 3 tasks!

[gcapes]

If you submit workflows using different versions of matflow, it (usually?) breaks matflow show. The fix is matflow manage clear-known-subs.
This normally happens to me when installing a newer version of matflow, but I guess it could happen either way. I regularly have to do this because I've got so many different versions of matflow installed all over the place 😫

[SJaffa]

This is blocked until we get the dummy workflows running with a pinned matflow/HPCflow version, and document their expected outputs

[SJaffa]

Progress: tested all dummy workflows on my laptop (two work, one fails), and one demo workflow (fails). Learned what the output looks like and where the log & error messages will go.
To do: Write up the expected output structure, fix the broken dummy workflows.
Blockers/problems: LightForm-group/matflow-user-documentation#40