Handling of directories

[samwaseda]

One thing I really like about pyiron_base is that I don't need to worry much about the directory structure, i.e. I simply write the files in self.working_directory, and afterwards when the job is erased there is no orphaned job directory. So I would like to suggest a couple of things here purely from the user perspective first, without distinguishing what should be offered by us and what the user should provide.

Let's assume I have a software tool that requires input files and exports output files. That means there's an input writer, and an output parser. Since I'm interested in the input parameters that I define, and the output parameters that I choose, I'm not really interested in the actual files, but just in case I would like to be able to look into them. That means I would need the following features:

1. A unique job folder should be generated for each job.
2. The job folder should in principle follow the same fate as the job itself, meaning it shouldn't be orphaned when the job is erased
3. It should lie inside the workflow folder, which is currently not automatically generated until save is called (and also mkdir is hard coded if I remember correctly)
4. I would like the WfMS to compress everything, since I usually don't need the files (compress & decompress from pyiron_base)

I'm getting the feeling that the right moment to create the workflow folder is maybe when as_path is called, and if it remains empty we can potentially erase it again after the run.

I guess I'm not totally done with this part, but I'm gonna come back to this and add more comments if something comes to my mind.

[liamhuber]

I'm fully on board with looking for ways to automate away more file system stuff.

So I would like to suggest a couple of things here purely from the user perspective first, without distinguishing what should be offered by us and what the user should provide.

On this note, there's also a middle ground where we do stuff in the standard corpus of nodes, but not in the engine itself. There's no guarantee we'll be able to solve our problems this way, but I like it because it provides a very natural way to extend power without adding complexity to the core engine.

It should lie inside the workflow folder

This is the tricky thing. Ultimately, we are pushing towards a function-based approach, and the writing and reading should be done by raw functions. These must necessarily take their working directories as inputs. While the operation of the function doesn't (indeed mustn't) know that it belongs to a node and a graph, we as users sure know it does and would love to exploit that.

It might seem at first that macros help us out of this, since they know about the fact that they are nodes -- indeed, the current implementation currently takes the self argument of the macro during the definition. Something to remember though, is that traditional macro definitions are recipe declarations and get run at instantiation (so that we have access to the graph object as early as possible) and do not get re-run at runtime.

Ultimately, what we want is some sort of feature which is based on the node objects, evaluated at runtime when the entire lexical graph path is known, and leveraged in the function calls of function nodes. There is no fundamental barrier to doing this by brute force, and it's relatively few lines if we always expose the path in question as an argument and just populate it later (cf. this example. I honestly don't think this is such an extreme burden, but I do understand and appreciate the drive to automate beyond that.

I don't currently see a way to accomplish this without going in and messing with the runtime logic of the engine. One approach might be some sort of macro like the for or while loops to accomplish this. Another would be to make a new type of function class, perhaps one that interjects between parsing inputs from the node input channels and passing input to the function the node owns.

Here is a running MWE of the latter case:

from typing import Any

import pyiron_workflow as pwf

# New infrastructure
class FunctionWithDirectory(pwf.api.Function):
    @property
    def run_args(self) -> tuple[tuple, dict]:
        _, input_kwargs = super().run_args
        return (self.as_path(),), input_kwargs

    def _on_run(self, *args, **kwargs):
        return self.node_function(*args, **kwargs)

    @classmethod
    def _build_inputs_preview(cls) -> dict[str, tuple[Any, Any]]:
        preview = super()._build_inputs_preview()
        preview.pop("working_directory")
        return preview


# Usage

## Normally we'd just decorate this
def just_say_where(working_directory, message):
    full_message = f"{message} {working_directory}"
    return full_message

## But this is an example so I haven't written the decorator infrastructre
class MakeThisFromADecoratorRatherThanTheSubclassButImLazyHere(FunctionWithDirectory):
    node_function = staticmethod(just_say_where)

## At least we get what we want?
wf = pwf.Workflow("parent_wf")
wf.n = MakeThisFromADecoratorRatherThanTheSubclassButImLazyHere("hello from")
wf()
>>> {'n__full_message': 'hello from /.../parent_wf/n'}

My big concern is this: automating working directories is great, but the number one complaint I hear about pyiron_workflow is that it's too complicated. There will certainly be some additional complication, and if we add this without removing other features, I'm worried that complication is too much.

One original client requirement for this code was the ability to instantiate nodes and connect them into a workflow without a parent macro/workflow. This is what leads to the complication in @samwaseda's first attempt, where the parent path was missing because the parent just wasn't there to start with. If we are able to guarantee that all nodes know their parentage at instantiation time, we might be able to find a more elegant solution than the one I gave above. But there's no free lunch and lines like wf.n = SomeNode(x=42) need to become wf.n = SomeNode(parent=wf, x=42) (or similar).

[samwaseda]

I don't think it's a particularly good idea to leave a comment after a glass of wine (or a particularly good one, who knows), but I try to write down some thoughts before going to bed.

Ultimately, what we want is some sort of feature which is based on the node objects, evaluated at runtime when the entire lexical graph path is known, and leveraged in the function calls of function nodes.

No not necessarily, because the directory is probably always going to be an input argument of the input parser node, meaning it will be accessible. The whole reason why I wanted to put them in the workflow directory was because that would allow me to compress the folder more easily. That means, it would not be a problem even if it is a totally different location, such as /tmp.

My big concern is this: automating working directories is great, but the number one complaint I hear about pyiron_workflow is that it's too complicated.

I totally understand it, but so far my implementation does not make it more complex, even if it's probably quite unintuitive.

But ok it's true that so far it doesn't sit very well as a concept. And at the same time the feeling for me grows that it has to be abstracted away. Maybe at this point, again I write down all the features that I would like to have:

1. Uniquely defined
2. Hidden
3. Accessible
4. Garbage-collectable (once the job is gone, it's by default automatically erased)
5. Compressible

Before I go farther, I would like to point out I can also accept the way pyiron_base defines two types of classes, one for python functions and the other for functions with executables, as a concept, because it probably satisfies all of the criteria above. The only problem is that it's not exactly compatible with the function approach, at least in the current form.

So while writing this text, I started feeling that there are the following three potential paths we could explore:

• Template Macro
  • This one is basically the pyiron_base solution. It is a macro, where the input writer, the executable and the output parser have to be written by the user. This one would indeed add a new concept and therefore more complexity to pyiron_workflow.
• Independent storage package
  • At the end of the day, dealing with the files is not much more than writing a few lines of code, so we might be able to come up with a well-defined storage package to be separated from pyiron_workflow (a bit like bagofholding?). Not sure about how to synchronize garbage-collection
• We keep the current code (compress to be added), put a few examples in the documentation page and hope for the best

[liamhuber]

Ultimately, what we want is some sort of feature which is based on the node objects, evaluated at runtime when the entire lexical graph path is known, and leveraged in the function calls of function nodes.

No not necessarily, because the directory is probably always going to be an input argument of the input parser node, meaning it will be accessible. The whole reason why I wanted to put them in the workflow directory was because that would allow me to compress the folder more easily. That means, it would not be a problem even if it is a totally different location, such as /tmp.

Yes, this sounds good to me. And indeed is also why I haven't been too stressed about getting nodes-as-paths available as runtime function input; because if I have a macro running a file-based executable, different function nodes in the macro are likely to want to be using a shared path and not their node path. Now, in that case it would still be convenient, perhaps, to use a path based off the owning macro's node path... but as you say, since it's accessible even a temp directory is sufficient.

I'm confident we could achieve your 5-points here, and if by "hidden" you mean that the user doesn't need to specify it (good default) or see it unless they go looking for it, then I think we can achieve that very quickly with a temp directory as the default directory.

Regarding your three options forward, I agree all are viable. Some more details in the realization would be

1. I think templates are analogous to the subclass-creator functions. Just like for_node takes a node class as input and returns a For subclass that builds a graph around the received node class, we could make something that takes a file read/writing node and returns a subclass of a new FileBasedMacro that makes sure that all happens nicely in the file-based-macro's node directory. This adds a bit of complication in that we introduce a new node type, but should be pretty straight forward.
2. We could, but I don't have anything here right now.
3. I can imagine that we want nodes in the standard library for creating directories (e.g. with your uuid bell and a tempdir whistle), another for compressing them, etc. So this seems wise in isolation from the question.

Actually, (1) is probably just a crystallized shortcut to doing our "best practices" of how to use stuff in (3). So we could start with (3) and see whether it's uncomfortable enough or strictly missing class features that necessitate (1), and if not stop early.

[liamhuber]

Garbage-collectable (once the job is gone, it's by default automatically erased)

This is a little bit trickier. When would we trigger it? If we have a database tool, we could hook it onto a "remove from database" function; if we make a subclass we could hook onto it when we remove the node from the parent graph, but that will probably make stuff more complicated.

I think it's doable, I just want to flag it as a pain.

[samwaseda]

Garbage-collectable (once the job is gone, it's by default automatically erased)

I was talking with @jan-janssen today, and I/we started wondering whether it's possible to take advantage of __getstate__ and __del__. The idea is to create a class that automatically erase the folder, unless __getstate__ or __setstate__ was called beforehand, which I presume would be the case if save is called. What do you think?

[liamhuber]

Yeah, I imagine that would work. It's implicit as hell though 😝

[samwaseda]

But isn't that rather a feature and not a problem?

[liamhuber]

I think "doesn't require attention" is the level of implicit that we want, eg nice default values, but IMO doing flag checks inside dunder methods on whether other dunder methods have been called during this interpreter session is treading into the territory of "hard to track down what went wrong".

[samwaseda]

I close this one since all functionalities have been implemented here, here and here