Review of the ATL solution

[azuendorf]

The ATL solution starts with some theory concerning the ATL approach to incremental model updates.
It shows that TL focuses on "commutative" incrementality, i.e. the final result is the same without regarding if you do an
incremental update or a full regeneration.
Interestingly, the labworkflow case requires to keep some information about jobs, that are already completed and these informations need to be considered on incremental update.
Thus, the case triggered some possible improvements for ATL which is great.
In general, the ATL rules provided in the github repository are ease to read and from my point of view somewhat better
comprehensible compared to the NMF solution.
The ATL solution also seems to be much faster than the NMF solution albeit not faster than the reference solution.
Nice.

[georghinkel]

I also find the ATL solution very interesting. The solution presentation is a bit weired as it is focused pretty much on very general things and not so much on the actual solution, which is only described at a very architectural level -- one has to dig into the source code to get an impression how it feels like to solve the case with ATL.

That said, the rule definitions are very straight-forward and comprehensible.

The fact that the next/previous pointers have not been tackled is a bit unfortunate, because I believe it is quite interesting and I would have liked to see how this could be done with ATL.

[fjouault]

Thanks for the nice reviews!

We agree that we could have described our solution at a more detailed level.

Regarding the next/previous references, @georghinkel 's remark convinced us to look into it.
Here is the commit that implements them: ESEO-Tech@153e5a1.
It is relatively simple, once the dispatching if-then-else-endif is factored out in a helper.

[georghinkel]

Hi Frederic,

I am not sure this really implements the next/previous properly (haven't executed the code, yet). I have the impression that it was not clear from the case description what these references are all about. Here is the description I gave Artur:

Actually, the previous/next references are a bit more sophisticated: A low-level job is a next job of another if the original high-level protocol step is next of the other and both jobs share at least one sample they are operating on.

That is, if a low-level job was responsible to distribute samples 1 and 2, then the add reagent to samples 3 and 4 is not a next job, even if the add reagent step was the next after the distribute. In reality, this sharpness is critical because time constraints cannot be met otherwise.

@fjouault

[fjouault]

Hi @georghinkel,

Thanks for the precision. Could you please point us to an example from the test set where this happens?

[georghinkel]

Hi Frederic,

unfortunately, not that often, only if you process more than 96 samples.

This is for instance in all scale_samples models from size 64.

Best,

Georg

[fjouault]

Ok, thanks. We managed to really compare our results to the Reference results only on smaller sizes, because of minor differences such as labware ordering.

[georghinkel]

Yeah. The current design of the benchmark is very permissive, I basically wanted to leave some freedom to implementations. Otherwise, I could have used the NMF model signature checks to check for equality, which does ignore the order for unordered collections.

[fjouault]

With @TheoLeCalvar, we gave some thought to result validation. Our current idea is that the most permissive way to do it without ignoring issues would probably be to simulate job execution.

[fjouault]

Regarding ordering, ordered collections is the default for Ecore (see ETypedElement.isOrdered()). Therefore, I believe that all of the case's collections are ordered in the provided metamodels (hence our remark about job ordering in our YAMTL review). However, should I infer from your remark that that was not the intent?

[georghinkel]

I thought I had switched the collections to unordered, but actually, you are right, they are ordered right now.

[arturboronat]

Thank you all for the discussion. As a summary, it does not matter how jobs are processed as long as prev/next links respect data dependencies. Let me know if this interpretation is not correct.

[georghinkel]

@arturboronat One might run into the case that a liquid transfer can be deleted entirely because all samples are failed. In that case, you could (and should) remove it from any prev/next pointers (it does not make sense to wait for a job that does not do anything).

[arturboronat]

Thanks for the clarification, is this case included in the benchmark correctness check?

[fjouault]

@georghinkel the ATL solution now produces the same initial previous/next values as the Reference solution, after the following change: ESEO-Tech@521ef7d

We only check whether consecutive jobs use the same microplate or not. This is simpler than what both NMF-based solutions do, and works on all the benchmark models, but we have not checked what would happen when adding samples, which we do not support yet anyway.