Review of NMF Synchronizations

[arturboronat]

NMF Synchronizations are used to implement the synchronization between the two versions of the data model involved in each scenario. Two solutions are provided: a specific one, where overlaps between models need to be declared, and a generic one, where only overlaps that require some sort of custom transformation are declared.

Expressiveness

Correctness criteria has not been ported to .NET and this aspect of the benchnmark could not be checked. For example, in scenario3-forward of the Generic solution, the change to the instance V2 is performed before the initial synchronization. Is the change in the target instance ignored? The benchmark provides two test cases for each scenario.

The generic solution, by generating blocks, has been developed explicitly for the round-trip case and it does not support the full expressivity of the modeling language (e.g., inheritance, multivalued attributes and references).

Comprehensibility

Understanding the generic solution involves the use of reflection in the .NET framework, and this part is a bit verbose. The specification of mappings, in both solutions, is fairly intuitive but there are features of NMF Synchronizations that have not been explained (the method shouldCorrespond, the methods Coalesce and CoalesceBack).

Bidirectionality

The solution is fully bidirectional, including the reversal of expressions involving operations (like difference). This is possible because NMF provides a mechanism for defining the reversal of operations.

Re-usability

NMF provides superimposition for reusing NMF synchronization blocks in scenario 2 and the generation of blocks enables the reuse of migration logic in scenario 1.

Performance

The author argues that the generic solution needs not be computationally more expensive than the specific one because .NET compiles expressions built through reflection to object code. This is an interesting point. However, runtimes have not been analysed because the performance tests have not been ported to .NET.

[georghinkel]

Regarding performance tests: The test framework simply executed the super-tiny test models a bunch of times, isn't it? To me, this does not qualify as performance measurement because the results are rather ridiculous.

[dstrueber]

The solution relies on NMF's concept of synchronisation block. The paper gives a brief primer for a formalization of synchronization block based on lenses, and proceeds to present the solution in two parts: A specific solution (which relies on an explicit specification of the metamodel differences), and a generic solution (which generates synchronisation rules automatically, based on what could be described as a simple programmatic "diffing" of the metamodels).

Paper: Even though the formalization is interesting and potentially useful, it left me somewhat puzzled, as I don't have much background knowledge on lenses (which is required for understanding the formalization). Instead, I would have preferred an informal high-level overview of the mechanics involved in the solution (components with inputs and outputs).
The notion of synchronization rules (as opposed to synchronization blocks) seems crucial for the examples, but is not explained.
It's also a bit confusing terminology that the solution consists of two solutions (specific and generic).

Tool: Following the instructions, I was able to check out the repository and run the solution executable. Still, I was not able to actually apply the solution to the test cases, after the provided specification for the input parameters was somewhat vague. There was no reply to my request for a usage example (asked 1,5 days before the deadline in a GitHub issue).

Assessment:

1. Expressiveness: According to the paper, all scenarios could be expressed (although I couldn't validate that claim, see above). A minus is the use of hardcoded year data in the age-to-date transformation, which renders the solution obsolete after 2020.

2. Comprehensibility: It's a bit hard to compare the solution to the reference solution, since the solution does not implement the provided Task interface (which was harder in this case due to using a programming language outside the Java ecosystem). In the paper, the code shown for Scenario 1 looks compact and easily understandable, whereas, due to my missing background on lenses, the relevant part of Scenario 3 is hard to understand for me. The motivation for the generic solution makes sense, but the code is very hard to understand and not well-explained (much of the code relies on lambda functions which are not mentioned in the text).

3. Bidirectionality: The solution is bidirectional by design, an inherent advantage of NMF Synchronizations.

4. Re-usability: The solution involves the use of NMF Synchronizations's superimposition concept, which seems to support the reuse of synchronisation logic beyond the context of individual scenarios (the paper could have been a bit more explicit about reuse over different scenarios).

5. Performance: The solution does not implement the provided benchmark framework (which was harder in this case due to using a programming language outside the Java ecosystem). In the paper, the author doubts the usefulness of the performance benchmark.