[dou] Model-Driven Trace Diagnostics

[grammarware]

Submitted by @weidou to https://github.com/modelsconf2018/artifact-evaluation/tree/master/dou

[moji1]

@weidou please add a link to your paper

[grammarware]

2 @weidou: as usual, if you don't want to expose it to the web, email me and I'll forward it to @moji1 @TheoLeCalvar @md2manoppello. This is just for the purpose of reviewing. Thanks in advance!

[moji1]

Hi @weidou
I have tried the TemPsy-Report using Eclipse DSL 4.8.0, and OCL 6.4 and It worked after a few changes in Eclipse project configuration and source code. The following issues for compiling and execution of the artifact need be fixed:
1- The eclipse project has dependencies to EMF and OCL libraries which are not configured by default and users require to add them manually.

2- The path for traces and properties are hardcoded, and their default values are incorrect. I changed the path in experiment method source code to ./inputs/properties/ for compiling the code.

3- The detail instruction to reproduce the evaluation results should be provided. Currently, the instruction briefly explains the tools (TemPsy-Report and trace_generator.sh) without any detail. Without a detail instruction, it is hard to validate that the submitted artifact is compatible with the paper. For instance, the following figure shows a partial output of TemPsy-Report for generated traces in my pc, but I do not know how to interpret the results.

4- No information is provided regarding Tempsy-Violation-Visualization which is discussed in Section 5.

Thanks
Moji

[weidou]

Hi @moji1
thanks for your feedback.
Indeed, the EMF and OCL libraries need to be configured manually. I have added a notice into the README file.
I have revised the Main method (for your 2 and 3 comments) and I hope these changes can answer your third comment.
The link to Tempsy-Violation-Visualization has also been instered into the README file. But please notice that Tempsy-Violation-Visualization does not impact the evaluation of our approach to model-driven trace diagnostics (see section 6 of our paper).

Thanks,
Wei

[moji1]

Hi @weidou

Thanks for the update.
For the requirement section, that would be great if you add a list of required jar files. The following figure shows the libraries I added manually.

Also, I have an issue with redoing the evaluation. As shown in the following figure the execution time of P3 is much higher than what you reported in the paper. Any idea what can cause this issue?
My pc has 8 GB Ram and 2.7 Core i5 processor.

Thanks
Mojtaba

[jdirocco]

Summary
The artifact consists of a software program, called TemPsy-Report, that runs a model-driven procedure for trace diagnostics of TemPsy properties. The software program is presented as Eclipse project that includes: the java source code, the temporal property files (XMI files) and the shell script to generate the traces for the TemPsy properties.

(1) Is the artifact consistent with the paper?
Yes, the artifact is consistent with the paper. It contains the same 12 temporal properties (TemPsy properties) used for the evaluation presented in the paper. In the source code the authors indicate exactly which part of the code produces the result presented in which figure of the paper, I appreciate that.

(2) Is the artifact as complete as possible?
The artifact is quite complete. Just a little missing is the library used to generate the temporal property files (the XMI files): the library, as indicated by the authors, is part of a previous version of the tool. Anyway, that is not the focus of the paper and the authors provided the link to the library, so not a big missing in my opinion.

(3) Is the artifact well-documented?
After the suggestions of another reviewer, the authors improved accordingly the documentation. In this version, it is clear enough to a new reader like me.

(4) Is the artifact easy to (re)use?
It is not a nightmare to install and run the project, but unfortunately, it is required some manual operations to add the required libraries. Maybe some more clear indications of the steps to add them should be nice to have.

My overall opinion is positive with this artifact. It needs just some little improvements, I suggest an acceptance.

Kudos to @mircofranzago for helping me in making this review.

[weidou]

Hi @moji1
I have updated the artifact by adding EMF & OCL libraries, and revising the main method to evaluate NSOR and WTO traces.
However, the execution time of checking P3 is fine on my machine (see screenshot below).

Thanks,
Wei

[weidou]

Hi @md2manoppello
I am sorry for the inconvenience. The previous versions of the artifact was developed long time ago when I was doing my PhD and unfortunately the last stable version was not pushed to my github. I hope the updated version can make it easy to reproduce the evaluation of our approach.

Thanks,
Wei