[shin] Hardware-in-the-Loop Testing of CPS

[grammarware]

Submitted by @seungyeob to https://github.com/modelsconf2018/artifact-evaluation/tree/master/shin

Paper: http://orbilu.uni.lu/handle/10993/36092

[zolotas4]

Hi,

In the https://figshare.com/s/240acc73c40ec949f27a artefact repository I cannot find any instructions on how to work with the artefact. There is only a small description for each of the files provided in the zip. Could you please provide instructions on how to run the artefacts (software needed, dependencies, how to import, how to execute, etc.)?

Thanks,
Thanos

[grammarware]

Ping @seungyeob

[seungyeob]

Dear Thanos (@zolotas4),

Our artifacts include the HITECS language grammar and case study models since this is the main contribution of our paper. We did not include any runnables in our artifacts. Specifically, the zip file that you mentioned contains the (sanitized) HITECS specifications of our case study which are readable by using a text editor. The artifact shows how to use the HITECS language via concrete examples and helps readers of our paper to have a better understanding.

Regards,
Seung Yeob

[zolotas4]

Hi Seung (@seungyeob),

Thanks for your reply.

So, if I would like to use your proposed language how can I do? If I want to replicate your experiments how can I do?

Regards,
Thanos

[jdirocco]

Hi @zolotas4,
By creating a new xtext project, the provided Hitecs xtext grammar (with xtext extension) allows generating editor and facilities that enable the creation of Hitecs specifications. Look at https://www.eclipse.org/Xtext/documentation/102_domainmodelwalkthrough.html for generating Hitecs editor.

[jdirocco]

The artefacts provided are part of the paper presenting an approach to help engineers systematically specify and analyze CPS HiL test cases.
Summarizing the artifact contribution consists of three parts:

• the Hitecs grammar specification (Xtext),
• evaluation reports (2 xlsx file), and
• archive file that contains Hitecs specifications. These specifications are conform to Hitecs grammar.

========================================================================
(1) Is the artifact consistent with the paper?
The artifacts are consistent with the paper since the Hitecs grammar provided with the artifacts can be used to define Hitecs specifications.
I suggest providing the whole project that contains Hitecs grammar instead of just the xtext file. However, by creating a new xtext project, the provided Hitecs xtext grammar allows generating editor and facilities that enable the creation of Hitecs specifications. At last, the generated editor permits to formulate Hitecs specifications defined in the archive.

(2) Is the artifact as complete as possible?
No, it doesn't permit to replicate the evaluation experiments that support the approach. Moreover, as I previously said, the xtext projects is partial. It just includes the grammar definition.

(3) Is the artifact well-documented?
The documentation is good and each artifact is provided with a dedicated description. If the xtext grammar is given, the artifact documentation should be improved by providing information on how one can create an editor that permits to create HITECT specifications.
(4) Is the artifact easy to (re)use?
It needs the Xtext knowledge to generate the corresponding editor. Moreover, The excel files don't provide any suggestions to replicate the experiment with the artifacts defined into repository.

[zolotas4]

Thanks @md2manoppello!

[seungyeob]

Dear Thanos (@zolotas4) and Juri (@md2manoppello),

We were not aware that the artifact evaluation requires executable tools. At this moment, the repository contains our case study material and some additional artifacts that help readers to have a better understanding about our paper. We are currently working on writing a tool paper to be submitted to the MODELS 2018 tools and demonstrations track based on the approach presented in our research paper. We will also try to update the repository to include executables and the related instructions for running the executables.

Thanks and best regards,
Seung Yeob