[tolvanen] Effort Used to Create DSMLs

[grammarware]

Submitted by @mccjpt to https://github.com/modelsconf2018/artifact-evaluation/tree/master/tolvanen

[hernanponcedeleon]

@mccjpt could you please provide a version of the paper?

[mccjpt]

Currently out of office, but will upload it later this evening. Juha-Pekka On Tue, Jul 17, 2018 at 10:09 AM +0300, "Zeta" <notifications@github.com<mailto:notifications@github.com>> wrote: @mccjpt<https://github.com/mccjpt> could you please provide a version of the paper? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#9 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AYLIXs9j_7S_415jEmNYJkAh8XPf-fs7ks5uHY2mgaJpZM4VOH4b>.

[mherzberg]

Artefact summary

The submitted artefact accompanies the paper "Effort Used to Create
Domain-Specific Modeling Languages", which presents a survey of 10 case studies
which analyzes the amount of effort spent on creating domain-specific models and
languages. The artifact consists of an excel spreadsheet that provides details
on the time spent per activity for two of the case studies.

Consistency with the paper

The excel sheet contains some of the graphs of the paper along with the formulas
used to generate them. The artefact is therefore consistent with the paper.

Completeness of artefact

The artefact only contains data and graphs regarding the two big case studies.
The data regarding the 8 smaller studies, which has been used for figure 6,
seems to be missing; however, each case study is recorded only with a single
data point (the number of days invested), which is contained in the paper.

Artefact documentation

The artefact is undocumented. Some information can be inferred from looking at
the contained log event names from the tool, which are listed with time stamps
and serve as the bases for the graphs.

Ease of reuse

The case studies, as described in the paper, seem reproducible.

The data, in the form present in the artefact, is difficult to interpret and
therefore not reusable.

[mccjpt]

Was made available the same evening at github. Do you need it via email? Juha-Pekka On Thu, Jul 19, 2018 at 9:33 AM +0300, "Zeta" <notifications@github.com<mailto:notifications@github.com>> wrote: @mccjpt<https://github.com/mccjpt> if still waiting for a copy of the paper. @mherzberg<https://github.com/mherzberg> did you find the paper online somewhere? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#9 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AYLIXoieP-NHJz2dGIrTyqtU8xM5mlJNks5uIChEgaJpZM4VOH4b>.

[mccjpt]

Please find it at github Juha-Pekka On Thu, Jul 19, 2018 at 9:33 AM +0300, "Zeta" <notifications@github.com<mailto:notifications@github.com>> wrote: @mccjpt<https://github.com/mccjpt> if still waiting for a copy of the paper. @mherzberg<https://github.com/mherzberg> did you find the paper online somewhere? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#9 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AYLIXoieP-NHJz2dGIrTyqtU8xM5mlJNks5uIChEgaJpZM4VOH4b>.

[hernanponcedeleon]

@mccjpt yes ... I found it immediately after writing the comment. Thanks!

[hernanponcedeleon]

Summary

This artifact accompanies a paper presenting a study in the investment needed to create DSLs, generators and related tooling.
The study is based on ten use cases: two cases implemented by the authors and 8 cases coming from different companies and based on industrial reports.
The study focus on languages created using the MetaEdit+ tool.

The artifact consist on an excel sheet presenting the data used in their experiments. Data is also shown as graphs

Assessment

Packaging: Met expectations

The artifact is packed as a single (excel) file. For such kind of artifact the packaging is ok.

Reproducibility and Consistency: (partially) Met expectations

The data is consistent with the results display in the paper.
For the two uses cases performed by the authors, I cannot really think about a way of reproducing the results. However, for the 8 cases coming from different industrial reports, the authors could have provided in the artifact the reports back-uping the data.

Documentation: Fell below expectations

There is no documentation at all. An excel sheet is normally self-explanatory, but the sheet is quite messy with a lot of information.

[TheoLeCalvar]

Summary

In the paper, the authors want to assess the time and effort needed to build ten different fully functional DSL/DSM.
Eight of them come from industrial reports and two from the author themselves.
This artefact consists in a spreadsheet containing detailed data about the effort invested in the creation of these two complete DSM.

Is the artefact consistent with the paper?

The artefact is consistent with the data presented in the paper.
It contains formulas used to generates the graphs used in the article.

Is the artefact as complete as possible?

Yes, the artefact contains the durations of each task of both case study developed by the authors.
However a sheet could have been added with durations of industrial cases.
Even if not detailed it could have been interesting to gather all data into a single shareable document.

Is the artefact well-documented?

The spreadsheet is not documented however its content is self explanatory.

Is the artefact easy to (re)use?

Given the type of artefact I doubt it is easily reusable as is.
However, given the descriptions of the case study is given they can be reproduced.

[grammarware]

Dear @mccjpt,

Based on all the comments and the reviews provided by the members of the Artifact Evaluation Committee of MoDELS 2018, we have reached the conclusion that this artifact conforms to the expectations and is hereby approved. Please use the badge instructions page to add the badge of approval to your article, and add the link to the artifact to the camera ready version of the paper.

Thank you very much for putting extra effort into the preparation and finalising of the artifact. If any of the comments above are still not addressed, please try to accommodate them before the conference.

In particular, we would recommend you to focus on two major points:

• Submit a snapshot of your artifact to ReMoDD, FigShare, Zenodo or any similar service, for archival purposes. Both GitHub and MetaCase are mature and stable, but even they are not immune to URI changes and similar trouble. It is also more reuser-friendly to package the entire artifact in one place (now there are three).
• If some documentation, even the most basic one, can be added, it could improve usability of the artifact. Our reviewers did not think that this is a major problem, but if you have the means, please consider accommodating.