Artifact Evaluation for ICESS
Have you ever complained that you cannot reproduce results from a paper?
Have you ever wished you could have someone else validate your work?
Have you ever tried to validate someone else’s work?
Artifact evaluation (AE) is intended to help check experimental results from the accepted papers (see more information in the links below). AE is not used to prove formally artifacts are correct and are not a way to publish software. Artifacts are kept confidential and under control of the authors. Notifications of “passing” evaluation will be sent to the authors approximately 4 weeks after artifact is submitted.
This process is relatively straightforward. Participation in the AE is optional by submitting all related research material (a guide/README, simulators, tools, benchmarks, data sets, configuration files, …) necessary to check claims and results of their accepted paper. The guide is a README-style file that describes the artifact and experimental procedure for their artifact. Two reviewers will follow your guide, install your software, rerun your experiments, recreate your graphs, and send you a report with the overall assessment of your artifact. At this time, due to the difficulty of validating software on special embedded hardware, we only accept simulation results or experimental results that do not require special hardware.
The AE webpage will recognize only all the artifacts that passed evaluation. The discussion period is to enable authors to improve their artifacts for eventual sharing, which is the goal of this exercise. The AE chairs will mediate all communication to keep the reviewers’ identities confidential. The chairs also offer help in packaging the artifact to the first 5-10 authors that request help, using the EASE system.
|Artifacts due:||15th June - 1st July|
|Artifact review:||1st July - 22nd July|
Feedback step/Discussion period: In the 3 weeks following submission, the evaluators and authors will potentially engage in a helpful discussion. Evaluators give initial feedback to authors about any problems in the first few days, let the authors fix them, and then resume the evaluation. This is to make sure reviewers can unpackage and run the artifact before doing a deeper evaluation.