How to (not) Artifact your code
Art of Artifacting
Some of my experience during my research work on building the artifact for our paper.
Disclaimer: Purely out of my own experiences which is likely to refine/modify as I go along!
- List the requirements of software/OS/driver and hardware.
- Any library dependencies, version, etc. You must mention the environment it is tested on!
- Perform all the baseline computations on the same hardware platform.
- Include all the testcases or input instances or have makefile to wget form URL.
- Get instances from published work URLs. Eg. CVRP LIB, STEINLIB, TSPLIB, DIMACS.
- Mostly, your code should handle different input formats (or parts) without editing inputs.
- In the worst case, if we are modifying the inputs, we should have provided the converter
- Executable should’ve help/man options – list all runtime args+their default values if not set.
- The output should’ve meaningful labels than printing out just numbers.
- We should print the time, solution value and solution (optional).
- The validation routine can be included (in or separate) that prints if the solution is in/valid.
- We can enable/disable via arg some or part of the output.
- This holds for approx algorithms and parallelizing algorithms from sequential implementation to match/check the expected output.
- We must bundle the artifact after submission.
- We must have git/zenodo URL with
READMEmust have instructions to build and reproduce your/baseline results.
- We must have a script that runs all the inputs.
- We must have a script to run each/all baselines.
- Artifact. Source code of a research paper to get its plot/fig/times
- baseline. Code/comparison paper
- exe. Executable.
- testcase. Input instances or file for the program.