How to (not) Artifact your code

Art of Artifacting

Some of my experience during my research work on building the artifact for our paper.

Disclaimer: Purely out of my own experiences which is likely to refine/modify as I go along!

  1. List the requirements of software/OS/driver and hardware.
  2. Any library dependencies, version, etc. You must mention the environment it is tested on!
  3. Perform all the baseline computations on the same hardware platform.
  4. Include all the testcases or input instances or have makefile to wget form URL.
  5. Get instances from published work URLs. Eg. CVRP LIB, STEINLIB, TSPLIB, DIMACS.
  6. Mostly, your code should handle different input formats (or parts) without editing inputs.
  7. In the worst case, if we are modifying the inputs, we should have provided the converter
  8. Executable should’ve help/man options – list all runtime args+their default values if not set.
  9. The output should’ve meaningful labels than printing out just numbers.
  10. We should print the time, solution value and solution (optional).
  11. The validation routine can be included (in or separate) that prints if the solution is in/valid.
  12. We can enable/disable via arg some or part of the output.
  13. This holds for approx algorithms and parallelizing algorithms from sequential implementation to match/check the expected output.

More points.

Legends

★ 2 min read · Rajesh Pandian M · research , artifact