iros-artifact-evaluation

IROS Artifact Evaluation

IROS has no formal artifact-evaluation track or badge, so "artifact" here means the evidence bundle a

reviewer could inspect and that a careful author prepares anyway. The audience is an embodied-systems

reviewer who trusts a logged trajectory over a claimed one, and who cannot rerun your robot but can read

your logs, configs, and code.

The robotics artifact stack

• Hardware ledger: platform, sensors (make/model/firmware), actuators, onboard computer, and power

budget — the embodiment the result depends on.

• Logs: time-stamped rosbag-equivalent recordings for representative trials, so a claim about a

contact force or trajectory is auditable.

• Configs: the exact parameters, calibration, and launch files that produced the reported runs.
• Code: the system as run, with an entry point and a one-minute orientation in the README.
• Data: any dataset or object set, with provenance and licensing; for restricted data, enough detail

for credible reproduction without violating terms.

What an IROS reviewer inspects first

| Claim type | First artifact inspected | Common failure caught |

|---|---|---|

| "Runs onboard at rate X" | Timing logs and the compute/power spec | Rate measured on a desktop, not the robot |

| "Reliable across trials" | The trial log and failure record | Success rate with resets and failures omitted |

| "Transfers from sim" | Paired sim and real logs | Only sim logs exist; the gap is asserted |

| "Beats prior system" | Baseline configs on the same platform | Baseline run with different hardware or tuning |

Because a reviewer will read a log far sooner than rebuild a robot, make the logs and configs

legible first, and polish the code second.

Review-time versus acceptance-time states

• Review time (double-anonymous): strip organization names, cluster paths, calibration files tagged

with a lab, commit authorship, and any URL carrying a lab identity. If you link code, use an anonymized

mirror or omit the link and describe the artifact.

• Acceptance time: replace anonymized stand-ins with a public, licensed, citable release; test every

link from a logged-out browser; and add the DOI or archival link the IEEE Xplore record can point to.

Worked vignette: packaging a field-robot result

A submission claims a lidar-inertial system runs onboard across three outdoor sites.

• Ship one representative rosbag per site plus the launch/config that processed it, so a reviewer can see

the sensor streams and the estimated trajectory line up.

• Record the compute-and-power measurement method, not just a headline rate.
• Emit the trajectory-error tables directly from the logged runs so the paper numbers and the artifact

cannot drift apart.

• State which site is easy and which stresses the system, since that mapping is what a reviewer grades.

Artifact layout:
  /hardware.md      platform, sensors, compute, power
  /logs/<site>/     time-stamped bags for representative trials
  /config/          calibration, params, launch files
  /src/             system as run + README (1-minute orientation)
  /data/            provenance + license (or access instructions)

Output format

[Artifact role] review-time anonymous / acceptance-time public
[Contents] <hardware/logs/configs/code/data>
[Anonymity risks] <paths/metadata/URLs/calibration tags>
[Auditability] logged / scripted / described / asserted
[Fixes before upload] <ordered list>