icra-reproducibility

ICRA Reproducibility

Reproducibility at a robotics conference means something different than at an ML

venue: no reader can rerun your hardware trials without your robot, your room,

and your objects. The achievable standard is therefore two-tier — rerunnable

(code, simulation, analysis) and auditable (hardware evidence logged well

enough that a skeptic can verify you did what you claim). ICRA papers earn trust

by being explicit about which tier each result sits in.

The specification ledger

A robotics result is unreproducible if the platform is underspecified. The paper

(body, since there is no appendix escape) plus released materials should pin:

| Layer | Must specify | Example of "enough" |

|---|---|---|

| Robot | model, DOF, payload, firmware version | "UR5e, PolyScope 5.11" |

| End effector | model, mods, wear state if relevant | "Robotiq 2F-85, stock pads" |

| Sensors | model, resolution, rate, mounting frame | "D435i @ 640×480, 30 Hz, wrist" |

| Compute | on-board vs off-board, GPU, latency path | "off-board 3080, 5G-free wired" |

| Middleware | ROS distro, key package versions | "ROS 2 Humble, MoveIt 2.5.4" |

| Control | loop rates, gains or their source | "impedance @ 500 Hz, gains in repo" |

| Calibration | method + residual error | "hand-eye via [ref], 1.8 mm RMS" |

| Objects/scenes | identity, source, dimensions | "YCB subset listed in Tab. II" |

Standard object sets (YCB and similar) exist precisely so that "graspable

household objects" can be a checkable claim; use them or publish the object

specs.

Logging: the rosbag is the lab notebook

• Record every hardware session — including failed ones — with topic-complete

rosbags (or vendor-equivalent logs) plus an untouched camera view.

• One log per trial, named by protocol: trial_<cond>_<idx>_<date>.bag; the

analysis pipeline should consume these logs, so paper numbers regenerate from

raw data with one command.

• Keep a session sheet: date, operator, firmware state, anomalies. Two months

later, this distinguishes "the gripper was recalibrated between Tables II and

III" from unexplainable drift.

• The video attachment should be cut from these recorded sessions, which

guarantees footage corresponds to logged, reported trials.

Determinism where it exists, honesty where it doesn't

Hardware trials are not seed-reproducible — contact, cables, lighting, and

thermal drift see to that. Split the claims accordingly:

• Deterministic tier: training runs, simulation experiments, and analysis

scripts get pinned seeds, pinned dependency versions, and containerized

environments. State seeds and repetition counts.

• Stochastic tier: hardware trials get protocol reproducibility — the

documented procedure another lab could follow — plus dispersion reporting.

• Never write "results are fully reproducible" over a hardware table; write

"code and simulation reproduce Tables I-II; hardware protocol and logs for

Tables III-IV are released for audit."

Release scaffold

repo/
├── README.md            # claim → command map (Table III = run_hw_analysis.sh)
├── env/                 # Dockerfile or rosdep/lockfiles, pinned
├── sim/                 # simulation experiments, seeds fixed
├── control/             # controllers/planners as ROS packages
├── hw_protocol.md       # step-by-step trial procedure, success criterion,
│                        #   reset procedure, safety notes
├── logs/                # rosbags or download script (size!), session sheets
├── analysis/            # regenerates every figure/table from logs/
└── video/               # raw clips ↔ trial IDs mapping

Anonymize the repository for review under the double-anonymous policy (2026

cycle onward): anonymized hosting or an artifact stripped of names — an org URL

is an identity leak (see icra-submission).

Compute and latency reporting

Robotics adds a real-time dimension absent from ML reproducibility norms:

• Report where each computation ran (on-board CPU/embedded GPU vs off-board

workstation) and the link between them; an off-board 3090 over gigabit wire

is a materially different system than on-board inference.

• State end-to-end latency per pipeline stage (sensing → perception → decision

→ actuation) at least once; "runs in real time" is unverifiable without it.

• Training compute (GPU-hours, hardware) belongs in the paper for learned

components — cheap to report, increasingly expected by review forms.

Availability statement patterns

Honest, specific statements outperform boilerplate promises:

• "Code, simulation environments, and trial logs: <link> (anonymized for

review; public on acceptance)."

• "The gripper design files are released; the arm is commercial (UR5e)."
• "Raw rosbags total 1.2 TB; per-trial extracted features and a 40 GB sample

are hosted, full logs on request."

• Avoid: "code will be released upon publication" with nothing at review time —

reviewers discount unverifiable promises, and some cycles ask availability

questions directly on the review form (verify the current form).

Pre-submission reproducibility pass

1. Specification ledger complete in-paper (table above, compressed)?
2. Every reported number regenerates from logs via one documented command?
3. Seeds/versions pinned for the deterministic tier; container builds clean?
4. Hardware protocol written so a stranger could run trial 1 tomorrow?
5. Release scaffold anonymized; links leak-checked?
6. Availability statement matches what actually exists today?

Output format

[Tier split] rerunnable: <tables/figs> | auditable: <tables/figs>
[Spec ledger] missing rows: <list or none>
[Log coverage] trials logged <n>/<n>, analysis regenerates: y/n
[Determinism] seeds/versions pinned for sim+training: y/n
[Release] scaffold complete? anonymized? size plan?
[Statement] honest-specific / boilerplate-risk — rewrite: <line>