hri-reproducibility

HRI Reproducibility

Reproducibility means something specific — and harder — at HRI than at a benchmark-driven venue. You

cannot "re-run" a study of an embodied robot and real people the way you re-run a model on a test

set: the participants, the room, the exact robot, and the moment are gone. So HRI reproducibility is

about making the procedure, stimulus, and analysis transparent enough that another team could

replicate the method and readers can audit your inferences. This skill builds that story; it

pairs with the shared kit in [../../resources/code/README.md](../../resources/code/README.md).

What "reproducible" can and cannot mean here

Be honest about the limits, and design the artifact around what is reproducible:

• Reproducible (aim for this): the study protocol, the robot behavior/stimulus, the

measures, the analysis code, and the de-identified data — enough to re-analyze your

results and to run the same study elsewhere.

• Not reproducible (say so): the exact human responses. A replication is a new sample; framing

it otherwise overclaims. State this plainly rather than let a reviewer catch the overclaim.

The replication package for an HRI study

Assemble, and reference from the paper:

• Study materials — consent-form summary, instructions/scripts, questionnaires with exact items

and scoring, interview guides, and coding schemes for qualitative work.

• Robot-behavior specification — the behaviors/conditions the robot exhibited, at enough

granularity to reproduce the stimulus: for autonomous behavior, the software version and

parameters; for Wizard-of-Oz, the wizard's action space, protocol/script, and error rates

(see hri-experiments).

• De-identified data — participant-level data with identifiers removed, plus a codebook. Scrub

free-text and metadata that could re-identify participants.

• Analysis code — scripts that reproduce every reported statistic, figure, and table from the

shared data, with fixed seeds and pinned dependencies.

• Pre-registration — a link/ID if you pre-registered the hypotheses and analysis, with any

deviations documented.

Report the things that make a study re-runnable

• Sample and procedure in reproducible detail: recruitment, inclusion criteria, N and how it was

determined, session flow, timing, compensation.

• Apparatus: the robot platform and version, sensors, environment, and anything that would change

the interaction if swapped.

• Wizard-of-Oz disclosure and constraints, so the stimulus is not an unlogged human performance.
• Analysis decisions: exclusions and why, transformations, the exact tests, and the

confirmatory/exploratory split.

De-identification and the video

HRI artifacts contain people, which raises obligations a code-only artifact does not:

• De-identify data before sharing; get consent for data sharing at study time — retrofitting it

is usually impossible.

• The video figure and any recordings need consent for the use you are making (review-time

sharing, public archival, or conference presentation are different permissions). For review, blur

faces and mute identifying audio if consent for public use was not obtained (see

hri-supplementary).

• Vulnerable populations (children, older adults, clinical) demand extra care in what is shared at

all.

Openness at HRI (calibrated to the venue)

HRI's open-science and artifact-badge culture is less formalized than SIGSOFT's — there is not a

long-standing, uniformly enforced ACM badge track for every edition (待核实 per cycle). So treat

openness as a credibility and community-norm move, not a checkbox:

• Share what ethics allows via a DOI-issuing archive (e.g., OSF, Zenodo, figshare); avoid a personal

homepage link that both rots and de-anonymizes.

• Where confidentiality or human-subjects constraints prevent full sharing, say what and why —

an honest limitation reads far better than silence or "available on request."

• If the edition does run an artifact/reproducibility evaluation, follow it and see

hri-artifact-evaluation.

Anonymization for double-blind

Everything above must be anonymized at review time: strip author/institution names from

materials and code, host the archive behind an anonymizing view, and scrub the video and data of

identity leaks. See hri-submission for the mechanical sweep.

Anti-patterns HRI reviewers flag

• Overclaiming reproducibility — implying a human study can be exactly re-run.
• Unlogged Wizard-of-Oz — no way to reproduce the stimulus.
• "Data available on request" — treated as not available.
• Identifiable data or video shared without consent for that use.
• Analysis not reproducible from the shared data — figures that no script regenerates.

Output format

[Repro scope] procedure/stimulus/analysis reproducible? human responses correctly framed as not?
[Package] materials · robot-behavior spec (incl. WoZ) · de-identified data + codebook · analysis code
[Pre-registration] link/ID · deviations documented?
[De-identification] data scrubbed · video/audio consent for the use made?
[Openness] DOI archive · honest statement of what cannot be shared and why
[Anonymization] materials/code/archive/video anonymized for review?
[Fix queue] <ordered>