uist-reproducibility

UIST Reproducibility

Reproducibility at UIST means something different from rerunning a training script:

the question is whether a competent lab could **rebuild the artifact and reproduce

its measured behavior**. That decomposes into three ledgers — build, measurement,

and study — and most UIST papers under-specify a different one than they think.

UIST posts no reproducibility checklist (none found for 2026 — 待核实), so this

discipline is self-imposed and reviewer-enforced.

The build ledger: could they make one?

Everything the system's behavior depends on, pinned:

• Software: exact framework and driver versions, OS, lockfiles; for anything

learned, model checkpoints and training data provenance.

• Hardware: part numbers (not "an IMU" but which IMU), mechanical tolerances

that matter, firmware version, and the calibration routine with expected outputs.

• Environment: the physical conditions the system assumes — lighting range,

acoustic environment, mounting geometry, surface materials.

• Magic numbers: every threshold, filter coefficient, debounce window, and

gain, with how each was set (tuned by hand? on which data?). These constants are

where re-implementations actually fail.

The measurement ledger: could they get your numbers?

A latency or accuracy figure is reproducible only with its protocol:

| Reported number | Must be pinned |

|---|---|

| Latency | Measurement boundary (sensor-to-photon? software-only?), instrument, event count, load conditions |

| Recognition accuracy | Dataset splits, per-user vs pooled, session separation, chance level |

| Tracking error | Ground-truth apparatus and its own accuracy, spatial sampling grid |

| Throughput / bitrate | Task, phrase set or corpus, session and rest structure |

| Power / weight / cost | Configuration measured, currency and date for cost |

Automate the protocol: a measurement harness checked into the supplement converts

"trust me" into "run this" (see uist-artifact-evaluation for packaging).

# eval/rerun.sh — regenerate every reported number from raw logs
python analyze_latency.py logs/latency_10k.jsonl --out tables/table1.csv
python analyze_accuracy.py logs/study/ --split per-user --seed 17 --out tables/table2.csv
diff -u tables/table1.csv paper_tables/table1.csv   # drift check against the PDF

Log at the event level with timestamps and raw sensor values, not just computed

outcomes — future you, rebuttal you (see uist-author-response), and replicating

labs all consume the same logs.

The study ledger: could they rerun the human part?

For any user evaluation: full task instructions and stimuli, counterbalancing

scheme, practice/rest structure, apparatus placement (photograph it),

inclusion/exclusion criteria, compensation, and the analysis scripts from raw logs

to reported statistics. Share instruments even when raw human data cannot leave the

IRB envelope — protocol transparency and data availability are separable, and

saying so precisely is the honest move.

The availability statement

Write one even though UIST does not require a template, and make it specific:

GOOD: "Firmware, PCB design files, BOM, and the measurement harness are at
      <archive-DOI> (tag: as-published). The gesture corpus (14 of 16
      participants consented to release) is included; per-participant raw
      video is withheld under IRB #—. The two commercial tracking SDKs
      required are named in BUILD.md with tested versions."
BAD:  "Code available upon reasonable request."

Papers with learned components inherit ML reporting norms too — seeds, training

configs, compute — and can borrow the shared kit's checklists (see

[../../resources/code/README.md](../../resources/code/README.md)).

Learned components inherit ML norms

When the system embeds recognition or generation models, the ML reporting

conventions stack on top of the systems ledgers:

• Seeds, splits, and training configs pinned; per-user vs pooled evaluation made

explicit (interaction data is brutally user-dependent, and pooled splits

inflate accuracy).

• For third-party or hosted models: exact model identifiers and versions, the

full prompt set, decoding parameters, and dated transcripts — hosted models

drift, so the logged behavior is the only permanent record of what reviewers

saw.

• Report the interactive costs alongside accuracy: per-inference latency on the

deployment hardware, and per-session cost for metered APIs; a technique that is

reproducible but unaffordable to run is only half-replicable.

• The shared kit's checklists cover this lane; the adapter in

resources/code/README.md scopes what it can and cannot check.

The one-page REPRO.md

Compress the three ledgers into a single file at the archive root:

# Reproducing <SystemName> (paper §6-7)
## Rebuild        parts: BOM.csv · firmware: v2.3 · calibration: docs/calib.md
## Environment    tested: indoor 200-800 lux, 18-26°C · assumes: mounted per Fig 4
## Constants      thresholds in config.yaml — tuned on pilot data (n=4), §5.2
## Remeasure      eval/rerun.sh regenerates Tables 1-2 from logs/ (or raw capture)
## Study          instruments/ · counterbalancing: latin square, §7.1 · IRB #—
## Known drift    accuracy drops outdoors (§9); p95 latency sensitive to BLE stack

The "known drift" line is the credibility multiplier: it tells replicators you

know where the envelope ends before they find out.

Replication drift and its uses

When a rebuilt system misses the paper's numbers, the causes rank: unstated

environmental assumptions, hand-tuned constants, part substitutions, then genuine

bugs. Pre-empt the first two by stating the envelope and the tuning story in the

paper body — it costs three sentences and buys the paper years of credibility (and

Lasting Impact eligibility runs on decade-scale credibility).

Output format

[Build ledger] pinned / gaps: <software · hardware · environment · constants>
[Measurement ledger] protocols pinned for <k>/<n> reported numbers
[Study ledger] instruments · counterbalancing · analysis scripts — present?
[Availability statement] drafted? honest about withholdings?
[Top drift risk] <the unstated assumption most likely to break replication>