acmmm-reproducibility

ACM MM Reproducibility

Use this to make an ACM Multimedia result rebuildable — both for main-track credibility and

for the dedicated Reproducibility track, which routes artifacts through ACM's

badging pipeline. Multimedia adds a wrinkle: the data is often video, audio, or interactive

media, and "run the code" is not enough if a reviewer cannot obtain or render the media.

What reproducibility means here

ACM's artifact model distinguishes availability, evaluation, and reproduction. Map your goal

to the badge you are actually pursuing:

| Badge (ACM terminology) | What it asserts | What you must ship |

|---|---|---|

| Artifacts Available | The artifact is publicly, permanently retrievable | A DOI/archived repository with the code and media pointers |

| Artifacts Evaluated (Functional/Reusable) | Reviewers ran it and it works / is reusable | Build + run instructions, environment, documentation |

| Results Reproduced | An independent team reproduced the paper's results | A pipeline that regenerates the reported numbers/media |

Confirm the exact badge set offered for the current cycle on the Reproducibility-track call;

ACM's badge names and criteria evolve.

The multimodal reproducibility ledger

Keep a single record that ties each reported result to the code, data, and config that

produced it:

result: Table 2, row "full model"
  code commit: <hash>
  config: configs/full.yaml
  data: <dataset name + version + anonymous mirror for review>
  media preprocessing: <fps, sample rate, caption source>
  seed(s): <list>
  hardware: <GPU/CPU, hours>
  expected output: results/table2_full.json

Media and data access

• Provide an anonymous, working path to the data during double-blind review — a mirror

that a reviewer can actually download, not a placeholder.

• State the license and any consent/usage terms; user-generated media often cannot be

redistributed, so document how a reviewer obtains it.

• Pin preprocessing: frame rate, resampling, transcription source, and alignment — small

differences here silently break multimodal results.

Determinism where it is achievable

• Fix and log seeds; note where nondeterminism is irreducible (e.g., some GPU kernels) and

report variance instead of pretending to bit-exactness.

• Version the environment (container or lockfile) and record hardware, since media models are

often memory- and throughput-sensitive.

Reproducibility-track readiness pass

• The Reproducibility and Open Source tracks are single-blind, so the artifact carries

its real identity — but the main-track review artifact must still be anonymous.

• Package for a stranger: a reviewer with your README and nothing else should build, run, and

hit an expected-output check within a bounded time.

• Include a smoke check (see [../../resources/code/README.md](../../resources/code/README.md))

that verifies structure and media rendering before you submit.

Where multimodal pipelines silently break

Multimedia reproduction fails in places pure-code reproduction does not:

• Codec and container drift — a video re-encoded with a different codec changes pixel values

and breaks frame-exact results; pin the decode path.

• Sample-rate and resampling — audio resampled by a different library shifts features; record

the exact resampler and rate.

• Caption/transcript source — if captions come from an ASR system or a platform, name the

version; a different transcript is a different input.

• Frame sampling — "every k-th frame" depends on the container's frame rate; state fps and

the sampling rule.

A reproduction package that omits these looks complete but regenerates different numbers, which

is worse than an honest gap.

Anonymous review vs. public artifact

The review artifact and the release artifact have different rules, and conflating them causes

anonymity leaks or dead links:

• During review (double-blind tracks): anonymous repository, anonymous data mirror, no author

names in code comments, media metadata, or commit history.

• At release (camera-ready): the public, de-anonymized repository with a permanent archive

(DOI), the license, and the final media — replacing, not merely supplementing, the anonymous

mirror.

Output format

[Badge target] Available / Evaluated / Results Reproduced
[Ledger] complete / gaps: <which results lack a trace>
[Data access] anonymous + licensed / broken or unlicensed
[Media preprocessing] pinned / underspecified
[Determinism] seeds+env logged / gaps
[Track blinding] correct for chosen track / mismatch
[Top fixes] <ordered>