mobisys-reproducibility

MobiSys Reproducibility

Use this before submission and again before the artifact-evaluation deadline. On-device

results are fragile across hardware and framework versions, so reproducibility at MobiSys is

mostly about provenance: recording enough of the device and measurement context that

someone on a different phone can rebuild the result.

Evidence map

• Map each latency, energy, throughput, and accuracy claim to a verifiable location in the

paper, appendix, or artifact package.

• For each on-device result, record device model, SoC, OS build, framework/runtime version,

model checkpoint hash, power source, ambient temperature, and thermal state at run start.

• Record the power-instrument setup and the energy boundary so an energy figure can be

re-derived rather than trusted.

• For stochastic results, report seeds, repeated runs, and the spread; a single number from a

single phone is not reproducible evidence.

• Explain missing data, firmware, or hardware honestly, and describe how a reader could

reproduce the result in principle on a different device.

• Keep the artifact consistent with the manuscript; a number in the PDF that the artifact

cannot regenerate is a review-risk multiplier and an AEC failure.

What survives a different device

| Provenance item | Turnkey answer | Common failure caught |

|---|---|---|

| Device + SoC + OS build | Named per experiment, in a table | "on a mobile device" with no model |

| Framework / runtime version | Pinned with hashes | Result drifts silently across a minor version |

| Energy instrument + boundary | Instrument, sampling rate, envelope stated | "efficient" with no measured joules |

| Thermal state | Ambient + steady-state trace | 30-second benchmark hides throttling |

| Model / cache version | Checkpoint hash, cache size | Accuracy irreproducible from a different weight file |

Marking a device result "reproducible" without its provenance is a recognizable MobiSys red

flag, because the AEC and reviewers cross-check the artifact against the PDF and read a gap as

carelessness about the rest of the paper.

Vignette: an on-device inference paper

Consider a runtime that keeps CNN latency stable under load, validated on four phones. Its

reproducibility spine: the four device models and OS builds, the runtime version and model

checkpoint hashes, the power-monitor model and sampling rate, the ambient temperature and the

steady-state thermal traces, the workload trace, the seeds and run counts — plus one honest

sentence about the device it could not test and why the result may differ there.

Degrees of reproducibility

• Turnkey: one script regenerates each figure from logged runs on the target device.
• Scripted: scripts exist but need the specific hardware or manual device setup.
• Descriptive: prose detailed enough that a competent reader could rebuild the pipeline on

a comparable device.

For MobiSys, the on-device measurement path should be as turnkey as the hardware allows, and a

hardware-optional downscaled or emulator path should let an evaluator without your exact

device reach at least Functional (mobisys-artifact-evaluation). State the achieved level

honestly rather than overpromising turnkey behavior that fails on a different phone.

Legal and privacy provenance

• Decide early what traces, firmware, and datasets can ship; on-device data often carries

user, sensor, or vendor-firmware restrictions.

• Strip identifying content from screenshots, logs, and demo media before release.
• If data cannot ship, provide a synthetic or public substitute and name which results it

cannot reproduce.

Output format

[Claim inventory] <claim -> evidence location>
[Provenance status] complete / partial / missing (device/OS/runtime/energy/thermal)
[Reproducibility gaps] <device pinning / seeds / energy boundary / model version>
[Paper fixes] <must appear in main PDF>
[Artifact fixes] <appendix or artifact additions, incl. hardware-optional path>