mobisys-artifact-evaluation

MobiSys Artifact Evaluation

Use this for the MobiSys artifact submission. MobiSys has a strong, script-first artifact

culture: the single-blind AEC runs the authors' workflow scripts on the reviewers' own

systems, so an artifact that only works on your bench is not an artifact.

The three ACM badges

Authors may seek one, two, or all three independent badges; passing badges are printed on the

paper and recorded in the ACM DL:

| Badge | What it certifies | What the AEC does |

|---|---|---|

| Artifacts Available | The artifact is publicly archived at a stable location | Checks the archival link and license |

| Artifacts Evaluated — Functional | The artifact runs and does what the paper says | Runs your workflow scripts on the AEC's machines |

| Results Reproduced | Key paper results are reproduced independently | Regenerates headline numbers/figures from your package |

Choose badges as claim calibration: Available is a release commitment; Functional

needs a runnable package; Results Reproduced needs the headline device results to come back

on machines you do not control — the hardest badge for on-device work.

Artifact plan

• Decide what evidence supports each badge: source, build scripts, model files, workload

traces, and the measurement/plotting pipeline.

• Provide a top-level workflow script and a README that orients an evaluator in one minute:

environment, dependencies, one command to smoke-test, one to reproduce a headline figure,

expected outputs, and runtime.

• Anonymize repository history, paths, screenshots, notebook metadata, license headers, and

device serial numbers for the review round; de-anonymize only at camera-ready.

• Pin framework, runtime, and model checkpoint hashes; on-device results drift across

framework minor versions.

The hardware-optional path (the MobiSys-specific move)

On-device artifacts fail Functional most often because the evaluator lacks the exact phone,

wearable, or board. Design for that from the start:

Hardware-optional package layout:
  /device      full pipeline for the exact hardware (Results Reproduced target)
  /emulated    downscaled or emulator path an evaluator can run without the device
  /logs        recorded runs so plots regenerate even if nothing executes
  /README      states which claims each path can and cannot reproduce

State plainly which headline numbers the emulated path cannot reproduce (typically absolute

energy and thermal behavior) so the AEC grades the right badge instead of failing the whole

package.

What AEC reviewers open first

• The README and the top-level workflow script — if these do not run cleanly, the badge stalls

regardless of how good the system is.

• The one-command smoke test — make it fast and hermetic.
• The plotting pipeline for a headline figure — emit figures directly from logged results so

the paper numbers and artifact numbers cannot drift.

Calibration anchors

• Artifact evaluation is single-blind and separate from paper review; the badge names and the

artifact deadline are cycle-specific — verify against the current Artifact Evaluation page

rather than past years.

• Absolute on-device energy and latency are the least portable results; scope

Results Reproduced claims to what survives on another evaluator's hardware.

Output format

[Badges sought] Available / Functional / Results Reproduced
[Package contents] <source/scripts/models/traces/plotting>
[Hardware-optional path] present? which claims it cannot reproduce
[Anonymity risks] <paths/screenshots/serials/repo owners>
[Reproduction level] turnkey / scripted / descriptive / weak
[Fixes before upload] <ordered list>