mobisys-experiments

MobiSys Experiments

A MobiSys result becomes believable only when it is **measured on real hardware, driven to

steady state, and reported in latency, energy, memory, and thermal terms**. Simulation-only or

single-run numbers read as under-done. The evaluation axis here is the device and platform —

compute, energy, latency, memory, heat — not the radio or channel a wireless venue grades, and

not the traffic-and-topology an infrastructure venue grades. Build the evaluation as one

device-behavior claim per experiment.

Turn each claim into one device experiment

The clean structure gives each subsection a claim about the running system, then the one

experiment that settles it. The opposite habit — run a benchmark suite and describe whatever

survives — is the unfocused evaluation MobiSys reviewers push back on. The claim-to-experiment

map for a running mobile system:

| Claim about the system | Experiment that settles it | Metrics |

|---|---|---|

| It meets the target in the motivating scenario | on-device run under the real workload | p50/p95 latency, throughput, task success |

| Its energy cost is acceptable | instrumented run on a power monitor | energy-per-operation, average and peak power |

| It survives sustained use | multi-minute run to thermal steady state | frame-rate stability, temperature trace, throttle onset |

| Each mechanism earns its place | component breakdown / ablation | per-mechanism contribution |

| It fits within device memory | peak-memory and storage accounting | peak RSS, model and cache size |

| Its failure mode is understood | hot, low-memory, low-battery device states | the state where a baseline wins |

What the setup must state

• The full device context: phone/board model, SoC, OS build, framework/runtime

version, battery vs. wall power, and ambient temperature. A number without its device is not

interpretable.

• Define the energy boundary: the power instrument (on-device rail, external monitor,

shunt + DAQ), the sampling rate, and what is inside the measured envelope (SoC only? whole

device?). Do not let "efficient" stand in for a measured joules figure.

• Make load explicit: the workload trace, its duration, and whether the run reached

thermal steady state; a 30-second benchmark hides the throttling that a 20-minute run shows.

• Report ground truth honestly: for a service or inference claim, how correctness was

established and its own error.

Making energy and heat auditable

Battery, energy, and thermal claims are common at MobiSys and are held to instrument-level

scrutiny:

Energy/thermal report checklist:
  instrument: power monitor / on-device rail / shunt + DAQ, sampling rate
  quantity: energy-per-operation (mJ), not just average power (mW)
  boundary: what is inside the measured envelope (SoC? whole device?)
  thermal: skin/SoC temperature trace and whether steady state was reached
  battery: state-of-charge span, or wall power stated explicitly

An energy or thermal claim that cannot be re-derived from a described setup should not survive

your own audit.

Comparators a systems reviewer will trust

• Compare against the system people actually run, configured the way its own documentation

prescribes — an untuned baseline is the most common credibility wound in systems reviewing.

• Include the do-less baseline: the trivial fix (a smaller model, a fixed low frame rate,

more aggressive quantization). If the system cannot beat it at equal quality, that is the

finding.

• When a competitor cannot be run (proprietary runtime, unavailable device), reimplement and

label it a reimplementation, or compare on published numbers with the configuration deltas

stated.

Report the spread, never a lone peak

• Report percentiles and confidence intervals for latency, energy, and throughput; a

single "up to N×" without the distribution is a review risk (mobisys-writing-style).

• Show tails and CDFs for headline latency results, and repeat runs across enough

devices, thermal states, or battery levels to expose run-to-run spread.

• On-device results are device- and state-dependent — a number from one phone at full

charge in a cool room is not a claim about the system until the spread is characterized.

Pre-submission evaluation pass

• [ ] Every evaluation subsection answers a named question.
• [ ] Device, OS, runtime version, power source, and ambient fully specified per experiment.
• [ ] Energy measured with a described instrument and boundary, not estimated.
• [ ] Sustained-load run reaches steady state; thermal behavior reported.
• [ ] Incumbent-grade baseline present and tuned; do-less baseline present.
• [ ] Percentiles + tails for headline latency; multi-device or multi-state spread reported.
• [ ] At least one device state the system does not win, discussed rather than buried.
• [ ] Numbers in abstract/intro regenerate from the recorded runs.
• [ ] A hardware-optional or downscaled variant exists for artifact evaluators

(mobisys-artifact-evaluation).

Output format

[Evidence form] on-device measurement / deployment / user study / trace / simulation (claimed vs actual)
[Question map] question -> experiment -> metric (gaps flagged)
[Device+energy] setup and boundary specified? y/n per headline experiment
[Sustained load] steady state reached and thermal reported? y/n
[Baseline audit] incumbent tuned? do-less present?
[Break condition] device state where the system loses: <named or MISSING>
[Priority additions] ordered by review-risk reduction per bench-day