mobicom-experiments

MobiCom Experiments

MobiCom's evidence culture is physical: a mobile/wireless mechanism is believed when it is

measured over the air, on real hardware, under conditions that resemble deployment. A

simulation-only or single-run evaluation reads as under-done here. Design the evaluation as

a set of questions about the mechanism, then build the smallest measurement campaign that

answers them on real radios.

Questions before measurements

Write the evaluation's subsection titles as questions first — *Does the mechanism hold under

mobility? What does it cost in energy at rest? When does the channel defeat it?* — then

design one experiment per question. The inverted approach (run everything, narrate the

survivors) produces the benchmark tour that MobiCom reviews call unfocused.

A minimal matrix for a wireless-mechanism paper:

| Question | Experiment | Metrics that answer it |

|---|---|---|

| Does it work in the motivating condition? | over-the-air run under the target channel/mobility | delivery rate, goodput, SNR/BER distribution |

| What does it cost when idle? | baseline with no stress | energy-per-bit, power draw, CPU/airtime overhead |

| Why does it work? | component breakdown / ablation | per-mechanism contribution |

| Does it scale? | node / distance / density sweeps | knee location, per-node or per-meter curve |

| When does it break? | interference, deep fades, high mobility | the regime where baselines win |

| Does it hold over time? | multi-hour or multi-day runs | drift, stability, tail behavior |

Measurement methodology reviewers check

• State the RF setup completely: radio/SDR model and firmware, carrier frequency,

bandwidth, transmit power, antenna and gain, and receiver chain. A number without its

radio context is not interpretable.

• Name the channel condition: distance, line-of-sight vs multipath, ambient

interference, and how you characterized it (RSSI/CSI traces, coherence time). Do not let

"in our lab" stand in for a channel description.

• Make mobility explicit: walker paths, speeds, and schedules for mobility experiments,

and a stationary control. "Under mobility" without the traces is unverifiable.

• Report ground-truth honestly: how position, gesture, or decode-correctness truth was

obtained, and its own error, since a sensing result is only as good as its reference.

Energy is a first-class metric

Battery and harvested-energy claims are common at MobiCom and are held to instrument-level

scrutiny:

Energy report checklist:
  instrument: power monitor / shunt + DAQ, sampling rate
  quantity: energy-per-bit or per-operation, not just average power
  boundary: what is inside the measured envelope (radio only? whole tag?)
  budget: for harvested/batteryless designs, the source and the duty cycle

An energy claim that cannot be re-derived from a described measurement setup should not

survive your own audit.

Baselines on tuned hardware

• Compare against the incumbent people actually run, configured the way its own

documentation prescribes — an untuned baseline is the most common credibility wound in

systems and wireless reviewing.

• Include the do-less baseline: the trivial fix (more power, a fixed high rate, another

antenna). If the mechanism cannot beat it at equal cost, that is the finding.

• When a competitor cannot be run (proprietary radio, unavailable hardware), reimplement and

label it a reimplementation, or compare on published numbers with the configuration deltas

stated.

Distributions, not superlatives

• Report percentiles and confidence intervals for delivery, latency, and throughput

claims; a single "up to N×" without the distribution behind it is a review risk

(mobicom-writing-style).

• Show CDFs for headline results and repeat runs across enough channel realizations,

days, or walker paths to expose run-to-run spread; state what varies between repeats.

• Wireless results are time- and place-dependent — a result from one room at one hour is

not a claim about the mechanism until the spread is characterized.

Audit checklist

• [ ] Every evaluation subsection answers a named question.
• [ ] RF setup, channel condition, and mobility fully specified per experiment.
• [ ] Energy measured with a described instrument and boundary, not estimated.
• [ ] Incumbent-grade baseline present and tuned; do-less baseline present.
• [ ] Percentiles + distributions for headline metrics; multi-run spread reported.
• [ ] At least one condition the mechanism does not win, discussed rather than buried.
• [ ] Numbers in abstract/intro regenerate from the recorded runs.
• [ ] A hardware-optional or downscaled variant exists for artifact evaluators

(mobicom-artifact-evaluation).

Output format

[Evidence form] over-the-air testbed / deployment / trace / simulation (claimed vs actual)
[Question map] question -> experiment -> metric (gaps flagged)
[RF+channel] setup and conditions specified? y/n per headline experiment
[Energy] measured, with instrument and boundary? y/n
[Baseline audit] incumbent tuned? do-less present?
[Break condition] regime where the mechanism loses: <named or MISSING>
[Priority additions] ordered by review-risk reduction per testbed-week