percom-artifact-evaluation

PerCom Artifact Evaluation

Use this for reproducibility packaging. First, a cycle caveat: unlike SIGSOFT venues, PerCom

has not historically run a mandatory formal artifact-evaluation track with a fixed badge set, and

whether a given edition offers a reproducibility/badging track (e.g., **IEEE Open Research Objects

/ Results Reproduced) is 待核实** — confirm on the current call. Regardless of whether a badge

is offered, a well-packaged, de-identified sensing dataset and reproducible pipeline is a scored

strength in the double-blind review and a lasting community contribution.

What "reproducible" means for a ubicomp sensing paper

Two deliverables, kept distinct:

• The anonymized review package (at submission): dataset link and code scrubbed of owner,

testbed, and lab identity, for the double-blind reviewers.

• The public deposit (after acceptance): a de-identified dataset and code in a DOI-issuing

archive under an open license — the version others cite and reuse, and the version any badge

program evaluates.

What a ubicomp evaluator opens first

| Claim type | First thing inspected | Common failure caught |

|---|---|---|

| An activity/context recognizer | The script that regenerates cross-subject (LOSO) results | Only a pooled-accuracy script; no leave-one-subject-out path |

| A sensing dataset | The data itself + a datasheet (subjects, sensors, labels) | Link present, data missing; no de-identification described |

| A deployed system | A demo on bundled sample data | Only-runs-on-authors'-testbed; hardware not documented |

| A model result | Trained weights + inference on sample input | Requires the full raw dataset or private compute to run |

Assume an evaluator gives your package a bounded time budget on a clean machine with **no access to

your sensors or subjects**. Design for the first ten minutes — a demo on bundled, de-identified

sample data — to succeed.

Packaging plan

[Container]   ship a Dockerfile or a pinned environment (requirements/lockfile); avoid
              "install these 40 things by hand"
[README]      one-screen orientation: what it is, install, run the demo, reproduce each claim,
              expected runtime and outputs
[Datasheet]   a dataset datasheet: subjects (count, relevant demographics), sensors (device,
              firmware, sampling rate, placement), labels + protocol, and known biases
[Mapping]     an explicit table: paper claim -> script -> expected result (with the LOSO split)
[Data]        the de-identified dataset itself (or documented restricted-access), not just a query
[Ethics]      IRB/consent status and the de-identification performed before release
[License]     an open, DOI-issuing deposit (IEEE DataPort, Zenodo) so others can reuse and cite

De-identification is the ubicomp-specific bar

Human-subjects sensing data leaks identity in ways code does not: raw audio/video, GPS traces,

timestamps that pinpoint a home, and even accelerometer gait can re-identify. Before any release:

• Remove or transform direct identifiers and re-identifying traces; document exactly what you did.
• Confirm your consent and IRB approval permit public release of the de-identified data — some

approvals do not, and then the honest move is restricted access with a documented request path.

• Never ship a dataset that a subject did not consent to have published.

Worked vignette: a wearable-HAR dataset + recognizer

To make a HAR paper reproducible: ship a Docker image with the recognizer pre-built; a

run_demo.sh that classifies on a small bundled, de-identified sample in under a minute; a

reproduce/ directory whose scripts regenerate the leave-one-subject-out F1 table (not just a

pooled number) from logged features; a datasheet listing subjects, sensor placement, and sampling

rate; the de-identified dataset with a documented consent/IRB basis; and an open license with a

DOI. State honestly which results are turnkey and which need the full (slow) training run.

Calibration

• Whether a badge/reproducibility track runs, and which badges, is 待核实 per cycle — confirm

on the current call rather than assuming a SIGSOFT-style scheme.

• The DOI-issuing deposit and datasheet are portable value even when no badge is offered.
• Anonymize the review package; de-identify (and confirm consent for) the public dataset — these

are different obligations.

Output format

[Track status] formal reproducibility/badge track this cycle? yes/no/待核实
[Artifact role] anonymized review package / public de-identified deposit
[Contents] <recognizer/dataset/datasheet/scripts/ethics/license>
[Ten-minute test] does install + demo on bundled sample data succeed on a clean machine? yes/no
[Cross-subject reproduction] does a script regenerate the LOSO result? yes/no
[De-identification] documented + consent/IRB permits release? yes/no
[Fixes before deposit] <ordered list>