kdd-artifact-evaluation

KDD Artifact Evaluation

Use this while the submission is being assembled — not after. KDD's review mechanics

create one hard constraint that reorders all artifact work: **the rebuttal phase does

not allow hyperlinks**, so the anonymized repository referenced inside the submitted

PDF is the complete and final artifact channel for the whole review. There is no

"we'll share code if reviewers ask"; asking happens in a phase where you cannot answer

with a link.

Artifact strategy by track

| Track | Primary artifact | What reviewers actually probe | Non-shippable core, and its substitute |

|---|---|---|---|

| Research | Anonymized code + configs + data loaders | Can the headline table be regenerated? Does the scale claim have a runnable path? | Massive datasets → downsampled slice + full-scale download script |

| ADS | Measurement definitions + pipeline skeleton | Are post-launch metrics precisely defined? Is the eval window stated? | Production data/code → metric spec, schema, synthetic replay generator |

| Datasets & Benchmarks | The dataset itself + loaders + baseline harness | License, provenance, documentation, versioning | Nothing — the artifact is the paper |

Building the anonymized repository

• Create a fresh repository, never a scrubbed clone: git history, CI configs,

issue templates, and .git metadata leak identity that no README edit removes.

• Sweep for: usernames in paths, institutional cluster names, cloud bucket names,

internal package registries, license headers, notebook execution metadata, and

dataset names that only one company uses.

• Cite the repository in the paper body (and optionally the abstract); after the

deadline this reference is unchangeable, so double-check the URL resolves logged out.

• Structure for a 10-minute reviewer, because that is the realistic inspection budget:

anonymous-artifact/
├── README.md            # 1 screen: claim -> command -> expected output table
├── env/                 # lockfile or container spec, exact versions
├── configs/             # one config per reported table/figure row
├── data/
│   ├── get_data.sh      # public downloads, checksums
│   └── sample/          # small slice so the pipeline runs in minutes
├── run.sh               # regenerates the smallest headline result end-to-end
└── results/expected/    # committed reference outputs for diffing
# smoke-test on a clean machine:
docker run --rm -v $PWD:/w -w /w python:3.11 bash -c "pip install -r env/requirements.txt && bash run.sh --sample"

Scale claims need scale artifacts

KDD reviewers read "scales to billions of edges" as a checkable claim, not marketing:

• Ship the throughput/memory harness, not only accuracy scripts; a scalability figure

without its benchmark script is the least-trusted plot in the paper.

• Record hardware, dataset cardinalities (rows/edges/events), and wall-clock per run in

a machine-readable manifest, so the repo answers "on what?" precisely.

• Where the full-scale run needs a cluster no reviewer has, provide the mid-scale run

that completes on one machine plus the exact extrapolation logic the paper uses.

ADS evidence packaging without leaking production

The ADS track requires quantified post-launch performance, but production data almost

never ships. Reviewers accept that trade when the package contains:

• Exact metric definitions (numerator, denominator, unit, aggregation window) for

every post-launch number in the paper.

• The measurement design: A/B split or pre/post window, traffic share, duration, and

what guardrail metrics were tracked.

• A synthetic or replayed data generator that exercises the same pipeline shape, so

the method is runnable even though the evidence is observational.

• An honesty note distinguishing measured production numbers from simulated ones —

mixing them silently is a rejection pattern.

After acceptance

• Replace the anonymous mirror with the public, licensed, citable repository; the

camera-ready must carry the durable link (see kdd-camera-ready).

• Prefer an archival deposit (DOI-stamped release) over a bare git URL for the version

that the proceedings reference.

• Check the current cycle for artifact-badging or code-availability initiatives before

assuming none exists (待核实 — such programs change year to year).

Vignette: packaging a billion-edge graph paper

A Research Track paper claims its sampler trains GNNs on a 3B-edge graph on one

machine. What the artifact must contain for that claim to survive contact with a

skeptical reviewer:

• get_data.sh that downloads the public 3B-edge graph (or constructs it from

public parts) with checksums — a scale claim on an unfetchable graph is attested,

not rerunnable, and should be labeled accordingly (kdd-reproducibility).

• A --scale small|medium|full switch: small finishes on a laptop in minutes and

validates the pipeline; medium reproduces one main-table row on a single GPU

overnight; full documents the exact hardware used for the headline.

• The peak-memory tracker wired into every run, because the paper's "one machine"

claim is really a memory claim.

• results/expected/ with per-scale reference outputs, so a reviewer's partial rerun

has something to diff against.

What it must not contain: the 40GB of intermediate artifacts (regenerable), the

authors' cluster submission scripts (identity leak), or a README promising "full

instructions after acceptance" — that sentence tells reviewers the artifact is

theater.

Common artifact failures at this venue

| Failure | Why it is fatal at KDD specifically |

|---|---|

| Repo created but never cited in the PDF | The link ban makes it undiscoverable during rebuttal |

| Git history preserved from the lab repo | Identity leak → desk-level anonymity problem |

| Accuracy scripts only, no efficiency harness | The paper's scale/efficiency axis becomes unverifiable |

| Sample data missing, full data gated | Reviewer's 10-minute budget ends at the download wall |

| ADS package with raw production extracts | Confidentiality violation risk transferred to reviewers |

Output format

[Artifact channel] repo cited in PDF: yes/no (if no: unrecoverable after deadline)
[Track register] research-repro / ads-deployment-evidence / dataset-release
[Regeneration level] one-command sample / scripted / descriptive only
[Scale evidence] throughput+memory harness: present / missing
[Anonymity sweep] <paths/history/metadata findings>
[Post-acceptance plan] <public repo, license, archival DOI>