dac-experiments

DAC Experiments

Use this before the November deadline when the evaluation is not yet locked. At DAC the evaluation

is the paper: reviewers are EDA practitioners who decide acceptance mostly on whether the **QoR

comparison is fair, standard, and reproducible. The organizing principle is measured design

quality against the strongest baseline on recognized benchmarks** — not novelty in the abstract.

Evaluation audit

• Use standard benchmark suites. Match the suite to the task: ISPD placement/routing

contests, the EPFL combinational benchmark suite for logic synthesis, ISCAS'85/'89 and

ITC'99 for test/verification, the TAU contests for timing, CircuitNet/OpenABC-D and

similar for ML-for-EDA, and OpenROAD / OpenROAD-flow-scripts for full-flow experiments. A

private-benchmark-only evaluation is a scored weakness.

• Compare against the true state of the art, tuned with a documented, equal effort. An untuned

or outdated baseline is the most common DAC reject cause; the assigned reviewer often is the

author of the stronger tool you skipped.

• Report the whole suite, not a subset. Per-benchmark tables with the full circuit set; a

cherry-picked average invites "what happened on the circuits you dropped?"

• Report runtime and scalability, not just quality. EDA reviewers care whether the method scales

to realistic design sizes (millions of cells), so include the largest benchmarks and the compute

used.

• Ablate the mechanism. Isolate where the gain comes from — remove your key component and show

the QoR degrade — so the reviewer can attribute the improvement to your idea, not to tuning.

• Design threats in. Know before you run which designs, PDKs, or corners limit generality, and

instrument to bound them.

Claim-to-evidence design table

| DAC claim | Matching evidence | Reject pattern avoided |

|---|---|---|

| "Reduces wirelength / congestion" | Per-benchmark WL/DRC on ISPD vs a tuned SOTA placer | "Only averages; weak baseline" |

| "Closes timing better" | WNS/TNS across TAU/real designs, equal area/power | "Improved slack by hurting area silently" |

| "Fewer verification escapes / more coverage" | Coverage/bug-find on ISCAS/ITC or real RTL vs prior tool | "Toy circuits only" |

| "Scales to large designs" | Runtime/memory at million-cell scale | "Small benchmarks; scalability asserted" |

| "ML method predicts QoR" | Held-out designs, error metrics vs analytical/prior-ML baseline | "Trained and tested on the same designs" |

| "The new component drives the gain" | Ablation removing it | "Contribution and tuning entangled" |

PPA and QoR reporting floor

• Report the full PPA picture: a wirelength or timing win that silently costs area or power is

not a win — show the trade-off.

• Give per-benchmark numbers and the aggregate; state the geometric-mean convention you use.
• Report runtime for both your method and the baseline on the same hardware, and the hardware.
• For stochastic methods (simulated annealing, RL-based flows) report variance across seeds/runs,

not a single lucky run.

Contamination-aware ML-for-EDA evaluation

When a learner is in the loop, the reviewer's first questions are about leakage and fairness:

[Split integrity]  train and test on DIFFERENT designs/netlists; never leak a test design into
                   training. Report the split explicitly.
[Baseline]         compare against the strong non-ML tool (analytic placer, classical STA) AND the
                   prior-ML method, not just an untrained control
[Generalization]   evaluate on designs/technology nodes unseen in training; ML-for-EDA that only
                   works on its training distribution is a scored weakness
[Determinism]      report seeds and variance; a single run is not evidence for an RL flow
[Cost honesty]     report training cost and inference cost; a method needing per-design retraining
                   must say so
[Data provenance]  name the dataset (CircuitNet, OpenABC-D) and version; cache generated data

Vignette: evaluating a new global router

A paper claims a router that cuts congestion at equal wirelength. The matching plan: run on the full

ISPD routing benchmark set (not a subset); compare against the strongest published router

tuned to equal effort; report per-benchmark wirelength, DRC/overflow, and runtime on stated

hardware; include the largest circuits to show scaling; ablate the congestion-aware component to

show it, not parameter tuning, drives the gain; and state external validity (technology node,

macro density) as a bounded threat — every number traceable to a logged, re-runnable flow.

Output format

[Evaluation readiness]  strong / adequate / weak
[Benchmarks]            standard suite(s) named + full set reported? yes/no
[Baseline fairness]     strongest SOTA, tuned, equal effort, on same hardware? yes/no
[QoR completeness]      full PPA + runtime + variance reported? yes/no
[Ablation]              mechanism isolated from tuning? yes/no
[ML leakage]            train/test designs disjoint + unseen-node generalization? yes/no/NA
[Decision-critical run] the one experiment that would most strengthen the case