vis-experiments

VIS Experiments

Use this before submission when the evaluation is not yet locked. IEEE VIS reviewers judge whether

the evidence matches the contribution type — and visualization has several distinct contribution

types, each with its own evidence standard. The organizing principle is **evaluate the claim you

actually make**: a claim about perception needs a controlled study, a claim about scale needs a

benchmark, a claim about real-world usefulness needs a design-study validation or a deployment.

Evaluation audit

• Pick the evaluation to the contribution type, not by habit (see the table). The classic VIS

reject is a system paper "evaluated" only by an accuracy number, or a perceptual claim backed only

by author intuition.

• Design controlled studies properly: state hypotheses, a within/between design, a task from a

recognized task taxonomy, a power analysis justifying N, and report **effect sizes with

confidence intervals**, not just p-values. Consider preregistration for confirmatory studies

(vis-reproducibility).

• Justify encodings perceptually: color choices should be CVD-safe and appropriate to the

data type (sequential/diverging/categorical); channel choices should follow known effectiveness

rankings for the task. Reviewers check this explicitly.

• Benchmark techniques fairly: compare against the strongest existing technique and a reasonable

baseline on realistic data sizes, with runtime/quality reported and the code available.

• Hold qualitative and design-study work to method: coding schemes, multiple coders,

reflection across abstraction levels, and an audit trail — design studies are a first-class VIS

contribution, not a weak substitute for a controlled study.

• Pin provenance for datasets, stimuli, and rendering so the evaluation reproduces rather than

re-samples.

Contribution-type to evidence table

| Contribution type | Matching evidence | Reject pattern avoided |

|---|---|---|

| Perceptual/cognitive claim | Controlled experiment: real stimuli, power analysis, effect sizes + CIs | "Author intuition stands in for a perception result" |

| New encoding/interaction technique | Controlled study and/or task-based comparison vs. the conventional design | "Prettier, but no evidence it helps a task" |

| System / tool | Demonstration of real use, expert feedback, or a usage study | "Feature list with no evaluation of use" |

| Design study | Reflection + validation across data/task/encoding/algorithm levels | "A one-off tool with no transferable lesson" |

| Algorithm (layout/rendering) | Benchmark: quality + runtime vs. strong baselines on realistic sizes | "Toy inputs only; no comparison" |

| Data/model contribution | Characterization + a task the data enables, with the data shared | "Dataset dumped with no analysis or task" |

Controlled-study design floor

[Hypotheses]  stated before analysis; confirmatory vs. exploratory labeled
[Design]      within/between justified; counterbalancing; the task from a known taxonomy
[Power]       an a-priori power analysis justifies N; do not stop at "we recruited 20"
[Stimuli]     real or realistic; the exact stimuli archived
[Measures]    accuracy AND time AND (where relevant) preference/confidence; define each
[Statistics]  effect sizes + CIs; appropriate tests; corrections for multiple comparisons
[Reporting]   report what you found, including null and exploratory results, honestly

Perceptual and accessibility checks

• Color: use CVD-safe palettes; match palette type to data (sequential for ordered, diverging

for a meaningful midpoint, categorical for nominal); never encode magnitude on hue alone.

• Channel effectiveness: prefer position/length for quantitative comparison; justify any use of

area, angle, or color for a precise task.

• Legibility: ensure figures read in grayscale and at print size; a result a reviewer cannot see

is a result you cannot claim.

Vignette: evaluating a new time-series encoding

Suppose the paper claims a new encoding reads trends faster than a line chart. The matching plan: a

controlled within-subjects study; trend-reading tasks drawn from a task taxonomy; real time-series

stimuli, archived; an a-priori power analysis fixing N; accuracy and completion-time as measures;

effect sizes with CIs comparing the new encoding to a tuned line-chart baseline; a CVD-safe palette

justified against the task; and honest reporting of any task where the line chart won — every number

traceable to the archived analysis notebook.

Output format

[Evaluation readiness] strong / adequate / weak
[Contribution type] perceptual / technique / system / design-study / algorithm / data
[Evidence match] <contribution type -> evidence chosen -> appropriate? yes/no>
[Study rigor] <hypotheses? power analysis? effect sizes + CIs? preregistered?>
[Encoding validity] <CVD-safe? channel matched to task? grayscale-legible?>
[Provenance] <stimuli/data/rendering archived and reproducible? yes/no>
[Decision-critical next run] <one study or benchmark to add>