optimize-python-parallelism

AtomLane Python Advisor

Improve a program only after separating three questions: where time is spent,

whether iterations are semantically independent, and whether the proposed

executor is likely to beat its overhead. A long runtime is a reason to inspect,

not proof that parallel execution is legal or useful.

Keep analysis and execution separate

Use python_parallel_advisor for bounded source analysis. It reads strict

project-local UTF-8 Python, builds a conservative same-module call/effect

summary, and may return a source-hash-bound rewrite preview. It never imports

or executes target code and never changes files.

Read [references/python-program-ir.md](references/python-program-ir.md) before

interpreting or applying a candidate. It defines classification, proof gates,

GIL/spawn constraints, rewrite validity, and the verification certificate.

Call the advisor with:

• an absolute project_path;
• concrete paths when the entrypoint is known, otherwise bounded discovery;
• caller-observed hotspots only when they are real serial measurements;
• the actual execution_context, so an inner pool is not multiplied by an

AtomLane or native worker pool;

• an explicit worker ceiling only as a ceiling, never as a safety override.
• target_platform when the optimized program will deploy somewhere other

than the analysis host.

Do not run a workload merely to obtain a profile when repeating it may mutate

state, incur cost, or affect an external system.

Treat each classification precisely

• reviewable_rewrite is the strongest static result, not a runtime proof.

Its patch is still conditional on pickling, import, memory, correctness, and

measured-performance checks.

• advisory_only identifies a plausible I/O, network, subprocess, or otherwise

conditional opportunity. Explain the missing guarantees; do not apply its

outline as an automatic transformation.

• prefer_native means vectorization or a library-owned worker pool should be

considered before another Python pool.

• already_parallel requires one coordinated outer/inner resource budget.
• blocked remains serial until every hard blocker is removed by evidence or a

semantics-preserving redesign. A confidence score cannot override a blocker.

If the current task does not authorize editing source, stop at advice. A

request to analyze performance alone does not authorize a refactor.

Apply only a minimal, current rewrite

Before using a preview, recompute the source SHA-256 and require an exact match.

If source, runtime evidence, executor choice, or resource assumptions changed,

discard the preview and analyze again.

For the initial supported CPU pattern, preserve ordered map semantics with

ProcessPoolExecutor.map; require a module-level worker and a spawn-safe

if __name__ == "__main__" boundary. Do not move logging, file writes,

database work, randomness, environment reads, or unknown calls into speculative

workers. Keep exception, cancellation, result order, and deterministic merge

behavior explicit.

Choose by workload rather than syntax:

• pure Python CPU work: processes, after spawn and pickling review;
• blocking read-only I/O: bounded threads, after exception and resource review;
• established async call chains: bounded async tasks and cancellation design;
• NumPy, BLAS, OpenMP, PyTorch, and similar native kernels: vectorize or use

their own workers, then cap outer concurrency;

• subprocess batches: prefer AtomLane task atoms with declared effects when

source modification adds no value.

Verify before claiming an improvement

After an authorized edit:

1. Compile the source without importing it.
2. Run focused unit/integration tests.
3. Differentially compare serial and parallel return values, ordering,

exceptions, stdout/stderr, files, hashes, random seeds, and numeric tolerance.

1. Exercise process candidates with an explicit multiprocessing spawn

context on every platform. Windows process pools have a 61-worker ceiling;

treat it as an upper bound, not a useful default.

1. For safe repeatable work, compare multiple serial and parallel samples and

report p50/p90, throughput, peak memory, worker count, and break-even size.

1. Revert only the newly proposed refactor if correctness fails or measured

performance regresses; preserve unrelated user changes.

Use the existing AtomLane plan/execution path when several validation commands

are independently runnable, but treat a Python program that owns an inner pool

as one native-parallel compound atom unless the combined budget proves nested

concurrency safe.

Finish with a parallelization certificate containing the source and candidate

hashes, satisfied and unresolved proof obligations, executor/resource choice,

correctness evidence, measured or explicitly modeled benefit, limitations, and

rollback boundary. Never describe a modeled projection as measured speedup.