django-queryset-batch-processing

Django QuerySet Batch Processing

Use this skill when Django code processes many rows. The goal is to avoid loading unnecessary model instances, avoid queryset result-cache blowups, and move writes into set-based database operations when behavior allows.

Workflow

1. Identify the per-row work.

• Is it read-only export/reporting?
• Does it need model methods, validation, or signals?
• Can the database compute or update the value directly?

1. Choose the read pattern.

• Use values() or values_list() for scalar exports and reports.
• Use iterator(chunk_size=...) when model instances are needed but queryset caching is not.
• Keep ordering deliberate; unnecessary ordering costs work.

1. Choose the write pattern.

• Use QuerySet.update() with F() or expressions for uniform updates.
• Use bulk_update() when each object has a different value.
• Use bulk_create() for inserts, with conflict options only when the project supports their semantics.
• Fall back to per-instance save() only when hooks, validation, side effects, or signals are required.

1. Control batch size.

• Keep transactions bounded.
• Avoid huge IN lists and oversized CASE updates.
• Monitor locks, replication lag, and memory for production jobs.

See [batch-patterns.md](references/batch-patterns.md) for examples and caveats.

Safety Notes

• Bulk update/delete operations do not call each model instance's save() or delete() methods.
• Bulk operations can skip application-level side effects and signals.
• Long transactions can hold locks and delay vacuum or replication.

Verification

Measure rows processed per second, query count, memory, transaction duration, and correctness on a representative batch.