Skip to content

Make partition expression evaluation stateless#3664

Open
dossett wants to merge 3 commits into
apache:mainfrom
dossett:stateless-expression-evaluator
Open

Make partition expression evaluation stateless#3664
dossett wants to merge 3 commits into
apache:mainfrom
dossett:stateless-expression-evaluator

Conversation

@dossett

@dossett dossett commented Jul 15, 2026

Copy link
Copy Markdown

I used CODEX to analyze this problem and create this PR. I've reviewed the code and tests and stand by them. This summary is written completely by a human (me) other than very light copy editing by an LLM.

Currently, because _ExpressionEvaluator can't be safely shared one is created per file. That's expensive relative to actually evaluating the expression. This change makes _ExpressionEvaluator shareable by moving the mutable parts into a new _ExpressionEvaluationVisitor which stays local to the per-file work of evaluation.

These benchmarks are showing improvements in sub-second times, but in the production workloads I tested this on it did translate to real wall clock improvement. But irrespective of performance gains the new _ExpressionEvaluator is safer since nothing was preventing its accidental concurrent reuse in the future.

CODEX-generated summary follows:

Partition pruning currently binds the same projected expression for every data file because _ExpressionEvaluator stores the current record as mutable instance state. That repeated preparation is expensive, but sharing the existing evaluator across workers would allow concurrent calls to mix records.

This separates preparation from per-record evaluation. _ExpressionEvaluator binds the expression once, while every call creates a private _ExpressionEvaluationVisitor containing that call's record. Manifest planning can therefore prepare one evaluator per partition spec and safely share it across manifests and workers. This provides the construction-reuse benefit targeted by #3656 without depending on files within a manifest remaining sequential, and it also benefits one-file manifests.

Summary

  • Split the prepared expression evaluator from the call-local mutable visitor.
  • Reuse one prepared partition evaluator per partition spec during manifest planning.
  • Add deterministic concurrent-use, prepared-state, and planner-sharing coverage.
  • Add a benchmark using a realistic 15-leaf predicate across dense and one-file manifests.

Performance

I compared this branch with main using the partition-evaluator workload in this PR. It evaluates 1,000 files with two identity partition fields and a 15-leaf predicate modeling five (event_day range AND region_id) branches. Timings are medians from seven samples of five iterations.

Manifest layout main This PR Speedup
1 manifest × 1,000 files 187.928 ms 14.413 ms 13.04×
1,000 manifests × 1 file 188.000 ms 14.584 ms 12.89×

The improvement comes from binding the projected partition expression once per partition spec instead of once per file. This is an isolated partition-pruning benchmark rather than an end-to-end scan-planning measurement.

Testing

  • pytest tests/expressions/test_visitors.py tests/table/test_partition_evaluator_planning.py tests/table/test_init.py (200 passed)
  • pytest tests/benchmark/test_partition_evaluator_benchmark.py -m benchmark (2 passed)

@dossett
dossett marked this pull request as ready for review July 15, 2026 17:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant