z3/doc/range_set_integration_plan.md

Range-Set Integration — Handoff Plan

Status: planning / clean slate Branch: veanes/range-set-integration (based on master) Author of range-set infrastructure: Margus Veanes (veanes), 2026

This document is the starting point for follow-up work on tighter integration of range-sets into the regex/sequence solver. It summarizes the current state in master (after the "Derive with ranges" merge, PR #9965), lays out the two candidate integration approaches we discussed (the invasive re.ranges AST node vs. an internal-only deepening), and gives a concrete evaluation plan so the branch can be measured against master on the same benchmarks used during the derivative work.

---

1. Current state in master (baseline)

The "Derive with ranges" work already landed a self-contained range-set value type and a translator, but range-sets are not first-class in the regex AST. They live only as a transient internal representation at the smart-constructor boundary.

1.1 range_predicate — the range-set value type

• Files: src/ast/rewriter/seq_range_predicate.h, seq_range_predicate.cpp
• A canonical subset of the unsigned character domain [0, max_char], stored as a sorted vector of disjoint, non-adjacent ranges [(lo_0,hi_0), ...] with hi_i + 1 < lo_{i+1}.
• Canonical: two predicates over the same domain are extensionally equal iff their internal vectors are elementwise equal.
• Full Boolean algebra, all linear in the number of ranges: operator| & - ^ ~, plus empty/top/singleton/range, contains, cardinality, equals, operator<, hash, display.
• Pure value: no ast_manager allocation in construction or Boolean ops. Conversion to/from expr* is delegated to a separate translator.

1.2 seq_range_collapse — the regex <-> range_predicate translator

• Files: src/ast/rewriter/seq_range_collapse.h, seq_range_collapse.cpp
• regex_to_range_predicate(u, r, out): recognizes the boolean-combination-of-ranges fragment (re.empty, re.allchar, re.range, re.union, re.inter, re.diff, and inter with a complement operand → difference) and folds it into a canonical range_predicate. Returns false (caller falls back to the generic path) for anything outside the fragment.
• range_predicate_to_regex(u, p, seq_sort): materializes a predicate back into AST as a right-associated union of single re.range nodes, sorted by expression id to match the canonical shape seq_rewriter::merge_regex_sets expects.

1.3 Where it is used today (the only integration point)

• src/ast/rewriter/seq_rewriter.cpp, smart constructors:
  • try_collapse_re_union (~3690): pa | pb
  • try_collapse_re_inter (~3703): pa & pb
  • try_collapse_re_diff (~3866): pa - pb
• Pattern in each: collapse both operands to range_predicate, run the Boolean op on the value type, materialize back to AST. If either operand is outside the fragment, bail out and use the generic regex constructor.
• seq_derive.cpp does not use range_predicate directly — the derivative engine still works over regex AST + ITE path conditions; range-sets only touch the union/inter/diff smart constructors.

1.4 Latent AST vehicle already present

• OP_RE_RANGE (re.range): the single-range character class.
• OP_RE_OF_PRED (re.of.pred, is_of_pred): a regex that accepts a single character satisfying a predicate — already declared and signed in seq_decl_plugin.cpp (sig at ~240, build at ~1206, nullable/printer cases at ~410/1111/1694). This is the natural existing hook for a first-class character-class / range-set node.

1.5 Key invariants that any integration MUST preserve

• Soundness: 0 sat/unsat mismatches vs master on the corpus (this held through all derivative work; it is the gate).
• Domain guard: the range algebra only models length-1 character classes over [0, max_char]. Reject non-string regex element sorts up front (see regex_to_range_predicate guard) — never fabricate a char-class for a regex over (Seq Int) etc.
• re.complement is sequence-level, not character-level. Its language includes epsilon, length >= 2 strings, and length-1 strings outside the operand. It must not be collapsed to char-level ~. De-Morgan / empty / full special cases are handled in mk_re_complement. (See the long NOTE in seq_range_collapse.cpp.)
• Canonical id-sorted union shape on materialization, or merge_regex_sets produces wrong unions.
• Derivative-kind gating (from prior work, now in master): intersection- over-union and concat-over-union distribution are gated on derivative_kind::antimirov_t; brzozowski mode (used by regex_bisim and the emptiness enumeration) keeps intersections above unions. Do not regress this.

---

2. Candidate approaches

Approach A — first-class re.ranges node (most invasive)

Make a range-set a single regex AST node, either by introducing a new internal op (e.g. _OP_RE_RANGES) or by reusing OP_RE_OF_PRED with a range-set encoded predicate so a length-1 character class is one node carrying a range_predicate (or a canonical predicate term).

Potential benefits:

• Eliminates the repeated collapse <-> materialize round-trips; a char class is created, hashed, and shared once.
• Removes the union-of-re.range blowup in the stored AST: one node instead of N union leaves; structural sharing + hash-consing give O(1) equality on char classes.
• Derivative head conditions / OneStep predicates become single nodes; bisim and emptiness compare char classes by pointer/hash instead of walking unions.
• Cleaner normal form: union/inter/diff/complement of char classes fold into the node directly.

Costs / risks (this is the invasive part):

• Touches every site that pattern-matches the regex AST: seq_rewriter, seq_derive, seq_regex (theory), seq_decl_plugin (nullable, derivative, pretty-printer, well-formedness), sls_seq_plugin, NNF/translation, model construction, and any is_union/is_range matcher that currently assumes the union-of-ranges shape.
• Needs nullable + derivative rules for the new node (a length-1 class is never nullable; its derivative wrt c is epsilon if c in set else empty).
• Sub-decision: internal-only (never exposed to SMT-LIB / API / models; rewritten away at the boundary) vs fully exposed (parser, printer, proofs, model values, API, pretty-printing all updated). Internal-only is strongly preferred as the first step — much smaller blast radius.

Approach B — deepen internal range-set use, keep it out of the AST (least invasive)

Keep range_predicate as a transient value type (no AST node), but use it in more places and cache it harder.

Possible moves:

• Memoize regex_to_range_predicate results keyed on expr* id so the collapse cost is paid once per char-class subterm.
• Use range_predicate for the char-class arithmetic inside derivative path conditions / OneStep instead of building ITE/union AST and re-collapsing.
• Extend the recognized fragment (carefully — respecting the re.complement caveat) and short-circuit more constructor cases through the value type.

Benefits: low risk, incremental, no AST/API/printer/model changes, all existing matchers keep working; easy to land and measure piecewise. Costs: the collapse/materialize cost at the AST boundary remains; the union-of-ranges blowup still exists in any AST that is stored or compared structurally (bisim/emptiness still walk unions).

Recommended phasing

1. Land Approach B's caching/memoization first as a measurable baseline — it is cheap and de-risks the corpus.
2. Prototype Approach A internal-only via OP_RE_OF_PRED on the single hottest path (derivative char-class conditions or the union/inter/diff constructor output), keeping it rewritten-away at the SMT-LIB/API boundary.
3. Only consider full exposure (parser/printer/model/API) if the internal-only prototype shows a clear, sustained win on the corpus.

---

3. Evaluation plan

Goal: measure this branch against master on the same benchmarks used during the derivative work, with soundness as the gate (0 mismatches).

3.1 Benchmarks

• Regex-equivalence corpus: C:\git\bench\inputs\regex-equivalence (1523 .smt2 files). Key sub-dirs: cade29[_equiv]\parametric, cade29[_equiv]\boolean_and_loops, mutations[_equiv], realworld\{email,url,snort}.
• QF_S: C:\git\bench\inputs\QF_S (e.g. 20230329-automatark-lu, 20230329-woorpje-lu, 20240318-omark, 2019-Jiang, ...).

3.2 Harness (already present, untracked)

• C:\git\z3\.regtests\run_branch_only.ps1 — runs one solver over a directory, emits CSV file, branch_result, branch_ms. Param: -Out <csv> (NOT -OutCsv), -Timeout <sec> (default 10), -Dir <path>.
• C:\git\z3\.regtests\run_regex_equiv.ps1 — runs both solvers, emits file, branch_res, branch_ms, master_res, master_ms, status.
• Baseline master numbers do not change — reuse the cached master columns rather than re-running master.

3.3 Metric / comparison method (proven during derivative work)

• Normalize verdicts to sat / unsat / to (timeout/unknown/empty/error annotations stripped) before comparing — CSV-quoting of (error ...) self- check noise from mutation benchmarks otherwise causes false MISMATCH.
• Report:
  • branch-only timeouts vs master (branch times out, master solves) — the primary regression signal,
  • branch wins (master times out, branch solves),
  • soundness mismatches — must be 0.
• Reference numbers from the derivative branch at hand-off (regex-equivalence, 10s timeout): the shipped config had 14 branch-only timeouts and 13 branch wins vs master, 0 mismatches. Use that as the bar to beat; a range-set change should not increase branch-only timeouts.

3.4 Build (Windows / CMake + Ninja)

& $env:ComSpec /c 'call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvars64.bat" >nul && cd /d C:\git\z3\build\release && ninja shell test-z3'

Unit tests: C:\git\z3\build\release\test-z3.exe /a (expect all pass). (The 'vswhere.exe' is not recognized warning is harmless.)

3.5 Procedure

1. Build shell + test-z3; test-z3 /a green.
2. run_branch_only.ps1 over regex-equivalence -> CSV; compare to cached master baseline with the normalized-verdict comparison.
3. Repeat over a QF_S subset (start with 20230329-automatark-lu).
4. Gate: 0 mismatches; branch-only timeouts <= 14 on regex-equivalence.
5. Spot-check the historically hard cases: flat_vs_loop_018_n20, det_blowup_024 (direct + equiv), a few mut_* sat cases, and the 5721/5728/5731 regressions.

---

4. Pointers / file index

Area	File	Notes
Range-set value type	src/ast/rewriter/seq_range_predicate.{h,cpp}	canonical Boolean algebra, no AST alloc
Translator	src/ast/rewriter/seq_range_collapse.{h,cpp}	regex <-> range_predicate, fragment + caveats
Constructor integration	src/ast/rewriter/seq_rewriter.cpp ~3690/3703/3866	try_collapse_re_{union,inter,diff}
Derivative engine	src/ast/rewriter/seq_derive.{cpp,h}	derivative-kind gating; no range_predicate yet
Bisim	src/ast/rewriter/seq_regex_bisim.cpp	brzozowski cofactors
Theory	src/smt/seq_regex.cpp	emptiness/membership, get_derivative_targets
AST ops	src/ast/seq_decl_plugin.{h,cpp}	OP_RE_RANGE, OP_RE_OF_PRED (is_of_pred)

---

5. Open questions for the author

• For Approach A: new internal _OP_RE_RANGES op, or reuse OP_RE_OF_PRED with a range-set predicate? (Reuse minimizes new surface; a dedicated op is clearer and lets range_predicate ride directly.)
• Internal-only first (rewritten away at the boundary), or commit to full SMT-LIB/API/model exposure?
• How invasive is acceptable for the bisim/emptiness comparison path — is a pointer/hash equality on char classes worth the AST churn?
• Which is the single hottest path to prototype on (derivative char-class conditions vs. constructor output)?