mirror of
https://github.com/Z3Prover/z3
synced 2026-08-08 15:02:09 +00:00
249778137d
Fixes #10375. ## Analysis of [run 30786942559](https://github.com/Z3Prover/z3/actions/runs/30786942559) The job is green, but `make test` exited 2. Every failing test is an **expected-output mismatch**, and all 70 of them differ only by the `--proof_recovery` banner: ``` - The SMT solver could not prove the query. + - This query was retried due to the --proof_recovery option, yet it still + failed on all attempts. ``` That text comes from `--proof_recovery` in the workflow's own `fstar_otherflags` default, which F*'s `.expected` files do not carry — so these are configuration mismatches, not Z3 regressions. There are also failures that never reach the solver (`hello.__all`, `dune`, extraction diffs on `RemoveUnusedTypars_B.fs` / `Bug3865.out`). None of that was reconstructible from the artifact: the run uploaded 526 `.smt2` files, almost all logged failing queries from negative tests that are *supposed* to fail, and nothing else. No logs, no produced output, no `.expected` oracle, no diff. ## Changes * **`Resolve FStar options`** — computes `OTHERFLAGS` once (removing the duplicated `--z3version` extraction) and enforces `--log_failing_queries`. Scheduled runs get no `workflow_dispatch` inputs, so the options that make F* emit `.smt2` files are hard-wired instead of assumed to come from the inputs. The effective flags are recorded in the artifact, the job summary and the discussion. * **`Build FStar`** now tees to a log, as the test step already did. * **`Collect FStar failure artifacts`** replaces `Collect generated SMT2 files` and produces: * `logs/` — build log, test log, versions, commit, effective flags, and a `failing-tests.txt` summary of mismatched outputs plus failed make targets; * `smt2/` — the logged failing queries, as before; * `test-output/` — for every expected-output test whose result differs from its oracle: `<name>.actual`, `<name>.expected` and a unified `<name>.diff`. * **Upload always runs.** Previously, if no `.smt2` file existed the collect step short-circuited and the upload was skipped, so the hardest failures produced no artifact at all. * The failure summary is surfaced in the **job summary** and in the **discussion**, so a mismatch is visible without downloading anything. Applied to the run above, `test-output/` would hold the actual/expected/diff triple for each of the 70 mismatches and `failing-tests.txt` would list them alongside `hello.__all` and the other non-SMT failures. ## Validation The workflow parses as YAML; every `run:` block passes `bash -n` and the `github-script` body passes `node --check`. The new steps were executed locally against a fixture reproducing the run's failure shapes: * mismatched `.output` / `.json_output` / `.ideout` / `.fs` / `.out` files are collected with correct diffs; the generated diff for `Basic.fst.output` reproduces the annotation in the run exactly; * matching outputs, `_output` files with no `.expected`, and files over 4 MB are correctly skipped; * failed make targets are parsed from both logs; * with no F* tree at all (build failed before clone) the collector still exits 0 and the artifact still contains the logs; * option resolution was checked with the default flags, with `--log_failing_queries` absent, with empty flags, and with an unparseable `z3 --version`; * the rendered discussion body is 53 040 characters in the worst case, below GitHub's 65 536 limit. Behaviour deliberately unchanged: the build and test steps keep `continue-on-error`, so a broken F* master still does not block the report. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| shared | ||
| a3-python.lock.yml | ||
| a3-python.md | ||
| academic-citation-tracker.lock.yml | ||
| academic-citation-tracker.md | ||
| agentics-maintenance.yml | ||
| android-build.yml | ||
| api-coherence-checker.lock.yml | ||
| api-coherence-checker.md | ||
| build-warning-fixer.lock.yml | ||
| build-warning-fixer.md | ||
| build-z3-cache.yml | ||
| ci.yml | ||
| clang-tidy-warning-report.yml | ||
| code-conventions-analyzer.lock.yml | ||
| code-conventions-analyzer.md | ||
| code-simplifier.lock.yml | ||
| code-simplifier.md | ||
| compare-stats-anomaly-reporter.lock.yml | ||
| compare-stats-anomaly-reporter.md | ||
| copilot-setup-steps.yml | ||
| coverage.yml | ||
| cross-build.yml | ||
| csa-analysis.lock.yml | ||
| csa-analysis.md | ||
| docs.yml | ||
| fstar-master-build.yml | ||
| issue-backlog-processor.lock.yml | ||
| issue-backlog-processor.md | ||
| mark-prs-ready-for-review.yml | ||
| memory-safety-report.lock.yml | ||
| memory-safety-report.md | ||
| memory-safety.yml | ||
| msvc-static-build-clang-cl.yml | ||
| msvc-static-build.yml | ||
| nightly-validation.yml | ||
| nightly.yml | ||
| ocaml.yaml | ||
| ostrich-benchmark.lock.yml | ||
| ostrich-benchmark.md | ||
| pyodide-pypi.yml | ||
| release-notes-updater.lock.yml | ||
| release-notes-updater.md | ||
| release.yml | ||
| smtlib-benchmark-finder.lock.yml | ||
| smtlib-benchmark-finder.md | ||
| specbot-crash-analyzer.lock.yml | ||
| specbot-crash-analyzer.md | ||
| tactic-to-simplifier.lock.yml | ||
| tactic-to-simplifier.md | ||
| tptp-benchmark.lock.yml | ||
| tptp-benchmark.md | ||
| wasm-release.yml | ||
| wasm.yml | ||
| Windows.yml | ||
| wip.yml | ||
| workflow-suggestion-agent.lock.yml | ||
| workflow-suggestion-agent.md | ||
| zipt-code-reviewer.lock.yml | ||
| zipt-code-reviewer.md | ||