mirror of
https://github.com/Z3Prover/z3
synced 2026-08-08 15:02:09 +00:00
249778137d
Fixes #10375. ## Analysis of [run 30786942559](https://github.com/Z3Prover/z3/actions/runs/30786942559) The job is green, but `make test` exited 2. Every failing test is an **expected-output mismatch**, and all 70 of them differ only by the `--proof_recovery` banner: ``` - The SMT solver could not prove the query. + - This query was retried due to the --proof_recovery option, yet it still + failed on all attempts. ``` That text comes from `--proof_recovery` in the workflow's own `fstar_otherflags` default, which F*'s `.expected` files do not carry — so these are configuration mismatches, not Z3 regressions. There are also failures that never reach the solver (`hello.__all`, `dune`, extraction diffs on `RemoveUnusedTypars_B.fs` / `Bug3865.out`). None of that was reconstructible from the artifact: the run uploaded 526 `.smt2` files, almost all logged failing queries from negative tests that are *supposed* to fail, and nothing else. No logs, no produced output, no `.expected` oracle, no diff. ## Changes * **`Resolve FStar options`** — computes `OTHERFLAGS` once (removing the duplicated `--z3version` extraction) and enforces `--log_failing_queries`. Scheduled runs get no `workflow_dispatch` inputs, so the options that make F* emit `.smt2` files are hard-wired instead of assumed to come from the inputs. The effective flags are recorded in the artifact, the job summary and the discussion. * **`Build FStar`** now tees to a log, as the test step already did. * **`Collect FStar failure artifacts`** replaces `Collect generated SMT2 files` and produces: * `logs/` — build log, test log, versions, commit, effective flags, and a `failing-tests.txt` summary of mismatched outputs plus failed make targets; * `smt2/` — the logged failing queries, as before; * `test-output/` — for every expected-output test whose result differs from its oracle: `<name>.actual`, `<name>.expected` and a unified `<name>.diff`. * **Upload always runs.** Previously, if no `.smt2` file existed the collect step short-circuited and the upload was skipped, so the hardest failures produced no artifact at all. * The failure summary is surfaced in the **job summary** and in the **discussion**, so a mismatch is visible without downloading anything. Applied to the run above, `test-output/` would hold the actual/expected/diff triple for each of the 70 mismatches and `failing-tests.txt` would list them alongside `hello.__all` and the other non-SMT failures. ## Validation The workflow parses as YAML; every `run:` block passes `bash -n` and the `github-script` body passes `node --check`. The new steps were executed locally against a fixture reproducing the run's failure shapes: * mismatched `.output` / `.json_output` / `.ideout` / `.fs` / `.out` files are collected with correct diffs; the generated diff for `Basic.fst.output` reproduces the annotation in the run exactly; * matching outputs, `_output` files with no `.expected`, and files over 4 MB are correctly skipped; * failed make targets are parsed from both logs; * with no F* tree at all (build failed before clone) the collector still exits 0 and the artifact still contains the logs; * option resolution was checked with the default flags, with `--log_failing_queries` absent, with empty flags, and with an unparseable `z3 --version`; * the rendered discussion body is 53 040 characters in the worst case, below GitHub's 65 536 limit. Behaviour deliberately unchanged: the build and test steps keep `continue-on-error`, so a broken F* master still does not block the report. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| actions | ||
| agents | ||
| aw | ||
| prompts | ||
| scripts | ||
| skills | ||
| workflows | ||
| CI_MIGRATION.md | ||
| CI_TESTING.md | ||
| copilot-instructions.md | ||
| dependabot.yml | ||
| mcp.json | ||