mirror of
https://github.com/Z3Prover/z3
synced 2026-05-28 12:56:28 +00:00
Update compare-stats anomaly reporter to read benchmark stats from /z3/ (#9650)
This updates the compare-stats anomaly reporter to retrieve benchmark data from `http://mtzguido.tplinkdns.com:8081/z3/` instead of the old `compare_stats.html` endpoint. The workflow prompt and generated lock file now consistently reference the root benchmark stats URL. - **Workflow source** - Repoint the benchmark source URL from `.../compare_stats.html` to `.../z3/` - Update prompt text to describe the source as benchmark statistics rather than a specific HTML file - Rename the temporary fetched artifact from `compare_stats.html` to `benchmark_stats.html` for consistency - **Generated workflow** - Regenerate `compare-stats-anomaly-reporter.lock.yml` so the compiled workflow matches the markdown source - Refresh the embedded workflow description and prompt payload to reference the new endpoint - **Report output** - Update the discussion template’s source link to point at the root benchmark stats page ```md Source URL: `http://mtzguido.tplinkdns.com:8081/z3/` curl -fsSL --max-time 60 "http://mtzguido.tplinkdns.com:8081/z3/" \ -o /tmp/gh-aw/agent/benchmark_stats.html ``` --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
This commit is contained in:
Copilot
GitHub
parent
1564e00215
commit
eb4c3a0756
2 changed files with 22 additions and 22 deletions
30
.github/workflows/compare-stats-anomaly-reporter.lock.yml
generated
vendored
30
.github/workflows/compare-stats-anomaly-reporter.lock.yml
generated
vendored
|
|
@ -1,4 +1,4 @@
|
|||
# gh-aw-metadata: {"schema_version":"v3","frontmatter_hash":"2a5cfa12c70a857202c704c628cf174628068934dbfcc3caf526c8901f372c93","compiler_version":"v0.76.1","agent_id":"copilot"}
|
||||
# gh-aw-metadata: {"schema_version":"v3","frontmatter_hash":"ba7cc0883e671716a71c8e76bcbdff5a80ec7bea87f96a19f5419c019ab96fdd","compiler_version":"v0.76.1","agent_id":"copilot"}
|
||||
# gh-aw-manifest: {"version":1,"secrets":["COPILOT_GITHUB_TOKEN","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GITHUB_TOKEN"],"actions":[{"repo":"actions/checkout","sha":"de0fac2e4500dabe0009e67214ff5f5447ce83dd","version":"v6.0.2"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/setup-node","sha":"48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e","version":"v6.4.0"},{"repo":"actions/upload-artifact","sha":"043fb46d1a93c77aae656e7c1c64a875d1fc6a0a","version":"v7.0.1"},{"repo":"github/gh-aw-actions/setup","sha":"v0.76.1","version":"v0.76.1"}],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.25.55"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.25.55"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.25.55"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.3.19"},{"image":"ghcr.io/github/github-mcp-server:v1.0.4","digest":"sha256:e3816a476a977cfb836e7d221510011436c654d11861db66ecfd826601aba6a4","pinned_image":"ghcr.io/github/github-mcp-server:v1.0.4@sha256:e3816a476a977cfb836e7d221510011436c654d11861db66ecfd826601aba6a4"},{"image":"node:lts-alpine","digest":"sha256:d1b3b4da11eefd5941e7f0b9cf17783fc99d9c6fc34884a665f40a06dbdfc94f","pinned_image":"node:lts-alpine@sha256:d1b3b4da11eefd5941e7f0b9cf17783fc99d9c6fc34884a665f40a06dbdfc94f"}]}
|
||||
# ___ _ _
|
||||
# / _ \ | | (_)
|
||||
|
|
@ -22,7 +22,7 @@
|
|||
#
|
||||
# For more information: https://github.github.com/gh-aw/introduction/overview/
|
||||
#
|
||||
# Analyze compare_stats.html for the latest 30 hours and publish bug/crash/anomaly summary as a GitHub Discussion
|
||||
# Analyze benchmark statistics from the latest 30 hours and publish bug/crash/anomaly summary as a GitHub Discussion
|
||||
#
|
||||
# Secrets used:
|
||||
# - COPILOT_GITHUB_TOKEN
|
||||
|
|
@ -188,20 +188,20 @@ jobs:
|
|||
run: |
|
||||
bash "${RUNNER_TEMP}/gh-aw/actions/create_prompt_first.sh"
|
||||
{
|
||||
cat << 'GH_AW_PROMPT_e3de1425a791a61a_EOF'
|
||||
cat << 'GH_AW_PROMPT_fa9bfce3ee233090_EOF'
|
||||
<system>
|
||||
GH_AW_PROMPT_e3de1425a791a61a_EOF
|
||||
GH_AW_PROMPT_fa9bfce3ee233090_EOF
|
||||
cat "${RUNNER_TEMP}/gh-aw/prompts/xpia.md"
|
||||
cat "${RUNNER_TEMP}/gh-aw/prompts/temp_folder_prompt.md"
|
||||
cat "${RUNNER_TEMP}/gh-aw/prompts/markdown.md"
|
||||
cat "${RUNNER_TEMP}/gh-aw/prompts/safe_outputs_prompt.md"
|
||||
cat << 'GH_AW_PROMPT_e3de1425a791a61a_EOF'
|
||||
cat << 'GH_AW_PROMPT_fa9bfce3ee233090_EOF'
|
||||
<safe-output-tools>
|
||||
Tools: create_discussion, missing_tool, missing_data, noop
|
||||
</safe-output-tools>
|
||||
GH_AW_PROMPT_e3de1425a791a61a_EOF
|
||||
GH_AW_PROMPT_fa9bfce3ee233090_EOF
|
||||
cat "${RUNNER_TEMP}/gh-aw/prompts/mcp_cli_tools_prompt.md"
|
||||
cat << 'GH_AW_PROMPT_e3de1425a791a61a_EOF'
|
||||
cat << 'GH_AW_PROMPT_fa9bfce3ee233090_EOF'
|
||||
<github-context>
|
||||
The following GitHub context information is available for this workflow:
|
||||
{{#if github.actor}}
|
||||
|
|
@ -230,12 +230,12 @@ jobs:
|
|||
{{/if}}
|
||||
</github-context>
|
||||
|
||||
GH_AW_PROMPT_e3de1425a791a61a_EOF
|
||||
GH_AW_PROMPT_fa9bfce3ee233090_EOF
|
||||
cat "${RUNNER_TEMP}/gh-aw/prompts/github_mcp_tools_with_safeoutputs_prompt.md"
|
||||
cat << 'GH_AW_PROMPT_e3de1425a791a61a_EOF'
|
||||
cat << 'GH_AW_PROMPT_fa9bfce3ee233090_EOF'
|
||||
</system>
|
||||
{{#runtime-import .github/workflows/compare-stats-anomaly-reporter.md}}
|
||||
GH_AW_PROMPT_e3de1425a791a61a_EOF
|
||||
GH_AW_PROMPT_fa9bfce3ee233090_EOF
|
||||
} > "$GH_AW_PROMPT"
|
||||
- name: Interpolate variables and render templates
|
||||
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
|
||||
|
|
@ -447,9 +447,9 @@ jobs:
|
|||
mkdir -p "${RUNNER_TEMP}/gh-aw/safeoutputs"
|
||||
mkdir -p /tmp/gh-aw/safeoutputs
|
||||
mkdir -p /tmp/gh-aw/mcp-logs/safeoutputs
|
||||
cat > "${RUNNER_TEMP}/gh-aw/safeoutputs/config.json" << 'GH_AW_SAFE_OUTPUTS_CONFIG_ab9f6dd567ca7d62_EOF'
|
||||
cat > "${RUNNER_TEMP}/gh-aw/safeoutputs/config.json" << 'GH_AW_SAFE_OUTPUTS_CONFIG_4d0c9ad86b64b83a_EOF'
|
||||
{"create_discussion":{"category":"agentic workflows","close_older_discussions":true,"expires":168,"fallback_to_issue":true,"max":1,"title_prefix":"[Compare Stats] "},"create_report_incomplete_issue":{},"missing_data":{},"missing_tool":{},"noop":{"max":1,"report-as-issue":"false"},"report_incomplete":{}}
|
||||
GH_AW_SAFE_OUTPUTS_CONFIG_ab9f6dd567ca7d62_EOF
|
||||
GH_AW_SAFE_OUTPUTS_CONFIG_4d0c9ad86b64b83a_EOF
|
||||
- name: Generate Safe Outputs Tools
|
||||
env:
|
||||
GH_AW_TOOLS_META_JSON: |
|
||||
|
|
@ -648,7 +648,7 @@ jobs:
|
|||
|
||||
mkdir -p /home/runner/.copilot
|
||||
GH_AW_NODE=$(which node 2>/dev/null || command -v node 2>/dev/null || echo node)
|
||||
cat << GH_AW_MCP_CONFIG_87d3e139f223cdea_EOF | "$GH_AW_NODE" "${RUNNER_TEMP}/gh-aw/actions/start_mcp_gateway.cjs"
|
||||
cat << GH_AW_MCP_CONFIG_1d647de00ba486c1_EOF | "$GH_AW_NODE" "${RUNNER_TEMP}/gh-aw/actions/start_mcp_gateway.cjs"
|
||||
{
|
||||
"mcpServers": {
|
||||
"github": {
|
||||
|
|
@ -689,7 +689,7 @@ jobs:
|
|||
"payloadDir": "${MCP_GATEWAY_PAYLOAD_DIR}"
|
||||
}
|
||||
}
|
||||
GH_AW_MCP_CONFIG_87d3e139f223cdea_EOF
|
||||
GH_AW_MCP_CONFIG_1d647de00ba486c1_EOF
|
||||
- name: Mount MCP servers as CLIs
|
||||
id: mount-mcp-clis
|
||||
continue-on-error: true
|
||||
|
|
@ -1167,7 +1167,7 @@ jobs:
|
|||
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
|
||||
env:
|
||||
WORKFLOW_NAME: "Compare Stats Bug/Crash/Anomaly Reporter"
|
||||
WORKFLOW_DESCRIPTION: "Analyze compare_stats.html for the latest 30 hours and publish bug/crash/anomaly summary as a GitHub Discussion"
|
||||
WORKFLOW_DESCRIPTION: "Analyze benchmark statistics from the latest 30 hours and publish bug/crash/anomaly summary as a GitHub Discussion"
|
||||
HAS_PATCH: ${{ needs.agent.outputs.has_patch }}
|
||||
with:
|
||||
script: |
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
---
|
||||
description: Analyze compare_stats.html for the latest 30 hours and publish bug/crash/anomaly summary as a GitHub Discussion
|
||||
description: Analyze benchmark statistics from the latest 30 hours and publish bug/crash/anomaly summary as a GitHub Discussion
|
||||
|
||||
on:
|
||||
schedule:
|
||||
|
|
@ -36,14 +36,14 @@ safe-outputs:
|
|||
|
||||
Your name is ${{ github.workflow }}. You are a Z3 benchmarking analysis agent for `${{ github.repository }}`.
|
||||
|
||||
Analyze the benchmark comparison page below, focusing on results from the last 30 hours, then create a GitHub Discussion with a concise but actionable summary of:
|
||||
Analyze the benchmark statistics page below, focusing on results from the last 30 hours, then create a GitHub Discussion with a concise but actionable summary of:
|
||||
|
||||
- Bugs
|
||||
- Crashes
|
||||
- Anomalies
|
||||
|
||||
Source URL:
|
||||
`http://mtzguido.tplinkdns.com:8081/z3/compare_stats.html`
|
||||
`http://mtzguido.tplinkdns.com:8081/z3/`
|
||||
|
||||
Note: this endpoint is currently HTTP-only. Treat fetched data as non-sensitive benchmark telemetry and do not include secrets in requests or reports.
|
||||
Note: the workflow runs every 12 hours but analyzes 30 hours intentionally to provide overlap and avoid missing transient failures between runs.
|
||||
|
|
@ -53,16 +53,16 @@ Overlapping windows are expected; `close-older-discussions: true` keeps only the
|
|||
|
||||
### 1) Fetch and save the source page
|
||||
|
||||
Use bash to fetch the page into `/tmp/gh-aw/agent/compare_stats.html`.
|
||||
Use bash to fetch the page into `/tmp/gh-aw/agent/benchmark_stats.html`.
|
||||
|
||||
Try this first:
|
||||
```bash
|
||||
curl -fsSL --max-time 60 "http://mtzguido.tplinkdns.com:8081/z3/compare_stats.html" -o /tmp/gh-aw/agent/compare_stats.html
|
||||
curl -fsSL --max-time 60 "http://mtzguido.tplinkdns.com:8081/z3/" -o /tmp/gh-aw/agent/benchmark_stats.html
|
||||
```
|
||||
|
||||
If that fails, retry once with:
|
||||
```bash
|
||||
wget -q -T 60 -O /tmp/gh-aw/agent/compare_stats.html "http://mtzguido.tplinkdns.com:8081/z3/compare_stats.html"
|
||||
wget -q -T 60 -O /tmp/gh-aw/agent/benchmark_stats.html "http://mtzguido.tplinkdns.com:8081/z3/"
|
||||
```
|
||||
|
||||
If both fail, still create a discussion that explains the fetch failure, includes stderr output, and marks the report as incomplete.
|
||||
|
|
@ -133,7 +133,7 @@ Use this structure:
|
|||
```markdown
|
||||
### Compare Stats Analysis Report
|
||||
|
||||
**Source**: [compare_stats.html](http://mtzguido.tplinkdns.com:8081/z3/compare_stats.html)
|
||||
**Source**: [benchmark statistics](http://mtzguido.tplinkdns.com:8081/z3/)
|
||||
**Workflow Run**: [#${{ github.run_id }}](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }})
|
||||
**Analysis Time (UTC)**: <timestamp>
|
||||
**Window**: last 30 hours (or fallback mode)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue