Skip to content

feat: pr-reviewer agent, claude-review CLI and GitHub Action - #5099

Open
GGnom wants to merge 1 commit into
claude-builders-bounty:mainfrom
GGnom:agent-pr-reviewer
Open

GGnom wants to merge 1 commit into
claude-builders-bounty:mainfrom
GGnom:agent-pr-reviewer

Conversation

@GGnom

@GGnom GGnom commented Oct 31, 2026

Copy link
Copy Markdown

Closes #4

What

pr-review-agent/ — a Claude Code agent that reviews a real PR diff and returns a structured Markdown review, plus a CLI and a GitHub Action that posts it.

claude-review --pr https://github.com/pallets/flask/pull/5918
claude-review --pr <url> --post --output review.md
claude-review --diff-file change.diff --title "Add invoices"   # offline

Acceptance criteria

Criterion Where
Works via CLI pr-review-agent/claude-review (accepts /files, /commits, ?diff=split suffixes people paste from the browser)
OR GitHub Action with workflow YAML pr-review-agent/.github/workflows/pr-review.yml — drop it in, add ANTHROPIC_API_KEY, done
Summary (2-3 sentences) ## Summary
Identified risks (list) ## Risks, each with severity, path:line and the offending lines
Improvement suggestions (list) ## Suggestions, each with path:line
Confidence: Low / Medium / High ## Confidence + one-line justification
Tested on 2 real PRs with outputs samples/pr-5096-changelog-skill.md, samples/pr-5918-flask-options.md
README with setup and usage pr-review-agent/README.md

The two real reviews (dogfooding)

Both were produced from the actual fetched diffs, not written by hand as a demo.

#5096 (bash changelog generator, +350/-0) — found the real defects in that submission:

  • bash 3.2 + set -u breaks on empty array expansion (README claims bash 3.2 support on macOS),
  • the fallback git log silently turns a typo'd tag into a full-history changelog,
  • classify() is dead code duplicated by the inline case,
  • the root script is a byte-for-byte copy of the one under generate-changelog/scripts/.

pallets/flask#5918 (automatic OPTIONS routing, +56/-30):

  • except Exception: pass around the internal rule addition hides every failure, leaving provide_automatic_options=True with no OPTIONS route while flask routes still prints OPTIONS,
  • methods = rule.methods or set() in cli.py mutates the live Rule.methods set,
  • rule.methods no longer contains OPTIONS for the view rule — an observable change that needs a CHANGES.rst entry,
  • the _automatic_options endpoint name becomes reserved, and make_default_options_response(ctx) changes a public call shape without a deprecation path.

The first sample reviews my own PR and reports my own bugs. That is the point: it is a diff-level review, not a pitch.

Why the output can be trusted enough to post

  • Validated before delivery: missing section or a confidence level that is not Low/Medium/High -> exit 3, nothing is posted. An empty or chatty answer cannot reach the PR.
  • Posts COMMENT, never APPROVE — model output must not satisfy a required review or unblock a merge.
  • Truncation is explicit: the diff is cut by whole diff --git blocks at 3000 lines and the prompt says how much is missing, so the review states its own scope instead of guessing.
  • Prompt is adversarial to flattery: 'none found' is required over filler, style that a formatter owns is out of scope, and the model is forbidden from claiming it ran the code.

Security of the Action

Uses pull_request_target (needed to comment on fork PRs) and is safe only because it never checks out PR code: it checks out the base repo with sparse-checkout: pr-review-agent and reads the diff over the API. The README says explicitly not to add a head-SHA checkout to it. Permissions are contents: read, pull-requests: write.

Tests

python3 -m unittest discover -s tests -v -> 19 tests, no network: URL parsing, prompt assembly, whole-file truncation, description cap, fence stripping, section/confidence validation, offline end-to-end, output file, --dry-run (asserts the model is never called), exit codes, --post requiring --pr, and the missing-claude error path.

Live path verified against GitHub: fetched #5096 (+350/-0), #5097, flask#5918 (+56/-30), flask#5665 (+8/-18) through --dry-run, which exercises URL parsing, API fetch, file list and diff truncation without a model call.

Notes

  • Found and fixed a Windows bug during testing: print() to a cp1251 console crashed on UTF-8 diff content (UnicodeEncodeError). stdout/stderr are now reconfigured to UTF-8 with errors="replace".
  • The CLI takes the token from GITHUB_TOKEN/GH_TOKEN or gh auth token, never writes it anywhere, and the only write it performs is the review itself.

- .claude/agents/pr-reviewer.md: agent definition with severity model and
  strict output contract
- claude-review: fetch PR diff via API, truncate by whole files, call claude -p,
  validate the four sections, print or post as a COMMENT review
- GitHub Action on pull_request_target (API-only, never checks out PR code)
- samples: two real PRs reviewed (+350/-0 and +56/-30)
- 19 offline unittest tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BOUNTY $150] AGENT: Claude Code sub-agent that reviews a PR and posts a structured comment

1 participant