Copilot Triage

A maintainer's copilot for GitHub: it triages issues, discussions, and pull requests, and keeps a board of what actually needs you.

copilot-triage-v018-1080p60.mp4

Download the video

Why I built it, with the numbers: I get 215 GitHub notifications a day, so I built a bot to triage them.

Open source maintenance is mostly reading. Copilot Triage reads for you. When someone opens an issue, starts a discussion, sends a pull request, or comments, an agent investigates with read-only tools and decides what helps:

• Issues. Labels, and one useful reply or none: an answer from your docs, a released fix, a duplicate, one essential question, or a recap of a long report. Silence is a valid decision; most follow-ups need no reply.
• Discussions. Questions get answered from your docs. A discussion that is really a bug report or feature request moves to an issue.
• Pull requests. Requests a Copilot code review when a change deserves one, reads Copilot's verdict when it arrives, checks your contribution policy (screenshots, an issue first, scope), and explains or closes changes your documented scope rules out.
• Closing. Closes what is plainly finished, each time with a comment that says why: duplicates, issues a release fixed, issues the reporter says are resolved, and requests your documented scope rules out. Merging and every judgment call stay with you.
• Your board. Every issue and pull request lands on a GitHub project board sorted by what it needs from you: Sign off, Decide, Do, or nothing because it is someone else's move. Each card has a priority and a one-line next step, your review is requested on the pull requests that need it, and urgent issues are assigned to you. Triage builds the board itself on an empty project.

So you can turn off GitHub's email for everything and open the board instead.

It runs on your Copilot allowance by default, or on your own API key through RubyLLM: a company key, OpenRouter, a cloud provider, or any OpenAI-compatible endpoint. We measure models before recommending them, and publish the results.

It runs on every one of our repositories, and was shaped by the ones that get the most issues and pull requests: RubyLLM (about 900), Spotifast (about 500), and ZapFast (about 350), more than one maintainer can read. It works the same in yours.

Why this exists

We used GitHub Agentic Workflows to triage RubyLLM and Spotifast. In RubyLLM alone, the compiled workflow was 2,035 lines of YAML: tool gateways, agent jobs, a separate threat detector, safe-output jobs, failure-reporting machinery. Then it started opening issues about its own failures and commenting about its detector failing. The bot became another thing to maintain, and another source of email.

So we removed the platform and kept the job: one agent, a system prompt, and small read-only tools. The agent chooses what to search, reads evidence, and decides whether to help. Ruby validates its structured decision and publishes it. No classifier calls, no keyword gates, no failure tickets: problems go to the job summary.

Quick start

1. Token. Add a COPILOT_GITHUB_TOKEN repository secret: a fine-grained token with Copilot Requests and an available Copilot allowance (see Copilot authentication). To request Copilot reviews of pull requests, also give it Pull requests: Read and write on the repository. Using your own API key instead? See Models and cost.
2. Policy. Save examples/triage.yml as .github/triage.yml on your default branch and adapt the labels, replies, source paths, and instructions to your project. Until it is on the default branch, triage and the sweep skip every run with a notice.
3. Workflow. Add .github/workflows/triage.yml. It names the events that start triage and calls Copilot Triage's own workflow, which holds the permissions, queueing, and inputs and updates with the v0 tag:

name: Triage
on:
  issues:
    types: [opened, reopened, closed]
  issue_comment:
    types: [created]
  discussion:
    types: [created]
  discussion_comment:
    types: [created]
  pull_request_target:
    types: [opened, reopened, ready_for_review, synchronize, closed]
  pull_request_review:
    types: [submitted]
  workflow_dispatch:
    inputs:
      kind:
        type: choice
        options: [issue, discussion, pull_request]
      number:
        required: true
      dry_run:
        type: boolean
        default: true

permissions:
  contents: read
  issues: write
  discussions: write
  pull-requests: write

jobs:
  triage:
    uses: crmne/copilot-triage/.github/workflows/assess.yml@v0
    secrets: inherit
    with:
      reviews: copilot # private, or off: when Copilot reviews pull requests
      kind: ${{ inputs.kind || '' }}
      number: ${{ inputs.number || '' }}
      dry_run: ${{ inputs.dry_run || false }}

reviews decides when the action asks Copilot to review a pull request: copilot in every repository, private only in private ones (public ones get free reviews elsewhere, such as CodeRabbit), or off. It lives in the workflow, not in .github/triage.yml, because GitHub reads it before starting a job: a push to a pull request, which only asks for that review, starts nothing where Copilot does not review. Pass the same value to the sweep. A policy that still sets pull_requests.reviews keeps working when the workflow leaves reviews out.

It reads COPILOT_GITHUB_TOKEN, and if you set them, TRIAGE_PROJECT_TOKEN for the board and OPENROUTER_API_KEY for the fallback. To pick inputs the shared workflow does not pass, such as engine or model, call the action directly with uses: crmne/copilot-triage@v0 in a job of your own.

The shared workflow asks for cache-mode: write, because GitHub gives issue and comment events a read-only cache by default and the bot keeps its conversation state there. Leave cache-mode unset in your workflow, or set it to write: a caller limited to read cannot start it. If you call the action directly, set cache-mode: write on that job.

That's it. To add the board, see The board.

The action checks out your default branch (never contributor code), restores small conversation state, installs Copilot CLI, and runs the assessment on GitHub's Ubuntu runners. The v0 tag follows tested releases; pin a commit SHA if you need an immutable version. main is development work.

Models and cost

Copilot (default)

The default model is gpt-5.6-luna at low reasoning effort, billed to the token's Copilot allowance. A typical assessment reads 10,000 to 60,000 input tokens over a few model turns, around a cent or less at GitHub's listed price. Change it with the model input.

When the Copilot allowance runs out

Give the action a fallback key and triage keeps working after your Copilot allowance is spent:

      - uses: crmne/copilot-triage@v0
        with:
          copilot-token: ${{ secrets.COPILOT_GITHUB_TOKEN }}
          fallback-api-key: ${{ secrets.OPENROUTER_API_KEY }}

Before each run, triage checks the remaining premium requests. When fewer than 50 are left and overage is off, it uses the fallback model instead, openai/gpt-5.6-luna through OpenRouter by default, the same model the Copilot engine runs (fallback-provider and fallback-model change it, for example openai and gpt-5.6-luna with an OpenAI key). A fallback run triages, replies and places the card like a Copilot run, but requests no Copilot reviews until the allowance returns.

Your own API key

Set engine: rubyllm to run a RubyLLM agent with the same system prompt, tools, evidence budget, and decision checks:

      - uses: crmne/copilot-triage@v0
        with:
          engine: rubyllm
          provider: openrouter
          model: openai/gpt-oss-120b
          api-key: ${{ secrets.OPENROUTER_API_KEY }}

provider is any RubyLLM provider slug, such as openai, anthropic, gemini, openrouter, mistral, deepseek, xai, bedrock, azure, or ollama, and model is that provider's model ID. For a server that speaks the OpenAI API, use provider: openai with api-base set to its URL; its models need not be in RubyLLM's registry. Replies show the run's cost when RubyLLM knows the model's price. The action installs the ruby_llm gem only for this engine, cached between runs.

About free models: free and small models vary widely at tool calling and judgment, and the small local ones we tried fail most of our cases (see below). Free tiers also have tight rate limits, and some free providers log or train on prompts. Issue text from private repositories goes to whichever provider you choose, so use one whose data terms you accept. Measure a model before you rely on it.

Evaluations

An evaluation corpus of real and synthetic reports checks both help and silence: stop requests, thank-yous, already answered questions, documented answers, released fixes, duplicates, recaps. A case passes only with the expected outcome, the required content, and within its call budget. Every change runs it offline; before each release, and before changing the default model, we run it live against real models and publish the results here.

Latest results, measured on October 1, 2026 with Copilot CLI 1.0.83 and RubyLLM 2.0.0. Copilot models ran two rounds of the 15 cases, local models one. Costs use GitHub's list prices without cache discounts.

Model	Engine	Passed	Unneeded replies	Missed help	Input tokens per case	Cost per assessment
GPT-5.6 Luna (default)	Copilot	29/30 (97%)	0	1	18,591	$0.0039
GPT-6 Luna	Copilot	23/30 (77%)	3	2	21,847	$0.0023
Claude Haiku 4.5	Copilot	19/30 (63%)	1	10	16,750	$0.0228
gpt-oss 20B (local, Ollama)	RubyLLM	6/15 (40%)	2	1	not reported	free (your hardware)
Qwen3 8B (local, Ollama)	RubyLLM	5/15 (33%)	2	4	not reported	free (your hardware)
GPT-5 mini	Copilot	8/30 (27%)	9	0	23,518	$0.0087
Gemini 3.6 Flash	Copilot	0/15: every case hit the 90-second limit

The default stays GPT-5.6 Luna: GPT-6 Luna costs less per assessment but misses more help and replies when it should not. Claude Haiku stays silent when the docs have an answer, GPT-5 mini replies to almost everything, and the small local models fail most cases even with the same tools and limits. Gemini 3.6 Flash never finished within Copilot CLI's 90-second limit.

Run it yourself against any model, with Copilot or through RubyLLM:

EVAL_LIVE=true TRIAGE_MODEL=gpt-5.6-luna bundle exec rake ruby_llm:eval
EVAL_LIVE=true TRIAGE_ENGINE=rubyllm TRIAGE_PROVIDER=openrouter TRIAGE_MODEL=openai/gpt-oss-120b TRIAGE_API_KEY=... bundle exec rake ruby_llm:eval

Issues

On a newly opened issue of at least 100 words, not counting logs or code, the agent writes one concise recap when it has no better answer: the problem, relevant environment, and key evidence, so you need not read the whole report. It starts with "Recap:", so nobody mistakes it for a question or an answer. A short report needs only labels, unless triage can help, for example when the request is already on the default branch but not yet released. After that, replies must add new help:

Forwarding is already available: right-click the message or picture, choose Forward, then select the destination chat.

Does restarting the app pick up the system theme?

Replies stay short, normally under 60 words, with no headings or status chatter. Ruby turns any em dash in a reply into a comma. Replies are for the reporter. One that talks to the maintainer, such as "please investigate", "needs a product decision", or "confirmed bug", is dropped before posting: the labels and board still apply, and the job summary says why. Facts learned through tools carry citations to evidence the agent actually read, and Ruby renders the links. A released-fix claim needs release notes, not code on main or a closed issue. Your docs and code show the default branch, so every file the agent reads is marked with how many commits it is ahead of your latest stable release, and before citing one the agent checks what landed since, and tells users when something is not in a release yet instead of calling it available. Map documentation files to your public site:

documentation:
  docs/*.md: https://example.com/guides/%{name}/

Every comment ends with a footer naming the model, the tokens used, and a link to the run, added by Ruby. Successful assessments get a 🎉 reaction. When a maintainer or bot spoke last, the agent does not reply again. After two triage replies since a maintainer last commented, counted from the thread, follow-ups get no reply: labels and the board still update, and a card that would wait on the reporter goes to Decide.

Duplicates and related issues

The agent searches open and closed issues and must read a candidate in full before proposing a relationship.

duplicates: suggest # or close, or 'off'

suggest posts the link and keeps the report open:

See also #325. That issue covers the mini player's taskbar entry; this request concerns the separate Milkdrop window.

close also closes clear duplicates with GitHub's native duplicate reason, but only against an older issue, never a reopened or maintainer-authored one, and never after a maintainer has joined the conversation. Both reports are fetched again before anything changes.

Neither mode closes or proposes a duplicate of an issue closed as completed: the bug came back, so the agent links it as related and calls it a recurrence. The same goes for an error report from a report_bots tracker that reopened it because the error recurred, and for a closed issue by the same author, who filed it again: the new report is the one to keep.

A link the conversation already has is not posted again: a related issue the report or a comment mentions, any issue the bot linked before, or the issue a pull request closes ("Fixes #42"). A duplicate the reporter only suspected is still confirmed. When the comment had nothing else to say, the decision is silent. The agent cannot cite the report itself as evidence.

An issue's report also lists the pull requests in your repository that reference it, open or merged in the last 90 days, read from GitHub's closing references and cross-references (up to 10 of each). When one may fix the issue and nobody has linked it in the conversation, the agent names it:

Possible fix: #31. It has merged; could you try a build from the default branch and say whether the import works?

When the pull request has merged and the reporter has not tried it, the reply asks them to retry with a build from the default branch. A pull request the conversation already mentions is not linked again.

Closing finished issues

closing: suggest # or auto

The agent proposes closing an issue for one of three reasons, always with a comment that explains it:

• Fixed: a published release fixed it, citing the release notes that name the fix. A fix on main that no release contains is not enough.
• Resolved: the reporter says it is solved, in their latest comment. When someone else spoke last, the decision is refused and nothing is posted.
• Out of scope: your documented scope rules out the request itself, citing the document.

suggest posts the comment and puts the issue in Sign off on your board. auto also closes it, as completed or, for out of scope, as not planned, but never an issue a maintainer opened, joined, or reopened; those go to Sign off instead.

After a release

Issues a release fixed often stay open because nobody went back to them. Add .github/workflows/released-fixes.yml to check them when you publish a release:

name: Released fixes
on:
  release:
    types: [published]
  workflow_dispatch:
    inputs:
      tag:
        description: Release tag whose fixed issues to settle
        required: true
      dry_run:
        type: boolean
        default: true

permissions:
  contents: read
  issues: write
  pull-requests: read

jobs:
  settle:
    uses: crmne/copilot-triage/.github/workflows/settle.yml@v0
    secrets: inherit
    with:
      tag: ${{ inputs.tag || github.event.release.tag_name }}
      dry_run: ${{ inputs.dry_run || false }}

Without a model, it lists the open issues the release may have fixed: those that pull requests merged since the previous stable release would close or mention as #123, and those its release notes mention. Each is then assessed like any other report, told which release and pull requests it is checked against, and the agent proposes closing it as fixed only when the release notes establish the fix. An issue that tracks several pieces of work, whose pull requests are now all merged and released, gets one question whether anything is left. Otherwise it stays silent. closing decides whether the issue closes or goes to Sign off. One run assesses at most 10 issues, two at a time; run it again by hand for the same tag to continue, and issues triage already answered since the release are skipped. Prereleases are left alone. A release published by a workflow's own GITHUB_TOKEN starts no workflows, so run it by hand for that tag.

Follow-ups and commands

Every eligible human comment reaches the agent, which decides whether a reply helps. followups: off limits triage to explicit commands and manual runs. Bot comments and ordinary maintainer comments are skipped before any work.

• /triage or /triage reassess asks for a fresh assessment.
• /triage mute stops the bot in that conversation; a natural-language "please stop" works too, without a public apology.
• /triage unmute (maintainers only) turns it back on.

Comment runs wait 10 seconds (debounce-seconds, 0 to 60) so a burst of comments is assessed once. A comment that arrives during an assessment invalidates it; the next run sees the whole conversation.

Error reports from bots

Issues opened by bots are skipped unless you list them:

report_bots:
  - honeybadger[bot]

Error reports are assessed for the maintainer using the exception and backtrace, which works in private repositories too.

Discussions

Discussions can stay a place for questions and community. The agent answers questions your docs establish, in the thread they were asked in. A new question on a discussion closed as resolved still gets that answer when the agent can cite one, and nothing else: no questions back, no issue links, and the discussion stays closed. Discussions closed as duplicates or moved to issues are left alone.

When a new discussion is really a reproducible bug report or a concrete feature request that no issue tracks yet, it moves to an issue. GitHub has no API to convert a discussion, so the action creates the issue itself: same title, the original text under a "Moved from" line that mentions the author (which subscribes them), and the agent's labels. It replies in the discussion with the link and closes it as outdated. The same run then triages the new issue as if the author had just opened it, so it gets its answer, labels, and board card there, and later comments on it are triaged too. The new issue is opened by the workflow's bot, so the author cannot edit it. Questions, ideas still being explored, anything already tracked, and anything the agent is unsure about stay put. So does a request your documented scope rules out, a maintainer already declined, or the agent would label wontfix, out-of-scope, or invalid: the agent answers it in the discussion with the citation, rather than opening an issue only you can close. To never move discussions:

discussions:
  move_to_issues: false

Pull requests

pull_requests:
  review_min_lines: 100 # lines of code a change needs before Copilot reviews it
  out_of_scope: suggest # or close
  visual_paths:         # files whose changes need screenshots, if your rules ask for them
    - src/ui/**

With this in your policy, pull requests are assessed when they open, reopen, become ready for review, or get a comment. The agent sees the changed files and reads patches on demand; it never checks out or runs contributor code, which is what makes pull_request_target safe here.

• Reviews. When a change touches behavior, public API, security, data handling, or non-trivial logic, and changes at least review_min_lines lines of code, the action requests a review from Copilot. Documentation, translations, workflows, lockfiles, and media do not count toward those lines, and pull requests still waiting on a process step are skipped. In a study of 693 review findings, Copilot found little that mattered on smaller changes. GitHub bills a Copilot review to whoever requests it: the review-token input, or copilot-token. After a new push, a fresh review of the new commit is requested without calling the model, once per commit. Drafts wait until they are ready. Set reviews in your triage workflow (see below), and a push starts no job at all where Copilot does not review. With the board, the daily sweep also asks Copilot to review the latest commit of any pull request it reviewed before but not since, in case a push was missed. It asks nothing of a pull request Copilot is already reviewing, and stops while fewer than 50 premium requests are left and overage is off. How thorough Copilot's review is, and what it costs, is a setting in GitHub; on changes this size, Balanced found about twice as many real problems as Lite.
• Template checkboxes. A required checkbox the author left unticked in your pull request template, such as taking responsibility for the code, is asked about before review. Optional items are left alone.
• Screenshots. When a pull request changes a file under visual_paths and neither its description nor its author's comments hold an image or video, the agent asks for the visual evidence your contribution rules require, citing them, and the author has the next move.
• Findings. The agent reads every finding Copilot and other review bots, such as CodeRabbit, leave on the latest commit. The bots catch different problems, so it weighs each one: races, data loss, broken migrations, security problems, and findings the bots rate high or major send the pull request back to its author; documentation, translation, and style remarks, nitpicks, and low-rated findings do not. A review that only says the bot ran out of quota counts as no review. A review from either bot runs triage, like a new comment. So does CodeRabbit's comment that it reached its review limit or paused reviews: the agent then judges the change without it, and the card's next step says when to ask CodeRabbit again.
• Your own pull requests are not assessed and never cost a model call: the board places them from their checks, conflicts, and reviews.
• Policy. Replies only when the author needs something: a requirement from your contribution policy, one essential question, or a scope explanation. It does not summarize changes or review code line by line: a reply that quotes code, names an identifier, or points at a line is refused. To check your contribution rules, such as keeping review screenshots out of the repository or approving git dependencies, the agent is told which binary and media files the pull request adds and which dependency manifests it changes, with any added lines that point a dependency at a git repository or a local path, which the action reads from those manifests' patches. When a documented rule is broken, it replies naming the rule and the card goes to Their move.
• State of the change. The action also tells the agent which files a conflicting pull request and its base branch both changed, which checks failed on the latest commit or that none ran, which open pull requests it is stacked on, which merged ones it still carries the original commits of (a stack whose lower pull request was squash-merged, which needs git rebase --onto), which others change the same files, and when it changes translation catalogs or lockfiles that look unrelated. The agent asks the author to rebase or fix failing checks, gives the merge order of a stack and names overlapping or superseded pull requests in the card's next step, and asks to leave out regenerated catalogs and unrelated lockfiles. Before welcoming a new feature, it checks whether the default branch already has it. These come from a few extra read-only GitHub requests; when one fails, its fact is left out.
• Out of scope. Only when your documented scope rules out the change itself, never for a missing process step, code quality, or tests, and always with a cited explanation. suggest stops at the explanation; close also closes the pull request, never one opened by a maintainer.

The board

A GitHub project board of everything that needs you, across every repository that uses it, public and private. Each column names what an item needs from you, so you can clear Sign off in minutes on your phone and save Do for focus time:

Column	What it asks of you	What lands there
Sign off	Say yes to a prepared result	Pull requests ready to merge, and closures triage proposes but may not make itself
Decide	Use your judgment	Feature requests, scope and design calls, answers only you can give
Do	Use your hands	A confirmed bug, a pull request worth reading closely, your own unfinished work, a failing default branch
Their move	Nothing	A contributor, reporter, reviewer, or upstream has the next step, or checks are still running
Not now	Nothing until you choose	Accepted but not scheduled, your own notes and roadmap; yours to set
Done	Nothing	Closed or merged since the last daily sweep, which archives it

Finished work moves to Done the moment it closes or merges, and the daily sweep archives it, so Done shows what finished today. Set archive_after_days to keep it there longer (7 keeps a week). An open item you drag to Done yourself is archived the same way, without closing it, for work you consider finished. Archived cards stay searchable in the project, and one comes back onto the board when its issue or pull request is reopened or someone comments on it.

Your own comments move cards too, without a reply: ask the reporter for a pull request and the card goes to Their move, say "not now" and it goes to Not now, take the work and it goes to Do. On a pull request, the sweep keeps that placement until the author comments or pushes, even when the pull request is approved and mergeable. The same holds when triage's own reply asks the author for something, such as screenshots: the card stays in Their move.

In Sign off, an issue is a closure triage proposes: close it if you agree, or answer in the issue if you don't. Triage never proposes closing an issue that was reopened or that you have commented on, and the sweep places such a card again from its conversation: back to the reporter when you spoke last, to you when they did. Each card also gets a Priority (Urgent, High, Normal) and a Next step written for you, such as "Reproduce from the Windows backtrace; likely the path join in loader.rb".

Set it up

1. Create an empty project for your account or organization, for example with gh project create --owner your-name --title "Maintainer board". Keep it private unless you want the public to see your process; labels and replies stay the public signal on each issue.
2. Create a classic personal access token with the project scope, plus repo for private repositories and for requesting your review on pull requests (fine-grained tokens cannot reach user-owned projects yet). Save it as a TRIAGE_PROJECT_TOKEN secret.
3. Add the board to your policy:

board:
  project: https://github.com/users/your-name/projects/1
  maintainer: your-name   # requests your review and assigns urgent issues
  archive_after_days: 0   # optional: days finished work stays in Done before the sweep archives it

4. Add a daily sweep as .github/workflows/board.yml. The sweep uses no model. It reads the same secrets as triage: TRIAGE_PROJECT_TOKEN for the board, and COPILOT_GITHUB_TOKEN to ask Copilot to review new pushes:

name: Board
on:
  schedule:
    - cron: '17 6 * * *'
  workflow_dispatch:
    inputs:
      dry_run:
        type: boolean
        default: true

permissions:
  contents: read
  pull-requests: write # requests your review
  actions: write       # sends fork pull requests to triage

jobs:
  sweep:
    uses: crmne/copilot-triage/.github/workflows/sweep.yml@v0
    secrets: inherit
    with:
      reviews: copilot # the same as in the triage workflow
      dry_run: ${{ inputs.dry_run || false }}
      triage_workflow: triage.yml

Run it once by hand with dry_run turned off. It builds the board on the empty project: the six columns in order with their colors, the Priority and Next step fields, and an All repositories board view. Every sweep keeps that shape, archives finished work, and fixes nothing that is already right. A column of your own survives. Columns of earlier versions are renamed in place, so their cards stay put: Approve becomes Sign off, Answer or decide becomes Decide, Fix becomes Do, Waiting on others becomes Their move, and Backlog becomes Not now. Review is retired, and its cards are placed again.

To pick a repository from a list beside the board, open the view's menu and choose Slice by › Repository. GitHub's API cannot set a slice, so this one click is yours; the counts next to each repository show where work piles up.

How cards move

On each assessment, the agent picks the item's next move, a priority, and the next step. An issue moves to Their move only when that run asked the reporter something, and a closed one goes to Done. Priority only rises. Urgent issues are assigned to maintainer, the one notification GitHub sends you.

Pull requests follow GitHub's facts on their latest commit, without a model. Drafts, failing checks, conflicts, and a person's requested changes that the author has not pushed after are the author's move (Do for your own), and an approved, green, mergeable pull request goes to Sign off. Another review bot's approval of a ready pull request, such as CodeRabbit's when it found nothing, also goes to Sign off; when it requests changes or leaves findings, the agent weighs them, since bots ask for changes over nitpicks too. With CodeRabbit, set request_changes_workflow: true in .coderabbit.yaml so it approves clean pull requests instead of only summarizing them. Copilot's Approval recommended goes to Sign off, Changes recommended to Their move, and a pull request whose review or checks are still running waits in Their move, including right after a new push; a card already placed keeps its column meanwhile, so your review is not withdrawn and requested again. Needs a closer look is the agent's call, because it means two things: when Copilot only says a change is broad or risky, the pull request is ready for you (Sign off, or Do when it deserves a close read); when Copilot names something still wrong, it goes back to its author. Triage runs when Copilot posts its review on a same-repository pull request; for forks, whose review runs get no secrets, the sweep sends the pull request to the workflow named by triage-workflow, once per review. That workflow needs a workflow_dispatch trigger with kind, number, and dry_run inputs, as in Preview a report.

A contributor's pull request that nothing reviewed on its latest commit, such as when CodeRabbit ran out of reviews, never goes to Sign off, whatever the agent proposes: it goes to Do, since it needs your reading first.

While a pull request sits in Sign off or Do, your review is requested by the workflow, so GitHub's review-requested list is your queue; it is withdrawn when the card moves on. Issues are not assigned to you outside urgent ones, since being assigned subscribes you to every comment.

The sweep also places issues by GitHub facts: new ones go to Decide, or Do when labeled bug; to Their move when a maintainer spoke last; and your own untouched issues to Not now. An issue with an open pull request that would close it follows that pull request's card. A card in Their move comes back when the reporter answers, and goes to Decide after 21 quiet days since your reply, with no pull request that would close it, such as an approved plan whose pull request never came: nudge them, or mark it up for grabs. The sweep posts nothing there, and a card you drag back waits another 21 days. Cards in Not now or a column of your own stay where you put them. While the default branch fails its checks, an urgent card in Do says so, because every pull request inherits the failure; it is archived once the branch passes.

Labels

Labels stay public and simple: the types in your policy, such as bug, enhancement, documentation, and question, on issues and pull requests alike. Triage creates any of them a repository lacks. Priority, next steps, and columns stay private on the board.

bug, enhancement, question, and documentation are type labels. Once a report has one, whether a maintainer or an earlier run set it, the agent adds no other type, only labels such as wontfix. It never pairs enhancement with bug, wontfix, or out-of-scope.

A label that asks the reporter for something, such as needs-info: "One specific missing technical fact prevents investigation.", comes off an issue once the reporter supplies it. The agent names the label to remove, and Ruby removes it only when the issue has it, your policy lists it, and the reporter wrote the latest comment. When the answer is partial, the label stays and the reply says what is still missing.

Preview a report

Add manual inputs to the workflow:

  workflow_dispatch:
    inputs:
      kind:
        type: choice
        options: [issue, discussion, pull_request]
        default: issue
      number:
        description: Issue, discussion, or pull request number
        required: true

and these inputs to the action step:

          kind: ${{ inputs.kind || (github.event.discussion && 'discussion' || (github.event.pull_request || github.event.issue.pull_request) && 'pull_request' || 'issue') }}
          number: ${{ inputs.number || github.event.issue.number || github.event.discussion.number || github.event.pull_request.number }}
          dry-run: ${{ github.event_name == 'workflow_dispatch' }}

Add || inputs.number to the concurrency group's number expression. A manual run shows its decision in the job summary without changing anything, including for closed reports. Previews spend model tokens.

To assess existing reports after installing, dispatch the same workflow for each open issue and pull request with dry_run off. Pass quiet: true (to assess.yml, beside dry_run, or to the action step) to sort them onto the board without comments, labels, closures, or Copilot reviews, so nobody is pinged about an old thread.

Configuration

Action inputs:

Input	Default	Purpose
copilot-token		Copilot Requests token for the copilot engine
github-token	github.token	Reads reports, publishes labels and comments
config	.github/triage.yml	Policy file on the default branch
engine	copilot	copilot or rubyllm
model	gpt-5.6-luna	Model ID; for rubyllm, as the provider names it
reasoning-effort	low	Copilot reasoning effort, none or low; default for models without it, such as Claude Haiku
provider, api-key, api-base		Provider, key, and optional endpoint for rubyllm
fallback-api-key, fallback-provider, fallback-model	openrouter, openai/gpt-5.6-luna	Fallback once the Copilot allowance is spent
review-token	copilot-token	Token that requests Copilot reviews, billed to its owner
project-token		Classic token with the project scope, for the board
triage-workflow	triage.yml	For the sweep, the triage workflow to run on fork pull requests Copilot asks a human to look at
mode	triage	triage, sweep for the board, or release to list the issues a release may have fixed
release, release-pulls		The release tag, and in triage the pull requests in it that reference the issue, for After a release
kind, number	from the event	What to assess, for manual runs
reviews	policy, else copilot	When Copilot reviews pull requests: copilot, private, or off
dry-run	false	Show the decision without changing GitHub
debounce-seconds	10	Wait for nearby comments, 0 to 60

Several repositories can share settings: start a policy with extends: owner/repo:path/to/policy.yml, and it inherits that file's settings, section by section, with its own keys winning. A shared policy that cannot be read is reported and skipped. Review settings are the exception: reviews goes in the workflow, since GitHub must know it before starting a job.

Policy keys in .github/triage.yml: labels (at most two per issue), replies (optional reply templates), sources (globs the agent may read), documentation (links to your site), instructions (your project's policy), duplicates, closing, followups, report_bots, discussions, pull_requests, and board. See examples/triage.yml.

Safety

• No code execution. The model has no shell, no file writes, no arbitrary URLs, and no GitHub write tools. It gets six tools:

  Tool	Returns
  search_repository	Up to ten literal matches in your configured sources
  search_issues	Up to five same-repository issue search results
  list_releases	Five published releases with short notes
  search_changes	Release notes and unreleased default-branch commits that mention a phrase
  read_evidence	A paged file, issue, release, or pull request patch, up to 6 KB
  submit_decision	A schema-checked proposal; no GitHub mutation

• Ruby publishes. Labels, comments, closures, moves, reviews, and board changes happen in Ruby after validating the decision, re-reading the report, and re-checking cited evidence. A report that changed during assessment is left alone.

• Untrusted input. Report text, comments, code, and links are evidence, never instructions. Comments cannot contain mentions, HTML, raw URLs, or Markdown links written by the model.

• Tokens stay out of the model. The GitHub, project, review, and API tokens are removed from the Copilot CLI environment and redacted from logs. Copilot runs with isolated settings and repository instructions disabled.

• Bounded. At most 12 evidence calls, 90 seconds (Copilot) or 20 turns and 120 seconds (RubyLLM). The agent reads the whole conversation, up to the latest 100 comments, because long threads are where it helps most. Only a single text over 30 KB, such as a pasted log, is shortened around a visible marker, and a starting prompt over 200 KB (about 50,000 tokens) is left for a maintainer.

Failures

Model failures and invalid decisions appear in the job summary and leave the report unchanged. They never become issues, comments, or apologies. A Copilot session that ends without calling a single tool, which happens when many runs start at once, changed nothing, so it is tried again after 20 and then 40 seconds; the failure message includes the model's final text. A model that finishes without submitting a decision, even after being reminded and made to call submit_decision, gets one fresh run before the assessment fails; the metrics count the model calls, tokens, and cost of both. GitHub write failures fail the job without marking the assessment complete, so it is retried. A failed board update or review request fails the job after the reply is published, so it never causes a repeated reply. Exhausted Copilot credits need a reset or a budget; the action never changes billing settings.

Conversation state (mute, processed updates, prior replies, review history) lives in a small Actions cache. If it is evicted, up to 500 older comments are recovered; an unchanged issue may then be assessed again.

Development

See CONTRIBUTING.md. Tests use fake GitHub and model responses; the Copilot CLI integration test uses a local fake provider and spends no credits.

bundle install
bundle exec rubocop
bundle exec rspec
bundle exec rake ruby_llm:eval

This independent project uses GitHub Copilot CLI and RubyLLM and is not an official GitHub product. MIT licensed.