Skip to content

Store and consult denylist for repeated failures - #65

Open
sgallagher wants to merge 2 commits into
fedora-eln:mainfrom
sgallagher:denylist
Open

Store and consult denylist for repeated failures#65
sgallagher wants to merge 2 commits into
fedora-eln:mainfrom
sgallagher:denylist

Conversation

@sgallagher

@sgallagher sgallagher commented Aug 12, 2026

Copy link
Copy Markdown
Member

Once a build (identified by its SCM URL) has been part of a batch and has failed, save it to the database in a denylist. When starting up a batch, check each requested build to see if it matches the denylist and exclude it if it does.

This will avoid endless retries in the periodic catch-up task.

Fixes: #64

Summary by CodeRabbit

  • New Features

    • Failed builds are now tracked and automatically excluded from future rebuild attempts.
    • Rebuild checks support batched processing for improved reliability with large sets of builds.
    • Superseded build triggers are completed and recorded instead of being removed silently.
  • Bug Fixes

    • Empty rebuild batches are now skipped safely without causing errors.
    • Improved handling for missing or outdated build state.
  • Documentation

    • Added guidance for failed-build filtering, persistence, rebuild behavior, and database batch-size configuration.

@sgallagher sgallagher self-assigned this Aug 12, 2026
@coderabbitai

coderabbitai Bot commented Aug 12, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@sgallagher, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 43 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 64e45b93-7c1e-4019-a469-a75d36253dc5

📥 Commits

Reviewing files that changed from the base of the PR and between 7a42047 and f39b48d.

📒 Files selected for processing (2)
  • .flake8
  • README.md
📝 Walkthrough

Walkthrough

Changes

Failed-build denylist workflow

Layer / File(s) Summary
Failed-build persistence
elnbuildsync/db_models.py, README.md
Adds the failed_builds model and batched lookup and recording helpers. Documents the database configuration and architecture.
Batch admission and failure recording
elnbuildsync/rebuildbatch.py, elnbuildsync/batching.py, README.md
Filters failed or URL-less triggers, skips empty batches, records failed SCM URLs, and updates related documentation.
Trigger completion semantics
elnbuildsync/buildtrigger.py, elnbuildsync/batching.py
Centralizes idempotent completion and logs superseded triggers using component and build identifiers.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant RebuildBatch
  participant db_models
  participant failed_builds
  participant batching
  RebuildBatch->>db_models: Query failed SCM URLs
  db_models->>failed_builds: Read denylist in batches
  failed_builds-->>db_models: Return denylisted URLs
  db_models-->>RebuildBatch: Filter eligible triggers
  RebuildBatch->>batching: Skip empty batch
  RebuildBatch->>db_models: Record failed SCM URLs
  db_models->>failed_builds: Insert unique URLs
Loading

Possibly related PRs

Suggested reviewers: bhoy-troy

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: storing and consulting a denylist for repeated build failures.
Linked Issues check ✅ Passed The changes implement denylist persistence and filtering to stop periodic catch-up from repeatedly retrying completely failed builds [#64].
Out of Scope Changes check ✅ Passed The code and documentation changes support denylist persistence, filtering, failure handling, and related batch completion behavior.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
elnbuildsync/rebuildbatch.py (1)

226-233: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Keep task failure metadata separate from SCM URLs.

RebuildBatchSlice.run() can return "Task: {task_id}" or None alongside URL strings. RebuildBatch.run() treats every entry as an SCM URL. The task label can enter the denylist as a fake URL, and None can break email formatting. Preserve the structured failure data and pass only valid SCM URLs to URL-based consumers.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@elnbuildsync/rebuildbatch.py` around lines 226 - 233, Update
RebuildBatch.run() to preserve the structured failure metadata returned by
RebuildBatchSlice.run() while filtering all_successes and all_failures before
URL-based consumers. Exclude None and task labels such as “Task: {task_id}” from
record_failed_build_urls and any URL/email formatting, passing only valid SCM
URL strings; retain task failure information separately for its intended
consumers.

Source: Learnings

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@elnbuildsync/rebuildbatch.py`:
- Around line 79-98: Update the SCM lookup handling in the loop around
BuildTrigger.get_scmurl(): treat InfoUnavailableError as the isolated
missing-URL case, but stop completing triggers for that or other lookup
failures. Propagate failed Koji requests out of the loop so
process_message_batch() leaves queued triggers uncompleted for retry, while
preserving successful triggers in triggers_with_scmurl.

In `@README.md`:
- Around line 39-43: Fix the malformed Markdown table rows in README.md: lines
39-43, keep the Denylist entry within one table row or use <br> inside its
second cell; lines 93-101, apply the same formatting to every When row. Ensure
no continuation text remains outside the table cells.

---

Outside diff comments:
In `@elnbuildsync/rebuildbatch.py`:
- Around line 226-233: Update RebuildBatch.run() to preserve the structured
failure metadata returned by RebuildBatchSlice.run() while filtering
all_successes and all_failures before URL-based consumers. Exclude None and task
labels such as “Task: {task_id}” from record_failed_build_urls and any URL/email
formatting, passing only valid SCM URL strings; retain task failure information
separately for its intended consumers.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 4bd796c7-db79-43c4-b58d-85b9a658c878

📥 Commits

Reviewing files that changed from the base of the PR and between a7dd109 and 7a42047.

📒 Files selected for processing (5)
  • README.md
  • elnbuildsync/batching.py
  • elnbuildsync/buildtrigger.py
  • elnbuildsync/db_models.py
  • elnbuildsync/rebuildbatch.py

Comment thread elnbuildsync/rebuildbatch.py
Comment thread README.md Outdated
Once a build (identified by its SCM URL) has been part of a batch and
has failed, save it to the database in a denylist. When starting up a
batch, check each requested build to see if it matches the denylist and
exclude it if it does.

This will avoid endless retries in the periodic catch-up task.

Signed-off-by: Stephen Gallagher <sgallagh@redhat.com>
Signed-off-by: Stephen Gallagher <sgallagh@redhat.com>
level=logging.WARNING,
)
except Exception:
logger.exception(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the double logging necessary?

@bhoy-troy bhoy-troy left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

Just a question re potential double logging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stop retrying builds in periodic catch-up

2 participants