[CRCR] Implement CRCR upstream check run management for L3/L4 jobs

[can-gaa-hou]

Summary

• The current implementation focuses on L3/L4 levels defined in the RFC: [RFC] Cross-Repository CI Relay for PyTorch Out-of-Tree Backends rfcs#90
• For detailed design for L3/L4, please refer to Adding L1-L4 design to RFC-0050 rfcs#93

Architecture

• webhook function:
  • Create PR label handling function for L3 repo (refer to the 3 scenario cases mentioned in Adding L1-L4 design to RFC-0050 rfcs#93)
    • Scenario 1: Label arrives before the downstream workflow is triggered. So we cached this information in Redis using mark_check_run_wanted and let the callback lambda create this check run.
    • Scenario 2: Label arrives during the downstream workflow is running. The callback lambda will cache workflow information beforehand so it can immediately create the in_progress check run.
    • Scenario 3: Label arrives after the downstream workflow is done. If the cached workflow information is still alive in Redis (3 days by default, could be set by OOT_STATUS_TTL), it will create a completed check run immediately. Otherwise, it will not create a check run.
  • Dispatch function will check for L3 label or L4, and store this information in Redis for the callback to check whether a check run is needed.
  • Create check run and check suite handling functions for the downstream workflow re-run mechanism within the check run.
• callback function:
  • Handle upstream check-run creation for L3/L4
    • Scenario 1 & L4: Check in Redis by calling is_check_run_wanted to see if this PR needs a check run. If so, immediately create one.
    • All Scenario: Store workflow information in Redis for check run creation in the webhook.

Changes

aws/lambda/cross_repo_ci_relay/
├── tests/                         # Add more unit tests
├── allowlist.py                   # Update utils function for L3
├── redis_helper.py                # Set more keys for L3
├── gh_helper.py                   # Update utils function for L3
├── misc.py                        # Update utils function for L3
├── event_handler.py               # Create PR label/check run/check suites handling function
└── callback_handler.py            # Handle upstream check-run creation

Verification

We performed the following scenario verification on our AWS Lambda instance:

• L3:

  • L3 labels named ciflow/crcr/{device} are added immediately after the PR is created, and show up in the corresponding check-run on the PR with the name crcr/{repo}/{workflow_name}.
  • After clicking into the check-run, the corresponding information is correct.
  • L3 labels are added while the workflow is running, which should show up the in_progress check-run.
  • L3 labels are added after the workflow is done, which should show up the completed check-run.
  • Check run is updated when the PR with L3 labels is reopened or synchronized.

• L4:

  • Check run should be created after the PR is opened.
  • Check run is updated when the PR is reopened or synchronized.

• Re-run

  • Clicking the Re-run button in each failed check run will trigger the corresponding downstream workflow to re-run and update the check run status to in_progress.
  • Clicking the Re-run all jobs or Re-run all failed jobs button will trigger the corresponding downstream workflows in the check suite and update the corresponding check run status to in_progress.

Unit Tests

• Unit Tests (Mock)

TODO

• Modify PyTorchBot for L3 non-blocking merge: [WIP][CI][CRCR] Update mergebot rule to adapt CRCR L3 pytorch#185612
• Add a new feature in PyTorchBot that automatically mentions related on-call maintainers in L3/L4 repos when check-run fails: [CRCR] Implement on-call bot and allowlist functionality for downstream CI failures #8183

cc @albanD @fffrog @KarhouTam @atalman @huydhn @zxiiro @subinz1 @jewelkm89

[vercel]

@can-gaa-hou is attempting to deploy a commit to the Meta Open Source Team on Vercel.

A member of the Team first needs to authorize it.

[subinz1]

upstream_token is minted for config.upstream_repo (always pytorch/pytorch), but it's called inside the for downstream_repo in l3_repos loop. If a device maps to multiple repos, this creates redundant installation tokens for the same upstream repo on every iteration.

Consider hoisting it above the loop:

    upstream_token = gh_helper.get_repo_access_token(
        config.github_app_id,
        config.github_app_private_key,
        config.upstream_repo,
    )

    created: list[str] = []
    for downstream_repo in l3_repos:
        ...

This also gives a clean fail-fast — if the token mint fails, it would fail identically on every iteration anyway, so there's no benefit to retrying it per-repo inside the loop.

[can-gaa-hou]

Good catch! Fixed with lazy mint since when job_info is None (mostly happens when the label is added before the workflow starts), we don't even need to call the GitHub API.