Fix internal preview deployment environment

[bingran-you]

Summary

• bump main from the final 0.6.2 release version to the next preview line, 0.6.3.dev0
• move integration-eval provider secrets to a dedicated integration-eval-secrets environment so CI deployments stop polluting pypi-internal-preview
• document the separate GitHub environments and why integration eval must not reuse the PyPI preview environment

Why

pypi-internal-preview was showing as inactive because integration-eval also used that environment. GitHub Actions creates deployment records for every job environment, so PR/main integration runs mixed with real PyPI preview deployments and could mark them inactive. The release workflow was also skipping publication because main still carried final version 0.6.2; preview publication only resumes once main is on a plain .dev version.

Validation

• uv lock --check
• uv run python -m pytest tests/test_release_version.py -q
• uv run python tools/release_version.py internal-preview --run-number 123 -> publish=true, version=0.6.3.dev123
• uv run ruff check tools/release_version.py tests/test_release_version.py
• uv run ty check tools/release_version.py tests/test_release_version.py

Follow-up

Repo admin should create integration-eval-secrets and copy the integration provider secrets there if GitHub Models fallback is not sufficient. The current token cannot create environments (403 Must have admin rights).

[bingran-you]

Current blocker for this PR:

• The code-side fix is in place: main moves to 0.6.3.dev0, and integration eval moves off the PyPI publishing environment to integration-eval-secrets.
• CI failed at Select integration provider because the new environment does not yet have the provider secrets that currently live under pypi-internal-preview (DeepSeek/GLM/Qwen are missing; repo-level fallbacks returned LiteLLM 401, OpenAI 429, GitHub Models 403).
• I attempted to create/populate integration-eval-secrets, but the current GitHub token lacks repo-admin permission (403 Resource not accessible by personal access token).

To unblock: a repo admin should create integration-eval-secrets and copy the integration provider secrets from pypi-internal-preview into it, at minimum one working provider key/base-url pair; recommended: DEEPSEEK_API_KEY, DEEPSEEK_BASE_URL, GLM_API_KEY, GLM_BASE_URL, QWEN_API_KEY, QWEN_BASE_URL, LITELLM_API_KEY, LITELLM_BASE_URL, OPENAI_API_KEY, OPENAI_BASE_URL, and DAYTONA_API_KEY where available. Then rerun CI on this PR.

Alternative: move these provider secrets to repo/org-level Actions secrets and remove environment: from integration eval entirely. That avoids creating GitHub deployment records for CI jobs, but gives up the environment-scoped secret boundary.

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
benchflow-bff148e7	🟢 Ready	View Preview	Jun 15, 2026, 6:01 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

[bingran-you]

Automation status update (2026-06-16):

Progress: the integration-eval-secrets GitHub Actions environment now exists (created since the last CI run). The code side is complete (main → 0.6.3.dev0, integration-eval moved off pypi-internal-preview).

Remaining blocker (needs repo-admin): integration-eval-secrets is empty — none of the 7 provider secrets are populated, so eval-and-judge still fails at Select integration provider (DEEPSEEK/GLM/QWEN all missing). I attempted to populate them from the local .env but the automation PAT is maintain/not admin, so env-secret writes return 403.

Exact action needed — copy these 7 secrets into integration-eval-secrets (same values already live in pypi-internal-preview):

• DEEPSEEK_API_KEY, DEEPSEEK_BASE_URL
• GLM_API_KEY, GLM_BASE_URL
• QWEN_API_KEY, QWEN_BASE_URL
• DAYTONA_API_KEY

Once populated, re-run eval-and-judge and this is mergeable. Keeping status:blocked.

[bingran-you]

Current-state update (2026-06-18, automation):

This PR has two parts, and one is now obsolete while the other is blocked:

1. Version bump → 0.6.3.dev0 is now moot. main has already shipped 0.6.3 (final) and v0.6.4 is in flight (chore: release v0.6.4 #801), so the "main still on final 0.6.2 → preview publish skipped" rationale no longer applies. This part should be dropped on rebase.
2. integration-eval-secrets environment split is still novel (not yet on main) and remains the real value here — but it is blocked: moving integration-eval to a dedicated environment makes eval-and-judge fail-fast (it fails in ~10s) because the new integration-eval-secrets environment exists but is empty. The automation PAT has maintain, not admin, so it cannot populate environment secrets. An admin must copy the 7 provider secrets (present in .env / pypi-internal-preview) into integration-eval-secrets, then re-run.

The PR is also now CONFLICTING/DIRTY (pyproject/uv.lock version drift from the 0.6.3 release).

Decision needed (owner): either (a) admin populates the 7 secrets in integration-eval-secrets, then I rebase off the version bump and re-run; or (b) close this PR if the env-split is no longer wanted. Holding status:blocked.

[bingran-you]

Automation triage (2026-06-20): moving this to review:changes-requested rather than keeping it ready for review. The branch is now stale against the current split integration workflow structure: it still modifies deleted .github/workflows/integration-eval.yml, conflicts in release/version files, and its old provider-secret-environment blocker is obsolete because current main already uses integration-light, integration-scope, and integration-final-review against pypi-internal-preview. Recommended action is close/supersede #784 or rebuild the preview-env fix from current main; do not merge this branch as-is.

[bingran-you]

Closing as superseded by the current split integration workflows. This branch is now conflicted against main, still targets the old integration-eval shape, and its original internal-preview env-secret blocker has been handled by the newer workflow/secret path. If we still want an env-split follow-up, it should be rebuilt from current main as a fresh narrow PR.