Skip to content

ci: drop PyTorch 2.0-2.5 and raise the minimum supported version to 2.6 - #3449

Merged
bhimrazy merged 13 commits into
Lightning-AI:masterfrom
bhimrazy:ci/drop-torch-2.0-2.5
Aug 31, 2026
Merged

ci: drop PyTorch 2.0-2.5 and raise the minimum supported version to 2.6#3449
bhimrazy merged 13 commits into
Lightning-AI:masterfrom
bhimrazy:ci/drop-torch-2.0-2.5

Conversation

@bhimrazy

@bhimrazy bhimrazy commented Aug 9, 2026

Copy link
Copy Markdown
Collaborator

What does this PR do?

Drops the end-of-life PyTorch minors and raises the floor to 2.6, keeping the latest five supported minors (2.6–2.10). Mirrors Lightning-AI/pytorch-lightning#21851.

  • CPU matrix: drop the 2.0–2.5 rows, retarget oldest/macOS/Windows to 2.6.0 — 23 → 17 jobs
  • Docker: drop the 2.0.1–2.5.1 CUDA images — 14 → 8 builds
  • GPU lanes: oldest image moves to torch 2.6
  • Requirements: torch >=2.6.0, torchaudio >=2.6.0, torchvision >=0.21.0, pytorch-lightning >=2.5.1 (first release CI-tested against torch 2.6)

The GPU lanes hardcoded TORCH_VER == "2.0" to gate the oldest requirements, which silently goes stale when the floor moves on the CPU side. Now derived from requirements/base.txt, with a fail-fast check if an image sits below the declared floor.

How the floor check works

TORCH_MIN is read from requirements/base.txt and used both to gate the oldest-requirements branch and to reject a mismatched image:

ERROR: image provides torch 2.4, below the 2.6 floor in requirements/base.txt.
       Bump the images in .lightning/workflows/ together with the floor.

The comparison goes through packaging.version.Version rather than a string compare, so 2.10 > 2.6 orders correctly. It runs before adjust-torch-versions.py, which rewrites the >= pin to an exact version.

Dependency floors raised for the newer images

The GPU and docker images move from Python 3.10 to 3.11, so a few packages need floors that actually publish cp311 wheels — each carries an inline note explaining why:

  • onnxruntime >=1.12.0>=1.15.0
  • scikit-image >=0.19.0>=0.20.0
  • mecab-ko >=1.0.0>=1.0.1

PIP_BREAK_SYSTEM_PACKAGES=1 was added to integrations.yml, since the newer pytorch/pytorch images ship a PEP 668 "externally managed" system Python.

Docker, docs, and a bug fixed along the way
  • dockers/ubuntu-cuda/Dockerfile — defaults moved to CUDA 12.4.1 / Python 3.11 / torch 2.6.
  • dockers/README.md — the build example pointed at a dockers/base-cuda/Dockerfile path that no longer exists, and at an unsupported CUDA 11.7.1 / torch 2.0 combination.
  • .lightning/workflows/unittests.yml referenced an undefined lowercase ${cuda_version} when computing CUDA_VERSION_M_M, silently producing an empty CUDA_VERSION_MM and a broken wheel index URL (.../whl/cu/torch_stable.html). Fixed to ${CUDA_VERSION%.*}.

Follow-ups: docs/support policy, then removing the dead _TORCH_GREATER_EQUAL_2_1..2_5 paths.

…t device

`Metric.__init__` gated the use of `torch.get_default_device()` on torch >= 2.3.
However, until torch 2.8 that function only reflected `set_default_device()` and
ignored an active `with torch.device(...)` context manager, so metrics created
inside such a block were assigned the CPU instead of the context device.

torch 2.8 (pytorch/pytorch v2.8.0) added the `DeviceContext` lookup to
`get_default_device`, so gate on 2.8 and keep the `torch.empty(0).device` probe
for older versions, which does respect the context manager.

Verified on torch 2.6.0: inside `with torch.device("meta")`,
`get_default_device()` returns cpu while `torch.empty(0).device` returns meta.

Adds a CPU-runnable regression test using the meta device; the existing coverage
for this path was GPU-only.
Drops the end-of-life PyTorch minors and raises the floor to 2.6, keeping the
latest five supported minors (2.6-2.10), matching what Lightning did in #21851.

- CPU matrix: drop the 2.0-2.5 rows and retarget the oldest/macOS/Windows
  entries to 2.6.0 (23 -> 17 jobs).
- Docker: drop the 2.0.1-2.5.1 CUDA images (14 -> 8 builds); 2.6.0/cu12.4.1 is
  the new oldest. Dockerfile ARG defaults follow it.
- GPU lanes: move the oldest image to torch 2.6 (cu12.4.1 for unittests,
  pytorch/pytorch:2.6.0-cuda12.4 for integrations).
- Requirements: torch >=2.6.0, torchaudio >=2.6.0, torchvision >=0.21.0.
- pytorch-lightning floor 1.9.0 -> 2.5.1, the first release CI-tested against
  torch 2.6.

The GPU lanes previously hardcoded `TORCH_VER == "2.0"` to decide whether to
apply the oldest requirements. That silently goes stale whenever the floor moves
on the CPU side, so derive it from requirements/base.txt instead and fail fast
if an image is below the declared floor.
@codecov-commenter

codecov-commenter commented Aug 9, 2026

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 36%. Comparing base (d184220) to head (02307a0).
⚠️ Report is 12 commits behind head on master.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@          Coverage Diff           @@
##           master   #3449   +/-   ##
======================================
- Coverage      37%     36%   -0%     
======================================
  Files         349     349           
  Lines       19901   19910    +9     
======================================
- Hits         7264    7263    -1     
- Misses      12637   12647   +10     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

- state the version check in positive polarity instead of relying on
  bool -> exit-code coercion, so it reads the same way as the error below it
- shorten the guard comments to the one-liner style used in these files, and
  note the adjust-torch-versions ordering in both workflows, not just one
- drop a redundant `pip install -q packaging`, already installed at the top
- keep the _integrate.txt note about our own floor rather than upstream's
  test matrix, which we cannot verify here
@bhimrazy bhimrazy changed the title [wip] ci: drop PyTorch 2.0-2.5 and raise minimum to 2.6 ci: drop PyTorch 2.0-2.5 and raise the minimum supported version to 2.6 Aug 21, 2026
@mergify mergify Bot added the ready label Aug 21, 2026
@bhimrazy
bhimrazy enabled auto-merge (squash) August 25, 2026 02:22
@mergify

mergify Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

@bhimrazy
bhimrazy merged commit 8d008de into Lightning-AI:master Aug 31, 2026
65 checks passed
@bhimrazy
bhimrazy deleted the ci/drop-torch-2.0-2.5 branch August 31, 2026 13:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants