[bdemirb]

Set HF_HOME to a writable per-job cache in trunk.yml before the torchao checkpoint job runs. Keep a matching fallback in test_torchao_huggingface_checkpoints.sh so direct callers do not try to write into OSDC's read-only
/mnt/hf_cache mount.

cc @digantdesai @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell @rascani

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24604

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 49 Pending

As of commit 612028f with merge base 3d5172b ():

NEW FAILURE - The following job has failed:

• pull / test-llama-runner-linux-android / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[bdemirb]

@pytorchbot label "partner: arm"

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[zingo]

A rebase/update-branch will probably fix the Arm error. A fix as merged recently.

[zingo]

Ci seem to be stuck in a bad state I rebased to get a proper re-run

[shoumikhin]

@claude review effort=max

[claude]

Claude finished @shoumikhin's task in 13m 21s —— View job

---

Requesting changes: setting only HF_HOME most likely doesn't move hf download off OSDC's read-only HF_HUB_CACHE, and making the redirect work would bypass the pinned, seeded cache that #24524 (already in this branch) added for the same failure.

PR Review: #24604

Summary

The PR redirects HF_HOME to a per-job writable directory in the test-torchao-huggingface-checkpoints job, and adds a fallback in the script. The trunk.yml change most likely has no effect on hf download, because OSDC sets HF_HUB_CACHE separately. If it did take effect, it would undo the seeded-cache fix from #24524. The script change does nothing in CI and makes local runs worse.

Infrastructure

• trunk.yml:610 and script :50-55: the redirect doesn't reach hf download (blocker). huggingface_hub uses HF_HUB_CACHE when it is set, and falls back to $HF_HOME/hub only when it isn't. The repo already documents that OSDC sets it:

  • script :59-61 (from Pin Hugging Face revisions in test_torchao_huggingface_checkpoints.sh #24524): "On OSDC runners HF_HUB_CACHE is a read-only mount"
  • vulkan.yml:180: "Runners pin HF_HUB_CACHE to a read-only mount independently of HF_HOME". That is why vulkan.yml:184 also exports HF_HUB_CACHE.
  • .ci/scripts/tests/vulkan/test_export_vulkan_llm.py:37-85 tests exactly this case, with both variables set to read-only paths.

  The block here was copied from cuda.yml:984-988, which sets only HF_HOME. So the hf download calls at :79, :102 and :125 still use the read-only mount, and any snapshot missing from it still fails with "Read-only file system". The new comments at trunk.yml:608 and script :48-49 also contradict the existing comment at script :59-61.

  The six checkpoint jobs passing in trunk run 42605229963 doesn't show the redirect works. With Pin Hugging Face revisions in test_torchao_huggingface_checkpoints.sh #24524's pins every snapshot is already in the seeded cache, so nothing needs to be written. The set -x line HF_MODEL_DIR=... in those logs shows which cache served the files: a path under /mnt/hf_cache means the redirect did nothing. I couldn't check this myself because the log download API returned a server error.

  If a writable cache really is needed, vulkan.yml:180-185 has a version that works: it also exports HF_HUB_CACHE="${HF_HOME}/hub" and HF_HUB_DISABLE_XET=1. Every sibling script that downloads into a fresh cache sets the latter, for example download_hf_hub.sh:3-4, test_lora.sh:10 and test-rocm-voxtral.sh:31. Read the next point first, though.

• A working redirect would undo Pin Hugging Face revisions in test_torchao_huggingface_checkpoints.sh #24524 (major). Pin Hugging Face revisions in test_torchao_huggingface_checkpoints.sh #24524 (237c4635, an ancestor of this PR's commit 9bf3913f) fixed the same LFM2.5 "Read-only file system" failure. It pinned revisions so the seeded mount serves them, and documented the ci-refresh-hf-cache label as the way to bump them (script :59-64; test_lora.sh:60-64 uses the same scheme). Pointing the hub cache at an empty ${RUNNER_TEMP}/hf_cache would mean:

  • All 6 matrix entries (3 models, each on xnnpack/x86 and torchao/arm64) download their full multi-GB checkpoints from the Hub on every trunk trigger. The block is unconditional, and trunk runs on pushes to main and release/* and on PRs touching .ci/scripts/**.
  • The downloads are anonymous, because the job has no secrets-env (trunk.yml:597-602).
  • The pin-and-seed comment at script :59-64 no longer describes this job.

  Please name the failure that still happens after Pin Hugging Face revisions in test_torchao_huggingface_checkpoints.sh #24524 and link its log. If there isn't one, this PR looks superseded. If a pinned snapshot is missing on one runner pool, the documented fix is to seed it with the ci-refresh-hf-cache label from a pytorch/executorch branch, not to bypass the cache.

Backward Compatibility

• test_torchao_huggingface_checkpoints.sh:50-55 does nothing in CI and changes the default cache for local runs (major). trunk.yml:622 is the script's only caller. By the time the script runs, HF_HOME is already ${RUNNER_TEMP}/hf_cache, so the if at :51 never fires in CI. The PR calls the two blocks "matching", but one always overrides HF_HOME and the other only sometimes does. The block only changes anything for direct callers, who usually have neither HF_HOME nor RUNNER_TEMP set:

  • HF_HOME becomes /tmp/hf_cache instead of ~/.cache/huggingface (or $XDG_CACHE_HOME/huggingface). Checkpoints that are already downloaded get fetched again into /tmp, which is often tmpfs.
  • The saved login token at $HF_HOME/token is no longer found.
  • Every user on a host shares /tmp/hf_cache. If another user created it first, mkdir -p succeeds and the download then fails with permission denied.
  • A caller-set HF_HOME that can't be created is silently replaced (:54) instead of failing.

  Drop the script hunk. cuda.yml and vulkan.yml keep this redirect in the workflow. Fix this →

Recommendation

Request Changes

As written, the trunk.yml hunk most likely has no effect on hf download, and the script hunk only changes behaviour for local callers, for the worse. Before putting more work into the redirect, please confirm which failure remains after #24524. If it's a missing cache entry, seeding it with the label is the documented fix.

[bdemirb]

The issues had been fixed in an earlier PR.