Skip to content

Run stubtest and pyrefly coverage check in CI (and tox) - #794

Merged
amoffat merged 4 commits into
amoffat:developfrom
jorenham:typing/stubtest-in-ci
Jul 23, 2026
Merged

amoffat merged 4 commits into
amoffat:developfrom
jorenham:typing/stubtest-in-ci

Conversation

@jorenham

@jorenham jorenham commented Jul 23, 2026 •

Copy link
Copy Markdown
Contributor

As promised in #791:

I plan on working on adding stubtest to the CI pipeline so we can ensure that the stubs stay in sync with the runtime API.

... and so it shall be done.

Oh and while we're at it, it might also be a good idea to ensure 100% type coverage (i.e. ensure that everything in the public API is annotated). For that we could use pyrefly coverage check (docs; but don't mind the "experimental" status, which doesn't really apply anymore, and I think I'll go ahead and open a PR to remove that right now and I opened a PR for that: facebook/pyrefly#4269):

$ uvx pyrefly coverage check src/sh
 INFO type coverage 100.00% (178 of 178 typable)

When I remove a random annotation in the stubs, here's what you'll see:

$ uv run stubtest sh
Success: no issues found in 1 module

$ uvx pyrefly coverage check src/sh
 WARN `sh.glob` is not fully typed [coverage-partial]
    --> src/sh/__init__.pyi:1393:1
     |
1393 | def glob(path, *args: Any, **kwargs: Any) -> list[str]: ...
     | -----------------------------------------------------------
     |
ERROR type coverage 99.44% (177 of 178 typable) is below the 100.00% threshold

And FWIW; we're also using it in numpy and scipy-stubs, and it has been very helpful, and was able to catch things that stubtest didn't catch.

So if you want I could also add that as well?

@jorenham
jorenham requested a review from amoffat as a code owner July 23, 2026 21:46
@jorenham

Copy link
Copy Markdown
Contributor Author

CI failure seems unrelated:

coveralls.exception.CoverallsException: Could not submit coverage: 500 Server Error: Internal Server Error for url: https://coveralls.io/api/v1/jobs

@amoffat

amoffat commented Jul 23, 2026

Copy link
Copy Markdown
Owner

So if you want I could also add that as well?

Yeah, let's do it 👍 Thank you! This is awesome. People have had gripes about sh typing in modern IDEs for a long time, and your work here really buttons it up.

@jorenham jorenham changed the title Run stubtest in CI (and tox) Run stubtest and pyrefly coverage check in CI (and tox) Jul 23, 2026
@amoffat
amoffat merged commit 823ac7f into amoffat:develop Jul 23, 2026
23 checks passed
@jorenham
jorenham deleted the typing/stubtest-in-ci branch July 23, 2026 22:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants