Skip to content

Fix embedding sizes for unused DataFrame categories - #1177

Closed
ska2704 wants to merge 1 commit into
skorch-dev:masterfrom
ska2704:fix/categorical-signature-vocabulary
Closed

ska2704 wants to merge 1 commit into
skorch-dev:masterfrom
ska2704:fix/categorical-signature-vocabulary

Conversation

@ska2704

@ska2704 ska2704 commented Oct 18, 2026 •

Copy link
Copy Markdown

Fixes #1176.

A filtered pandas categorical column can retain codes such as [0, 2]. describe_signature counts two observed codes and suggests an embedding with two entries, which then raises IndexError for code 2.

Use the declared category count so the suggested embedding covers the full vocabulary. The regression tests cover unused middle and final categories, empty subsets, and ordered categories, including embedding calls on both the subset and the full vocabulary. The docstring and changelog describe the corrected behavior.

This is separate from #1172, which preserves inferred integer codes between batches and leaves this count unchanged.

Validation on Python 3.12.11, pandas 3.0.6 and PyTorch 2.14.1 (macOS arm64):

  • python -m pytest skorch/tests/test_helper.py --no-cov -q: 201 passed. The six new cases all fail on unchanged master; the unchanged helper suite passes all 195 tests.
  • python -m pytest skorch/tests/test_helper.py skorch/tests/test_utils.py --no-cov -q: 306 passed, 21 skipped.
  • sphinx-build -b html docs ../skorch-docs: succeeded with the same 15 warnings as unchanged master.
  • flake8 --max-line-length=88 skorch/helper.py skorch/tests/test_helper.py: two existing test-file diagnostics, both reproduced on unchanged master.
  • pylint skorch: reports repository-wide diagnostics. Comparing the touched files against unchanged master leaves one added no-self-use style warning on the new test method, which follows the surrounding class convention.
  • git diff --check: passed.

The full local suite was interrupted during the GPT-2 download after 338 passed and 29 skipped. This is an incomplete run, not a full-suite pass. The Linux CI workflow is awaiting maintainer approval and has not started any jobs.

@BenjaminBossan

Copy link
Copy Markdown
Collaborator

Thanks for the PR. As maintainers of skorch, we decided to update the status of the skorch project. As a consequence, we will close this PR. Your effort is still appreciated.

@ska2704
ska2704 deleted the fix/categorical-signature-vocabulary branch November 5, 2026 01:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DataFrameTransformer.describe_signature undercounts unused categories

2 participants