Skip to content

Improve documentation for framework iterators and readers - #6435

Open
rostan-t wants to merge 1 commit into
NVIDIA:mainfrom
rostan-t:improve-fw-iterator-doc
Open

Improve documentation for framework iterators and readers#6435
rostan-t wants to merge 1 commit into
NVIDIA:mainfrom
rostan-t:improve-fw-iterator-doc

Conversation

@rostan-t

Copy link
Copy Markdown
Collaborator

Category:

Other (e.g. Documentation, Tests, Configuration)

Description:

Rewrites the size / reader_name / last_batch_padded docs shared by all framework iterators, and documents the name operator argument (already in the generated signature, never in the docstring).

Fixes two statements that contradicted the code:

  • size=-1 without reader_name does not disable auto_reset. base_iterator.py resets
    on StopIteration when self._size < 0 and self._auto_reset == "yes".
  • reader_name does not set last_batch_policy to PARTIAL on DALIRaggedIterator.

Additional information:

Affected modules and functionalities:

Documentation.

Key points relevant for the review:

Tests:

  • Existing tests apply
  • New tests added
    • Python tests
    • GTests
    • Benchmark
    • Other
  • N/A

Checklist

Documentation

  • Existing documentation applies
  • Documentation updated
    • Docstring
    • Doxygen
    • RST
    • Jupyter
    • Other
  • N/A

DALI team only

Requirements

  • Implements new requirements
  • Affects existing requirements
  • N/A

REQ IDs: N/A

JIRA TASK: DALI-2967

Signed-off-by: Rostan Tabet <rtabet@nvidia.com>
@greptile-apps

greptile-apps Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Improves framework iterator and sharding documentation by:

  • Clarifying size, reader_name, last_batch_padded, and last_batch_policy behavior.
  • Documenting operator instance names in generated ops and fn API documentation.
  • Adding a concrete named-reader iterator example.

Confidence Score: 5/5

The PR appears safe to merge because the documentation changes agree with the existing generated API and iterator behavior.

Operator naming is supported by generated non-dynamic signatures and instance processing, while the revised iterator descriptions match reader metadata extraction, unbounded iteration, and framework-specific policy behavior.

Important Files Changed

Filename Overview
dali/python/nvidia/dali/ops/_docs.py Adds generated documentation for the supported operator-instance name parameter while excluding the dynamic API where pipeline naming does not apply.
dali/python/nvidia/dali/plugin/base_iterator.py Corrects shared iterator documentation to match existing reader metadata, padding, policy, and unbounded-iteration behavior.
dali/python/nvidia/dali/plugin/jax/clu.py Aligns JAX CLU iterator documentation with the shared iterator contract and existing JAX policy restrictions.
dali/python/nvidia/dali/plugin/jax/iterator.py Aligns JAX iterator and decorator documentation with the implementation.
dali/python/nvidia/dali/plugin/paddle.py Updates Paddle iterator documentation consistently without changing runtime behavior.
dali/python/nvidia/dali/plugin/pytorch/init.py Updates PyTorch iterator documentation and removes the incorrect claim that reader lookup changes the ragged iterator's last-batch policy.
docs/advanced_topics_sharding.rst Reorganizes iterator option guidance and adds a valid example showing matching reader and iterator names.

Reviews (1): Last reviewed commit: "Improve documentation for framework iter..." | Re-trigger Greptile

@rostan-t

Copy link
Copy Markdown
Collaborator Author

!build

@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [59986895]: BUILD STARTED

@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [59986895]: BUILD PASSED

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants