Improve documentation for framework iterators and readers - #6435
Open
rostan-t wants to merge 1 commit into
Open
Conversation
Signed-off-by: Rostan Tabet <rtabet@nvidia.com>
Contributor
|
| Filename | Overview |
|---|---|
| dali/python/nvidia/dali/ops/_docs.py | Adds generated documentation for the supported operator-instance name parameter while excluding the dynamic API where pipeline naming does not apply. |
| dali/python/nvidia/dali/plugin/base_iterator.py | Corrects shared iterator documentation to match existing reader metadata, padding, policy, and unbounded-iteration behavior. |
| dali/python/nvidia/dali/plugin/jax/clu.py | Aligns JAX CLU iterator documentation with the shared iterator contract and existing JAX policy restrictions. |
| dali/python/nvidia/dali/plugin/jax/iterator.py | Aligns JAX iterator and decorator documentation with the implementation. |
| dali/python/nvidia/dali/plugin/paddle.py | Updates Paddle iterator documentation consistently without changing runtime behavior. |
| dali/python/nvidia/dali/plugin/pytorch/init.py | Updates PyTorch iterator documentation and removes the incorrect claim that reader lookup changes the ragged iterator's last-batch policy. |
| docs/advanced_topics_sharding.rst | Reorganizes iterator option guidance and adds a valid example showing matching reader and iterator names. |
Reviews (1): Last reviewed commit: "Improve documentation for framework iter..." | Re-trigger Greptile
Collaborator
Author
|
!build |
Collaborator
|
CI MESSAGE: [59986895]: BUILD STARTED |
Collaborator
|
CI MESSAGE: [59986895]: BUILD PASSED |
rostan-t
commented
Aug 4, 2026
Comment on lines
+55
to
+73
| ``size`` | ||
| Provides the size of the shard for an iterator or, if there is more than one | ||
| shard, the sum of all shard sizes for all wrapped pipelines. | ||
|
|
||
| ``last_batch_padded`` | ||
| Whether the reader pads the last batch by repeating its last sample | ||
| (``True``) or continues into the next epoch (``False``). It applies when | ||
| the shard size is not a multiple of the batch size. | ||
|
|
||
| ``last_batch_policy`` | ||
| Determines the handling of the last batch when the shard size is not divisible | ||
| by the batch size. It affects batches only partially filled with the data. See | ||
| :meth:`~nvidia.dali.plugin.base_iterator.LastBatchPolicy` enum for possible | ||
| values. | ||
|
|
||
| ``fill_last_batch`` | ||
| Deprecated in favour of ``last_batch_policy``. Determines whether the last | ||
| batch should be full, regardless of whether the shard size is divisible by the | ||
| batch size. |
Collaborator
Author
There was a problem hiding this comment.
Rewritten as a definition list for clarity in the rendered doc but for the most part it doesn't change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Category:
Other (e.g. Documentation, Tests, Configuration)
Description:
Rewrites the
size/reader_name/last_batch_paddeddocs shared by all framework iterators, and documents thenameoperator argument (already in the generated signature, never in the docstring).Fixes two statements that contradicted the code:
size=-1withoutreader_namedoes not disableauto_reset.base_iterator.pyresetson
StopIterationwhenself._size < 0 and self._auto_reset == "yes".reader_namedoes not setlast_batch_policytoPARTIALonDALIRaggedIterator.Additional information:
Affected modules and functionalities:
Documentation.
Key points relevant for the review:
Tests:
Checklist
Documentation
DALI team only
Requirements
REQ IDs: N/A
JIRA TASK: DALI-2967