Skip to content

Improve documentation for framework iterators and readers #6435

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rostan-t wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Improve documentation for framework iterators and readers #6435

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion dali/python/nvidia/dali/ops/_docs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,18 @@ def _get_rng_doc():
)


def _get_name_doc():
"""Return documentation for the operator name used in pipeline mode."""
return _numpydoc_formatter(
"name",
"str, optional, default = None",
"Name of the operator instance. Names must be unique within a pipeline.\n\n"
"Set an explicit name when another API needs to refer to the operator, for\n"
"example through an iterator's ``reader_name`` argument. If omitted, DALI\n"
"generates a name automatically.",
)


def _get_kwargs(schema, api="ops", args=None):
"""
Get the numpydoc-formatted docstring section for keywords arguments.
Expand All @@ -155,7 +167,7 @@ def _get_kwargs(schema, api="ops", args=None):
schema : OpSchema
Schema of the operator to be documented
"""
ret = ""
ret = _get_name_doc() + "\n" if api != "dynamic" else ""
for arg in schema.GetArgumentNames():
if args is not None and arg not in args:
continue
Expand Down
35 changes: 16 additions & 19 deletions dali/python/nvidia/dali/plugin/base_iterator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,17 +69,18 @@ class _DaliBaseIterator(object):
for data or label correspondingly.
output_names should be distinct.
size : int, default = -1
Number of samples in the shard for the wrapped pipeline (if there is more than one
it is a sum). Providing -1 means that the iterator will work until StopIteration
is raised from the inside of iter_setup(). The options `last_batch_policy`,
`last_batch_padded` and `auto_reset` don't work in such case. It works with only
one pipeline inside the iterator.
Mutually exclusive with `reader_name` argument
Number of samples in the shard. For multiple pipelines, this is the sum of
their shard sizes.
Mutually exclusive with `reader_name`. When left at -1 without
`reader_name`, a single-pipeline iterator reads until the pipeline raises
`StopIteration`, for example when an external source is exhausted;
`last_batch_policy` and `last_batch_padded` do not apply.
reader_name : str, default = None
Name of the reader which will be queried for the shard size, number of shards, and
all other properties necessary to count properly the number of relevant and padded
samples that iterator needs to deal with. Sets `last_batch_padded`
accordingly to the reader's configuration (`pad_last_batch` reader argument)
Name of the reader operator that determines the iterator length and
last-batch padding. It must match the reader's `name` argument in every
supplied pipeline.
When set, `size` and `last_batch_padded` are determined automatically and
must not be provided. It does not change `last_batch_policy`.
auto_reset : string or bool, optional, default = False
Whether the iterator resets itself for the next epoch or it requires reset() to be
called explicitly.
Expand All @@ -104,15 +105,11 @@ class _DaliBaseIterator(object):
What to do with the last batch when there are not enough samples in the epoch
to fully fill it. See :meth:`nvidia.dali.plugin.base_iterator.LastBatchPolicy`
last_batch_padded : bool, optional, default = False
Whether the last batch provided by DALI is padded with the last sample
or it just wraps up. In the conjunction with `last_batch_policy` it tells
if the iterator returning last batch with data only partially filled with
data from the current epoch is dropping padding samples or samples from
the next epoch. If set to False next
epoch will end sooner as data from it was consumed but dropped. If set to
True next epoch would be the same length as the first one. For this to happen,
the option `pad_last_batch` in the reader needs to be set to True as well.
It is overwritten when `reader_name` argument is provided
Whether the reader pads the last batch by repeating its last sample
(`True`) or continues into the next epoch (`False`).
Without `reader_name`, set this to the same value as the reader's
`pad_last_batch` argument. With `reader_name`, it is determined
automatically and must not be provided.
prepare_first_batch : bool, optional, default = True
Whether DALI should buffer the first batch right after the creation of the iterator,
so one batch is already prepared when the iterator is prompted for the data
Expand Down
72 changes: 32 additions & 40 deletions dali/python/nvidia/dali/plugin/jax/clu.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,18 +58,18 @@ class DALIGenericPeekableIterator(DALIGenericIterator):
of those names.
Each name should be distinct
size : int, default = -1
Number of samples in the shard for the wrapped pipeline (if there is more than
one it is a sum)
Providing -1 means that the iterator will work until StopIteration is raised
from the inside of iter_setup(). The options `last_batch_policy` and
`last_batch_padded` don't work in such case. It works with only one pipeline inside
the iterator.
Mutually exclusive with `reader_name` argument
Number of samples in the shard. For multiple pipelines, this is the sum of
their shard sizes.
Mutually exclusive with `reader_name`. When left at -1 without
`reader_name`, a single-pipeline iterator reads until the pipeline raises
`StopIteration`, for example when an external source is exhausted;
`last_batch_policy` and `last_batch_padded` do not apply.
reader_name : str, default = None
Name of the reader which will be queried for the shard size, number of shards and
all other properties necessary to count properly the number of relevant and padded
samples that iterator needs to deal with. It automatically sets `last_batch_padded`
accordingly to match the reader's configuration.
Name of the reader operator that determines the iterator length and
last-batch padding. It must match the reader's `name` argument in every
supplied pipeline.
When set, `size` and `last_batch_padded` are determined automatically and
must not be provided. It does not change `last_batch_policy`.
auto_reset : string or bool, optional, default = False
Whether the iterator resets itself for the next epoch or it requires reset() to be
called explicitly.
Expand All @@ -85,15 +85,11 @@ class DALIGenericPeekableIterator(DALIGenericIterator):
to fully fill it. See :meth:`nvidia.dali.plugin.base_iterator.LastBatchPolicy`.
JAX iterator does not support LastBatchPolicy.PARTIAL
last_batch_padded : bool, optional, default = False
Whether the last batch provided by DALI is padded with the last sample
or it just wraps up. In the conjunction with `last_batch_policy` it tells
if the iterator returning last batch with data only partially filled with
data from the current epoch is dropping padding samples or samples from
the next epoch. If set to ``False`` next
epoch will end sooner as data from it was consumed but dropped. If set to
True next epoch would be the same length as the first one. For this to happen,
the option `pad_last_batch` in the reader needs to be set to True as well.
It is overwritten when `reader_name` argument is provided
Whether the reader pads the last batch by repeating its last sample
(`True`) or continues into the next epoch (`False`).
Without `reader_name`, set this to the same value as the reader's
`pad_last_batch` argument. With `reader_name`, it is determined
automatically and must not be provided.
prepare_first_batch : bool, optional, default = True
Whether DALI should buffer the first batch right after the creation of the iterator,
so one batch is already prepared when the iterator is prompted for the data
Expand Down Expand Up @@ -323,18 +319,18 @@ def peekable_data_iterator(
of those names.
Each name should be distinct
size : int, default = -1
Number of samples in the shard for the wrapped pipeline (if there is more than
one it is a sum)
Providing -1 means that the iterator will work until StopIteration is raised
from the inside of iter_setup(). The options `last_batch_policy` and
`last_batch_padded` don't work in such case. It works with only one pipeline inside
the iterator.
Mutually exclusive with `reader_name` argument
Number of samples in the shard. For multiple pipelines, this is the sum of
their shard sizes.
Mutually exclusive with `reader_name`. When left at -1 without
`reader_name`, a single-pipeline iterator reads until the pipeline raises
`StopIteration`, for example when an external source is exhausted;
`last_batch_policy` and `last_batch_padded` do not apply.
reader_name : str, default = None
Name of the reader which will be queried for the shard size, number of shards and
all other properties necessary to count properly the number of relevant and padded
samples that iterator needs to deal with. It automatically sets `last_batch_padded`
accordingly to match the reader's configuration.
Name of the reader operator that determines the iterator length and
last-batch padding. It must match the reader's `name` argument in every
supplied pipeline.
When set, `size` and `last_batch_padded` are determined automatically and
must not be provided. It does not change `last_batch_policy`.
auto_reset : string or bool, optional, default = False
Whether the iterator resets itself for the next epoch or it requires reset() to be
called explicitly.
Expand All @@ -350,15 +346,11 @@ def peekable_data_iterator(
to fully fill it. See :meth:`nvidia.dali.plugin.base_iterator.LastBatchPolicy`.
JAX iterator does not support LastBatchPolicy.PARTIAL
last_batch_padded : bool, optional, default = False
Whether the last batch provided by DALI is padded with the last sample
or it just wraps up. In the conjunction with `last_batch_policy` it tells
if the iterator returning last batch with data only partially filled with
data from the current epoch is dropping padding samples or samples from
the next epoch. If set to ``False`` next
epoch will end sooner as data from it was consumed but dropped. If set to
True next epoch would be the same length as the first one. For this to happen,
the option `pad_last_batch` in the reader needs to be set to True as well.
It is overwritten when `reader_name` argument is provided
Whether the reader pads the last batch by repeating its last sample
(`True`) or continues into the next epoch (`False`).
Without `reader_name`, set this to the same value as the reader's
`pad_last_batch` argument. With `reader_name`, it is determined
automatically and must not be provided.
prepare_first_batch : bool, optional, default = True
Whether DALI should buffer the first batch right after the creation of the iterator,
so one batch is already prepared when the iterator is prompted for the data
Expand Down
72 changes: 32 additions & 40 deletions dali/python/nvidia/dali/plugin/jax/iterator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,18 +43,18 @@ class DALIGenericIterator(_DaliBaseIterator):
of those names.
Each name should be distinct
size : int, default = -1
Number of samples in the shard for the wrapped pipeline (if there is more than
one it is a sum)
Providing -1 means that the iterator will work until StopIteration is raised
from the inside of iter_setup(). The options `last_batch_policy` and
`last_batch_padded` don't work in such case. It works with only one pipeline inside
the iterator.
Mutually exclusive with `reader_name` argument
Number of samples in the shard. For multiple pipelines, this is the sum of
their shard sizes.
Mutually exclusive with `reader_name`. When left at -1 without
`reader_name`, a single-pipeline iterator reads until the pipeline raises
`StopIteration`, for example when an external source is exhausted;
`last_batch_policy` and `last_batch_padded` do not apply.
reader_name : str, default = None
Name of the reader which will be queried for the shard size, number of shards and
all other properties necessary to count properly the number of relevant and padded
samples that iterator needs to deal with. It automatically sets `last_batch_padded`
accordingly to match the reader's configuration.
Name of the reader operator that determines the iterator length and
last-batch padding. It must match the reader's `name` argument in every
supplied pipeline.
When set, `size` and `last_batch_padded` are determined automatically and
must not be provided. It does not change `last_batch_policy`.
auto_reset : string or bool, optional, default = False
Whether the iterator resets itself for the next epoch or it requires reset() to be
called explicitly.
Expand All @@ -70,15 +70,11 @@ class DALIGenericIterator(_DaliBaseIterator):
to fully fill it. See :meth:`nvidia.dali.plugin.base_iterator.LastBatchPolicy`.
JAX iterator does not support LastBatchPolicy.PARTIAL
last_batch_padded : bool, optional, default = False
Whether the last batch provided by DALI is padded with the last sample
or it just wraps up. In the conjunction with `last_batch_policy` it tells
if the iterator returning last batch with data only partially filled with
data from the current epoch is dropping padding samples or samples from
the next epoch. If set to ``False`` next
epoch will end sooner as data from it was consumed but dropped. If set to
True next epoch would be the same length as the first one. For this to happen,
the option `pad_last_batch` in the reader needs to be set to True as well.
It is overwritten when `reader_name` argument is provided
Whether the reader pads the last batch by repeating its last sample
(`True`) or continues into the next epoch (`False`).
Without `reader_name`, set this to the same value as the reader's
`pad_last_batch` argument. With `reader_name`, it is determined
automatically and must not be provided.
prepare_first_batch : bool, optional, default = True
Whether DALI should buffer the first batch right after the creation of the iterator,
so one batch is already prepared when the iterator is prompted for the data
Expand Down Expand Up @@ -444,18 +440,18 @@ def data_iterator(
of those names.
Each name should be distinct
size : int, default = -1
Number of samples in the shard for the wrapped pipeline (if there is more than
one it is a sum)
Providing -1 means that the iterator will work until StopIteration is raised
from the inside of iter_setup(). The options `last_batch_policy` and
`last_batch_padded` don't work in such case. It works with only one pipeline inside
the iterator.
Mutually exclusive with `reader_name` argument
Number of samples in the shard. For multiple pipelines, this is the sum of
their shard sizes.
Mutually exclusive with `reader_name`. When left at -1 without
`reader_name`, a single-pipeline iterator reads until the pipeline raises
`StopIteration`, for example when an external source is exhausted;
`last_batch_policy` and `last_batch_padded` do not apply.
reader_name : str, default = None
Name of the reader which will be queried for the shard size, number of shards and
all other properties necessary to count properly the number of relevant and padded
samples that iterator needs to deal with. It automatically sets `last_batch_padded`
accordingly to match the reader's configuration.
Name of the reader operator that determines the iterator length and
last-batch padding. It must match the reader's `name` argument in every
supplied pipeline.
When set, `size` and `last_batch_padded` are determined automatically and
must not be provided. It does not change `last_batch_policy`.
auto_reset : string or bool, optional, default = False
Whether the iterator resets itself for the next epoch or it requires reset() to be
called explicitly.
Expand All @@ -471,15 +467,11 @@ def data_iterator(
to fully fill it. See :meth:`nvidia.dali.plugin.base_iterator.LastBatchPolicy`.
JAX iterator does not support LastBatchPolicy.PARTIAL
last_batch_padded : bool, optional, default = False
Whether the last batch provided by DALI is padded with the last sample
or it just wraps up. In the conjunction with `last_batch_policy` it tells
if the iterator returning last batch with data only partially filled with
data from the current epoch is dropping padding samples or samples from
the next epoch. If set to ``False`` next
epoch will end sooner as data from it was consumed but dropped. If set to
True next epoch would be the same length as the first one. For this to happen,
the option `pad_last_batch` in the reader needs to be set to True as well.
It is overwritten when `reader_name` argument is provided
Whether the reader pads the last batch by repeating its last sample
(`True`) or continues into the next epoch (`False`).
Without `reader_name`, set this to the same value as the reader's
`pad_last_batch` argument. With `reader_name`, it is determined
automatically and must not be provided.
prepare_first_batch : bool, optional, default = True
Whether DALI should buffer the first batch right after the creation of the iterator,
so one batch is already prepared when the iterator is prompted for the data
Expand Down
Loading