-
Notifications
You must be signed in to change notification settings - Fork 215
[REVIEW] New Dataset API Clarifying Ownership #1846
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
HowardHuang1
wants to merge
65
commits into
NVIDIA:release/26.08
Choose a base branch
from
HowardHuang1:HH-Dataset-API
base: release/26.08
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 53 commits
Commits
Show all changes
65 commits
Select commit
Hold shift + click to select a range
f221341
Add Dataset API for CAGRA (host vs device / owning vs view / standard…
HowardHuang1 badb837
Merge remote-tracking branch 'upstream/release/26.08' into HH-Dataset…
HowardHuang1 41a6f22
Fix failing hnsw host index serialize_to_hnswlib test case
HowardHuang1 b406041
squash all Tarang commits to remove 26.10 ancestry. Should be left wi…
HowardHuang1 23e4114
Merge pull request #1 from tarang-jain/hh-dataset-api
HowardHuang1 0882b6c
merge API updates
aamijar 56cb7e1
c-serialize
divyegala e2d7b53
Merge remote-tracking branch 'upstream/release/26.08' into HH-Dataset…
HowardHuang1 b9dd705
rust: make CAGRA dataset ownership explicit
yan-zaretskiy 2354487
record type and layout in serialization
divyegala b28cc72
merge and reconcile conflicts
divyegala 07f91a1
rename make verb + noun
aamijar 5e334b2
skip dataset when reading graph only
divyegala dab5c6b
Add CAGRA build heuristics independent of HNSW parameters (#2345)
achirkin 195b04c
Merge remote-tracking branch 'upstream/release/26.08' into HH-Dataset…
HowardHuang1 11af0a6
Fix failing rocky clib tests on CI
HowardHuang1 a306dc9
remove cuvsCagraIndex_t from dataset.h
aamijar 7e68c56
finish audit
divyegala 5f7b90b
Fix regression introduced by upstream merge still using old build() f…
HowardHuang1 fdfab57
review bug
divyegala 21481e0
unify dataset make and view factories
aamijar 6d534e2
Revert "unify dataset make and view factories"
aamijar 2b753ac
unify dataset make and view factories 2
aamijar b4f7d3e
Fix failing Java test cases. Java test call sites were outdated and m…
HowardHuang1 2ffe67b
Fix merge conflicts. Factory names were using old factory names prior…
HowardHuang1 ed911bf
conflicts
divyegala c503887
Merge remote-tracking branch 'upstream/release/26.08' into HH-Dataset…
HowardHuang1 38cf6fc
Have downstream wrappers mirror dataset factory consolidation in C AP…
HowardHuang1 80905ea
Fix merge conflict
HowardHuang1 96ff183
Merge remote-tracking branch 'upstream/release/26.08' into HH-Dataset…
HowardHuang1 a6dd4eb
Fix merge conflict that brought back compression param in a ann_cagra…
HowardHuang1 089808a
merge
divyegala f52a0a7
merge
divyegala 501d066
undo java and python changes
divyegala 469f912
Fix merge conflict where upstream introduced a new cpp test case that…
HowardHuang1 33ce2bd
Update get_dlpack.cmake
divyegala 679553e
Merge pull request #2 from divyegala/c-serialize
HowardHuang1 8b20957
Fix styles. Fix copyright issue from merge
HowardHuang1 ed188de
target_mem_type argument in C API
aamijar 598998c
restore java behavior
divyegala 04649cb
Update get_dlpack.cmake
divyegala 4f5ebd5
Update get_dlpack.cmake
divyegala fe7a121
add datasets back
divyegala f046dd2
Merge remote-tracking branch 'origin/java-serialize' into java-serialize
divyegala 7f81391
update python api
tarang-jain c8df0d4
Add new extend API contract. Users no longer pass in 2 separate datas…
HowardHuang1 e916441
Merge remote-tracking branch 'upstream/release/26.08' into HH-Dataset…
HowardHuang1 5581ffd
Merge pull request #3 from divyegala/java-serialize
HowardHuang1 8140998
Fix merge conflicts. Upstream introduced new test cases that used old…
HowardHuang1 62a8da6
More test case call site fixes to resolve merge conflicts and port th…
HowardHuang1 06b02ad
remove explicit view type in C API
aamijar 378ac4e
bool is_owning
aamijar a08714f
Revert "update python api"
tarang-jain 55bc6e8
Merge branch 'HH-Dataset-API' of https://github.com/HowardHuang1/cuvs…
tarang-jain 02963e1
remove unused header in dataset.h
aamijar 1d23a34
fix python api
tarang-jain 09b3334
Increase binary limits to pass CI
HowardHuang1 9f90009
new updates after pr reviews
tarang-jain 640ab06
update the python tests
tarang-jain 84eeb19
address pr review
tarang-jain 3a3c116
Fix failing NEIGHBORS_MG_TEST which was calling MG with ace params. T…
HowardHuang1 15a50a7
new c getters
tarang-jain a7594d0
add c tests for dataset
tarang-jain 1763e0b
update pydocs
tarang-jain 09256be
Merge pull request #4 from tarang-jain/hh-dataset-api-2
HowardHuang1 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,98 @@ | ||
| /* | ||
| * SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| */ | ||
|
|
||
| #pragma once | ||
|
|
||
| #include <cuvs/core/c_api.h> | ||
|
|
||
| #include <dlpack/dlpack.h> | ||
| #include <stdbool.h> | ||
| #include <stdint.h> | ||
|
|
||
| #ifdef __cplusplus | ||
| extern "C" { | ||
| #endif | ||
|
|
||
| /** | ||
| * @brief Generic dataset layout kind for C API dataset handles. | ||
| */ | ||
| typedef enum { | ||
| CUVS_DATASET_LAYOUT_STANDARD = 0, | ||
| CUVS_DATASET_LAYOUT_PADDED = 1 | ||
| } cuvsDatasetLayout_t; | ||
|
|
||
| /** | ||
| * @brief Memory space holding a C API dataset handle's data. | ||
| */ | ||
| typedef enum { | ||
| CUVS_DATASET_MEM_TYPE_HOST = 0, | ||
| CUVS_DATASET_MEM_TYPE_DEVICE = 1 | ||
| } cuvsDatasetMemType_t; | ||
|
|
||
| /** | ||
| * @brief Dataset handle representing owning storage or a non-owning view. | ||
| * | ||
| * `addr` points to C++ dataset storage or view metadata managed by the C API. `mem_type` | ||
| * identifies the memory space, `layout` identifies the data layout, and `is_owning` indicates | ||
| * whether the handle owns its backing data. | ||
| */ | ||
| typedef struct { | ||
| uintptr_t addr; | ||
| void (*destroy_addr)(void*); | ||
| DLDataType dtype; | ||
| cuvsDatasetMemType_t mem_type; | ||
| cuvsDatasetLayout_t layout; | ||
| bool is_owning; | ||
| } cuvsDataset; | ||
| typedef cuvsDataset* cuvsDataset_t; | ||
|
|
||
| /** | ||
| * @brief Create an empty owning dataset handle. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We can improve the docs here slightly to also mention that the underlying data is not allocated, only the handle. |
||
| * | ||
| * The dataset storage, memory type, layout, and dtype are populated by the operation that fills | ||
| * this handle. | ||
| */ | ||
| CUVS_EXPORT cuvsError_t cuvsDatasetCreate(cuvsDataset_t* dataset); | ||
|
|
||
| /** | ||
| * @brief Create an owning padded dataset in the requested memory space. | ||
| * | ||
| * The source tensor may reside in host- or device-accessible memory. Its contents are copied into | ||
| * newly allocated padded storage in `target_mem_type`. | ||
| * | ||
| * @param[in] res cuVS resources | ||
| * @param[in] dataset source tensor | ||
| * @param[in] target_mem_type memory space in which to allocate the padded dataset | ||
| * @param[out] padded_dataset newly allocated owning padded dataset | ||
| */ | ||
| CUVS_EXPORT cuvsError_t cuvsDatasetMakePadded(cuvsResources_t res, | ||
| DLManagedTensor* dataset, | ||
| cuvsDatasetMemType_t target_mem_type, | ||
| cuvsDataset_t* padded_dataset); | ||
|
|
||
| /** | ||
| * @brief Create a non-owning padded dataset view from a host- or device-resident tensor. | ||
| * | ||
| * Memory residency is inferred from the tensor. | ||
| */ | ||
| CUVS_EXPORT cuvsError_t cuvsDatasetMakePaddedView(cuvsResources_t res, | ||
| DLManagedTensor* dataset, | ||
| cuvsDataset_t* padded_dataset); | ||
|
|
||
| /** | ||
| * @brief Create a non-owning standard dataset view from a host- or device-resident tensor. | ||
| * | ||
| * Memory residency is inferred from the tensor. | ||
| */ | ||
| CUVS_EXPORT cuvsError_t cuvsDatasetMakeStandardView(cuvsResources_t res, | ||
| DLManagedTensor* dataset, | ||
| cuvsDataset_t* standard_dataset); | ||
|
|
||
| /** @brief Destroy a dataset handle created by a `cuvsDatasetMake*` function. */ | ||
| CUVS_EXPORT cuvsError_t cuvsDatasetDestroy(cuvsDataset_t dataset); | ||
|
|
||
| #ifdef __cplusplus | ||
| } | ||
| #endif | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will this be expanded later with compressed layouts?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes. This enum is not final.