Skip to content

Update to DSI search; fix to sync.index(); docs updates - #295

Merged
jpulidojr merged 29 commits into
mainfrom
minor_sync_changes
Oct 27, 2026
Merged

jpulidojr merged 29 commits into
mainfrom
minor_sync_changes

Conversation

@Vedant1

@Vedant1 Vedant1 commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator
  • Change abs path check for remote paths in sync.index() to just make sure no ".."
  • update duckdb summary's sql statement to exclude NaN/Infinite values since they throw error with std dev function
  • DSI search now first tries a params search and if specific error, falls back to old combo of find cell/table/col
    • params search is for webserver backends so users can do an additional filtered search from online -- currently only do it on init which isn't enough
    • input 'query' is a dict with minimum requirement of a specific dataset/collection to retrieve from the repo
      • ex: {'dataset_id" : 1} ---> to get metadata for all entries within that dataset
      • ex: {'collection_id" : 1} ---> to get metadata for all entries within that collection_id
      • ex: {'dataset_id" : 1, "entry_id" : 1} ---> to get all data data for that entry in that dataset
    • first key name is specific to that backend -- might be dataset_id/collection_id/etc
    • second key name is also specific to that backend
    • returns a pandas DataFrame of all entries for a specific dataset OR all data pointers for a specific entry
    • backend search() includes extra arg for DSI to download the data, but not included in user api yet
  • updates to docs to include recent backend additions
  • updated core to dynamically check if webserver backend is valid

@Vedant1

Vedant1 commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator Author

add minor code to support webserver backends that are doing 2 tiered metadata retrieval

@Vedant1

Vedant1 commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator Author

update dsi docs with new backends

@Vedant1 Vedant1 changed the title change abs path check for remote paths in index() Update to DSI search; fix to sync.index(); docs updates Oct 23, 2026
@jpulidojr jpulidojr linked an issue Oct 26, 2026 that may be closed by this pull request
@Vedant1

Vedant1 commented Oct 27, 2026

Copy link
Copy Markdown
Collaborator Author

need to add minor changes to conduit and pfcp copy commands

@jpulidojr
jpulidojr self-requested a review October 27, 2026 04:06

@jpulidojr jpulidojr left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added misc fixes, tested on CI and Sync changes work.

@jpulidojr
jpulidojr merged commit 6f744d4 into main Oct 27, 2026
17 checks passed
@jpulidojr
jpulidojr deleted the minor_sync_changes branch October 27, 2026 04:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI unit testing for backends, readers, and tutorial

2 participants