[naglepuff]

Switches all asset deletion in the CLI to the bulk asset deletion endpoint added in dandi/dandi-archive#2920 (POST /dandisets/{id}/versions/{version}/assets/bulk-delete/). Issuing many per-asset DELETE requests in quick succession can overload the server.

Draft: this depends on dandi/dandi-archive#2920. On servers without the endpoint, the CLI falls back to deleting assets one at a time with the single-asset DELETE, so it keeps working against older archives. CI here runs against the published dandiarchive/dandiarchive-api Docker image, which doesn't have the endpoint yet, so the integration tests exercise the fallback. They're written to pass against servers with or without the endpoint.

Changes

1. RemoteDandiset.delete_assets() / iter_delete_assets() (new):
   • Deletes assets from a draft version in batches of BULK_DELETE_BATCH_SIZE (100), one request at a time. Input is deduplicated.
   • Returns a BulkDeleteResult with deleted and absent lists.
   • The endpoint deletes the requested assets that belong to the version, ignores the rest, and returns the IDs it deleted. Requested assets missing from the response are recorded as absent. This covers concurrent deletion and retries of a request that already succeeded. A response body that isn't a list of IDs raises RuntimeError.
   • With skip_missing=False, it raises AssetsNotFoundError (with missing_asset_ids) once the batch is processed. The rest of the batch has already been deleted at that point, and the error lists those assets in deleted_asset_ids.
   • On a 404 from the endpoint, the client fetches the version. If the version doesn't exist, it raises NotFoundError. Otherwise the server predates the endpoint, and the batch is deleted one asset at a time with the single-asset DELETE, with the same skip_missing handling (except that with skip_missing=False it stops at the first missing asset). The client remembers this, so later batches and calls skip the bulk request.
2. RemoteAsset.delete() now goes through the bulk endpoint with a single ID and skip_missing=False. This also covers move and other callers.
3. dandi delete sends assets in batches instead of one request per asset through pyout's worker pool:
   • Assets that are already gone are reported as Skipped.
   • A failed batch marks only that batch's assets as Error and continues. That's accurate because the server rolls back the whole request.
   • On servers without the endpoint, each asset is deleted with its own request, one at a time, and its status is reported as soon as it's processed. A failure only marks that asset as Error, as on master, but without the concurrency.
   • The removed Deleter.process_assets_pyout() / process_assets_debug() are replaced by a single process_assets(). The --devel-debug output now has one dict per asset with its final status (no Deleting lines).
   • The jobs argument of delete() is now ignored (the dandi delete command doesn't expose it).
4. upload --sync deletes remote assets that no longer exist locally in batches, and logs separate counts for deleted and already-absent assets. On servers without the endpoint it behaves as on master (sequential single-asset deletes, stopping at the first failure), except that assets that are already gone are skipped instead of causing an error.
5. request() gained a quiet_statuses parameter, so the expected 404s from the bulk endpoint, the version check and the fallback are logged at DEBUG instead of ERROR.

Not included

• dandi move still renames and overwrites one asset at a time, so an overwrite is still one delete request per asset. It's now through the bulk endpoint, though. Batching those deletes up front would risk local and remote diverging after a partial failure, and the renames cost more than the deletes anyway.

Testing

• Mocked (responses) unit tests cover:
  • batching and deduplication
  • missing assets, with and without skip_missing
  • matching the returned IDs, and rejecting unexpected response bodies
  • other error statuses (400/403/405), which are raised without falling back
  • falling back to single-asset deletion, including the client remembering it, and a missing version
  • non-draft versions and assets from another Dandiset
  • logging levels
• Integration tests cover delete, upload --sync and RemoteAsset.delete(), including multiple batches and assets deleted concurrently. They pass against servers with or without the endpoint:
  • Where the expected requests differ, a bulk_delete_supported() test helper probes the server with a bulk deletion request for no assets. A server with the endpoint rejects it with a 400; one without it responds with a 404.
  • One dandi delete test emulates a server without the endpoint and checks that a failure only affects that asset. Another emulates a server with it, so that reporting successful bulk requests is covered on either server.
• Against a local archive running the final design of #2920 (bulk-delete-asset-endpoint-sync at 08ed76ef):
  • test_dandiapi.py, test_delete.py and test_upload.py: 191 passed, 1 xfailed, 0 failed, before the last commit (per-asset reporting in the fallback).
  • After that commit, test_delete.py and the deletion tests in test_dandiapi.py: 63 passed, 0 failed.
  • After making the integration tests pass with or without the endpoint (63f2947e), the delete- and sync-related tests in those three files: 76 passed, 0 failed. The server-without-endpoint path of these tests is exercised by this PR's CI.
• dandi-archive's cli-integration workflow passed against #2920's final design with this branch at a7148196: https://github.com/dandi/dandi-archive/actions/runs/41507477207 (attempt 3; 318 passed, 2 xfailed).

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 98.31776% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.44%. Comparing base (54e842c) to head (63f2947).

Files with missing lines	Patch %	Lines
dandi/tests/fixtures.py	70.00%	3 Missing ⚠️
dandi/tests/test_dandiapi.py	99.11%	2 Missing ⚠️
dandi/tests/test_delete.py	98.56%	2 Missing ⚠️
dandi/tests/test_upload.py	94.44%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1960      +/-   ##
==========================================
+ Coverage   91.18%   91.44%   +0.26%     
==========================================
  Files          95       95              
  Lines       14740    15209     +469     
==========================================
+ Hits        13440    13908     +468     
- Misses       1300     1301       +1     

Flag	Coverage Δ
unittests	91.44% <98.31%> (+0.26%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[yarikoptic]

quick view

1. provide compatibility with older dandi-archive's (until EMBER updates; if it already did -- let me know...)
2. keep in mind, as per our recent conversation, this is just one of the end-points which causes server-side slow down due to absent "debouncing", but there are others too! (I forgot already what it was but IIRC triggring manifests updates upon addition of assets etc). So I rather see addition of this particular API interface as "band-aid": ideally there should be review of server tasks allocations dependent on adjustment of the state of assets in the dandiset to address the core issue! Implementation of "bulk" operations would still be welcome but would not address it as users could still "delete in bulks of 1"!

[waxlamp]

1. provide compatibility with older dandi-archive's (until EMBER updates; if it already did -- let me know...)

We would work directly with @NEStock to make sure EMBER receives this same update. Since it's adding a new endpoint and major data migrations, etc., won't be required, we anticipate this should be relatively straightforward. That makes the problem of compatibility much easier.

2. keep in mind, as per our recent conversation, this is just one of the end-points which causes server-side slow down due to absent "debouncing", but there are others too! (I forgot already what it was but IIRC triggring manifests updates upon addition of assets etc). So I rather see addition of this particular API interface as "band-aid": ideally there should be review of server tasks allocations dependent on adjustment of the state of assets in the dandiset to address the core issue! Implementation of "bulk" operations would still be welcome but would not address it as users could still "delete in bulks of 1"!

Rather than "debouncing" what we want here is "throttling", which we implement in a web service via rate limiting. Since we're retaining the single-asset delete endpoint, a rate limit on that endpoint prevents a recurrence of the original issue. And putting a rate limit on the bulk deletion endpoint prevents problems of "deleting in bulks of 1".

This problem arose due to the scale of a specific dandiset, and the proper fix for that problem really is bulk-delete It is not just a band-aid: deletion of many assets due to a change in the local state of a dandiset can be orders of magnitude faster with bulk-delete, and in that sense this is indeed a change in "server task allocations".

[naglepuff]

@yarikoptic this is in a state where you are free to unleash the robot reviewers.