fix(datasource): perform real sync deletions scoped per data source - #2690
Merged
Conversation
- deleted connector items remove the matching KB knowledge, scoped per data source via metadata datasource_id + external_id - SyncDeletions=false neither deletes nor counts - stream handler classifies deleted/failed items like the batch loop
- applyFetchedItem now performs the KB deletion instead of only counting - lookup is scoped to datasource_id + external_id so identical external IDs from two data sources cannot collide or overwrite each other - ingestItem update-path lookup uses the same scoped query
- SyncResult.DeletionFailed tracks deletion failures (subset of Failed) - streaming partial message warns that failed deletions only retry on a full sync, since the connector cursor is checkpointed past the deleted item
- raw lookup/delete errors go to server logs only, with full context - UI samples carry a humanized "see server logs" fallback - add datasource.syncError.* i18n keys Note: ko-KR / ru-RU syncError translations were generated without a native-speaker review.
mdrkrg
force-pushed
the
feat/ds-sync-deletion
branch
from
August 13, 2026 07:33
b60d6ae to
7e43b29
Compare
Sync-internal deletions now call HardDeleteKnowledge after the soft-delete cascade so tombstones cannot block re-sync; subtree sweeps and URL metadata attach failures are handled with datasource scoping and explicit errors.
Install via scripts/install-git-hooks.sh (sets core.hooksPath). Pre-commit runs whitespace/gofmt/golangci-lint on staged files; pre-push mirrors app/frontend/cli workflows on the diff since origin/main.
Pre-push now gofmts the full PR diff and runs go vet on all app packages (like app.yml), not only Go files in the latest commit. Previous pushes skipped checks via --no-verify and because hook-only commits had no .go files.
Use three-dot diff against the open PR baseRefOid (same as CI) instead of a two-dot merge-base range; run full go vet on every push with Go PR changes.
qbsoft
pushed a commit
to qbsoft/TeKnowra
that referenced
this pull request
Aug 20, 2026
…encent#2690) * test(datasource): cover scoped sync deletions - deleted connector items remove the matching KB knowledge, scoped per data source via metadata datasource_id + external_id - SyncDeletions=false neither deletes nor counts - stream handler classifies deleted/failed items like the batch loop * fix(datasource): perform real sync deletions scoped per data source - applyFetchedItem now performs the KB deletion instead of only counting - lookup is scoped to datasource_id + external_id so identical external IDs from two data sources cannot collide or overwrite each other - ingestItem update-path lookup uses the same scoped query * fix(datasource): note failed deletions retry only on full sync - SyncResult.DeletionFailed tracks deletion failures (subset of Failed) - streaming partial message warns that failed deletions only retry on a full sync, since the connector cursor is checkpointed past the deleted item * fix(datasource): humanize deletion errors, keep raw detail in logs - raw lookup/delete errors go to server logs only, with full context - UI samples carry a humanized "see server logs" fallback - add datasource.syncError.* i18n keys Note: ko-KR / ru-RU syncError translations were generated without a native-speaker review. * fix(datasource): hard-delete synced rows and tighten deletion scoping Sync-internal deletions now call HardDeleteKnowledge after the soft-delete cascade so tombstones cannot block re-sync; subtree sweeps and URL metadata attach failures are handled with datasource scoping and explicit errors. * chore: add git hooks aligned with CI and PR checks Install via scripts/install-git-hooks.sh (sets core.hooksPath). Pre-commit runs whitespace/gofmt/golangci-lint on staged files; pre-push mirrors app/frontend/cli workflows on the diff since origin/main. * fix: gofmt test file; tighten pre-push to match CI app workflow Pre-push now gofmts the full PR diff and runs go vet on all app packages (like app.yml), not only Go files in the latest commit. Previous pushes skipped checks via --no-verify and because hook-only commits had no .go files. * fix(hooks): align pre-push diff with GitHub PR base via gh Use three-dot diff against the open PR baseRefOid (same as CI) instead of a two-dot merge-base range; run full go vet on every push with Go PR changes. * style: gofumpt datasource_service_test.go --------- Co-authored-by: wizardchen <wizardchen@tencent.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
同步删除
sync_deletions（目前默认开启）在文档里写“源端删除文档时同步删除本地知识”，但此前实现只做了计数，不会删除本 PR 让删除生效，同时确保删除/更新定位不会跨数据源误删
假如认为同步删除有风险，可以默认关闭同步删除（UI 侧也改掉默认值）
预期行为
sync_deletions开启时，同步从知识库移除对应条目。关闭时不删除或计数Type of Change
Related Issue
部分解决 #2623
注意：用户删除 knowledge 时数据源跳过同步的 PR #2684 使用软删除作为 tombstone 跳过标记。如果先合入，这个分支的同步删除需要补充。
可能关联与合并冲突：#1643, #2632
Testing
新增测试
Checklist
git diff --check origin/main...HEADpassesgolangci-lint run --new-from-rev=origin/main ./...)docs/, Swagger annotations, etc.)Screenshots / Recordings