Skip to content

[Fix][admin] Fix resource same-name overwrite not refreshing local rs cache - #4565

Merged
aiwenmo merged 3 commits into
DataLinkDC:devfrom
fancy27:Fix-rs-local-cache-overwrite
Oct 20, 2026
Merged

aiwenmo merged 3 commits into
DataLinkDC:devfrom
fancy27:Fix-rs-local-cache-overwrite

Conversation

@fancy27

@fancy27 fancy27 commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Invalidate tmp/rs local cache after resource same-name overwrite / delete so OSS/HDFS updates are picked up
  • Rewrite rs:// local files atomically (delete + temp download + move) and disable URL caches to avoid stale JarFile/ClassLoader content in Session modes
  • Fix parent directory size accounting on overwrite (use size delta instead of always adding)

Test plan

  • Upload a jar to Resource Center (OSS), run a Flink Jar task on kubernetes-session referencing rs:/...
  • Overwrite the same jar with a different version; confirm tmp/rs/... is removed/updated after upload
  • Re-run the task without restarting Dinky; confirm the new jar behavior is used
  • Delete the resource and confirm local tmp/rs cache for that path is cleared

@aiwenmo aiwenmo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@aiwenmo aiwenmo added the Fixed the bug has fixed label Oct 19, 2026
@aiwenmo aiwenmo moved this to Done in Dinky Roadmap Oct 19, 2026
@aiwenmo aiwenmo added this to the 1.3.0 milestone Oct 19, 2026
@aiwenmo

aiwenmo commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

Thanks for the patch, and for tracking down the stale tmp/rs cache. Updating the file size and description on overwrite, and adjusting parent directory size by the delta instead of always adding the new size, fixes a real accounting bug. Clearing the local cache on delete is the right direction as well.

A few cases are still open, so this does not fully fix Session jobs reusing an old jar:

  • setUseCaches(false) applies only to the rs:// download connection. RsURLConnection always calls readFile and does not use that cache. The stale content comes later, when PackagedProgram / URLClassLoader opens the same local file: URL. The JDK JarFileFactory caches by URL string, so deleting and rewriting the same tmp/rs/... path leaves the cached JarFile in place. A follow-up needs a new local path per version or content hash.
  • Cache invalidation and cache writes use different path rules. clearRsLocalCache uses the DB fullName, while toFile uses URL.getPath(). rs:/dir/app.jar matches, but getFile() builds rs:// + fullName, which parses dir as the host and writes tmp/rs/app.jar while invalidation deletes tmp/rs/dir/app.jar.
  • Every download uses the fixed sibling *.downloading. Two concurrent toFile calls for the same resource can interleave writes, and a later move can put the old bytes back after invalidation.
  • clearRsLocalCache runs inside the upload transaction, after the remote putFile. If the local delete fails, the transaction rolls back the DB size while OSS/HDFS already has the new object. Other Dinky nodes also keep their own tmp/rs copy.
  • The create-new-file branch still does x.getSize() + size, so a null parent size NPEs. There is no automated coverage for overwrite, delete, or concurrent download.

We will treat this as an incremental fix and merge it first. The Session classloader cache, a single path helper, unique temp files, and not rolling back a successful remote write should go into a follow-up PR.

… cache

Invalidate tmp/rs after overwrite/delete, rewrite local cache atomically on
rs:// load, and correct parent size delta on overwrite so session jobs do not
keep using stale jars.

Co-authored-by: Cursor <cursoragent@cursor.com>
@aiwenmo
aiwenmo force-pushed the Fix-rs-local-cache-overwrite branch from fe819bd to 7afabf7 Compare October 20, 2026 06:44
@aiwenmo
aiwenmo merged commit 6a230b9 into DataLinkDC:dev Oct 20, 2026
22 checks passed
@aiwenmo aiwenmo mentioned this pull request Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Fixed the bug has fixed

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants