Repository navigation
ProfileCredentialsProvider: concurrent callers get the previous profile file's credentials during a reload #7466
Description
Describe the bug
This follows up #5001, which was closed as stale while the maintainers waited for a self-contained reproduction. There's one below. It doesn't need an AWS account, a network or a file on disk.
ProfileCredentialsProvider.resolveCredentials() publishes the new profile file before it builds the delegate from it:
ProfileFile cachedOrRefreshedProfileFile = refreshProfileFile();
if (shouldUpdateCredentialsProvider(cachedOrRefreshedProfileFile)) {
synchronized (credentialsProviderLock) {
if (shouldUpdateCredentialsProvider(cachedOrRefreshedProfileFile)) {
currentProfileFile = cachedOrRefreshedProfileFile; // (1)
handleProfileFileReload(cachedOrRefreshedProfileFile); // (2)
}
}
}
return credentialsProvider.resolveCredentials();While one caller is between (1) and (2), a second caller's outer, unsynchronized shouldUpdateCredentialsProvider check sees currentProfileFile already set to the new file. The second caller skips the lock and resolves from the old delegate, so it returns the previous file's credentials even though the supplier has just handed it the new file.
The window is only as long as building the new delegate takes, so with a plain supplier you hit it now and then. ProfileFileSupplier.reloadWhenModified makes it much more likely. Its CachedSupplier lets one caller reload the file while the rest wait on the refresh lock. They then take that lock one at a time, get the cached new file, and reach resolveCredentials just behind the reloading thread, which puts them inside the window. Had they arrived alongside it, they would have blocked on credentialsProviderLock and been fine.
#3785 fixed the NPE this ordering caused on the very first call by adding the credentialsProvider == null check. Reloads still publish the file first.
The ordering causes trouble without any concurrency too. If building the new delegate throws (say, the new file has no credentials for the profile), that call fails, but currentProfileFile has already been updated. Every later call skips the reload and quietly returns the previous file's credentials, with no error. On 2.56.0, switching the supplier from a file with credentials to one without gives 1st: OLD, 2nd threw: SdkClientException, 3rd: OLD.
Regression Issue
- Select this option if this issue appears to be a regression.
Expected Behavior
Once the profile file supplier returns a new file, every resolveCredentials() call that receives it returns credentials from that file (or a newer one). A reload whose delegate cannot be built is attempted again on the next call rather than skipped.
Current Behavior
Some concurrent callers get credentials from the previous file. With the plain supplier in the reproduction below, about 4–6% of 8-thread bursts and 10–16% of 2-thread bursts returned the previous credentials on my machine; the rate varies between runs. The first hit usually comes within the first few iterations.
Reproduction Steps
The only dependency is software.amazon.awssdk:auth. Run it with java -cp <auth and its dependencies> ProfileReloadRace.java [threads] [iterations]. It exits with 1 if the race was hit.
import java.io.ByteArrayInputStream;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.CyclicBarrier;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.atomic.AtomicReference;
import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
import software.amazon.awssdk.core.util.VersionInfo;
import software.amazon.awssdk.profiles.ProfileFile;
/**
* Each iteration primes a provider with OLD, switches the supplier to NEW, then lets N threads call
* resolveCredentials() at once. Every call starts after the switch, so every call should return NEW.
*/
public class ProfileReloadRace {
public static void main(String[] args) throws Exception {
int threads = args.length > 0 ? Integer.parseInt(args[0]) : 8;
int iterations = args.length > 1 ? Integer.parseInt(args[1]) : 100_000;
System.out.println("SDK " + VersionInfo.SDK_VERSION + ", Java " + Runtime.version());
ProfileFile oldFile = profileFile("OLD");
ProfileFile newFile = profileFile("NEW");
ExecutorService pool = Executors.newFixedThreadPool(threads);
long staleCalls = 0;
long racyIterations = 0;
long firstHit = -1;
for (int i = 0; i < iterations; i++) {
AtomicReference<ProfileFile> current = new AtomicReference<>(oldFile);
var provider = ProfileCredentialsProvider.builder()
.profileFile(current::get)
.profileName("default")
.build();
provider.resolveCredentials();
current.set(newFile);
CyclicBarrier barrier = new CyclicBarrier(threads);
List<Future<String>> results = new ArrayList<>(threads);
for (int t = 0; t < threads; t++) {
results.add(pool.submit(() -> {
barrier.await();
return provider.resolveCredentials().accessKeyId();
}));
}
long stale = 0;
for (Future<String> result : results) {
if ("OLD".equals(result.get())) {
stale++;
}
}
if (stale > 0) {
staleCalls += stale;
racyIterations++;
if (firstHit < 0) {
firstHit = i;
}
}
}
pool.shutdown();
System.out.printf(
"threads=%d iterations=%d racyIterations=%d staleCalls=%d firstHitAtIteration=%d%n",
threads, iterations, racyIterations, staleCalls, firstHit);
System.exit(racyIterations > 0 ? 1 : 0);
}
private static ProfileFile profileFile(String accessKeyId) {
String content = "[default]\n"
+ "aws_access_key_id = " + accessKeyId + "\n"
+ "aws_secret_access_key = secret\n"
+ "aws_session_token = token-" + accessKeyId + "\n";
return ProfileFile.builder()
.content(new ByteArrayInputStream(content.getBytes(StandardCharsets.UTF_8)))
.type(ProfileFile.Type.CREDENTIALS)
.build();
}
}Possible Solution
Publish the delegate before the file it was built from:
if (shouldUpdateCredentialsProvider(cachedOrRefreshedProfileFile)) {
- currentProfileFile = cachedOrRefreshedProfileFile;
handleProfileFileReload(cachedOrRefreshedProfileFile);
+ currentProfileFile = cachedOrRefreshedProfileFile;
}A caller that reads the new currentProfileFile then also sees the new credentialsProvider. The delegate is written before the volatile write that caller reads, and credentialsProvider is read again after the check. With this order, a reload whose delegate fails to build leaves currentProfileFile unchanged, so the next call attempts it again. Holding the file and its delegate in a single volatile field also works.
Additional Information/Context
This happens in real production workloads when short-lived session credentials are written to the credentials file on a schedule, some time before the previous ones expire. The SDK checks the file only when credentials are resolved, and at most once per second. If nothing calls the service between the rewrite and the expiry of the previous token, the first calls after the expiry do the reload. When several arrive together, one or more of them are still signed with the expired token and fail with ExpiredToken.
AWS Java SDK version used
2.56.0
JDK version used
21.0.7 (Eclipse Temurin)
Operating System and version
Ubuntu 22.04.5 LTS (kernel 5.15)
Activity
adev-code commentedon Oct 24, 2026
Hi @sebastian-lakomy-duco, thanks for reaching out. I ran the
ProfileReloadRaceprogram on 2.56.0 and on the latest version (2.57.3) and reproduced it: some concurrent calls return the previous file's credentials. I also reproduced the case where a failed reload makes later calls return the previous credentials without an error.Found a workaround to wrap the provider so that only one call resolves credentials at a time:
and pass it to your client with
.credentialsProvider(new SynchronizedCredentialsProvider(provider)). With this wrapper, your program returned no previous credentials in my runs on both versions. Note that it doesn't change the failed-reload case. Let me know if this may work for you.Looks like there is an open PR for this, #7474. I'll bring the issue and the PR to the team and post updates here. If you have the
ExpiredTokenerror from production (stack trace and the SDK version), sharing it would help us understand the impact.sebastian-lakomy-duco commentedon Oct 26, 2026
Hi @adev-code, I'm sharing fragments of the expired token stacktrace:
SDK version:
2.41.19I think the workaround provided by you could work for us, although we settled for now with different kind of workaround - calling
resolveCredentialson a schedule from within the app, so that it's guaranteed to be called at least once after the old credentials are replaced, but before they expire. Anyway, we'd still be glad to see a fix directly in SDK.adev-code commentedon Oct 26, 2026
Thank you for the info @sebastian-lakomy-duco , I will update here as soon as there are updates.
github-actions commentedon Oct 28, 2026
This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
Publish reloaded profile file after its credentials provider is built (…
Publish reloaded profile file after its credentials provider is built (…