Skip to content

feat: Support new 2026 retries #3379

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
richardwang1124 wants to merge 16 commits into
base: version-3
Choose a base branch
from
Open

feat: Support new 2026 retries #3379

Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
8d1db59
Initial retry changes
richardwang1124 May 11, 2026
0d8d988
Update and add retry tests
richardwang1124 May 11, 2026
e869d9d
Update more tests
richardwang1124 May 11, 2026
e834f9d
Add 2026 retries gate
richardwang1124 May 12, 2026
9e49095
Add changelog and bump min core
richardwang1124 May 12, 2026
901300e
Merge branch 'version-3' into feature/new-retries
richardwang1124 May 12, 2026
17ca319
Fix changelog
richardwang1124 May 12, 2026
3f9005a
Fix default and tighten retry header value check
richardwang1124 May 12, 2026
09bb847
Update retry mode docstring
richardwang1124 May 12, 2026
87fcd9e
Revert min core to test SDK build
richardwang1124 May 12, 2026
51f180f
PR feedback
richardwang1124 May 12, 2026
1f58e9b
Use 1-based attempts
richardwang1124 May 13, 2026
8ceaa1f
Merge branch 'version-3' into feature/new-retries
richardwang1124 May 14, 2026
b90663c
Make JSON parse errors retryable and relax S3 error pattern
richardwang1124 May 14, 2026
aeb7e0c
PR feedback
richardwang1124 May 15, 2026
88d17f6
Revert parse error changes and add DynamboDB services changelog
richardwang1124 May 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions gems/aws-sdk-core/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,13 @@
Unreleased Changes
------------------

* Feature - Add `AWS_NEW_RETRIES_2026` environment variable to opt-in to updated `standard` retry mode with reduced backoff intervals.
Copy link
Copy Markdown
Contributor

@jterapin jterapin May 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, have we considered having a similar changelog entries for DynamoDB since they have specific retryable behavior or is that not needed?

Now that I think about it - are service-specific behaviors documented anywhere?

Copy link
Copy Markdown
Contributor

@jterapin jterapin May 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also have a question on the users who ARE already on standard mode for their retry_mode - do they feel any changes from this update?

Copy link
Copy Markdown
Contributor Author

@richardwang1124 richardwang1124 May 15, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can I add a DynamoDB changelog entry even though only core was updated? We could extend the core changelog entry to mention that DynamoDB defaults are changed if new retry behavior is enabled. I don't believe any service specific behaviors are documented anywhere yet. Externally I think the blogpost should mention this new DynamoDB behavior, internally I could add more comments or documentation?

Customers who are already on standard and opt in to new retries will feel a difference. Due to the updated backoff timing, retries will be much faster. Throttling behavior will be the same, and due to the updated retry quota draining, customers will fail faster during sustained service errors, but this is intentional to help services recover faster.

Copy link
Copy Markdown
Contributor

@jterapin jterapin May 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DynamoDB

I personally feel that blogpost is not enough for documentation. Not everyone will read blogpost releases nor aws-sdk-core's CHANGELOG entries.

Rethinking this...
This might be a good use case where we should have service-specific plugins. With plugins, you can be specific about this behavior + documentation. Now that I think about this - we might need to do something about the autogenerated config. See:

:max_attempts (Integer) — default: 3 — An integer representing the maximum number attempts that will be made for a single request, including the initial attempt. For example, setting this value to 5 will result in a request being retried up to 4 times. Used in standard and adaptive retry modes.

Above is what I see when I run codegen. Let's talk offline.

Customers who are already on standard ... will feel a difference.

We should probably add a separate entry about this. The way I read the above entry is like: "Ok so if I don't use that env var, i'm still on old standard retries mechanism"

Copy link
Copy Markdown
Contributor Author

@richardwang1124 richardwang1124 May 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure I can try adding a DynamoDB plugin instead for retries.

We should probably add a separate entry about this. The way I read the above entry is like: "Ok so if I don't use that env var, i'm still on old standard retries mechanism"

Your original understanding is correct, customers who are already on standard and opt in to new retries will feel a difference. If they do not set the environment variable, there will not be any differences. New retry behavior is disabled by default and opt in only.


3.247.0 (2026-05-13)
------------------

* Feature - Add YJIT & ZJIT tracking to user agent.

* Issue - Fix error messaging in SSO OIDC.

3.246.0 (2026-04-23)
Expand Down
3 changes: 3 additions & 0 deletions gems/aws-sdk-core/lib/aws-sdk-core/json/handler.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,9 @@ def build_body(context)

def parse_response(response)
response.data = parse_body(response.context)
rescue Json::ParseError => e
# make JSON parsing errors on 200-range responses retryable
response.error = Seahorse::Client::NetworkingError.new(e)
Copy link
Copy Markdown
Contributor

@jterapin jterapin May 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a new addition from last review so I have some questions:

  • Does other parser handlers need the same treatment? Such as xml? Is there a reason why JSON is being specifically targeted here?
  • If this is the final attempt, does this NetworkingErr gets surfaced to the user? Will this be helpful to the user?

Copy link
Copy Markdown
Contributor Author

@richardwang1124 richardwang1124 May 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The SCP tests didn't cover the other parser handlers, but I believe they should also be updated. I'll make the changes.

With these changes, the error message raised on the final attempt will be the NetworkingErr, but it wraps the parsing error message so the root cause will still be surfaced.

Copy link
Copy Markdown
Contributor

@jterapin jterapin May 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With these changes, the error message raised on the final attempt will be the NetworkingErr, but it wraps the parsing error message so the root cause will still be surfaced.

Is this how other SDKs are handling this case? Labeling it as a service error. Just curious 🤔

end

def parse_body(context)
Expand Down
19 changes: 11 additions & 8 deletions gems/aws-sdk-core/lib/aws-sdk-core/plugins/retries/retry_quota.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,11 @@ module Retries
# Used in 'standard' and 'adaptive' retry modes.
class RetryQuota
INITIAL_RETRY_TOKENS = 500
RETRY_COST = 5
RETRY_COST = 14
LEGACY_RETRY_COST = 5 # TODO: Remove when new retries become default
NO_RETRY_INCREMENT = 1
TIMEOUT_RETRY_COST = 10
THROTTLING_RETRY_COST = 5
TIMEOUT_RETRY_COST = 10 # TODO: Remove when new retries become default

def initialize(opts = {})
@mutex = Mutex.new
Expand All @@ -19,15 +21,16 @@ def initialize(opts = {})
end

# check if there is sufficient capacity to retry
# and return it. If there is insufficient capacity
# and return it. If there is insufficient capacity
# return 0
# @return [Integer] The amount of capacity checked out
def checkout_capacity(error_inspector)
@mutex.synchronize do
capacity_amount = if error_inspector.networking?
TIMEOUT_RETRY_COST
# TODO: Remove gate and keep only the new_retries branch
capacity_amount = if RetryErrors.new_retries?
error_inspector.throttling_error? ? THROTTLING_RETRY_COST : RETRY_COST
else
RETRY_COST
error_inspector.networking? ? TIMEOUT_RETRY_COST : LEGACY_RETRY_COST
end

# unable to acquire capacity
Expand All @@ -39,8 +42,8 @@ def checkout_capacity(error_inspector)
end

# capacity_amount refers to the amount of capacity requested from
# the last retry. It can either be RETRY_COST, TIMEOUT_RETRY_COST,
# or unset.
# the last retry. It can either be RETRY_COST,
# THROTTLING_RETRY_COST/TIMEOUT_RETRY_COST, or unset.
def release(capacity_amount)
# Implementation note: The release() method is called for
# every API call. In the common case where the request is
Expand Down
Loading
Loading