[abhiramshaji-cyber]

A deploy that got no answer from AWS used to wait until the job was killed, and now each silent call is dropped after 5 minutes and retried.

What this code is for

sdkRequestHandler in packages/@aws-cdk/toolkit-lib/lib/api/aws-auth/awscli-compatible.ts builds the HTTP settings that every AWS SDK client in the CLI and in the toolkit library uses. Both packages/aws-cdk/lib/cli/cli.ts (with the proxy agent, if any) and Toolkit.sdkProvider in toolkit.ts (with sdkConfig.httpOptions.agent, if any) call it, and the result is passed to every client next to a ConfiguredRetryStrategy that retries timeouts and other network errors. The one exception after this PR is SDK.lambda() in sdk.ts, which overrides the new idle timeout for the Lambda client.

Before this PR

A GitHub Actions job runs cdk deploy, and the connection under one DescribeStacks call goes silent after the request was sent, as described in #2097.

1. The CloudFormation client sends DescribeStacks through NodeHttpHandler with connectionTimeout: 10000 and requestTimeout: 300000.
2. The connection was already open, so connectionTimeout never applies.
3. After 300000 ms the requestTimeout passes. throwOnRequestTimeout is not set, so Smithy only logs [WARN] a request has exceeded the configured 300000 ms requestTimeout.
4. The request is never failed, so ConfiguredRetryStrategy never sees an error and never retries.
5. The CLI waits on that same request with no further output. In the report on (cli): deploy waits forever when an AWS API call gets no response, because requestTimeout only logs a warning #2097 that was almost 6 hours, until the GitHub job timeout cancelled it. Against a local server that accepts and never replies, I saw the same thing on main: one connection, the warning at 300 s, and still waiting at 330 s.

After this PR

Same job, same silent DescribeStacks call.

1. Same as above, but the settings now also carry socketTimeout: 300000.
2. Once no data has moved in either direction for 300 seconds, Smithy destroys the request and rejects it with a TimeoutError.
3. ConfiguredRetryStrategy treats TimeoutError as a temporary error and sends the call again on a new connection.
4. If the new connection answers, the deploy carries on. If every attempt stays silent, the command fails with that TimeoutError instead of waiting. There are 7 attempts in total, so that is about 35 minutes plus the short retry delays. Against the same silent local server with a 60 s value, I saw the call retried on connections 2 to 7 and then failed at about 467 s, which is the same path with a shorter timer.
5. A slow call that keeps sending or receiving data, such as a large asset upload, never goes 300 seconds without data, so it is not cut off.
6. A hotswap of a BucketDeployment invokes its Lambda function synchronously, and Lambda sends nothing back until the function returns, which can take up to 15 minutes. The Lambda client therefore gets socketTimeout: 905000, just above that maximum, so a long s3 sync is never dropped and started again while it is still running.

The diagram

flowchart LR

classDef box fill:#eef1ff,stroke:#4c5bd4,color:#111827
classDef bad fill:#ffd6d6,stroke:#cc0000,color:#111827,stroke-width:3px
classDef good fill:#c8f0c8,stroke:#008800,color:#111827,stroke-width:3px

subgraph BEFORE["BEFORE: waits until the job is killed, 1 connection"]
direction TB
  B1["cdk deploy calls DescribeStacks"] --> B2["NodeHttpHandler sends the request"]
  B2 --> B3["connection goes silent"]
  B3 --> B4{"requestTimeout 300000 passes"}
  B4 -->|"throwOnRequestTimeout not set"| B5["logs WARN and keeps waiting forever"]
  B4 -->|"response arrives"| B6["deploy continues"]
end

subgraph AFTER["AFTER: retried every 300 s, then fails with TimeoutError"]
direction TB
  A1["cdk deploy calls DescribeStacks"] --> A2["NodeHttpHandler sends the request"]
  A2 --> A3["connection goes silent"]
  A3 --> A4{"THE CHANGE: socketTimeout 300000"}
  A4 -->|"no data for 300 s"| A5["TimeoutError, retried on a new connection"]
  A4 -->|"data still moving"| A6["slow upload keeps going"]
end

B1 ~~~ A1

class B1,B2,B3,B6 box
class A1,A2,A3,A6 box
class B4,B5 bad
class A4,A5 good

linkStyle 3 stroke:#cc0000,stroke-width:5px
linkStyle 8 stroke:#008800,stroke-width:5px

style BEFORE fill:#fff5f5,stroke:#cc0000,color:#111827
style AFTER fill:#f3fff3,stroke:#008800,color:#111827

Loading

The change

• sdkRequestHandler now returns socketTimeout: 300000 next to the existing connectionTimeout and requestTimeout. This is the fix. I reused the 300 seconds that requestTimeout already has, because the Smithy docs for throwOnRequestTimeout say requestTimeout "was for a long time incorrectly being set as a socket idle timeout". So the CLI already ran with a 5 minute idle timeout that threw for years, and this brings that behavior back on purpose.
• SDK.lambda() in sdk.ts builds its client with socketTimeout: 905000 and keeps every other setting. The hotswap of a BucketDeployment calls Invoke synchronously and the function may run for the full 15 minutes, so a shorter idle timeout would drop it and the retries would start overlapping s3 sync --delete runs.
• RequestHandlerSettings in base-credentials.ts gets an optional socketTimeout field, so the value type checks and a custom IBaseCredentialsProvider receives it in clientConfig.requestHandler too.
• requestTimeout and its warning are left as they are. I did not turn on throwOnRequestTimeout, because that timer runs from the start of the request until the response headers arrive, so it would fail a healthy upload that takes longer than 5 minutes. @JimNero009 pointed this out in the issue, and their suggestion of a socketTimeout is what this PR follows. Thank you for the very clear report and reproduction.
• Tests in awscli-compatible.test.ts: one checks the settings sdkRequestHandler returns, including the agent on both httpAgent and httpsAgent, and fails without this change. Two more run a real STSClient against a local TCP server with a 100 ms socketTimeout. To be honest about what these two prove: they pass on main too, because they check how Smithy handles socketTimeout rather than this change. I kept them as a guard in case a future Smithy version changes that behavior, but I am very happy to drop them if you would rather not have a TCP server in a unit suite: a server that never answers gets a second connection and the call rejects with TimeoutError; a server that sends its whole response, headers included, in small pieces every 50 ms still succeeds on one connection. That second test fails if the settings use requestTimeout with throwOnRequestTimeout instead, which is why I went with socketTimeout.
• Tests in sdk.test.ts: one checks that the Lambda client gets socketTimeout: 905000 with the other settings unchanged, and fails without this change; one checks that the CloudFormation client still gets the shared settings.

I would really value your view on a few choices, and I am happy to change any of them:

• 300 seconds restores the old behavior. Would you prefer something shorter, such as the 60 to 120 seconds the issue suggested? If so, I would be glad to check the other clients for calls that can stay silent that long, the way the Lambda one does.
• The issue also asks for these timeouts to be configurable. I left that out to keep this small, but I can add it here or in a follow up if you think it belongs.
• The standalone cdk-assets CLI builds its clients in cdk-assets-lib/lib/aws.ts without any of these timeouts. Would you like the same change there as a separate PR?

What changes for a user

• A deploy, bootstrap or any other command whose AWS call goes silent no longer hangs. Each silent attempt is dropped after 5 minutes and retried.
• If AWS never answers at all, the command now fails with a TimeoutError after the existing retries, about 35 minutes, instead of running until the CI job is killed.
• Calls that answer normally, and slow calls that keep moving data such as large asset uploads, behave exactly as before.
• cdk deploy --hotswap and cdk watch with a BucketDeployment that takes many minutes keep working, because the Lambda client waits up to just over 15 minutes.
• Proxy users are covered, because the proxy agent goes through the same sdkRequestHandler.
• The requestTimeout warning after 5 minutes still appears for very long uploads, as it does today.

Fixes #2097

Checklist

• This change contains a major version upgrade for a dependency and I confirm all breaking changes are addressed
  • Release notes for the new version:

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license