[martincostello]

Description

Split out of #4566 as requested; this is the part of the chunked-upload work that is not covered by #4533 (which sizes the chunk buffers to the payload and is left untouched here).

Bug fix. The read-and-copy fill loops in ChunkedUploadWrapperStream (used when the wrapped stream encrypts) always asked the wrapped stream for a full BufferSize read, regardless of how much room was left in the chunk buffer. With a BufferSize that does not divide DefaultChunkSize (81920), or exceeds it, the Buffer.BlockCopy into the chunk buffer throws ArgumentException mid-upload. Each read is now clamped to the space left in the chunk, and the interim read buffer is allocated once per stream (at most a chunk in size) instead of once per fill.

Per-chunk work trimmed at the same time:

• The short final chunk is no longer copied into an exact-size array. SigV4 hashes only the bytes read through a new internal string-to-sign overload that takes an offset and count; the public BuildChunkedStringToSign keeps its contract and delegates to it. SigV4a signs a MemoryStream slice.
• The chunk header is encoded straight into the output buffer instead of going through a StringBuilder, a byte[] header and a byte[] trailer per chunk.
• The SigV4 signing key is captured once per stream instead of being cloned by GetSigningKey() for every chunk and again for the trailer.
• The empty-body hash in the string to sign uses the existing AWS4Signer.EmptyBodySha256 constant instead of hashing and hex-encoding an empty string per chunk.

Motivation and Context

Measured with a mocked-HTTP harness (no network) that drives a full PutObject through the pipeline and consumes the request body: branch vs development (9ab8dcf), three interleaved baseline/branch rounds of 20 × 2000 operations after warm-up, process pinned to the P-cores of an i7-13700H, .NET 10 runtime against the net8.0 build of the SDK. Timings are the best round per side; allocations are exact (GC.GetAllocatedBytesForCurrentThread) and identical across rounds. Ratios are branch ÷ baseline.

Operation	Baseline min / median / alloc	Branch min / median / alloc	Time ratio (min / median)	Alloc ratio
S3 PutObject, 40 B body (1 chunk)	53.0 µs / 56.1 µs / 214.56 KB	51.8 µs / 53.4 µs / 212.17 KB	0.98 / 0.95	0.99
S3 PutObject, 300 KB body (4 chunks)	1030.6 µs / 1045.8 µs / 286.36 KB	1017.4 µs / 1026.3 µs / 220.37 KB	0.99 / 0.98	0.77

The 300 KB upload saves 66 KB per request (the exact-size copy of the 54 KB final chunk, the per-chunk header and trailer arrays, and the signing-key clones); its time is dominated by hashing the body, so the time ratio moves little. Both rows still include the ~160 KB of full-size chunk buffers that #4533 addresses, which is why the small-upload allocation ratio is close to 1. The overflow fix does not show in the numbers; it is covered by the new unit test, which fails on development with ArgumentException: Offset and length were out of bounds for the array....

Benchmark code

The harness is not checked in. It is a console project that references the SDK projects directly, replaces the HTTP client with a canned-response handler that consumes the request body (so the aws-chunked signing stream really runs), and measures a full client call through the pipeline. Build it once from the default branch and copy bin/Release/net10.0 to CoreBench-base, then build it from the branch and run ab.sh "put putlarge" 3.

CoreBench.csproj

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>disable</Nullable>
    <LangVersion>latest</LangVersion>
    <ServerGarbageCollection>false</ServerGarbageCollection>
    <TieredPGO>true</TieredPGO>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="BenchmarkDotNet" Version="0.15.2" />
  </ItemGroup>
  <ItemGroup>
    <ProjectReference Include="D:\Coding\aws\aws-sdk-net\sdk\src\Core\AWSSDK.Core.NetStandard.csproj" />
    <ProjectReference Include="D:\Coding\aws\aws-sdk-net\sdk\src\Services\DynamoDBv2\AWSSDK.DynamoDBv2.NetStandard.csproj" />
    <ProjectReference Include="D:\Coding\aws\aws-sdk-net\sdk\src\Services\S3\AWSSDK.S3.NetStandard.csproj" />
    <ProjectReference Include="D:\Coding\aws\aws-sdk-net\sdk\src\Services\SQS\AWSSDK.SQS.NetStandard.csproj" />
  </ItemGroup>
</Project>

Benchmarks.cs

using Amazon;
using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.Model;
using Amazon.Runtime;
using Amazon.S3;
using Amazon.S3.Model;
using Amazon.SQS;
using Amazon.SQS.Model;
using BenchmarkDotNet.Attributes;

namespace CoreBench;

[MemoryDiagnoser]
public class PipelineBenchmarks
{
    private AmazonDynamoDBClient _ddb;
    private AmazonS3Client _s3;
    private AmazonSQSClient _sqs;
    private GetItemRequest _getItem;
    private PutObjectRequest _putObject;
    private PutObjectRequest _putLargeObject;
    private MemoryStream _largeBody;
    private ListBucketsRequest _listBuckets;
    private SendMessageRequest _sendMessage;

    [GlobalSetup]
    public void Setup()
    {
        var creds = new BasicAWSCredentials("AKIAIOSFODNN7EXAMPLE", "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY");
        var factory = new MockHttpClientFactory();
        _ddb = new AmazonDynamoDBClient(creds, new AmazonDynamoDBConfig { RegionEndpoint = RegionEndpoint.USEast1, HttpClientFactory = factory });
        _s3 = new AmazonS3Client(creds, new AmazonS3Config { RegionEndpoint = RegionEndpoint.USEast1, HttpClientFactory = factory });
        _sqs = new AmazonSQSClient(creds, new AmazonSQSConfig { RegionEndpoint = RegionEndpoint.USEast1, HttpClientFactory = factory });

        _getItem = new GetItemRequest
        {
            TableName = "BenchmarkTable",
            Key = new Dictionary<string, AttributeValue>
            {
                ["pk"] = new AttributeValue { S = "id-1" },
                ["sk"] = new AttributeValue { S = "sort-1" },
            },
            ConsistentRead = true,
        };
        _putObject = new PutObjectRequest
        {
            BucketName = "my-bucket",
            Key = "some/key.txt",
            ContentBody = "hello world, this is a small object body",
            ContentType = "text/plain",
        };
        // 300 KB: three full 80 KB chunks plus a short final chunk
        var largeData = new byte[300 * 1024];
        new Random(42).NextBytes(largeData);
        _largeBody = new MemoryStream(largeData);
        _putLargeObject = new PutObjectRequest
        {
            BucketName = "my-bucket",
            Key = "some/large-key.bin",
            InputStream = _largeBody,
            AutoCloseStream = false,
            AutoResetStreamPosition = true,
        };
        _listBuckets = new ListBucketsRequest();
        _sendMessage = new SendMessageRequest
        {
            QueueUrl = "https://sqs.us-east-1.amazonaws.com/123456789012/my-queue",
            MessageBody = MockHttpMessageHandler.SqsBody,
        };

        // warm once to fail fast on bad mock wiring
        DynamoGetItem().GetAwaiter().GetResult();
        S3PutObject().GetAwaiter().GetResult();
        S3PutLargeObject().GetAwaiter().GetResult();
        S3ListBuckets().GetAwaiter().GetResult();
        SqsSendMessage().GetAwaiter().GetResult();
    }

    [Benchmark] public Task<GetItemResponse> DynamoGetItem() => _ddb.GetItemAsync(_getItem);
    [Benchmark] public Task<PutObjectResponse> S3PutObject() => _s3.PutObjectAsync(_putObject);
    [Benchmark] public Task<PutObjectResponse> S3PutLargeObject() { _largeBody.Position = 0; return _s3.PutObjectAsync(_putLargeObject); }
    [Benchmark] public Task<ListBucketsResponse> S3ListBuckets() => _s3.ListBucketsAsync(_listBuckets);
    [Benchmark] public Task<SendMessageResponse> SqsSendMessage() => _sqs.SendMessageAsync(_sendMessage);
}

MockHttp.cs

using System.Net;
using System.Security.Cryptography;
using System.Text;
using Amazon.Runtime;

namespace CoreBench;

public sealed class MockHttpClientFactory : HttpClientFactory
{
    public override HttpClient CreateHttpClient(IClientConfig clientConfig)
        => new HttpClient(new MockHttpMessageHandler(), disposeHandler: true);

    public override bool UseSDKHttpClientCaching(IClientConfig clientConfig) => true;
}

public sealed class MockHttpMessageHandler : HttpMessageHandler
{
    public const string SqsBody = "hello world";
    public static readonly byte[] DdbGetItemBytes = Encoding.UTF8.GetBytes(
        "{\"Item\":{\"pk\":{\"S\":\"id-1\"},\"sk\":{\"S\":\"sort-1\"},\"payload\":{\"S\":\"" + new string('a', 256) + "\"},\"count\":{\"N\":\"42\"},\"flag\":{\"BOOL\":true}}}");
    private static readonly byte[] SqsSendMessage = Encoding.UTF8.GetBytes(
        "{\"MD5OfMessageBody\":\"" + Convert.ToHexString(MD5.HashData(Encoding.UTF8.GetBytes(SqsBody))).ToLowerInvariant() + "\",\"MessageId\":\"5fea7756-0ea4-451a-a703-a558b933e274\"}");
    private static readonly byte[] S3ListBuckets = Encoding.UTF8.GetBytes(
        "<?xml version=\"1.0\" encoding=\"UTF-8\"?><ListAllMyBucketsResult xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\"><Owner><ID>abc</ID><DisplayName>me</DisplayName></Owner><Buckets><Bucket><Name>bucket-a</Name><CreationDate>2020-01-01T00:00:00.000Z</CreationDate></Bucket><Bucket><Name>bucket-b</Name><CreationDate>2020-01-02T00:00:00.000Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>");
    private static readonly byte[] Empty = Array.Empty<byte>();

    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        // Consume the request body like a real server would, so request-side stream wrappers
        // (e.g. the aws-chunked signing stream) actually do their work.
        if (request.Content != null)
        {
            await request.Content.CopyToAsync(Stream.Null).ConfigureAwait(false);
        }

        var host = request.RequestUri.Host;
        HttpResponseMessage response;
        if (host.StartsWith("dynamodb", StringComparison.Ordinal))
        {
            response = new HttpResponseMessage(HttpStatusCode.OK) { Content = new ByteArrayContent(DdbGetItemBytes) };
            response.Content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/x-amz-json-1.0");
        }
        else if (host.StartsWith("sqs", StringComparison.Ordinal))
        {
            response = new HttpResponseMessage(HttpStatusCode.OK) { Content = new ByteArrayContent(SqsSendMessage) };
            response.Content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/x-amz-json-1.0");
        }
        else if (request.Method == HttpMethod.Put)
        {
            response = new HttpResponseMessage(HttpStatusCode.OK) { Content = new ByteArrayContent(Empty) };
            response.Headers.TryAddWithoutValidation("ETag", "\"9a0364b9e99bb480dd25e1f0284c8555\"");
        }
        else
        {
            response = new HttpResponseMessage(HttpStatusCode.OK) { Content = new ByteArrayContent(S3ListBuckets) };
            response.Content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/xml");
        }
        response.Headers.TryAddWithoutValidation("x-amzn-RequestId", "0123456789abcdef");
        response.Headers.TryAddWithoutValidation("x-amz-id-2", "abcdefghijklmnopqrstuvwxyz");
        response.RequestMessage = request;
        return response;
    }
}

Program.cs

using BenchmarkDotNet.Running;
using CoreBench;

if (args.Length > 0 && args[0] == "bench")
{
    // Hybrid CPU (P + E cores): keep the process on the performance cores so the single benchmark
    // thread is not scheduled onto an efficiency core mid-run. Override with BENCH_AFFINITY (hex mask).
    var mask = Environment.GetEnvironmentVariable("BENCH_AFFINITY");
    System.Diagnostics.Process.GetCurrentProcess().ProcessorAffinity =
        (IntPtr)(mask != null ? Convert.ToInt64(mask, 16) : 0xFFF);

    // Self-contained steady-state harness: time-based warm-up (so tiered compilation settles), then
    // measures blocks and reports min / median / mean per operation plus allocated bytes per operation.
    var b = new PipelineBenchmarks();
    b.Setup();
    var which = args.Length > 1 ? args[1] : "put";
    var warmupSeconds = args.Length > 2 ? double.Parse(args[2]) : 6;
    var blocks = args.Length > 3 ? int.Parse(args[3]) : 20;
    var blockIters = args.Length > 4 ? int.Parse(args[4]) : 2000;
    Func<Task> op = which switch
    {
        "ddb" => () => b.DynamoGetItem(),
        "put" => () => b.S3PutObject(),
        "putlarge" => () => b.S3PutLargeObject(),
        "list" => () => b.S3ListBuckets(),
        "sqs" => () => b.SqsSendMessage(),
        _ => throw new ArgumentException(which),
    };
    var sw = System.Diagnostics.Stopwatch.StartNew();
    long warmIters = 0;
    while (sw.Elapsed.TotalSeconds < warmupSeconds || warmIters < 20000)
    {
        op().GetAwaiter().GetResult();
        warmIters++;
    }
    var results = new List<double>();
    long allocBefore = GC.GetAllocatedBytesForCurrentThread();
    for (int k = 0; k < blocks; k++)
    {
        var t0 = System.Diagnostics.Stopwatch.GetTimestamp();
        for (int i = 0; i < blockIters; i++) op().GetAwaiter().GetResult();
        var t1 = System.Diagnostics.Stopwatch.GetTimestamp();
        results.Add((t1 - t0) * 1_000_000.0 / System.Diagnostics.Stopwatch.Frequency / blockIters);
    }
    long allocAfter = GC.GetAllocatedBytesForCurrentThread();
    results.Sort();
    var median = results[results.Count / 2];
    var mean = results.Average();
    var allocPerOp = (allocAfter - allocBefore) / (double)(blocks * blockIters);
    Console.WriteLine($"{which,-5} min {results[0],7:F2} us  median {median,7:F2} us  mean {mean,7:F2} us  alloc {allocPerOp / 1024,7:F2} KB  (warmup {warmIters} iters)");
    return;
}

// BenchmarkDotNet (use --inProcess so the saved baseline binaries are the ones measured)
BenchmarkSwitcher.FromAssembly(typeof(PipelineBenchmarks).Assembly).Run(args);

ab.sh

#!/bin/bash
# Usage: ab.sh "<ops>" [rounds] [warmupSeconds]
# Runs the harness for the baseline build (CoreBench-base, built from the default branch)
# and the candidate build alternately, so machine noise affects both the same way.
BASE="./CoreBench-base/CoreBench.exe"
NEW="./CoreBench/bin/Release/net10.0/CoreBench.exe"
OPS="$1"; ROUNDS="${2:-2}"; WARM="${3:-6}"
for op in $OPS; do
  for r in $(seq 1 $ROUNDS); do
    for v in base new; do
      if [ $v = base ]; then exe="$BASE"; else exe="$NEW"; fi
      echo "[$v r$r] $("$exe" bench $op $WARM 20 2000 | tail -1)"
    done
  done
done

Testing

• AWSSDK.Core.NetStandard.csproj builds for all targets with no warnings.
• AWSSDK.UnitTests.Core on net472, filtered to ChunkedUploadWrapperStream and AWS4Signer: 41 passed. The existing chunked-upload tests pin the emitted content byte for byte, so the header rewrite and the final-chunk change are verified against the previous output.
• The net8.0 test target compiles but could not be executed locally (MSTest adapter/framework mismatch in this environment), so CI is the first run of that target.
• New tests in ChunkedUploadWrapperStreamTests: a multi-chunk payload (DefaultChunkSize + 12345 bytes) round-trips with the expected chunk sizes and with every chunk signature, including the terminating chunk, recomputed independently from exact-size copies via the public string-to-sign helper; and an AES-encrypting wrapped stream read with a BufferSize of twice the chunk size, which overflows before this change.

Dry-runs

• DotNet Dry-run ID: N/A (external contribution)
  • Pending
  • Completed successfully
  • Failed
• PowerShell Dry-run ID: N/A (external contribution)
  • Pending
  • Completed successfully
  • Failed

Breaking Changes Assessment

1. No public API changes. The emitted aws-chunked content is unchanged (pinned by the existing tests and the new signature test). BuildChunkedStringToSign(string, string, string, string, int, byte[]) keeps hashing the whole buffer when dataLength > 0, exactly as before; only the new internal overload hashes a range.
2. Reviewer assignment is at the maintainers' discretion.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist

• My code follows the code style of this project
• My change requires a change to the documentation
• I have updated the documentation accordingly
• I have read the README document
• I have added tests to cover my changes
• All new and existing tests passed

License

• I confirm that this pull request can be released under the Apache 2 license

---

This change was generated by AI tools (Claude Code), and reviewed by Martin Costello (@martincostello).

🤖 Generated with Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

[boblodgett]

Closing this PR in favor of this one we had been working on: #4533

[martincostello]

There were some bits in this PR that weren't included in that one - I'll look to submit a new one with just the gaps on top.

[martincostello]

I'll look to submit a new one with just the gaps on top.

#4575