Skip to content

Accelerate CRC32 flexible checksum - #4560

Closed
jnunn-aws wants to merge 1 commit into
developmentfrom
jonatnun/fast-crc32-checksum
Closed

jnunn-aws wants to merge 1 commit into
developmentfrom
jonatnun/fast-crc32-checksum

Conversation

@jnunn-aws

@jnunn-aws jnunn-aws commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Description

Replace the byte-at-a-time Ionic.Zlib CRC32 loop with Crc32Ieee: PCLMULQDQ
folding on x86/x64 and ARM64 CRC32 instructions on .NET 8+, slicing-by-16
tables on netstandard2.0/netcoreapp3.1/net472. Covers Crc32Managed (flexible
checksums), CrcCalculatorStream (x-amz-crc32, event streams). Output is
bit-identical to the previous implementation.

PCLMULQDQ folding on x86/x64 and ARM64 CRC32 instructions on .NET 8+; slicing-by-16 tables on netstandard2.0/netcoreapp3.1/net472. The vectorized code is adapted from dotnet/runtime's System.IO.Hashing (MIT, under ThirdParty/RuntimeBackports).

Motivation and Context

The byte-at-a-time CRC32 dominated checksum cost on large uploads and x-amz-crc32 validation.

x86 ARM64
ops/CPU-sec, this change alone (71-case geomean) +2.3% +3.1%

The gain is concentrated in the large-payload cases.

Measurement: Numbers come from the serde ops/CPU-sec runner (sdk/test/Performance/SerdeBenchmarks, --cpu-time, stub transport), geomean of 71 cases, 3 interleaved runs. Instances: m7i.metal-24xl (x86) and m7g.metal (ARM64). Settings: net8.0, 20,000-iteration warmup, measured on development 8956121, which is the same code as this branch for these files. Each change is measured alone against that base. They will be re-measured at the harness's current default (100,000 warmup, net10.0, stub and https) before the PRs leave draft.

Testing

  • New Crc32IeeeTests (20): known vectors; slicing-by-16, PCLMULQDQ and ARM64 paths against a reference implementation over all lengths and offsets; incremental chunking vs one-shot; Crc32Managed, the checksum factory and CrcCalculatorStream against the reference.
  • Real-AWS check: S3's stored CRC32 matched an independently computed CRC32 for every upload that stores one.
  • Core unit tests pass on net8.0 and netcoreapp3.1.

Breaking Changes Assessment

None. Output is bit-identical.

Dry-runs

Combined dry-run of this series (jonatnun/perf-all-v3 @ 2a8cec6: #4557–#4563 together).

  • DotNet Dry-run ID: 438fa1b0-130d-4670-b74e-2df6c79bb7a1
    • Completed successfully
  • PowerShell Dry-run ID: c0049dbb-e11a-4a90-9fcd-087f2dc26e93
    • Completed successfully

Checklist

  • My code follows the code style of this project
  • I have added tests to cover my changes
  • All new and existing tests passed

License

  • I confirm that this pull request can be released under the Apache 2 license

Replace the byte-at-a-time Ionic.Zlib CRC32 loop with Crc32Ieee: PCLMULQDQ
folding on x86/x64 and ARM64 CRC32 instructions on .NET 8+, slicing-by-16
tables on netstandard2.0/netcoreapp3.1/net472. Covers Crc32Managed (flexible
checksums), CrcCalculatorStream (x-amz-crc32, event streams). Output is
bit-identical to the previous implementation.
@jnunn-aws

Copy link
Copy Markdown
Contributor Author

Closing: 4.1 covers this with System.IO.Hashing (#4492), plus slicing-by-16 for targets before .NET 10 (#4533).

@jnunn-aws jnunn-aws closed this Oct 23, 2026
@dscpinheiro
dscpinheiro deleted the jonatnun/fast-crc32-checksum branch October 23, 2026 21:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant