Skip to content

feat: add file-backed constraint system loading#1781

Draft
sergeytimoshin wants to merge 2 commits into
Consensys:masterfrom
sergeytimoshin:mmap-r1cs-loading
Draft

feat: add file-backed constraint system loading#1781
sergeytimoshin wants to merge 2 commits into
Consensys:masterfrom
sergeytimoshin:mmap-r1cs-loading

Conversation

@sergeytimoshin

Copy link
Copy Markdown

This PR adds file-backed loading for serialized constraint systems:

  • Adds generated ReadFromFile(path) support to curve-typed constraint systems.
  • Uses mmap on supported platforms to decode directly from file-backed pages.
  • Falls back to the existing ReadFrom path on unsupported platforms.
  • Adds public helpers:
    • groth16.ReadCSFromFile(curveID, path) for R1CS.
    • plonk.ReadCSFromFile(curveID, path) for SparseR1CS.
  • Keeps the existing serialization format unchanged.
  • Hardens malformed serialized-data handling in shared constraint-system decoding, integer-compression decoding, and generated coefficient-table decoding so invalid lengths return errors instead of panicking or attempting huge allocations.

Why

Today ReadFrom reads the whole serialized constraint-system payload into a heap []byte before decoding it. For large circuits this creates a temporary memory spike: the process holds both the serialized payload copy and the decoded constraint-system structures during load.

When the source is already a file, this PR removes that intermediate heap copy on mmap-capable platforms. The decoded R1CS/SparseR1CS still owns normal Go heap structures after loading, so the mmap can be closed before returning. This is intentionally a load-time peak-memory reduction, not a long-lived mmap-backed constraint-system representation.

This is useful independently of mmap-backed Groth16 proving keys: users who persist compiled R1CS or SparseR1CS artifacts can reduce load-time memory pressure even when using the existing proving-key formats.

API Notes

The PR does not add methods to constraint.ConstraintSystem. That avoids breaking external implementations of the interface. Instead, the fast file path is exposed through backend-level helpers next to the existing NewCS constructors, and the generated concrete types also get ReadFromFile.

Benchmark

Local benchmark on Apple M4 Max, darwin/arm64:

BenchmarkReadCSFromFile/ReadFrom-16          196069 ns/op   588948 B/op   4271 allocs/op
BenchmarkReadCSFromFile/ReadFrom-16          170360 ns/op   588941 B/op   4270 allocs/op
BenchmarkReadCSFromFile/ReadFrom-16          173465 ns/op   588968 B/op   4271 allocs/op

BenchmarkReadCSFromFile/ReadCSFromFile-16    160856 ns/op   425195 B/op   4272 allocs/op
BenchmarkReadCSFromFile/ReadCSFromFile-16    158764 ns/op   425179 B/op   4272 allocs/op
BenchmarkReadCSFromFile/ReadCSFromFile-16    162448 ns/op   425194 B/op   4272 allocs/op

The benchmark shows the expected reduction in heap bytes from avoiding the serialized-payload copy. Runtime is comparable; allocation count has two extra small allocations from file mapping.

Validation

Passed:

go generate ./internal/generator/backend
go test ./constraint/...
go test ./internal/backend/ioutils ./backend/groth16 ./backend/plonk
go test ./backend/groth16 -run '^$' -bench BenchmarkReadCSFromFile -benchmem -count=3
git diff --check

Checklist Notes

  • Documentation: public API comments were added on ReadCSFromFile and generated ReadFromFile; the PR body explains intended usage and limitations.
  • Generated files: this PR intentionally updates generated constraint-system marshal/test files and their templates.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant