Skip to content

Reject a composite modulus before a Pimpl exists, fixing a leak and a double free - #90

Merged
d-torrance merged 2 commits into
Macaulay2:masterfrom
d-torrance:pimpl-modulus-check
Sep 25, 2026
Merged

d-torrance merged 2 commits into
Macaulay2:masterfrom
d-torrance:pimpl-modulus-check

Conversation

@d-torrance

Copy link
Copy Markdown
Member

Both library constructors that validate a modulus allocated their Pimpl
first and checked second. A constructor that throws never runs its
destructor, so each one had to clean up by hand, and they got it wrong in
opposite directions.

1. GroebnerConfiguration leaked on a composite modulus

It allocated the Pimpl, then rejected the modulus with
mathic::reportError, and nothing freed it. Under ASan on master,
RejectsCompositeModulus alone reports

Direct leak of 120 byte(s) in 1 object(s) allocated from:
    ...
SUMMARY: AddressSanitizer: 660 byte(s) leaked in 10 allocation(s).

once per rejected modulus, five times, on every run of the suite.

2. StreamStateChecker double freed on a composite modulus

    try {
      MATHICGB_STREAM_CHECK(isPrime(modulus), "The modulus must be prime");
      MATHICGB_ASSERT(mPimpl->debugAssertValid());
    } catch (...) {
      delete mPimpl;
    }

No rethrow, so the constructor completed with mPimpl dangling and the
destructor freed it again. Every other catch (...) in the sources cleans up
and rethrows, so this was an omission. It is reachable only by constructing
mgbi::StreamStateChecker directly -- through the public interface the
modulus comes from a GroebnerConfiguration, which has already refused it --
but IdealStreamChecker<Stream> takes its modulus from any stream, so it is
not purely hypothetical either.

The fix: check before the Pimpl exists

Both checks move into their Pimpl's own constructor. A Pimpl whose
constructor throws is never completed and never destroyed, and the
new-expression frees its storage, so neither outer constructor has a cleanup
path left to get wrong. The try/catch goes.

For GroebnerConfiguration this is also what keeps Debug working.
Pimpl::~Pimpl asserts debugAssertValid(), which requires a nonzero
modulus; the leak was the only reason a Pimpl holding a rejected modulus had
never been destroyed. Freeing it after the check would trip that assert in
RejectsCompositeModulus.

The Pimpl pointers stay raw. std::unique_ptr was the first attempt, and
was dropped: mathicgb.h keeps its members free of standard library types so
that a caller and the library built against different STLs agree on the
layout, and once the checks move, unique_ptr fixes nothing. The public
header is unchanged.

One message for a composite modulus

The two checks also reported differently -- GroebnerConfiguration said
"Modulus N is not prime. MathicGB only supports prime fields." as a
MathicException, the checker said "The modulus must be prime" as an
invalid_argument without saying which. The same message was built by hand in
two more places, MathicIO::readBaseField and SparseMatrix::read, and the
matrix reader's had drifted to "The modulus N".

checkModulusIsPrime, beside isPrime in PrimeField.hpp, now owns it, and
all four sites call it. It is a template so that readBaseField, whose
coefficient type is long and whose scanner accepts a sign, still rejects
-7 as "Modulus -7 is not prime" rather than converting it to a huge
unsigned value first.

User-visible changes:

  • The checker's composite-modulus error is now a MathicException
    (runtime_error), like GroebnerConfiguration's. A composite modulus is
    bad input rather than a protocol violation; the checker's actual protocol
    errors are still invalid_argument.
  • A matrix file with a composite modulus reports "Modulus N is not prime"
    rather than "The modulus N is not prime".

Commits

  1. 583a683 -- the two constructors, the new function, and a regression test,
    StreamCheckerRejectsCompositeModulus.
  2. 04146b6 -- MathicIO and SparseMatrix switch to the function.

Verification

  • 254/254 in cmake Release and Debug, GCC 13.3, no build warnings.

  • ASan, master against this branch:

    master this branch
    RejectsCompositeModulus 5 x 120 bytes leaked clean
    StreamCheckerRejectsCompositeModulus double free passes
    StreamStateChecker(4, 2, 1), standalone program double free throws
  • mgb gb on an ideal file with modulus -7, and with 4, reports
    ERROR: Modulus -7 is not prime. MathicGB only supports prime fields. and
    the same for 4.

No autotools distcheck run locally; neither commit touches a build system and
CI covers it.

🤖 Generated with Claude Code

d-torrance and others added 2 commits September 25, 2026 13:07
GroebnerConfiguration and mgbi::StreamStateChecker each allocated
their Pimpl in the mem-initializer list and then checked the modulus
in the constructor body.  A constructor that throws never reaches its
destructor, so the Pimpl was already built and nothing owned it:

- GroebnerConfiguration leaked it.  RejectsCompositeModulus drives
  that path five times, so the test suite has leaked on every run
  since it was added.

- StreamStateChecker caught its own check, deleted the Pimpl and did
  not rethrow, so the constructor completed with a dangling pointer
  that the destructor freed a second time.  Only direct construction
  reaches this, since a GroebnerConfiguration has already refused a
  composite modulus by then; in Release it has been a double free all
  along.

Both checks now run in their Pimpl's own constructor.  A Pimpl whose
constructor throws is never completed and never destroyed, and the
new-expression frees its storage, so neither outer constructor has a
cleanup path to get wrong.  StreamStateChecker's try/catch goes, and a
composite modulus passed to it directly now throws, as the check always
said it should.  The new StreamCheckerRejectsCompositeModulus test
covers that, and on master trips ASan's double-free report.

The two checks were written differently: GroebnerConfiguration built
"Modulus N is not prime. MathicGB only supports prime fields." and
reported it with mathic::reportError, while StreamStateChecker threw
invalid_argument as a stream protocol error without saying which
modulus.  Both now call checkModulusIsPrime, a new function beside
isPrime in PrimeField.hpp, so they report the same message the same
way.  A composite modulus is bad input rather than a protocol
violation, so the checker's error is now a MathicException, the
runtime_error subclass GroebnerConfiguration already threw; its actual
protocol errors are still invalid_argument.

For GroebnerConfiguration the move is also what keeps Debug working
once the Pimpl is freed: its destructor asserts debugAssertValid(),
which requires a nonzero modulus, and the leak was the only reason a
Pimpl holding a rejected modulus had never been destroyed.

The Pimpl pointers stay raw rather than becoming unique_ptr.
mathicgb.h keeps its members free of standard library types so that a
caller and the library built against different STLs still agree on
the layout, and with the checks moved, unique_ptr would fix nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MathicIO::readBaseField and SparseMatrix::read each built the "is not
prime" message by hand, the last two copies of it now that the library
interface uses checkModulusIsPrime.  They had drifted: the matrix
reader said "The modulus N", every other site "Modulus N".

readBaseField's own charac < 2 guard goes too.  Its coefficient type is
long and the scanner accepts a sign, so a negative modulus reaches it,
and checkModulusIsPrime tests modulus < 2 in the caller's type before
converting to uint64 for exactly that reason.  -7 is still reported as
"Modulus -7 is not prime".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@d-torrance
d-torrance merged commit efb29a4 into Macaulay2:master Sep 25, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant