fix: Narrow conditions for load_value to give invalid address

[amjames]

Description

fixes: #6153

Objects initialized with cls.__new__(cls) (cls is a pybind11 bound type). Will not have the C++ object allocated. When hitting load_value storage is allocated but not initialized, calling a virtual method will load a garbage vptr and segfault. This is similar to #2152, but the guard in metaclass __call__ is only triggered when a subclass __init__ skips calling the parent init, the path blocked here can be hit with only a pybind exported type using __new__.

Impact

• The reproducer in [BUG]: Access to a __new__-created instance (C++ constructor never ran) dispatches through unconstructed storage instead of raising #6153 demonstrates that the bug can be hit by valid, albeit non-standard python code.
• This regression test added with pytorch-194647 demonstrates that a malformed pickle blob can also trigger the bug.

Review Notes

• A few options were considered the others (checking in dispatcher or adding a tp_getattro slot on pybind11_object were rejected because they leave other edge-cases unprotected, and they add more overhead than what has been implemented here.
• The additional check is nested below the vptr == nullptr branch. There is no additional work done when the object is fully initialized.

Authored with the assistance of claude

AI Generated Summary Below:

An instance created with __new__ has its Python object and value/holder slots
allocated, but no C++ object: make_new_instance() leaves the value pointer null
and simple_holder_constructed false. Using such an instance reached
type_caster_generic::load_value(), whose "lazy allocation" path handed out a
pointer to raw ::operator new storage. Reading a non-virtual member returned
garbage; calling a virtual one loaded a garbage vtable pointer and segfaulted.
The metaclass __call__ guard does not catch this, because __new__ reaches
tp_new directly without going through pybind11_meta_call().

That lazy allocation exists only to serve the deprecated old-style placement-new
__init__/__setstate__ idiom, which is handed a reference to uninitialized
storage on purpose. So gate it on being inside such a call rather than removing
it: cpp_function::dispatcher() marks the instance via a new
instance_construction_scope for constructor overloads (is_constructor already
covers both __init__ and __setstate__), and load_value() allocates only
while that mark is set. Otherwise it throws value_error, matching the wording
smart_holder already uses for a disowned instance.

__new__ itself stays unblocked, so pickle still works: the round trip and the
manual NEWOBJ + __setstate__ dance are both covered by the new tests.

The gate costs nothing on the hot path -- for a constructed instance the value
pointer is non-null, so the check sits in an already-cold branch. It also covers
every access path, not just attribute lookup: operator slots, property reads and
writes, and passing the instance to any bound function all funnel through
load_value().

sizeof(instance) is unchanged (the new flag takes a spare bit in the existing
bitfield byte) and the flag defaults to false, so no internals version bump.

---

📚 Documentation preview 📚: https://pybind11--6157.org.readthedocs.build/

[henryiii]

I pushed two fixes from Fable. Both started with failing tests, then fixed.

🤖 AI text below 🤖

Review complete: 12 candidates checked, 2 survived (both confirmed), 10 refuted.

1. Incomplete fix — failed old-style __init__ reopens the segfault (type_caster_base.h:1173, empirically reproduced, exit 139). If an old-style placement-new __init__ throws (or a later argument fails to convert) after self was loaded, load_value has already lazily allocated vptr but placement-new never ran. Nothing deallocates it on the error path, so the next method call sees vptr != nullptr, skips the new ValueError, and hands out uninitialized memory. The new construction_in_progress bit is exactly the hook to close this: on constructor-dispatch failure, free a value that was lazily allocated while the holder was never constructed.

2. Guard armed too broadly (pybind11.h:1008). The scope arms for every constructor chain, but new-style py::init never needs lazy allocation (self is injected directly), so reentrant access to a half-built instance during modern construction still gets silent garbage instead of the new diagnostic. Cheap narrowing: arm only when the chain has an old-style constructor (is_constructor && !is_new_style_constructor). Not a regression, but a missed improvement.

Notable refutations: cross-module ABI safe (layout unchanged, bit zero-filled by tp_alloc); the bitfield RMW race matches the accepted pre-existing pattern (has_patients, owned, etc.); throwing instead of overload fallthrough replaces UB, not working behavior; all three cleanup nits stand as written.