fix: do not run keep_alive for an overload that failed argument conversion

[jturney]

Description

Calling an overloaded function bound with py::keep_alive<0, N> segfaults whenever the call is not matched by the first overload tried. Minimal reproducer:

struct Tensor {};
struct Holder {
    Tensor t;
    Tensor &from_int(int)            { return t; }
    Tensor &from_string(std::string) { return t; }
};

PYBIND11_MODULE(min, m) {
    py::class_<Tensor>(m, "Tensor");
    py::class_<Holder>(m, "Holder")
        .def(py::init<>())
        .def("make", &Holder::from_int,    py::return_value_policy::reference, py::keep_alive<0, 1>())
        .def("make", &Holder::from_string, py::return_value_policy::reference, py::keep_alive<0, 1>());
}

>>> h.make(1)     # <min.Tensor object at 0x...>
>>> h.make("x")   # Segmentation fault

The crash is EXC_BAD_ACCESS in _Py_TYPE(ob=0x1). Swapping the registration order swaps which call crashes, so it is always the call that reaches an overload after an earlier one failed argument conversion. return_value_policy::reference is not required to trigger it, and removing keep_alive makes it go away.

Cause

The dispatch lambda in cpp_function::initialize runs the post-call hook unconditionally (pybind11.h:596-603):

auto result = call_impl<...>(call, ...);
process_attributes<Extra...>::postcall(call, result);

call_impl bails out at pybind11.h:504 when argument loading fails:

if (!args_converter.load_args(call)) { return PYBIND11_TRY_NEXT_OVERLOAD; }

and that sentinel is ((PyObject *) 1) (detail/common.h:367). For a keep_alive whose nurse or patient is index 0 the work happens in postcall, so keep_alive_impl receives the sentinel as ret, get_arg(0) returns it, and _Py_TYPE dereferences it.

The existing guards do not catch this. The sentinel is neither null nor Py_None, so it passes straight through the checks added in #341.

Fix

A single guard inside keep_alive_impl, beside those checks.

On scope: only the Nurse == 0 || Patient == 0 specialization does its work in postcall. The other specialization runs in precall, against a fully populated call.args, and I verified that keep_alive<1, 2> on the same setup is unaffected. keep_alive is also the only call policy with a non-trivial postcall; the other three in attr.h have empty bodies. So this guard covers every path that can observe the sentinel.

An alternative would be to guard the postcall call site so that no policy runs on the bail-out path. That may be more correct in general, since running a post-call hook for a call that never happened is questionable on its own terms, but it changes behaviour for call policies broadly. I went with the narrow fix and am happy to switch if you prefer the other.

Tests

Added to test_call_policies. The new test crashes the interpreter without the fix, which is rather the point: pybind11 currently has no test that exercises a call policy on a failing overload, and that gap is the likely reason this went unnoticed for so long.

Full suite on macOS/arm64, clang, Python 3.14, against master 5e9611a:

	result
master (baseline)	1326 passed, 23 skipped, 2 xfailed, 1 xpassed
this PR	1327 passed, 23 skipped, 2 xfailed, 1 xpassed

Notes

Reproduces on both v3.1.0 and current master. I could not find an existing issue or PR for this. I searched issues and PRs for keep_alive, postcall, and TRY_NEXT_OVERLOAD, and grepped the diffs of all 39 open PRs that touch pybind11.h.

[espressolee]

I independently built base 5e9611aacc0bdd2054aa36800055014ebcd8e805 and exact head 3536003c234e605df76bc3c70276072bfee65b66 on macOS/arm64 with CPython 3.14.6 and 3.14.6t, in both Debug and NDEBUG builds.

The patch does fix the reported sentinel crash. Both keep_alive<0, 1> (return is nurse) and the symmetric keep_alive<1, 0> (return is patient) crash on the base after the first overload rejects an argument; all four head configurations return normally. The PR test is 9/9 on 3.14 and 9/9 on 3.14t. The full exact-head Python suite is 1307 passed / 43 skipped / 2 xfailed / 1 xpassed, and the C++ suite is 34 cases / 11357 assertions passed. One CMake installed_function integration target aborts during Python 3.14.6 finalization, but the exact base reproduces the same failure, so I do not attribute that to this PR.

I found two adjacent pre-existing gaps that matter to the scope choice in the PR description:

1. Non-return keep_alive still runs for a rejected overload. process_attributes<...>::precall(call) runs at pybind11.h:588-590, before load_args(). With a first overload carrying keep_alive<1, 2> and rejecting its third argument, followed by a successful overload with no policy, the rejected overload still attaches patient 2 to nurse 1. On the exact head, one failed attempt retains the patient until the nurse dies; 1000 failed attempts retain 1000/1000 patients. A matched second-overload-only control retains 0/1000. This is identical on 3.14/3.14t and Debug/Release.

2. A failed return conversion still invokes postcall with a null result. A custom return caster that sets an exception and returns an empty handle behaves normally without keep_alive (TypeError: Unable to convert function return value...). With keep_alive<0, 1>, exact head aborts in Debug at pybind11_fail's assert(!PyErr_Occurred()); Release masks the conversion error with RuntimeError: Could not activate keep_alive!. The sentinel-only guard at pybind11.h:3364 does not cover this null/error result.

Both gaps are present on the base, so they are not regressions introduced by the patch. The narrow guard is sound if this PR is explicitly scoped to the return-index PYBIND11_TRY_NEXT_OVERLOAD crash and the two gaps are tracked separately. If the intended scope is the current broader title—do not run keep_alive for an overload that failed argument conversion—then the non-return specialization also needs to move after successful conversion.

As a feasibility check only, I moved all keep_alive activation to successful postcall and skipped null/error results in a scratch tree. That closed the 10-case normal/free-threaded Debug/Release matrix, preserved successful lifetime relationships and conversion fallback, and kept the existing call-policy tests at 9/9. I am leaving this as a commented review rather than a blocking review because the two residuals predate the submitted diff.

[henryiii]

Review with Fable:

🤖 AI text below 🤖

Code review is done. The fix itself is verified correct — the sentinel guard is right, and the test does crash without it. Six findings, ranked:

Correctness (same bug family):

1. pybind11.h:3364 — guard misses null ret. If the return value fails to cast (e.g. unregistered type), ret is a null handle, postcall still runs, and the user gets RuntimeError: Could not activate keep_alive! masking the real TypeError. One-line extension: if (!ret || ret.ptr() == PYBIND11_TRY_NEXT_OVERLOAD).

2. attr.h:661 — precall sibling is untouched. Arg-to-arg keep_alive<N,P> runs in precall, which happens before load_args, so it still fires for overloads that fail conversion (and can fire twice via the two-pass dispatcher). The commit title doesn't hold for that shape — fix it or scope it out explicitly in the PR.

Test robustness:

3. test_call_policies.cpp:125 — only keep_alive<0,1> is tested (sentinel as nurse). The keep_alive<1,0> shape (sentinel as patient → silent corruption via Py_INCREF((PyObject*)1)) is untested; a nurse-only guard refactor would keep the test green.

4. test_call_policies.cpp:122 — return self with keep_alive<0,1> makes the object its own patient: permanently uncollectable, leaks a patients-map entry per assertion. Return a distinct object, or switch this pair to keep_alive<1,0> (which also fixes finding 3).

Altitude / cleanup:

5. pybind11.h:603 — the fix is in the keep_alive leaf; any third-party process_attribute postcall that inspects ret still gets (PyObject*)1. Guarding at the dispatcher (line 603) covers the whole class, at the cost of precall-paired cleanup policies missing their postcall. If the narrow fix is intentional, a note at process_attribute_default::postcall would help.

6. test_call_policies.cpp:119 — the "reaching the SECOND overload" comment is inaccurate (a single non-overloaded def crashed identically; postcall runs before the dispatcher sees the sentinel), and the same explanation is repeated near-verbatim in three places. Keep the full version at the guard site, correct the claim, trim the test comments.

[henryiii]

Hi @jturney, you made a fork of a fork (loriab/pybind11), which means I can't push to your fork even though maintainer edits are on. You can do git pull https://github.com/henryiii/pybind11.git fix-keep-alive-failed-overload if you'd like my fixes for the above.