[WIP] Interoperability with other Python binding frameworks

[oremanj]

See also wjakob/nanobind#1140, the same feature for nanobind.

pymetabind is a proposed standard for Python <-> C-ish binding frameworks to be able to find and work with each other's types. For example, assuming versions of both pybind11 and nanobind that have adopted this standard, it would allow a pybind11-bound function to accept a parameter whose type is bound using nanobind, or to return such a type, or vice versa. Interoperability between different ABI versions or different domains of the same framework is supported under the same terms as interoperability between different frameworks. Compared to the _pybind11_conduit_v1_ API, this one also supports implicit conversions and to-Python conversions, and should have significantly less overhead.

The essence of this technique has been in use in production by my employer for a couple of years now to enable a large amount of pybind11 binding code to be ported to nanobind one compilation unit at a time. Almost everything that works natively works across framework boundaries too, at only a minor performance cost. Inheritance relationships and relinquishment (from-Python conversion of unique_ptr<T>) don't work cross-framework, and some of the more subtle corners of shared_ptr support probably don't transfer over; if you try to get a shared_ptr<T> from a foreign T, you'll either get a new shared_ptr control block whose deleter drops a pyobject reference, or a new reference to the enable_shared_from_this shared_ptr if T has one of those.

This PR adds pybind11 support for exposing pybind11 types to pymetabind for other frameworks to use ("exporting") and using other frameworks' types that they have exposed to pymetabind ("importing"). Types bound by a different framework than the internals version of pybind11 that an extension module links with are called "foreign" to that module. This PR does not introduce an ABI break, but there are some ways that it could be simplified and sped up if/when you're willing to take one. I know I just missed 3.0 so I'm guessing that won't be for a while. :-)

One notable impact of this PR is that pybind11 now generates copy and move constructors for every type, instead of generating them lazily when an instance of that type is returned from a bound function. This is needed because we might be asked to copy or move an instance of that type via the foreign bindings framework even though it's never returned from a pybind11-bound function. The new constructors can break code that previously worked: if a type has no copy constructor and none of its immediate members are non-copyable but some of their subobjects are non-copyable, it will look copyable to std type traits, but actually generating the copy constructor produces an error. Similarly, you can now get "definition of implicit copy constructor is deprecated because it has a user-declared destructor" warnings on types that previously didn't produce them. I'm not overly worried about this since it's no worse than what would already happen when returning even const Foo&, but we could consider a #define that suppresses the constructor generation if you think this is too much of an upgrade hazard.

Current status: nominally code complete and existing tests pass, but I haven't added interop-specific tests or public-facing docs yet.

Performance: Due to the inability to modify internals and type_info without an ABI break, once pybind11 knows about any foreign types, all failed casts incur an additional map lookup. Overhead should still be low for people who don't use the interoperability features at all. I haven't measured it.

Things that need to happen before this can be released:
[x] add unit tests
[x] add user-facing documentation
[ ] test correctness of nanobind/pybind11 interop
[ ] test performance
[ ] solicit feedback from maintainers of other binding libraries
[ ] release pymetabind v1.0, incorporating said feedback

Suggested changelog entry:

• pybind11 has adopted the new pymetabind standard for interoperating with other Python binding libraries (including other internals versions of pybind11). When the interoperability feature is activated, which in most cases is as simple as writing py::interoperate_by_default(), a function or method that is bound using pybind11 can accept and return values of types that were bound using other binding libraries that support pymetabind, notably including nanobind versions !TBD! and later. This is intended to replace the pybind11 "conduit" feature eventually, but the conduit is still supported for compatibility with existing extensions. Compared to the conduit feature, this interoperability mechanism supports both from- and to-Python conversions, and correctly handles implicit conversions, conversions to and from copyable holders, and conversions to unique_ptr.

---

📚 Documentation preview 📚: https://pybind11--5800.org.readthedocs.build/

[rwgk]

Wow.

https://github.com/hudson-trading/pymetabind

publish lightweight information about its bound types, along with function pointers that allow other frameworks to interact with them, to a shared data structure located via a Python capsule object;

That's awesome. I see @pablogsal is involved, so "standard for Python <-> C-ish binding frameworks" seems believable.

I'm not sure how much I can help here. This is a huge PR, I don't have a lot of spare time to review. But I sure like the direction.

This PR does not introduce an ABI break, but there are some ways that it could be simplified and sped up if/when you're willing to take one.

After this is merged and released, wait a year or so for adoption, then optimize. The ABI break won't matter as much anymore.

I'd also purge the conduit code at that time.

[pablogsal]

That's awesome. I see @pablogsal is involved, so "standard for Python <-> C-ish binding frameworks" seems believable.

I appreciate the kind words, but I only contributed a small typo fix in a macro after reading the code. All the credit for this great work goes to @oremanj.

[oremanj]

I have done a full pass of documenting and improving this change and it is ready for another round of review.

The terms "interop" and "foreign" are too generic and ambiguous. I think it'll be significantly clearer to always use "metabind" when it actually is about metabind.

I agree that "interop" on its own is too generic, since pybind11 itself is all about interoperability. I have been unable to find a better word than "foreign" though. It doesn't appear in pybind11's public API currently. It is used internally in a few places to refer to working with other pybind11 types that don't share internals with us; I think the use with pymetabind is a logical extension of that. It is also immediately suggestive of what it means, while "metabind" is a jargon term that someone would need to look up.

I have attempted to be careful to refer to "foreign frameworks", "foreign bindings", and "foreign-bound types", but not "foreign types" since the same type can be bound in multiple frameworks.

The new public API is just:

• Optional argument to PYBIND11_MODULE specifying default level: one of py::foreign_interop::full(), py::foreign_interop::import_only(), py::foreign_interop::export_only(), py::foreign_interop::on_request(), py::foreign_interop::disabled().
• py::import_foreign() for using a foreign binding that would not be used by default, or prioritizing the one you want if multiple are available
• py::export_to_foreign() for making a pybind11 type available to foreign frameworks when that would not occur by default