Factor out readable function signatures to avoid duplication #5857

swolchok · 2025-10-09T00:18:45Z

Description

This should provide a nice to have size optimization for programs that use pybind11 in general. I wrote it because it seems to reduce size costs of adding enum_-specific implementations of dunder methods, but since it seems generally useful, I thought it might be a good idea to send as a separate PR.

Concretely, for the pybind11_tests.cpython-313-darwin.so file built as part of pybind11's tests, I see the following size delta (output is sdiff of size -m):

pybind11_tests.cpython-313-darwin.so:                           pybind11_tests.cpython-313-darwin.so:
Segment __TEXT: 5668864                                       | Segment __TEXT: 5619712
        Section __text: 4660144                                         Section __text: 4660144
        Section __stubs: 4068                                           Section __stubs: 4068
        Section __stub_helper: 4008                                     Section __stub_helper: 4008
        Section __init_offsets: 288                                     Section __init_offsets: 288
        Section __const: 277048                               |         Section __const: 232528
        Section __gcc_except_tab: 543816                                Section __gcc_except_tab: 543816
        Section __cstring: 57906                                        Section __cstring: 57906
        Section __ustring: 32                                           Section __ustring: 32
        Section __unwind_info: 103928                                   Section __unwind_info: 103928
        Section __eh_frame: 280                                         Section __eh_frame: 280
        total 5651518                                         |         total 5606998
Segment __DATA_CONST: 98304                                     Segment __DATA_CONST: 98304
        Section __got: 968                                              Section __got: 968
        Section __const: 92840                                |         Section __const: 83016
        total 93808                                           |         total 83984
Segment __DATA: 16384                                           Segment __DATA: 16384
        Section __la_symbol_ptr: 2656                                   Section __la_symbol_ptr: 2656
        Section __data: 595                                             Section __data: 595
        Section __thread_vars: 120                                      Section __thread_vars: 120
        Section __thread_bss: 40 (zerofill)                             Section __thread_bss: 40 (zerofill)
        Section __bss: 2960 (zerofill)                                  Section __bss: 2960 (zerofill)
        total 6371                                                      total 6371
Segment __LINKEDIT: 114688                                      Segment __LINKEDIT: 114688
total 5898240                                                 | total 5849088

Full disclosure: Not all of the other .sos improved (though none regressed). For this to show a benefit, users' pybind11 extensions would have to define multiple pybind11 functions or methods with the same signature.

Suggested changelog entry:

Reduced size overhead by deduplicating functions' readable signatures and type information.

This seems to reduce size costs of adding enum_-specific implementations of dunder methods, but also should provide a nice to have size optimization for programs that use pybind11 in general.

…g 17+

swolchok · 2025-10-10T21:09:49Z

Hmm, I could've sworn I saw Apple Clang 15 raise -Wdeprecated-redundant-constexpr-static-def, but here it is, claiming not to know about that warning: https://github.com/pybind/pybind11/actions/runs/18418181095/job/52486700109?pr=5857#step:11:123
Perhaps the existence of the warning flag is gated on whether we're in C++17 mode or C++11.

Also, here is clang 16 calling it a plain old -Wdeprecated: https://github.com/pybind/pybind11/actions/runs/18418181095/job/52486699145?pr=5857#step:6:70

…s well.

rwgk

Looks fine to me. My suggestions are purely about naming and comments.

rwgk · 2025-10-12T20:56:11Z

include/pybind11/pybind11.h

+#define PYBIND11_READABLE_SIGNATURE_EXPR                                                          \
+    detail::const_name("(") + cast_in::arg_names + detail::const_name(") -> ") + cast_out::name
+
+// We centralize readable function signatures in a specific template


The word "centralize" here was throwing me off when I first looked at this PR. It led me to think you're making the source code more DRY, but apparently that's not what this PR does?

How about writing

// We factor out readable function signatures ...

?

// so that we don't duplicate them across different ...

Is it really "we"? It'd be the compiler? But a smart-enough compiler could do that regardless, maybe that's why some .sos are still the same size?

How about

// so that we can be sure compilers don't duplicate them across different ...

// We factor out readable function signatures ...

Yep, I think that is much more straightforward and standard wording. Thank you. Also revising the PR title.

a smart-enough compiler could do that regardless

While this is technically true (c.f. the LLVM MachineOutliner), I think automatic outlining of large portions of templated functions is generally beyond reasonable expectations for C++ compilers, and so we shouldn't blame them for duplication happening. Having to extract portions of templates to non-generic code to avoid bloat is usual practice (and pybind11 itself does that with initialize_generic in this same codepath!); the difference here is we are extracting to less-generic code (fewer template arguments) rather than non-generic code.

How about a neutral framing like

// so that they don't get duplicated across different...

rwgk · 2025-10-12T20:58:28Z

include/pybind11/pybind11.h

+};
+#undef PYBIND11_READABLE_SIGNATURE_EXPR
+
+// Prior to C++17, we don't have inline variables, so we have to provide an out-of-line definition


Wow, that's a pretty big block of pure ugliness, even by the usual C++ standards ...

Could you please add a comment to prominently suggest that this block is removed when C++11 and C++ 14 support is dropped?

rwgk · 2025-10-12T21:00:17Z

include/pybind11/pybind11.h

 #    define PYBIND11_COMPAT_STRDUP strdup
 #endif

+#define PYBIND11_READABLE_SIGNATURE_EXPR                                                          \


PYBIND11_READABLE_FUNCTION_SIGNATURE_EXPR

for consistency. — Since this is used only in a couple places, I think consistency is more valuable than trimming the length of the name.

rwgk · 2025-10-12T21:06:04Z

include/pybind11/pybind11.h

+template <typename cast_in, typename cast_out>
+class ReadableFunctionSignature {
+public:
+    using readable_signature_type = decltype(PYBIND11_READABLE_SIGNATURE_EXPR);


Within this scope shorter names would make this code more readable, e.g. simply sig_type here, and static constexpr sig_type sig(), kSig or similar below.

Good point. ReadableFunctionSignature::kReadableSignature is redundant.

rwgk · 2025-10-12T21:10:33Z

include/pybind11/pybind11.h

+
+public:
+    static constexpr readable_signature_type kReadableSignature = readable_signature();
+#if !defined(_MSC_VER)


Could you please add a comment to explain (or at least hint) why this works only with compilers other than MSVC?

rwgk

Looks good to me, thanks!

@oremanj It'd be great if you could give this a second-set-of-eyes review. (Unless you see a problem, I think it'll be a super quick review.)

oremanj · 2025-10-15T21:53:33Z

include/pybind11/pybind11.h

+};
+#undef PYBIND11_READABLE_FUNCTION_SIGNATURE_EXPR
+
+// Prior to C++17, we don't have inline variables, so we have to


C++17 also made static constexpr variables inline by default. Could you suppress the definitions in C++17 mode rather than adding the warnings? I think that might make this clearer

good idea. I will try that now.

…++17 compat code

swolchok added 5 commits October 8, 2025 17:01

Centralize readable function signatures to avoid duplication

058a02a

This seems to reduce size costs of adding enum_-specific implementations of dunder methods, but also should provide a nice to have size optimization for programs that use pybind11 in general.

gate disabling of -Wdeprecated-redundant-constexpr-static-def to clan…

3fa3d55

…g 17+

fix gating to include Apple Clang 15

cd1f09c

Make GCC happy with types

d3db87b

fix apple clang gating again. suppress -Wdeprecated for GCC

cb1027f

swolchok added 2 commits October 10, 2025 14:14

Gate warning suppressions to C++17. Suppress -Wdeprecated for clang a…

c5cbec7

…s well.

hopefully fix last straggler CI job

98bee98

rwgk reviewed Oct 12, 2025

View reviewed changes

swolchok added 2 commits October 15, 2025 11:10

Merge branch 'master' into centralize-function-signatures

b543acc

attempt to address readability review feedback from @rwgk

7af710e

swolchok changed the title ~~Centralize readable function signatures to avoid duplication~~ Factor out readable function signatures to avoid duplication Oct 15, 2025

swolchok requested a review from rwgk October 15, 2025 18:31

rwgk approved these changes Oct 15, 2025

View reviewed changes

oremanj approved these changes Oct 15, 2025

View reviewed changes

drop warning suppressions and instead just gate compilation the pre-C…

cc36bce

…++17 compat code

rwgk merged commit 1e5bc66 into pybind:master Oct 16, 2025
85 checks passed

github-actions bot added the needs changelog Possibly needs a changelog entry label Oct 16, 2025

Factor out readable function signatures to avoid duplication #5857

Factor out readable function signatures to avoid duplication #5857

Conversation

swolchok commented Oct 9, 2025

Description

Suggested changelog entry:

Uh oh!

swolchok commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rwgk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rwgk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

swolchok commented Oct 10, 2025 •

edited

Loading