Custom partitioner support

[sakshamgupta1]

Is there any documentation around using custom partitioners with producer?

[edenhill]

Custom partitioners are currently not exposed in the Python client due to threading issues. The partitioner may be called from an internal librdkafka thread and this does not play well with the Python interpreter. We're looking into alternatives.

[edenhill]

The next version of librdkafka (the underlying Kafka client) will expose the builtin partitioners as configuration properties, allowing you to change to an alternative builtin partitioner, such as the Java compatible murmur2_random partitioner.

Search for 'partitioner' here: https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md

Custom partitioners are not yet exposed in the Python client though.

[ltucker]

@edenhill does the partition argument of Producer.produce() work? (ie can I execute a routing strategy not in a callback assuming I know the partition count and have enough other info on hand? )

[edenhill]

Yes it does!

[rnpridgeon]

@edenhill Do you think exposing the built-in partitioners here is enough or do we intend to make upstream changes to librdkafka to make this possible. Unless there are any objections I say we mark this as closed in favor of #396 for the time being. If any compelling arguments can be made for customer partitioners beyond those which are provided we can reopen .

[edenhill]

Sounds good to me, close as wontfix with a reference to the built-ins. 2018-06-29 21:20 GMT+02:00 Ryan P <notifications@github.com>:

…

@edenhill <https://github.com/edenhill> Do you think exposing the built-in partitioners here is enough or do we intend to make upstream changes to librdkafka to make this possible. Unless there are any objections I say we mark this as closed in favor of #396 <#396> for the time being. If any compelling arguments can be made for customer partitioners beyond those which are provided we can reopen . — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#231 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAgCvj2831hjl1kfTBoCSBhSED1GX_IQks5uBn4KgaJpZM4O4Jyq> .

[rnpridgeon]

Marked as won't fix with the addition of #396 which exposes the built-in partitioners.

[Bec-k]

So there is still no support for custom partitioners and assignments?