Improve topic lookup for topics that have high number of partitions

[sijie]

Problem

Currently the client looks up topic location per topic partition. This would result in quite a lot of topic lookup requests when a topic has many partitions. It would be great to batch the topic location lookup requests, so a client doesn't have to send that many lookup requests.

[MarvinCai]

If no one is actively working on this issue I'll take it up.

[sijie]

I am not aware of anyone working on this. so feel free to pick it up

[MarvinCai]

Look into the code a bit and identify some places where change need to be made, also add proposed update to protobuf in the a doc.
@sijie can you please take a look and see if I'm on the right track? Feel free to leave any comment in the doc or in this thread. Thanks!

[sijie]

@MarvinCai awesome! since it is proposing a new set of wire commands, it would be good to finalize your google doc into a PIP and share it to dev@ mailing list. loop @jiazhai in since he did look into this area before he would have more context in helping you finalizing a good proposal so that you can share with dev@ mailing list.

[jiazhai]

glad to help