pub.search.handlers: Slow search: handler exceeded 200ms

[sortie]

I see a lot of these warnings in the search service logs. These messages obscure other warning level problems. We should either address the performance problem or demote the warning to a note.

However, dumping performance metrics to the logs is not the right solution. The right solution is to gather the performance data, plot that data, and monitor that instead and see if the probability distribution of latency is acceptable.

[isoos]

The right solution is to gather the performance data, plot that data, and monitor that instead and see if the probability distribution of latency is acceptable.

Is there a ready-made solution we can just plug our data into?

[sortie]

I'm sure there are, though I don't know them that well. Perhaps we can use things like google data studio for this?

[sigurdm]

This is still a problem (even got worse?)

We need profiling data. Where is all that time spent?

• Are we spending a lot of time loading package details for the listing?
  • these are in the worker output that potentially requires two fetches from cloud storage
    • should we preload this into datastore or redis?
• Do we need to move search into the frontend service to reduce the time in communication between services?
• How long time does the search itself take? (Can we create a dashboard for that)

[sigurdm]

From initial investigations it seems that most of the time is spent in the search service.

[sigurdm]

Still pending...

[sigurdm]

We should try profiling the search service - maybe we can find ways to optimize. I think we should be able to handle much higher request rates than we are.

[sigurdm]

Still an issue