[serve] Add cached metric for handle

[akyang-anyscale]

Why are these changes needed?

This PR caches the num handle requests value and updates the metric async instead of on the critical path.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[gemini-code-assist]

Code Review

This pull request introduces caching for handle request metrics, moving the updates to an asynchronous task to improve performance on the critical path. The implementation is sound and correctly plumbed through the necessary components. I've pointed out a couple of opportunities to refactor duplicated code to enhance maintainability. Overall, this is a solid improvement.