[Frontend] Pass API server count to each process

[DarkLight1337]

Purpose

Follow-up to #23018.

By passing API server count and rank instead of setting cache size to 0, this PR enables processor caching when API server scale-out is enabled. IPC caching is still disabled for internal LB though since there is no 1:1 relationship between API server and Engine Core processes.

Also these changes are required for #22070.

Test Plan

Should we add an endpoint to query the API server count and rank just to test that these arguments are passed correctly? Actually we already have /server_info for that, going to add a test.

cc @njhill

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request introduces the necessary changes to pass the API server count and rank to each server process. This is a crucial step for enabling more sophisticated caching and resource management in a scaled-out API server environment. The changes are well-implemented across the configuration, argument parsing, and process management layers. Key changes include adding api_process_count and api_process_rank to ParallelConfig, updating APIServerProcessManager to handle per-server arguments, and correctly disabling incompatible features like IPC cache when multiple API servers are active. The refactoring in EngineArgs.from_cli_args also improves robustness. Overall, this is a solid contribution that enhances the multi-process architecture of vLLM.

[hmellor]

Something like this to handle internal state which needs a public interface?

https://docs.python.org/3/library/dataclasses.html#dataclasses.InitVar

Suggested change

	api_process_count: int = 1
	"""[Internal] The number of API processes initialized."""
	api_process_rank: int = 0
	"""[Internal] The rank of this API process."""
	_api_process_count: int
	api_process_count: InitVar[int] = 1
	"""The number of API processes initialized."""
	_api_process_rank: int
	api_process_rank: InitVar[int] = 0
	"""The rank of this API process."""
	...
	def __post_init__(self, api_process_count, api_process_rank):
	...
	self._api_process_count = api_process_count
	self._api_process_rank = api_process_rank
	...

[DarkLight1337]

We still access those attributes as public attributes in our code. The "Internal" here refers to the fact that these are only supposed to be passed in CLI args via API server scale-out. Users should not set this flag

[hmellor]

Something like this works (although mypy will complain about redefinition):

# example.py
from dataclasses import InitVar, dataclass


@dataclass
class ParallelConfig:
    api_process_count: InitVar[int] = 1

    def __post_init__(self, api_process_count: int):
        self._api_process_count = api_process_count

    @property
    def api_process_count(self) -> int:
        return self._api_process_count

parallel_config = ParallelConfig(api_process_count=4)
print(parallel_config.api_process_count)
parallel_config.api_process_count = 2

$ python example.py
4
Traceback (most recent call last):
  File "/home/harry/vllm/demo.py", line 18, in <module>
    parallel_config.api_process_count = 2
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: property 'api_process_count' of 'ParallelConfig' object has no setter

[NickLucche]

Users should not set this flag

users can't set this flag already, probably worth it to make it read only as Harry suggested or in a similar fashion

[DarkLight1337]

I think this isn't worth the extra complexity, at least for now (especially since mypy doesn't even work with this)

[hmellor]

Ok, I was just trying to think of robust ways to have config that can be set on init but not later

[DarkLight1337]

To avoid confusion, I have updated the docstring to be more clear that "internal" refers to how the CLI arg is passed, rather than its usage in the code

[DarkLight1337]

I think most config attributes (not just this one) shouldn't be modified after construction tbh. We should try to fix their values at initialization time but it would take some refactoring.

[NickLucche]

This makes sense to me, thanks! Let's wait for tests

[NickLucche]

Users should not set this flag

users can't set this flag already, probably worth it to make it read only as Harry suggested or in a similar fashion

[DarkLight1337]

I much prefer full JSON format, but to avoid breaking BC, let's continue to use text format by default.

[njhill]

looks a bit weird to have the same variable name as method name here (any particular reason for introducing intermediate var here?)

[DarkLight1337]

It is because __enter__ returns the server list rather than the manager itself

[DarkLight1337]

I can change the return value of __enter__ if you want

[njhill]

same comment

[njhill]

Thanks @DarkLight1337

I guess I've lost track of why we need the new parameters in the config for this? given we are already passing the count/index to these places (AFAICT)

[njhill]

Is this used anywhere? or it's added to be used in future?

[DarkLight1337]

No, I just added this for consistency since client_index is being assigned

[DarkLight1337]

The new parameters are put in the config because the config is readily accessible in various parts of vLLM, which is needed for the next PR

[NickLucche]

yeah there's different instances of this behavior, I suppose at some point we could refactor this into a shared context (which isn't the forward one) to avoid abusing config.py changes.