Change `time-to-first-token` parameter to be based on number of request tokens

[mayabar]

Current state
TTFT is defined as a single value , not based on number of request tokens

Required state
Add a new parameter which defines prefill time per token instead of TTFT

Same for kv-cache-transfer-latency

[pancak3]

Can I confirm the following:

Implementation related

• the param will be called prefill-overhead
• time-to-first-token and prefill-overhead coexist
• time-to-first-token overrides prefill-overhead (forward compatibility)
• test cases
• docs and comments
• kv-cache-transfer-latency

[mayabar]

@pancak3 , Our initial intention was to move to calculation based on number of input tokens only, without option to define total TTFT, but it's a good idea to allow user to define both options.
What I'm thinking now is that in the calculation of total prefill time , in addition to part which depends on number of tokens, there is a part which is constant for any prompt length, please add this. Let's call it prefill-overhead.
Same for times defined for transfering kv-cache.
Pay attention that all times are defined by mean and standard deviation.

[mayabar]

@pancak3 do you plan to work on this issue?

[pancak3]

@pancak3 do you plan to work on this issue?

If this is not urgent, like, in days, sure

[mayabar]

great! thanks!

/assign @pancak3