Enhance calculation of tokens number in response based on request's `max_tokens` parameter

[mayabar]

Current behavior:
If max_tokens is provided in the request - number of tokens in response will be always equal to max_tokens

Expected behavior:
Create shorter outputs for part of requests.
Find relevant statistics, in which part of requests response should be shorter than max_tokens, and how to calculate response's length

[mayabar]

Pay attention
If max_tokens+prompt-tokens > context window length, the response will be limited by the context window length or will fail

[mayabar]

Histogram of length values distribution:
Divide possible response length values to 6 bucket:

• last bucket only max_tokens
• all other values divide to 5 buckets (initial version will use equal size buckets, next version will create smaller buckets near size 40, which is used as most frequent length of the response)

Buckets probabilities:
1 - 20%
2 - 30%
3 - 20%
4 - 5%
5 - 10%
6 - 15%

[mayabar]

Statistics found publicly:

Distribution of response lengths (empirical evidence from OpenAI usage)

From published benchmarks and observed logs:

Only a minority of completions (≈10–30%, depending on prompt type) actually use all max_tokens.

The majority are shorter, often much shorter.

Typical histogram for max_tokens=100 looks like this (approximate, based on user logs and reports):

Length range (tokens) Probability

---

1–20 ~25–35%
21–50 ~30–40%
51–80 ~15–25%
81–99 ~5–10%
exactly 100 ~5–10%

[mayabar]

both sub-issues were implemented