[TextGeneration] max token refactor

[dsikka]

For this ticket:

https://app.asana.com/0/1201735099598270/1205276886236972/f

Summary:

• Refactors the TextGeneration Constructor to remove the max_tokens argument and makes it part of the pipeline input
• Adds num_generated_predictions to the input as well, which dictates the number of sequences that are generated for a given input. Similar to the hugging face implementation, we repeat the input based on the number provided, defaulting to 1. When num_generated_predictions is > 1, the engine's deterministic property is togged to False
• In the case where the value is > 1, the output is a list of lists, where each list includes the generated sequences for a given prompt. This updates the sequences output to be one of str, List[str], and List[List[str]]

Testing

• Tested locally using the new input arguments
• Also added new tests to evaluate the num_generated_predictions

[dbogunowicz]

The good: much less additional code and complexity that I thought
The bad and ugly: could you add appropriate tests in tests/deepsparse/transformers/pipelines/test_text_generation.py ?

[mgoin]

Could max_tokens default to the prompt_length - sequence_length so we don't risk running out of kv cache context? I'm not sure what happens there actually, especially when using internal kv cache

[dsikka]

Could max_tokens default to the prompt_length - sequence_length so we don't risk running out of kv cache context? I'm not sure what happens there actually, especially when using internal kv cache

@dbogunowicz would like to get your opinion on this

[dbogunowicz]

@dsikka this is a very good idea.

[dsikka]

Could max_tokens default to the prompt_length - sequence_length so we don't risk running out of kv cache context? I'm not sure what happens there actually, especially when using internal kv cache

Talking to the MLE team, I think for now we want to keep the defaults as is and update them once we've established best practices.