Comparison of Different Fine-Tuning Techniques for Conversational AI

[ImamaDev]

Feature request

It would be incredibly helpful to have a clear comparison or support for various fine-tuning techniques specifically for conversational AI. This feature could include insights into their strengths, limitations, and ideal use cases, helping practitioners choose the right approach for their needs.

Here’s a list of techniques to consider:

LoRa
AdaLoRa
BONE
VeRa
XLora
LN Tuning
VbLora
HRA (Hyperparameter Regularization Adapter)
IA3 (Input-Aware Adapter)
Llama Adapter
CPT (Conditional Prompt Tuning)etc

Motivation

With the growing number of fine-tuning techniques for conversational AI, it can be challenging to identify the most suitable approach for specific use cases. A comprehensive comparison of these techniques—highlighting their strengths, limitations, and ideal scenarios—would save time, reduce trial-and-error, and empower users to make informed decisions. This feature would bridge the gap between research and practical application, enabling more effective model customization and deployment.

Your contribution

I’d be happy to collaborate on this! While I might not have a complete solution right now, I’m willing to contribute by gathering resources, reviewing papers, or helping organize comparisons. If others are interested in teaming up, we could work together on a PR to make this feature happen. Let’s connect and brainstorm how we can tackle this effectively!

[BenjaminBossan]

Thanks for coming up with this proposal. Indeed, this is something we have on our backlog for a long time. As you can imagine, providing objective and useful information on this is a huge undertaking, since relying on the paper results can often be problematic.

As a long term project, we plan to provide some kind of benchmark that compares all these methods in terms of runtime, memory usage, performance, etc. but I can't give any concrete date yet.

In the meantime, we have started to be more rigorous when new methods are being added in requiring a clear description of what the best use cases are. There is still a lot of room for improvement, especially when it comes to methods that were added some time ago.

If you (and others) want to contribute, I think a good place to start would be to go through the individual methods in the PEFT docs and help improve the descriptions. If we can make them more uniform, with more details on the best uses cases, pros and cons, this would already be a nice improvement.

There are other places that could benefit from such a clean up, e.g. the description of all the LoRA initialization methods.

[sparsh2]

I would be interested to contribute as well

[BenjaminBossan]

I would be interested to contribute as well

Thanks for the offer. As mentioned, as a first step, we could use some help with updating the "blurbs" of the PEFT methods. For this, it's often sufficient to read a couple of section from the paper. If anyone wants to work on one such method, please announce it here so that there is no duplicate work.

[imcoza]

How about having a sample fine-tuning script for each method and comparing different approaches for different tasks?

[BenjaminBossan]

How about having a sample fine-tuning script for each method and comparing different approaches for different tasks?

I'm not 100% sure what you mean, but let's start with a single task and then we can expand from there. We haven't come up with such a task yet, but we have some criteria:

1. It should be a task that is supported by all methods (most likely language model fine-tuning)
2. The task should be kinda realistic and practical
3. The task should not take too long to run and should not require expensive hardware
4. Training code should be easy to adapt for real training (should have example character)

Maybe we can find something from the trl examples that we can adopt.

[sirish-gambhira]

Hello @BenjaminBossan, @ImamaDev.

I hope this message finds you well. I am interested in contributing to this PR and would like your insights on the following:

1. "It should be a task that is supported by all methods (most likely language model fine-tuning)" - Could you please suggest suitable tasks where this comparison would be meaningful? A few options are: a) question-answering task (short input and output tokens) b) summarization task (long input and short output) c) code-generation (short input and long output). It would be great to know any preferred task / dataset to start with.
2. Could you please let me know any other metrics (runtime, memory usage, performance) that would be of interest to the community?

Thank you.

PS: I found a paper addressing the same issue: https://arxiv.org/pdf/2312.12148

[imcoza]

Sure @sirish-gambhira , lets connect over discord to discuss this further my user name is imcoza

[BenjaminBossan]

Thanks for the interest. We plan to do a mini sprint next week to make progress on this task. We will share any findings where the community could help contributing.

PS: I found a paper addressing the same issue: https://arxiv.org/pdf/2312.12148

Note that our goal is a bit different. We don't want to have a comprehensive literature overview of various methods on numerous datasets -- we wouldn't have the resources for that. Our goal is less academic and more practical: Giving PEFT users a simple starting guide to choose the method that is most likely to suite their needs.

[imcoza]

That would be great. I conducted a comparative study evaluating AdaLoRA, IA3, and LoRA for text generation tasks using a multi-step conversational dataset. The results indicate that IA3 outperforms the other methods, demonstrating superior efficiency by achieving convergence in fewer fine-tuning steps. Looking forward to see your findings.

[BenjaminBossan]

That would be great. I conducted a comparative study evaluating AdaLoRA, IA3, and LoRA for text generation tasks using a multi-step conversational dataset. The results indicate that IA3 outperforms the other methods, demonstrating superior efficiency by achieving convergence in fewer fine-tuning steps. Looking forward to see your findings.

Nice to hear that. I personally believe that users should also give other methods a chance instead of focusing only on LoRA. If you have anything to share, feel free to do so.

[imcoza]

That would be great. I conducted a comparative study evaluating AdaLoRA, IA3, and LoRA for text generation tasks using a multi-step conversational dataset. The results indicate that IA3 outperforms the other methods, demonstrating superior efficiency by achieving convergence in fewer fine-tuning steps. Looking forward to see your findings.

Nice to hear that. I personally believe that users should also give other methods a chance instead of focusing only on LoRA. If you have anything to share, feel free to do so.

Based on my comparative study, I fine-tuned Llama 3.1 8B Instruct on Hindi multi-turn dialogue conversational data using Lora,IA3 and Adalora to evaluate their performance.
IA3 demonstrated a smoother and more stable convergence, making it well-suited for resource-constrained environments. It maintained a smaller generalization gap, effectively reducing the risk of overfitting. In contrast, Adalora exhibited an aggressive initial optimization phase, causing a sharp drop in loss, but ultimately achieved a lower validation loss, indicating superior generalization. However, this rapid convergence introduces higher variance, necessitating careful learning rate tuning. If stability and robustness are the primary concerns, IA3 is the better choice. But if the objective is to maximize generalization and long-term adaptability, Adalora emerges as the superior method—provided there is sufficient training time and room for fine-tuning adjustments.

[BenjaminBossan]

Thanks for sharing your findings @imcoza. Are the code and the results publicly available?

We're still working on the method comparison framework, the progress is looking good so far. You can check for instance this PR.