LORA Continuos pre-training on 7B Instruct Model

[sinchanabhat]

Hi,

1. I was wondering if it is possible (or even makes sense) to try continuous pre-training using LORA for a 7b instruct model? I only have huge raw text, my hypothesis is if I train on LORA weights using the raw text, we could achieve domain adaptation to some level without losing the instruct behaviour.
2. Can I use the raw text as is in SFT Trainer and achieve this or should I use a larger LLM to convert these raw text into instructions?

Any suggestions would be really helpful.

Thanks

[mertbozkir]

Any comments?