Hi there, can anyone share their experiences on what they found to be the smallest (ideally locally runnable) model that can still reliably perform the command generation task? I tried LLama3.1 8B (Q8), but it was not really working well.
Hi @tozo ,
You can find some LLMs on huggingface that are fine-tuned for use with Rasa Pro CALM.
You run them locally like any llama model.
These LLMs are fine-tuned on rasa-calm-demo, but will likely work quite nice for your bot as well.
I recommend you try out:
Please let us know how it works out.
We will publish instructions soon how you can further fine-tune these LLMs on your own model. Stay tuned !
1 Like
Thank you! I will try it out as soon as I have time!
I personally used phi3:14b-instruct from Ollama and it worked decently well. However a point I noticed is that if you have multiple similar flows with only minor changes, the smaller models fail to capture the differences. But if its for entirely different tasks or conversations, Phi3 worked well for me.
Hi @Arjaan Could you provide more models like these that supports Rasa CALM well or the list/link where can we find them?