# Local LLM with RASA CALM **URL:** \ **Category:** Rasa CALM\ **Created:** [July 22, 2024, 3:25pm UTC](https://forum.rasa.com/t/local-llm-with-rasa-calm/62752 "2024-07-22T15:25:27Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![Shashanka](https://avatars.discourse-cdn.com/v4/letter/s/a9a28c/32.png) [@Shashanka](https://forum.rasa.com/u/Shashanka)\ **Post date:** [July 22, 2024, 3:25pm UTC](https://forum.rasa.com/t/local-llm-with-rasa-calm/62752/1 "2024-07-22T15:25:27Z")
Hello Community !! I have a perfectly working **RASA Pro CALM (v3.8.7)** chatbot which is dependent on **OpenAI** for tasks such as _‘Enterprise Search Policy’, ‘Intentless Policy’, ‘Command Generator’, ‘Contextual Response Rephraser’_ etc. Now that i need to make the chatbot completely independent, I need to use a local LLM. (Preferably via **Ollama** or any similar setup). I tried doing that but I wasn’t successful. I will share my previous config setup which I used with GPT and the new setup in which I’m trying to use the **OpenAI entry point** to use the local Ollama model. **Kindly help me with the right configuration and clear procedure of integrating local LLM with RASA Pro CALM which can replace OpenAI in every way.** Any help in regarding this is highly appreciated. _ **Old Config setup** _ ```auto recipe: default.v1 language: en pipeline: - name: LLMCommandGenerator llm: model_name: gpt-4-turbo policies: - name: FlowPolicy - name: EnterpriseSearchPolicy - name: IntentlessPolicy nlu_abstention_threshold: 0.6 assistant_id: 20240408-145223-unsorted-callable ``` _ **The Config setup that i’m trying** _ ```auto recipe: default.v1 language: en pipeline: - name: LLMCommandGenerator llm: model: 'llama2' type: openai openai_api_base: http://localhost:11434/v1 flow_retrieval: embeddings: type: "spacy" policies: - name: FlowPolicy llm: model: 'llama2' type: openai openai_api_base: http://localhost:11434/v1 embeddings: type: "spacy" - name: EnterpriseSearchPolicy llm: model: 'llama2' type: openai openai_api_base: http://localhost:11434/v1 embeddings: type: "spacy" - name: IntentlessPolicy nlu_abstention_threshold: 0.6 ``` ---
**Author:** ![Balowen](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/balowen/32/23878_2.png) [@Balowen](https://forum.rasa.com/u/Balowen)\ **Post date:** [July 23, 2024, 3:53pm UTC](https://forum.rasa.com/t/local-llm-with-rasa-calm/62752/2 "2024-07-23T15:53:52Z")
Hi @Shashanka, It seems like you are running ollama locally. Could you please try with the following change: ```auto - name: LLMCommandGenerator llm: type: ollama # defaults to locally running ollama server model: llama2 ``` I recommend you to try that with the newest rasa-pro 3.9.2 version. That’s what worked for me 🙂 You need to be aware that [FlowRetrieval](https://rasa.com/docs/rasa-pro/concepts/components/llm-command-generators#customizing-flow-retrieval) and [Rephraser](https://rasa.com/docs/rasa-pro/concepts/contextual-response-rephraser/#llm-configuration) component also use openai GPT models by default. You can change that in their configurations. ---
**Author:** ![Shashanka](https://avatars.discourse-cdn.com/v4/letter/s/a9a28c/32.png) [@Shashanka](https://forum.rasa.com/u/Shashanka)\ **Post date:** [July 24, 2024, 5:27pm UTC](https://forum.rasa.com/t/local-llm-with-rasa-calm/62752/3 "2024-07-24T17:27:38Z")
Hey @Balowen , Thank you for your response. I will try and get back to you about this. Meanwhile, could you please let me know the **embedding provider** that you are using ? **If possible, please do share the config and endpoint configuration here**. Many thanks in advance. ---
**Author:** ![Balowen](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/balowen/32/23878_2.png) [@Balowen](https://forum.rasa.com/u/Balowen)\ **Post date:** [July 26, 2024, 12:51pm UTC](https://forum.rasa.com/t/local-llm-with-rasa-calm/62752/4 "2024-07-26T12:51:31Z")
Hi, For this specific example I didn’t configure embeddings, thus I used the default openai. You don’t need to configure the endpoint for ollama – the default localhost will be used. Check out the documentation [here](https://rasa.com/docs/rasa-pro/concepts/components/llm-configuration#configuring-an-embeddings-provider) to see how to configure embeddings. I checked your config a little bit closer and there is one issue with it: - you shouldn’t configure `llm` and `embeddings` for FlowPolicy. It doesn’t use an LLM at all. The llm configuration for EnterpriseSearchPolicy should be similar as for the CommandGenerator. Check out vllm as an alternative to Ollama: [GitHub - vllm-project/vllm: A high-throughput and memory-efficient inference and serving engine for LLMs](https://github.com/vllm-project/vllm) ---
**Author:** ![MathSeven](https://avatars.discourse-cdn.com/v4/letter/m/90db22/32.png) [@MathSeven](https://forum.rasa.com/u/MathSeven)\ **Post date:** [July 28, 2024, 9:07pm UTC](https://forum.rasa.com/t/local-llm-with-rasa-calm/62752/5 "2024-07-28T21:07:57Z")
Hello Shashanka, I am looking for a tutor to help me in my Rasa X & Rasa Calm studies. Work online from home contact me at [omar@mathseven.com](mailto:omar@mathseven.com) Thanks ---
**Author:** ![Shashanka](https://avatars.discourse-cdn.com/v4/letter/s/a9a28c/32.png) [@Shashanka](https://forum.rasa.com/u/Shashanka)\ **Post date:** [July 30, 2024, 7:52pm UTC](https://forum.rasa.com/t/local-llm-with-rasa-calm/62752/7 "2024-07-30T19:52:11Z")
Hello Omar, Thank you for reaching out. Currently, I am still in the process of learning and exploring Rasa myself, so I wouldn’t be able to do that. Best of luck with your search and studies! ---
**Author:** ![MathSeven](https://avatars.discourse-cdn.com/v4/letter/m/90db22/32.png) [@MathSeven](https://forum.rasa.com/u/MathSeven)\ **Post date:** [July 31, 2024, 5:52am UTC](https://forum.rasa.com/t/local-llm-with-rasa-calm/62752/8 "2024-07-31T05:52:57Z")
Thank you ---
**Author:** ![Sanjukta.bs](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/sanjukta.bs/32/24453_2.png) [@Sanjukta.bs](https://forum.rasa.com/u/Sanjukta.bs)\ **Post date:** [July 31, 2024, 11:44am UTC](https://forum.rasa.com/t/local-llm-with-rasa-calm/62752/9 "2024-07-31T11:44:28Z")
Hello @Balowen i dont have access to open ai and I want a local llm. I am struggling to find a way on how to implement this. My data base is like a db for which i believe flow should work perfectly but I am not able to run since openAI is not allowed and blocked. Thanks in advance for help. ---
**Author:** ![Shashanka](https://avatars.discourse-cdn.com/v4/letter/s/a9a28c/32.png) [@Shashanka](https://forum.rasa.com/u/Shashanka)\ **Post date:** [July 31, 2024, 11:57am UTC](https://forum.rasa.com/t/local-llm-with-rasa-calm/62752/10 "2024-07-31T11:57:19Z")
Hey @Balowen, My bot is using following components. **LLMCommandGenerator, FlowPolicy, IntentlessPolicy, EnterpriseSearchPolicy, rasa.core.ContextualResponseRephraser**. Could you tell me which of these require **LLM configuration** and which of these demand **Embedding configuration**? I have been trying ever since. The ollama did not seem to work for me with the configuration that you had mentioned. **By the way, can you confirm if RASA expects embeddings in a fixed length or format ?** Thanks a lot !! ---
**Author:** ![Balowen](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/balowen/32/23878_2.png) [@Balowen](https://forum.rasa.com/u/Balowen)\ **Post date:** [August 5, 2024, 3:30pm UTC](https://forum.rasa.com/t/local-llm-with-rasa-calm/62752/11 "2024-08-05T15:30:04Z")
@Sanjukta.bsIf you have access to a powerful machine you could try CALM with a local llm like llama3. The easiest way to run it is with [ollama](https://ollama.com/library/llama3.1). You need to be aware though that models like llama 8B aren’t as powerful as gpt3.5 or gpt4 and won’t perform well in command generation, which is essential to understand the user and trigger correct flows. The components in Rasa Pro that require LLM configuration: - `LLMCommandGenerator` - `LLMIntentClassifier` - `ContextualResponseRephraser` - `EnterpriseSearchPolicy` - `IntentlessPolicy` Components that use embeddings: - `LLMCommandGenerator` (for [flow retrieval](https://rasa.com/docs/rasa-pro/concepts/components/llm-command-generators/#customizing-flow-retrieval:~:text=Customizing%20flow%20retrieval-,%23,-The%20ability%20to)) - `IntentlessPolicy` - `EnterpriseSearchPolicy` Refer to this page to check the supported embeddings providers: [LLM Providers](https://rasa.com/docs/rasa-pro/concepts/components/llm-configuration#other-llmsembeddings). For EnterpriseSearch, if Rasa doesn’t natively support a particular embedding model that you want to use, [custom information retrieval](https://rasa.com/docs/rasa-pro/concepts/policies/custom-information-retrievers) comes to the rescue. You can integrate local or fine-tuned embedding models of your choice to generate embeddings for search queries and documents. Could you share your whole config, how are you running the ollama and any error logs when running rasa with --debug? ---
**Author:** ![Sanjukta.bs](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/sanjukta.bs/32/24453_2.png) [@Sanjukta.bs](https://forum.rasa.com/u/Sanjukta.bs)\ **Post date:** [August 5, 2024, 4:59pm UTC](https://forum.rasa.com/t/local-llm-with-rasa-calm/62752/12 "2024-08-05T16:59:52Z")
Thanks so much for the reply. My config.yml file looks like this- ![image](https://europe1.discourse-cdn.com/flex013/uploads/rasa/original/3X/0/7/07fdd749c721bde2a9d90bcb704315f8e9c72afd.png) I am not able to include embeddings. As I am getting errors like- 2024-08-06 01:10:35 ERROR rasa.dialogue\_understanding.generator.llm\_based\_command\_generator - [error] Flow retrieval store isinaccessible. error=ValueError(“Unsupported embeddings type ‘ollama’”) event\_key=llm\_based\_command\_generator.train.failed And with flow retrieval as false, the model is getting trained but I am not able to use flows. It is not replying anything. I want to use a local llm, so which provider and embedding will be beneficial if you can recommend. If possible is there any github project available for rasa pro with local llm models? Thanks so much for the help. ---
**Author:** ![eyal-dialback](https://avatars.discourse-cdn.com/v4/letter/e/3be4f8/32.png) [@eyal-dialback](https://forum.rasa.com/u/eyal-dialback)\ **Post date:** [August 28, 2024, 8:49am UTC](https://forum.rasa.com/t/local-llm-with-rasa-calm/62752/14 "2024-08-28T08:49:50Z")
Hey guys, same config, same issue 2024-08-21 20:51:39 ERROR rasa.dialogue\_understanding.generator.llm\_based\_command\_generator - [error] Flow retrieval store isinaccessible. error=ValueError(“Unsupported embeddings type ‘ollama’”) event\_key=llm\_based\_command\_generator.train.failed ---
**Author:** ![muzzammil](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/muzzammil/32/24185_2.png) [@muzzammil](https://forum.rasa.com/u/muzzammil)\ **Post date:** [September 26, 2024, 11:33am UTC](https://forum.rasa.com/t/local-llm-with-rasa-calm/62752/15 "2024-09-26T11:33:50Z")
> [@Sanjukta.bs](#): > > And with flow retrieval as false, the model is getting trained but I am not able to use flows. It is not replying anything. > > I want to use a local llm, so which provider and embedding will be beneficial if you can recommend. This is true. Not sure if there has been a solution to this as well. Do you have any input on this please @Balowen ---
**Author:** ![Jyoti\_Prakash\_Behera](https://avatars.discourse-cdn.com/v4/letter/j/ecccb3/32.png) [@Jyoti\_Prakash\_Behera](https://forum.rasa.com/u/Jyoti_Prakash_Behera)\ **Post date:** [May 12, 2025, 3:41pm UTC](https://forum.rasa.com/t/local-llm-with-rasa-calm/62752/16 "2025-05-12T15:41:49Z")
I am using Rasa Pro version: 3.12.6. I am running llama3:8b model in my local using ollama. For me this is working: ```auto - name: SingleStepLLMCommandGenerator llm: model: llama3:8b provider: ollama api_base: http://localhost:11434 flow_retrieval: active: false ``` However, I am facing issue with EnterpriseSearchPolicy which needs either openai or azure as model provider, and this is not working with local LLM. ---
**Author:** ![m\_ashurkina](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/m_ashurkina/32/25068_2.png) [@m\_ashurkina](https://forum.rasa.com/u/m_ashurkina)\ **Post date:** [May 15, 2025, 11:41am UTC](https://forum.rasa.com/t/local-llm-with-rasa-calm/62752/17 "2025-05-15T11:41:59Z")
Hi @Jyoti_Prakash_Behera thank you for your response. Please check out this tutorial on how you can customise Enterprise Search to work with any model: [Handling FAQs with Rasa and Faiss: How to implement RAG | The Rasa Blog](https://rasa.com/blog/handling-faqs-with-rasa-and-faiss-how-to-implement-rag/) Here is an example: ```auto - name: EnterpriseSearchPolicy vector_store: type: faiss source: docs llm: model_group: gemini_flash_model embedding: model_group: gemini_embeddings prompt: prompts/enterprise-search-policy-template.jinja2 citation_enabled: true max_messages_in_query: 3 max_history: 3 ``` Please let me know if you have any questions.