Issue with Ollama LLM Integration - Port Binding and Quota Exceeded - RASA CALM

[alaa-sayed-ai-expert-flow]

Hi everyone,

I’m currently integrating Ollama as the LLM provider in RASA CALM. While my configuration seems to be correct, I’m encountering an issue where RASA still makes calls to OpenAI’s API and gives the following error:

"ProviderClientAPIException: RateLimitError: OpenAIException - Error code: 429 - {‘error’: {‘message’: ‘You exceeded your current quota, please check your plan and billing details.’} "

This happens despite specifying Ollama as the LLM provider in my config.yml. Here’s my configuration:

" pipeline:

• name: SingleStepLLMCommandGenerator

  llm:

  provider: ollama

  api_base: “http://localhost:11434/v1”

  model: llama3.1

"

It seems RASA still relies on OpenAI for some requests, and I want to ensure it only uses Ollama locally. Has anyone else experienced this, and do you know how to fully disconnect OpenAI while running a local LLM?

Thanks for your help!

[sk1382 (SrivathsavaKaza)]

Hey @alaa-sayed-ai-expert-flow try this:

llm:
  model: llama3.1
  type: openai
  openai_api_base: http://localhost:11434/v1
  openai_api_key: foobar

This helped me to manually override the OpenAI base to Ollama base.

[alaa-sayed-ai-expert-flow]

That is my config.yml file : I have done what you showed me :

But I Still get that error :

[alaa-sayed-ai-expert-flow]

@sk1382

model: llama3.1
  type: openai
  openai_api_base: http://localhost:11434/v1
  openai_api_key: foobar

I have used command “rasa train”

[sk1382 (SrivathsavaKaza)]

pipeline:
- name: NLUCommandAdapter
- name: MultiStepLLMCommandGenerator
  llm:
    model: "llama3"
    request_timeout: 10
    type: openai
    openai_api_base: http://localhost:11434/v1
    openai_api_key: foobar
  flow_retrieval:
    active: false
    embeddings:
      type: "huggingface"
      model_name: "sentence-transformers/all-mpnet-base-v2"
      task: "feature-extraction"

Try this and also make sure you are running the same model with ollama in your local.

[Sanjukta.bs (Sanjukta Biswas)]

I am using config file as

recipe: default.v1
language: en
pipeline:
- name: SingleStepLLMCommandGenerator
  llm:
    model: "llma3model"
    request_timeout: 10
    type: openai
    openai_api_base: http://localhost:11434
    openai_api_key: foobar
  prompt_template: prompt_templates/time_aware_prompt.jinja2
  flow_retrieval:
    active: false
    embeddings:
      type: "huggingface"
      model_name: "sentence-transformers/all-mpnet-base-v2"
      task: "feature-extraction"

but i am getting errors like 2024-09-12 12:03:21 WARNING langchain.llms.base - Retrying langchain.llms.openai.acompletion_with_retry.._completion_with_retry in 4.0 seconds as it raised APIError: Invalid response object from API: ‘404 page not found’ (HTTP response code was 404).

My ollama is running in docker and I have hosted in http://localhost:11434.

[stephens (Greg Stephens)]

It could be the response rephraser which uses openai got-3.5 by default. It’s documented here and I would disable it for now in your endpoints.yml.

If that’s not the issue, post the full logfile someplace and I will review.

[alaa-sayed-ai-expert-flow]

I am getting that error

ERROR rasa.dialogue_understanding.generator.llm_based_command_generator - [error ] llm_based_command_generator.llm.error error=ProviderClientAPIException(“\nOriginal error: litellm.APIError: APIError: OpenAIException - Error code: 500 - {‘error’: {‘message’: 'llama runner process no longer running: -1 ', ‘type’: ‘api_error’, ‘param’: None, ‘code’: None}})”)

and Let me Share with you The structure of my files first :

The structure of data files :

config.yml :

flows.yml:

nlu.yml:

patterns.yml:

rules.yml:

stories.yml:

domain.yml:

For Running command I am using that :

it needs OPENAI API KEY just for running and can’t run without it : export OPENAI_API_KEY=**************************************************** (I didn’t pay for it just for making RASA CALM project RUN)

export RASA_PRO_LICENSE=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJqdGkiOiJkYTViOGRmZC02NjRmLTQ5Y2ItYTFhNS02OTk3ZmIzN2JiNDMiLCJpYXQiOjE3MjU1NTgwMTEsIm5iZiI6MTcyNTU1ODAwOCwic2NvcGUiOiJyYXNhOnBybyByYXNhOnBybzpjaGFtcGlvbiIsImV4cCI6MTgyMDE2NjAwOCwiZW1haWwiOiJhbGFhLnNheWVkQGV4cGVydGZsb3cuY29tIiwiY29tcGFueSI6IlJhc2EgQ2hhbXBpb25zIn0.HgAZHBN00LJ1fv6rI5ZdTjXxiesdfdpNtrtDk0-TPq-4tWvnX7xWf8w_TloB1rCB3Gg_pU73vXzioCxtebS6rG_O3w-nXpRmntGsae-jI5y5jK2kqFDhEJ1gn5pX5Yzi0rhyt6AofgAm3SD6uiyvbLCnr0qia0HUQWFcZrF5YYsnqgIUOspVPRMH5S2X3Cu7wgMtZ0Ia3VIP0EqTVCtFYYSQsjr8pzSPJT02claDnJATzgqVq2QIqN1c1S4bMAHfb1h43KcPz22_GQbhSd8MbwuX-jZ1oOYptARxzYzXuve8lqVso-VrxHs1kHuf4Wxr4osYzv33-qSNJMIaJK9lcA

RASA train rasa shell

but I am getting the above error.

[alaa-sayed-ai-expert-flow]

@sk1382 @Sanjukta.bs @stephens