Issue connecting RASA PRO CALM with LLM on self-hosted vllm server: Hosted_vllmException - 'str' object has no attribute 'model_dump'

Rasa Pro CALM
1

Hi, I’m trying to connect a self-hosted Qwen model on vLLM server in SingleStepLLMCommandGenerator.

Below is my config.yml:

image
image1104×680 34.6 KB

But I’m getting the following error:

rasa.dialogue_understanding.generator.llm_based_command_generator - [error ] llm_based_command_generator.llm.error error=ProviderClientAPIException(“If you are using ‘provider=self-hosted’ to call a hosted vllm server make sure your config is correctly setup. You should have the following mandatory keys in your config: provider=self-hosted; model=‘’; api_base=‘your-hosted-vllm-serv’.In case you are getting OpenAI connection errors, such as missing API key, your configuration is incorrect.\nOriginal error: litellm.APIError: APIError: Hosted_vllmException - ‘str’ object has no attribute ‘model_dump’)”)

I’m on Linux 5.13.0 with Python 3.10.12

Below are the package versions:

rasa-pro==3.10.11

rasa-sdk==3.10.1

openai==1.54.5

litellm==1.52.16

The model is adhering to the OpenAI API format and I’m able to access the same model through chat completion endpoint. But it’s not working through RASA.

Thanks!

2

Hi Vamshi,

I asked the team and on first look it seems like you have model_name instead of model.

Can you try again with just model and see if this fixes the config setup?

3

It was a proxy issue between the VMs that hosted RASA and LLM which our networking team figured out and fixed it.

No issues from RASA config. Worked like a charm. Thanks @Lauren-Goerz . Appreciate your quick response :slight_smile:

1 Like