I was thinking of using openai api within a custom action and stream the streaming response coming from openai to my chatbot. Openai is giving me streaming response and i am doing “dispatcher.utter_message(openai_res_chunk)” from actions, but the response is not getting streamed in chunks to my bot and is available only after last chunk from openai. I am also passing “stream”: True while sending request to rasa server, but nothing is working. How to achieve this?
1 Like
Out of the box, the SDK CollectingDispatcher does not support streaming (see source code here).
Given that Rasa has an open source offering, you could extend the SDK to add streaming, however I don’t have any guidance for you on how to do so.
Is there any way to stream the response?