Send image file or file location along with text output

Hi,

I currently have a web chatbot UI that is connected to my RASA bot. For now I am able to recieve and send text output. But now, if the intent requires I want to be able to send an image file (or image file location) along with text output so that the image is rendered alongside the chatbot in the UI. The rendering part I can do. But I need to understand how to modify my action or the agent.handle_message() part to send text+image. For example I currently have an action like-

class ActionAchievementReport(Action):
   def name(self):
      # type: () -> Text
      return "action_achievement_report"

   def run(self, dispatcher, tracker, domain):
      # type: (Dispatcher, DialogueStateTracker, Domain) -> List[Event]

      my_period = tracker.get_slot('period')
      my_period_compare = tracker.get_slot('period_compare')
      my_bucket = tracker.get_slot('bucket')
   
      result =random.uniform(-0.5, 0.5) # eventually get from db
      ### 
      # Some code to generate image.png file
       ###
      if result > 0.0:
         dispatcher.utter_template("utter_achievement_up", tracker, achieve_amt=result,period=my_period)
         if result < 0.0:
         dispatcher.utter_template("utter_achievement_down", tracker, achieve_amt=result, period=my_period)
       
      return [SlotSet("achieve_amt", result if result is not None else []),SlotSet("report_list",report_list)]

so basically my utter_template takes care of response and when i run the model for this action my output is of the form-

rasa_output = [{"text": "You revenue is up by 0.4123."}]

Now i want this to have -

rasa_output = [{"text": "You revenue is up by 0.4123."},{"image":"image.png}"]

This “image” part can be empty or filled depending upon whether my action generated an image.

Where and what changes can I make to achieve this output? Any thoughts/ideas are greatly appreciated

Thank you, Shruti

Have you looked at the domain templates here? that’s how you can easily send an image Domain Format

Hi Akela,

Thank you for your response. Really silly of me to have missed that :slight_smile:

Regards, Shruti

HI Akela,

I tried to use the solution from domain format but what if I want to keep the image path variable based on input from my action for example I have-

utter_growth_up:
-text:"{period}revenue is up by {growth_amt}"
image:"{image_growth_up}"

In my utter_dispatch (called in the action), I fill the values for period and growth_amount. I tried doing the same for image_growth_up but it did not work.

Any suggestions about being able to change the {image_growth_up} variable. In my action I have-

image_growth_up = "/home/Downloads/static/res/histogram.png"
dispatcher.utter_template("utter_growth_up", tracker, growth_amt=result,period=my_period,image_growth_up=image_growth_up)

Thank you, Shruti

@akelad Do we have any solution to this ? {image_growth_up} that gets passed as a parameter in the utter_template shows as {image_growth_up} in the attachment tag in the json object rather than the URL. Any documentation how can we send a value to the parameter ?

I solved it by sending a dictionary object using utter_custom_message, this can be later parsed on the bot UI side.

snippet: tempvar = {‘text’:‘Any Custom Text’, ‘class’: ‘image’, ‘url’: URL}

dispatcher.utter_custom_message(tempvar)

1 Like

Hi Soumya,

Actually,my problem was not really to send it to utter_template. I have an chatbot where I want the relevant image to be along the bot window and only the text part gets written in bot window.

so my utter_template looks like- the temperature for today is {temp} || {img_temp_chart}

{imp_temp_chart} holds the path to the temperature chart file

So just before I sent the output to the chatbot window on the browser, I split the above output by ‘||’ , take first part and pass on to the bot window,where as the {img_temp_chart} part is used to grab the required file to be displayed along the bot window

Hope this helps. Let me know if you have any more questions.

Best, Shruti

1 Like

@akelad seems like the link you have attached is not working & i am facing similar issue. Can you please update the link.

Thank you

This is the new link: Domains

If we need to send a image dynamically instead of static one, we should give image url or make local image to run on server and send this local file path as dispatcher message. as shown below:

We are running Django Sever, and in its images folder, we are dynamically saving an image. This path is given as utterance of the dispatcher

plt.savefig(’./static/images/image1.png’)

dispatcher.utter_attachment(’./static/images/image1.png’)

I would suggest you to use image url as in some cases it is better to give url to UI. For me when i needed to send image to react UI so i had to give url for image preview as well as for storing it in mongoDB

I had same problem in showing different images based on different intents. Perfect solution. Saved my day. Thank you.

can you plz share the code in github or gitlab