Particular Image rendering in RASA

Hello everyone. I have a particular case where I’m making an API call and it’s gives me images with url’s as the one below: https://doc.lts.it/docsite/ImageRender.aspx?ID=1e6fc6d39fbcc48e160bb6159b138212

That’s a .NET ImageRender function and if you go to that URL then it will download the image, not actually display it. As such, my bot is not displaying the image as it should based on this template I have defined in my domain.yml file:

  utter_template:
    -  text: "all the rest of the api data goes here\n"
       image: "{image_url}"

I am filling in the {image_url} from my custom action that performs the api call.

Can anyone help me with this?

What channel are you seeing this behaviour on?

Facebook messenger

Also in the terminal it appears like this: 16

Not sure about this, but this means that it is not being set from the custom action. Can you use variables to fill an image slot in rasa, because it doesn’t recognize the expression inside the curly brackets as an image. I even tried with some other variables that I am filling in the - text part of my template and it still doesn’t work.

actually no as of now you can’t use variables for image urls, you can just define the image URL in your custom action though

By that you mean defining the image itself in the custom action? I can’t really understand, could you give me some sort of a simple example(if that’s not too much work).

(Small bump) Does anyone have any solution/fixes for this?

No I mean just using the url you need in the custom action, so using image = custom-url.com and then dispatcher.utter_attachment(image) from the custom action

I just tried this and by using dispatcher.utter_attachment(image) and in Messenger i just get something like this:

Is it the normal behaviour, because I am expecting the actual image to show up here. Here’s the custom action code snipppet: for item in result[‘Items’]: image = “https://python.rs/pylogo.png” dispatcher.utter_attachment(image) return []

What RASA means with image and buttons are just texts or tags for you to recognize and use them in your UI. When you set and get a image or button element in rasa, it just sends as key:value item in a JSON format. This is done so your UI can understand an image/button has been sent and you have to make the UI to take appropriate actions! RASA does not send any image file as it is. You just specify the url of the image file and let your UI take care of rendering it!

So if you have images in a location, enter the respective URL in the dispatcher.utter_attachment(imageurl). When you get the JSON reply make your UI, example Javascript, to extract the key:value in the JSON to display the image! IN cmdline this does not work as it cannot display images but rather displays that tag!

[
	{
		"recipient_id": "default",
		"text": "Here to help as always!"
	},
	{
		"buttons": [
			{
				"payload": "/where_is",
				"title": "Ask where we are!"
			},
			{
				"payload": "/get_form",
				"title": "ask for a form!"
			},
			{
				"payload": "/get_daily_list{\"list_value\":\" \"}",
				"title": "need fott or shipping lists?"
			},
			{
				"payload": "/what_can_i_do",
				"title": "need help just ask!"
			},
			{
				"payload": "/get_diagram{\"diagram_item\":\" \"}",
				"title": "get diagrams/images!"
			},
			{
				"payload": "/get_pass_info",
				"title": "ask for a pass"
			},
			{
				"payload": "/whats_the_weather",
				"title": "Weather!"
			},
			{
				"payload": "/how_to_contact",
				"title": "Contacts"
			}
		],
		"recipient_id": "default",
		"text": ""
	}
]

This is the response given by RASA server which is JSON. (I didnt use images but its the same as buttons) In the UI i take the values in button key and do the necessary function to display the buttons! The same goes for images! I hope you understood what I am trying to say…not very good at explaining things!

I think I sort of understand what you are trying to say, thing is that I only have the domain file to “define the UI” and then Messenger interpreting the messages.

So in this case, I would need a way to specify that my payload is an image and I can do that by using the image definition as specified in the domain format. The problem is that I can not control that from my custom action. So I need to use this utter_attachment(url) method. So far that’s what I have understood perfectly. As for what you said, I wouldn’t know how to tell my bot that the attachment coming from the action is an image.

That’s what I am saying! The response from RASA is a JSON content. In the JSON file you will have a key called “attachment”. Extract the value in that key (the image URL) and your UI will display the image!

As you can see, my action gives 2 attachments which is given in the JSON. The attachment key is extracted and you will get the image URL!

{
	"attachment": "blah.png",
	"recipient_id": "default"
},
{
	"attachment": "blah2.png",
	"recipient_id": "default"
}

So suppose you are using ajax and jquery to call your RASA chatbot. You can iterate over the successful response and extract all the attachment keys with its value. I haven’t used messenger so I don’t know how to display it in that.

I hope you got it

Thanks again Stalin. I get what you are saying, but I can’t seem to find a way to make this work on Messenger. After all if you are not using your own custom UI then extracting that key you get from the JSON response is not trivial. Anyways I’m to understand how can I get this to work but for the moment no solution.

Hey can you tell me how to get the rasa reponses as json