How to handle user input images in rasa

Rasa Open Source
1

Hey there! I’ve got some issues when handling the user input images. After connecting to a messaging channel like telegram, WhatsApp I want to get some images from users when running a form. As a example when I making a online pharmacy chatbot I had to get user inputs like name, location using rasa forms and finally I had to get the prescription as a image. But after inputting a image the bot stopped working. How can I handle this?
Another thing is I want to send images and pdf document via this messaging channels to users. In this case I able to the links of this images and documents… But I want to sent the image via WhatsApp and telegram. How can I do this? Wishing a powerful support from you

2

@Sandares Hey! is there any code you have developed or able to perform some steps?

@Sandares Please refer this hope this link rasa/telegram.py at main · RasaHQ/rasa · GitHub will help you, but you need to understand the code :wink: @Sandares Custom Connectors

1 Like
3

@nik202 Thanks a lot for your support

4

@Sandares welcome! please close this thread with the solution for others.

5

@nik202 Could you please elaborate the usage of how to take image or picture as user input from telegram bot and store ? I have used custom connectors for telegram unable to receive messages

6

@Hiranmai I’m afraid, I not personally implemented your use case, but it all depends on Telegram interface, if they allow to send or received images then for that you need to check the custom connector for that. But why you need that, can you share if you don’t t mind?

7

@nik202 In my usecase user need to upload an image from telegram bot and then it needs to be sent to other API from custom actions

8

@Hiranmai means you are extracting the information from the image or its just send by user and store in your database ?

9

@nik202 need to take the uploaded image from the bot and send it an API later it will be stored in database.

10

@Hiranmai means only stored ok.

11

Hi @Sandares, I hope you doing fine. Can I request you to please close this thread with the solution for others. Thanks.

12

Hi @nik202, I’m doing fine. I was unable to find out a way to handle user input images. And there’s simple way to send images to users. By adding image: tag after - text: in utters and given the link of the image is the easiest way send images. By adding the the telegram channel to credentials.yml file as the rasa docs it send images. But I couldn’t find the way to handle input images. Sorry for that. :expressionless:

1 Like
13

@Sandares I am sorry to hear that buddy! Please mark this thread as pre- solution :slight_smile: for others How to handle user input images in rasa - #12 by Sandares and keep safe!

14

@Sandares Might abit late on this but have a solution that worked for telegram.

There are 2 part to this and was suggest in the following 2 post: User information with Telegram integration Working with User Images Sent to the Bot

Solution:

  1. Edit the get_metadata function in channel.py (found in rasa>core>channels) to return the entire user message. Here is an example of the function as shared in the first link:

def get_metadata(self, request: Request) → Optional[Dict[Text, Any]]: metadata=request.json return metadata

The purpose of editing this function is to get the entire payload response set to the tracker object. That way if an image is share, the payload will contain the file_id value which can be used to download/process an image

  1. Add a condition to detect an image in the telegram connector, telegram.py(found in rasa>core>channels). In the TelegramInput class, add the following in the blueprint function:

Already in the script

            else:
                msg = update.message
                if self._is_user_message(msg):
                    text = msg.text.replace("/bot", "")
                elif self._is_location(msg):
                    text = '{{"lng":{0}, "lat":{1}}}'.format(
                        msg.location.longitude, msg.location.latitude
                    )

Add to the script

                elif msg.photo is not None: # check if message contains an image
                    text = 'img.jpg' # set the text to the file name

The condition msg.photo will get the connector to check if an image has been shared to the payload and return text = ‘img.jpg’ if there is an image. The text value can be any value you want to set.

After configuring the above, your bot will be able to detect an image shared via telegram.

In your nlu file, create an intent to detect when the image is shared. The nlu should contain the value ‘img.jpg’.

To get the file_id value, you can extract if from tracker.events[i][‘metadata’][‘message’][‘photo’][0][‘file_id’] where i is the number the events in the tracker. Iterate through all the events till you get the file_id.

Hope this helps :grinning: