How to handle user input images in rasa

Rasa Open Source
14

@Sandares Might abit late on this but have a solution that worked for telegram.

There are 2 part to this and was suggest in the following 2 post: User information with Telegram integration Working with User Images Sent to the Bot

Solution:

  1. Edit the get_metadata function in channel.py (found in rasa>core>channels) to return the entire user message. Here is an example of the function as shared in the first link:

def get_metadata(self, request: Request) → Optional[Dict[Text, Any]]: metadata=request.json return metadata

The purpose of editing this function is to get the entire payload response set to the tracker object. That way if an image is share, the payload will contain the file_id value which can be used to download/process an image

  1. Add a condition to detect an image in the telegram connector, telegram.py(found in rasa>core>channels). In the TelegramInput class, add the following in the blueprint function:

Already in the script

            else:
                msg = update.message
                if self._is_user_message(msg):
                    text = msg.text.replace("/bot", "")
                elif self._is_location(msg):
                    text = '{{"lng":{0}, "lat":{1}}}'.format(
                        msg.location.longitude, msg.location.latitude
                    )

Add to the script

                elif msg.photo is not None: # check if message contains an image
                    text = 'img.jpg' # set the text to the file name

The condition msg.photo will get the connector to check if an image has been shared to the payload and return text = ‘img.jpg’ if there is an image. The text value can be any value you want to set.

After configuring the above, your bot will be able to detect an image shared via telegram.

In your nlu file, create an intent to detect when the image is shared. The nlu should contain the value ‘img.jpg’.

To get the file_id value, you can extract if from tracker.events[i][‘metadata’][‘message’][‘photo’][0][‘file_id’] where i is the number the events in the tracker. Iterate through all the events till you get the file_id.

Hope this helps :grinning: