Working with User Images Sent to the Bot

Rasa Open Source
#1

Hey everyone!

We are currently working with something like this:

  1. user intent is to have information about their status
  2. bot utters message asking for their ID image
  3. they take a picture/send it
  4. image gets sent to an API - custom action
  5. bot utters an answer / returns user information

I was wondering if there is possible to recognize an user sent message like that (image)? (or in other words how to make it possible to use that message/image to make that next custom action?) I’m aware that rasa works with NLU and not image recognition, but I’m stuck in finding an solution to make that example above work while still working with rasa. Could anyone help with ideas?

Thanks a lot in advance! :relaxed:

1 Like
#2

Hey, right now I think we just ignore that input. You’d have to customise the relevant connector for the messaging platform to actually receive that image. Something like this in the telegram connector for example: rasa_core/telegram.py at master · RasaHQ/rasa_core · GitHub

#3

hii @akelad, rasa_core/telegram.py at master · RasaHQ/rasa_core · GitHub this shows 404 error not found while opening

#4

Here’s the updated link: rasa/telegram.py at master · RasaHQ/rasa · GitHub

#5

I am working on similar Rasa chat bot which has to take a picture of the document the user presents, and the bot has to click a picture via the webcam. I am new to Rasa and do not know how to work around with custom actions, if there are any implementations I can use for reference, it would be of great help.

Thanks in advance! :smiley:

#6

@carla.lmeida , did you find any solution?i have similar requiremnt with Telegram. Thanks

#7

@kabeer I made a custom action where the bot triggers the camera and takes a picture, when you hit a key on the keyboard.

photo.py (1.8 KB)