Can Rasa detect image as an input type and how to detect it?

Rasa Open Source
1

Hello,

I am new to rasa, and I’m working on a chatbot. The question is if Rasa can detect image as an input. If the customer responds with an image, can rasa detect the input type as an image for which a custom action can be triggered? The slot type “any” can help, but how to tell the bot that the input is an image. How to detect or identify input types which are images or links of webpages?

Thanks in advance!

2

I don’t think by default it accepts attachments but you can configure it in a custom channel for example

Consider creating a custom channel like rest.py Add a method that extracts the uploaded attachment. I would recommend passing the image as bytes in the attachment which reduces request size, extract it in the /webhook route of the channel near this line, once you have the image you can then save it in a shared object storage and pass the path in the metadata of the UserMessage component. This way your custom action will have the image path.

Alternatively you can do the same in your browser by saving the image in an object storage and providing the image path URI in the metadata of the RASA API so your custom action can access it.

Only difference between the two is that with the custom channel approach, you can do some post-processing to your image before providing the path to the custom action and in some cases, if you want to handle the image completely separately like in cases of validation(like upload your ID card during the flow to validate account creation), you can just save it with a correct tag so later KYC can process it.

smth like that

1 Like
3

@koaning, could you please also look into this question if any other trick or methodology can be added to the response given by Souvik?

4

I think Souvik is spot on there. To my knowledge Rasa does not allow attachments in messages, unless there’s a custom webhook in there.

1 Like