Handling files on the bot

Hello,

I have a feature where the user can start a conversation by sending a file (pdf or doc) on a platform say, Slack. I have a few questions for the same:

  1. How to detect that user has sent a file and create appropriate intents for the same?
  2. How do I extract the file from the chat and process it?
  3. Is there a way to save a file that the bot receives?

Thank :slight_smile:

File uploads wouldn’t be handled by Rasa. You’re UI component or a middleware layer in front of the Rasa channel would need to deal with the file and then initiate the dialog with Rasa.

The message to Rasa might include metadata that has the file name/url and state of the upload.

1 Like

Ahh! :frowning:

Can Rasa detect if the user input is of type file? I mean, if user sends a file to the Whatsapp bot, can Rasa detect that this is of type file and from there we can create some custom action to handle it?

Additionally, if the answer is yes, can we store the the input in slot and process it somehow?

Any lead would be helpful :slight_smile:

You’ll have to look into the options for WhatsApp. There is no direct integration between WhatsApp and Rasa that I’m aware of because of the restrictions WhatsApp places on the use of their API. Here’s a recent post from another user.

We’ve created and are maintaining a connector here: GitHub - praekeltfoundation/turn-rasa-connector: A Rasa Connector for https://www.turn.io/ , it’s very specific to the API provided by Turn: https://www.turn.io/ , but should work with any aggregator that provides the same API as a WhatsApp direct integration.

For documents, the connector will place the document details in metadata, and if there’s a caption, will place the caption as the message text.

While accessing this in a custom action should be easy, I don’t know if it’s possible to do using stories etc.

Maybe a custom mapping that runs a custom action if there’s a document attached might be a possible solution?

Thank you for the response. @rudi I have done the integration of the bot using Twilio already. The next requirement was to receive a file from the user and store it in the database. I was looking into this rasa/twilio.py at master · RasaHQ/rasa · GitHub to see if I can make some use something to get the file that was sent.

We can create an event handler using Twilio (GitHub - TwilioDevEd/whatsapp-media-tutorial-flask: A small Flask application to demonstrate receiving, saving, and replying with media to incoming WhatsApp media messages) rest api that can give us the file but I am not sure how should I incorporate it in the Rasa. Should the event handler listen to /webhooks/twilio/webhook/ for every message, and call and action whenever there is a media type associated with a message? How will that action be triggered specifically for the media file? I mean what should be the intent in the story file to trigger this action?

Stories.md

    *?
    - utter_capture_media_file

That’s where I have no idea. I don’t think that you can have an intent work on media, intents work on text content only.

I think the best way to do this is a custom mapping, so that if there’s an attachment you can always trigger a custom action, before it gets to all the ML things.

You can see here in the Rasa Mapping Policy, how they check if it matches, then they set the action, and set the probability to 1: rasa/mapping_policy.py at master · RasaHQ/rasa · GitHub . So I was thinking that you could do a similar thing. Create a custom policy, that if there’s an attachment, it triggers a custom action with probability of 1.

Hey @rudi, thanks for your suggestions. I achieved this by overriding the twilio connector.