Handle fileupload from Teams channel

Rasa Open Source
1

Hi,

I integrated rasa in Teams with Microsoft Bot Framework. This was super easy and I can chat with my bot. In our use-case we want to upload images within the conversation and store them in a database.

In a first test, I printed tracker.latest_message after an image was send, without any text. The result was:

{‘intent’: {‘name’: ‘bye’, ‘confidence’: 0.4087781012058258}, ‘entities’: [], ‘text’: ‘’, ‘message_id’: ‘628687d5083a469394a04ffce2c2e5fa’, ‘metadata’: {‘attachments’: [{‘contentType’: ‘application/vnd.microsoft.teams.file.download.info’, ‘content’: {'downloadUrl’: ’ → Download link to the image <----’, ‘uniqueId’: ‘8600ea17-739f-4722-b30f-9453cebf89e4’, ‘fileType’: ‘png’}, ‘contentUrl’: ‘…’, ‘name’: ‘version (1).png’}, {‘contentType’: ‘text/html’, ‘content’: ‘’}]}, ‘text_tokens’: [[0, 0]], ‘intent_ranking’: [{‘name’: ‘bye’, ‘confidence’: 0.4087781012058258},…

The downloadUrl allows me to download the image and store it in the databse. But the text of this message is empty and it got classified with the intent ‘bye’.

My Querstion is: How can I train my bot on these kind of messages?

My idea was to create an intent like this, with empty text:

- intent: Fileupload

  examples: |

    - ""

    - ''

    -

Is there a better way?

Thanks!

2

Create an inform intent to capture user responses that provide information and include examples of URL’s with an entity and slot. You can see an example of an inform intent in the Rasa financial demo here.

1 Like
3

Thanks, i will try this :slight_smile:

I use the DIETClassifier for entity extraction. So if I understood it correct, DIET can extract the entity from the text and (like in my case) additional from metadata?

4

I’m not sure what your front end UI looks like, but if you have a widget for the user where they select the image and this also sends the mesage to Rasa, it could send something like this:

\upload_image{"download_url":"https://host.com/myimage.png"}

This says, invoke the upload_image intent and set the slot download_url. You could then run a custom action to handle the image.

5

Hi,

so I use MS Teams as UI for the fileupload. If I press the upload icon, the explorer pops up and I can choose a file. After selecting the file, it gets send without any text, like in this example:

msTeams
msTeams1146×373 22.2 KB

If I print tracker.latest_message, after the file upoad, I get the following output:

outputMSG
outputMSG1501×262 35.6 KB

As you can see, the text of this file upload ist empty, therefore there are no entities (like an url) as well.

The download url, which I need, is attached to the metadata. (I know how to extract this in an custom action)

But my question is, how i can train my bot (nlu examples) on “empty messages”? The upload is classified to the wrong intent “nein”/“no”.

This was my idea, but it doesn´t work well…

- intent: Fileupload

  examples: |

    - ""

    - ''

    -
6

Did you resolve this issue?