Google Assistant Integration

Hi there,

This is a follow up of this issue: Custom Input Channel in Docker. I felt this is not really related, so I opened a new issue. If you are looking for any details of my implementation, I might have already shared it over there though.

I am trying to get the Google Assistant Integration into the Rasa framework working. I am using docker-compose and have a working Telegram Chatbot running in the same pipeline. To get an idea how to implement this I followed this guide: Going beyond ‘Hey Google’: building a Rasa-powered Google Assistant

The communication between the Bot and the Google API is working fine. If I answer with a static message hardcoded in the bot it works like a charm. But when invoking the on_new_message function, I get the following error:

rasa_1  | 2019-05-24 13:21:12 WARNING  rasa.core.agent  - Passing a text to `agent.handle_message(...)` is deprecated. Rather use `agent.handle_text(...)`.
rasa_1  |
rasa_1  | WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
rasa_1  | For more information, please see:
rasa_1  |   * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
rasa_1  |   * https://github.com/tensorflow/addons
rasa_1  | If you depend on functionality not listed there, please file an issue.
rasa_1  |
rasa_1  | [2019-05-24 13:21:12 +0000] [1] [ERROR] Exception occurred while handling uri: 'http://[censored].ngrok.io/webhooks/google_assistant/webhook'
rasa_1  | Traceback (most recent call last):
rasa_1  |   File "/usr/local/lib/python3.6/site-packages/sanic/app.py", line 917, in handle_request
rasa_1  |     response = await response
rasa_1  |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
rasa_1  |     return self.gen.send(None)
rasa_1  |   File "/app/custom/google_assistant_channel.py", line 48, in receive
rasa_1  |     await on_new_message(UserMessage(query, output_channel, user_id))
rasa_1  |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
rasa_1  |     return self.gen.send(None)
rasa_1  |   File "/usr/local/lib/python3.6/site-packages/rasa/core/channels/channel.py", line 68, in handler
rasa_1  |     await app.agent.handle_message(*args, **kwargs)
rasa_1  |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
rasa_1  |     return self.gen.send(None)
rasa_1  |   File "/usr/local/lib/python3.6/site-packages/rasa/core/agent.py", line 411, in handle_message
rasa_1  |     message, message_preprocessor=message_preprocessor, **kwargs
rasa_1  |   File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
rasa_1  |     return self.gen.send(None)
rasa_1  |   File "/usr/local/lib/python3.6/site-packages/rasa/core/agent.py", line 517, in handle_text
rasa_1  |     msg = UserMessage(text_message.get("text"), output_channel, sender_id)
rasa_1  | AttributeError: 'UserMessage' object has no attribute 'get'

I call it like follows (some parts are left out for simplicity):

class GoogleAssistantInput(InputChannel):
    def blueprint(self, on_new_message):
        """
        Sub-routing for Google Assistant Input Channel
        """
        google_assistant_webhook = Blueprint('google_assistant_webhook', __name__)

        @google_assistant_webhook.route('/webhook', methods=['POST'])
        async def receive(request):
            """
            Webhook endpoint that is called by Google API
            """
            payload = request.json
            logger.debug(f'The received request from the Google API was: {payload}')
            user_id = payload['user']['userId']
            intent = payload['inputs'][0]['intent']
            query = payload['inputs'][0]['rawInputs'][0]['query']

            if intent == 'actions.intent.MAIN':
                message = '<speak>Hello! <break time="1"/> You can start by saying start.</speak>'
            else:
                output_channel = CollectingOutputChannel()
                await on_new_message(UserMessage(query, output_channel, user_id))
                message = [message['text'] for message in output_channel.messages][0]
            ...

For the first message exchange with the Google API we enter the if part of the if-else block and this works fine as we simply return a static answer. For the next exchange we enter the else part and now want to invoke the bot by using the on_new_message function. With the await keyword I get the above error. If I do not use the await keyword the output_channel.messages are always empty.

I would love to know what I am doing wrong… :wink:

@erohmensing I have an update:

The problem seems to be in the handle_message function in the Agent class, where we have:

return await self.handle_text(
    message, message_preprocessor=message_preprocessor, **kwargs
)

message is of type UserMessage, where the handle_text function expects a string. It is also a bit strange that somewhere in the custom channel pipeline you still call handle_message even though it is deprecated.

Hm, handle message itself isn’t deprecated, just passing “text” to it is. They’re two different functions, one processes input of type text and the other of type UserMessage. I suppose the code from that Google Assistant tutorial is now out of date?

The strange thing is that you’re passing a UserMessage to handle_message, so it shouldn’t go into this logic where it hits this line 411

rasa_1  |   File "/usr/local/lib/python3.6/site-packages/rasa/core/agent.py", line 411, in handle_message

Yeah the code from the tutorial is quite outdated. This is why I am looking at the RestInput as an example. The error does not happen in my code, it happens when calling on_new_message somewhere downstream. And I tripple checked that I call it in the same way.

If you look at the stack trace you see that it goes through line 411 in the handle_message function, which explicitly checks that message is not of instance UserMessage but seconds later it crashes because it is a UserMessage object. I am baffled, how is this possible? Are there two classes called UserMessage?

I think I found the issue. Locally I am still using the rasa_core pypi package instead of rasa package. And something of instance rasa_core.channels.channel.UserMessage is not of instance rasa.core.channels.channel.UserMessage. This is why we enter this if block and the error arises.

If you are trying to use rasa, you should update any rasa_core imports to rasa.core. Why aren’t you running the same package/version locally? That doesn’t seem conducive to good testing

For my first tests I was running a version on my local machine with the rasa_core pypi package. After these tests I switched to a docker-compose setup so my local version does not matter except for import warnings in my IDE. And this is exactly what happened here. I copied the outdated code from the example and my IDE did not complain about the path as it is valid in this legacy version. But unfortunately this would also have happened if I had configured the docker-compose image as remote interpreter. When running the container it did not crash on importing rasa_core.channels.channel.UserMessage so the legacy package must be imported in the Docker image. This means that my IDE would not have complained about the path not existing either.

It is an ugly side effect of the migration from rasa_core to rasa in combination with careless code copying from an old example :wink:

Gotcha, yeah it’s always a little dangerous to copy old code that isn’t maintained within the repo cross-version :smile:

Have you been able to solve the issue then?

Yes :slight_smile: thanks again for your help! I will see if I can provide the Google Assistant Channel as an extension to the repo.

1 Like

@martyer did you get Google Assistant working with Rasa 1.0.x?