Google & Alexa Connector

Why does Rasa not have connectors for linking up to Google Assistant and Alexa projects for voice chatbots?

I’m going to guess that it has something to do with running it as an HTTP server, but what if I want to run it in the same way as Facebook, Slack and Telegraph, by using connectors?

Someone did share a Google Action connector over on the gitter community (let’s hope this forum works better than that) but I couldn’t get it to connect my bot to a google project. Can anyone point me in the right direction, share a url of an example or tutorial, or anything that might help?

I want to connect a Google Action to my Rasa bot, and then just pass everything through to Rasa.

1 Like

You would have to look into their APIs and create a custom channel for this. We do have it in our future plans to make a tutorial about connecting Rasa to Alexa!

1 Like

It used to say the words Google and Alexa on the Rasa homepage, talking about how to build these types of bots, but actually it still isn’t possible to do so without being a Python wizard.

Yes you do need to know some python to build a connector, “wizard” is probably a bit of a stretch :smiley: you can take a look at our example connectors and their APIs, that should help. Otherwise as I said there should be a tutorial on how to do it in the near future :slight_smile:

1 Like

Where will this tutorial appear, and is there a way to be notified when it is available?

Hey @davemac, we can’t give the specific date when the tutorial is going to be published, but it is definitely on top or our priority list, so ‘soon’ is the best word to give you an indication of when it’s going to be available. We will definitely share it on the forum as well as Rasa blog so keep an eye on these two :slight_smile:

Hey all, I’ll jump right in with a related question. I’ve just discovered rasa and am stilling trying to understand what it does. I am coming from a voice assistant background and so was wondering how the integration to, say, Alexa would look. Would I implement a catch all intent (with a search query slot) in my skill and pass the text to rasa? Or would I need to create some kind of Alexa voice service tool (hardware)?

I apologise for barging in like that but it seemed to fit at least halfway :wink:

Best

ch.

No dale, jump right in, this is exactly the same question I had, do I create a Google Action, which can now only be done in Dialogflow (I think) with a single intent which passes everything through to the rasa bot for processing? Same thing for Alexa. What is the architecture, how are things piped together, and how is Rasa setup and run (Python or HTTP server) to get all this working? Hopefully the upcoming tutorial can touch on all these points.

@dale-cooper @davemac So I’ve actually set up a Google Action with a Rasa backend so perhaps I can shed some light on the topic. You’re right in that it’s a little more complicated because Rasa doesn’t have a connector for it right now.

I unfortunately couldn’t figure out how the Rasa connectors work (probably because I really know nothing about Flask blueprints). But I was able to set up a custom server that makes calls to Rasa and works perfectly fine for my current dev environment. Obviously you’ll want to do something far more robust for a production server, but this should point you in the right direction.

Step 1: Create a Google Actions project that doesn’t use DialogFlow. See https://developers.google.com/actions/sdk/create-a-project for details.

Your action.json file (aka the action package) should look something like this:

{
  "actions": [
    {
      "description": "Default Welcome Intent",
      "name": "MAIN",
      "fulfillment": {
        "conversationName": "my project"
      },
      "intent": {
        "name": "actions.intent.MAIN",
        "trigger": {
          "queryPatterns": [
            "talk to my project"
          ]
        }
      }
    },
    {
      "description": "Default Intent",
      "name": "TEXT",
      "fulfillment": {
        "conversationName": "my project"
      },
      "intent": {
        "name": "actions.intent.TEXT",
        "trigger": {
          "queryPatterns": [
          ]
        }
      }
    }
  ],
  "conversations": {
    "my project": {
      "name": "my project",
      "url": "https://yoursite.com/webhook",
      "inDialogIntents": [{
        "name": "actions.intent.CANCEL"
      }]
    }
  },
  "locale": "en"
}

Step 2: Set up the Rasa stack. I’m assuming you know how to do this already but if not, @Juste has possibly the best tutorial on her website. It’s fantastic and I highly recommend you check it out: From zero to hero: Creating a chatbot with Rasa NLU and Rasa Core – Justina Petraitytė

Step 3: Run the following code on your server.

from flask import Flask, url_for, request, json, Response, send_file
from rasa_core.agent import Agent
app = Flask(__name__)


@app.route('/')
def api_root():
    return 'Welcome'

@app.route('/webhook', methods=['GET', 'POST'])
def api_webhook():
    if request.method == 'GET':
        data = {"Hello": "World"}
        js = json.dumps(data)
        resp = Response(js, status=200, mimetype='application/json')
        return resp

    if request.method == 'POST':
        user_input = ""
        user_input = request.get_json() # This returns a python dictionary in user_input
        user_query = user_input["inputs"][0]["rawInputs"][0]["query"]
        rasa_output = agent.handle_message(user_query)
        super_string = " ".join([x["text"] for x in rasa_output])
        expect_response = True
        if user_input["inputs"][0]["intent"] == 'actions.intent.CANCEL':
            expect_response = False
            data = {
                "conversationToken": "",
                "expectUserResponse": expect_response,
                "finalResponse": {
                    "richResponse": {
                        "items": [{
                            "simpleResponse": {
                                "ssml": '<speak>' + super_string + '</speak>'
                            }
                        }],
                        "suggestions": []
                    }
                }
            }
        else:
            data = {
                "conversationToken":
                "",
                "expectUserResponse":
                expect_response,
                "expectedInputs": [{
                    "inputPrompt": {
                        "richInitialPrompt": {
                            "items": [{
                                "simpleResponse": {
                                    "ssml": '<speak>' + super_string + '</speak>'
                                    # "textToSpeech": super_string,
                                    # "displayText": super_string
                                }
                            }],
                            "suggestions": []
                        }
                    },
                    "possibleIntents": [{
                        "intent": "actions.intent.TEXT"
                    }]
                }]
            }

        js = json.dumps(data)
        resp = Response(js, status=200, mimetype='application/json')
        resp.headers['Link'] = 'https://www.yoursite.com'
        return resp


if __name__ == '__main__':
    agent = Agent.load("./models/dialogue", interpreter="./models/nlu/default/yournlu")
    app.run(host='0.0.0.0')

Hope that helps! Let me know if you have any questions about the stuff I’ve posted! :slight_smile:

@ikhatri Thanks for your reply. I need to find some time to understand what you are doing, but on a quick read it has many items I know I had to do, but don’t know how to, or the details of how to, your reply ticks a lot of boxes.

In the meantime I have also found the google connector written by a clever guy on the rasa gitter forum. I cannot find his name, nor find anything in gitter because its such a poor interface, which is why I am so happy this forum has started, but if he ever shows up here he will know it was credit to him.

I couldn’t get this to work, mainly because google doesn’t supply all the tokens that are included, and I couldn’t work out how to wire it all together. Perhaps this could be a starting point for a community Google Connector?

Blockquote

from future import absolute_import from future import division from future import print_function from future import unicode_literals

import logging

from builtins import str from flask import Blueprint, request, jsonify

from rasa_core.channels.channel import UserMessage, OutputChannel from rasa_core.channels.rest import HttpInputComponent

logger = logging.getLogger(name)

class SlackBot(OutputChannel): def init(self, slack_verification_token, channel): self.slack_verification_token = slack_verification_token self.channel = channel

def send_text_message(self, recipient_id, message):
	from slackclient import SlackClient
	text = message
	# recipient = recipient_id
	
	CLIENT = SlackClient(self.slack_verification_token)
	CLIENT.api_call('chat.postMessage', channel = self.channel, text = text, as_user = True)

class SlackInput(HttpInputComponent): def init(self, slack_dev_token, slack_verification_token, slack_client, debug_mode): self.slack_dev_token = slack_dev_token self.debug_mode = debug_mode self.slack_client = slack_client self.slack_verification_token = slack_verification_token

def blueprint(self, on_new_message):
	from flask import Flask, request, Response
	slack_webhook = Blueprint('slack_webhook', __name__)
	
	@slack_webhook.route('/', methods = ['GET'])
	def health():
		return jsonify({'status':'ok'})
		
	@slack_webhook.route('/slack/events', methods = ['POST'])
	def event():
		logging.info('Received new event ' )
		logging.info(request.json.get('type'))
		logging.info(request.json.get('action'))
		logging.info(request.json.get('responseId'))
		queryResult = request.json.get('queryResult')
		action = queryResult.get('action')
		queryText = queryResult.get('queryText')
		logging.info(action)
		logging.info(queryText)


		if request.json.get('type') == 'url_verification':
			return request.json.get('challenge'), 200
			
		if request.json.get('token') == self.slack_client and request.json.get('type') == 'event_callback':
			data = request.json
			messaging_events = data.get('event')
			channel = messaging_events.get('channel')
			# user = messaging_events.get('user')
			text = messaging_events.get('text')
			bot = messaging_events.get('bot_id')
			if bot == None:
				on_new_message(UserMessage(text, SlackBot(self.slack_verification_token, channel)))
				
		return Response(), 200
		
	return slack_webhook

Hey @davemac no problem! I’m glad I could help. After you go through the code, let me know if you have any questions and I’d be happy to walk you through it.

As far as the blueprint you’ve posted goes, it looks like it’s just the slack connector, which would explain why google actions didn’t have the required tokens. I will take a look into it later though, and perhaps I’ll throw in a pull request for a google actions connector if I’m able to finish :slight_smile:

Hi Ishan,

Thank you for the flask code. I am trying to use it with Alexa. When I run the flask and call it from Alexa, the chatbot model returns a reply the first time but everytime after that , it returns empty response… unless i restart the flask. That means that I cannot have a conversation with the bot beyond its first response. Have you encountered this problem? Thank you, Shruti

Just adding to previous message- Basically is the bot can’t identify intent , it returns empty string(rasa_output=[]). I want it return some value (like “Sorry, I didn’t get you”) instead of empty response. Thank you, Shruti

Hi Shruti,

Firstly sorry for the late reply! I hope you’ve figured out your problem already. If not, there are a number of things that could be going wrong. It’s really hard to debug from the information you’ve provided but if I was in your shoes here are the things I’d try first.

  1. Print out the input string in the flask server file to ensure that your request is being delivered to the server and parsed correctly
  2. Print out the result of rasa’s agent.handle_message() function to make sure that rasa is providing an output to your server
  3. Use a third party service like API tester to make sure that your server is returning the output phrase in the proper format that Alexa can parse

I hope that helps!

- Ishan

Hi Ishan,

Thank you for the options to try. My flow works fine from Alexa. Its just the lack of training data for bot. I just need to configure a default intent for the bot so that it can fallback to it incase it can’t identify any of my specific intents. Any idea how I can do that?

Best, Shruti

Hi Shruti,

Firstly that should never happen.The ML model can never not identify the intent. It always has some guess, the confidence value will just be really really low if it’s totally unrelated to your data. The only thing is that if you have a fallback intent, instead of getting out some random gibberish when you ask your bot something unrelated, you’ll receive a fallback action when the confidence is too low.

Hopefully that made sense.

I will answer your question though. Here’s how to add a fallback. (also see: https://rasa.com/docs/core/fallbacks/)

In your training file where you call agent.train() add the following lines.

fallback = FallbackPolicy(fallback_action_name="utter_oops", core_threshold=0.15, nlu_threshold=0.15)
agent = Agent("your_domain.yml", policies = [MemoizationPolicy(max_history = 2), KerasPolicy(), fallback]) # all you're doing here is adding the fallback everything else should be the same as before
agent.train() # Just use whatever you had before for args to this function
1 Like

Hi Ishan,

If the model has to identify an intent everytime(even if crap), I am not sure why rasa_output is empty sometimes. Btw can I declare my fallback action in my domain file and action file and then while training give it the --fallback_action_name?

Cheers, Shruti

Again, sorry for the late response.

You are correct. I’m not sure why your rasa_output is empty either, you should look into debugging other parts of your code.

Yes I believe you can do that with your fallback action, though I’m not 100% sure. Give it a try and see if it works!

HI Ishan,

I want to move from Alexa skill to Google home action. In my current setup , when I say something to alexa, Alexa passes it as is to my rasa bot, bot responds and alexa speaks out that response.

Can I do this exact same thing with Google home? If yes, how do I do that? Is your above code part of the same solution I am looking for?

Thank you, Shruti

We just published a blog post about this the other day! https://blog.rasa.com/going-beyond-hey-google-building-a-rasa-powered-google-assistant/

4 Likes