How to interact with Rasa command prompt for the Two stage fallback policy buttons?

Rasa Open Source
1

I am trying to use Rasa’s two stage fallback policy by following the doc here.

I have a test “Joke” agent with following files:

domain.yml

intents:
- greet
- goodbye
- thanks
- deny
- joke
- name
- out_of_scope


entities:
- name


slots:
  name:
    type: text


actions:
- utter_name
- utter_thanks
- utter_greet
- utter_goodbye
- action_joke
- action_my_out_of_scope_handler_fallback


templates:
  utter_name:
  - text: "Hey there! Tell me your name."

  utter_greet:
  - text: "Nice to you meet you {name}. How can I help?"

  utter_goodbye:
  - text: "Talk to you later!"
  
  utter_thanks:
  - text: "My pleasure."

policies.yml

policies:
  - name: KerasPolicy
    epochs: 100
    max_history: 5
  - name: TwoStageFallbackPolicy
    nlu_threshold: 0.5
    core_threshold: 0.5
    fallback_core_action_name: "action_default_fallback"
    fallback_nlu_action_name: "action_default_fallback"
    deny_suggestion_intent_name: "out_of_scope"
  - name: MemoizationPolicy
    max_history: 5
  - name: FormPolicy

actions.py

# -*- coding: utf-8 -*-
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals

import logging
import requests
import json
from rasa_core_sdk import Action

logger = logging.getLogger(__name__)


class ActionJoke(Action):
    def name(self):
        # define the name of the action which can then be included in training stories
        return "action_joke"

    def run(self, dispatcher, tracker, domain):
        # what your action should do
        request = json.loads(
            requests.get("https://api.chucknorris.io/jokes/random").text
        )  # make an api call
        joke = request["value"]  # extract a joke from returned json response
        dispatcher.utter_message(joke)  # send the message back to the user
        return []

class ActionDefaultFallback(Action):
	def name(self):
		return "action_my_out_of_scope_handler_fallback"
		
	def run(self, dispatcher, tracker, domain):
		dispatcher.utter_message("Handing over to a human agent.")

I have trained NLU and Core and started the Rasa server.

However, when I try to interact with the Rasa bot in the console, after I reply with the button option (see below snippet), Rasa does not detect the choosed intent on subsequent similar utterance.

127.0.0.1 - - [2019-07-08 12:00:11] "POST /webhooks/rest/webhook?stream=true&token= HTTP/1.1" 200 335 1.657566
Your input ->  I want to go from A to B
Did you mean 'joke'?
Buttons:
1: Yes (/joke)
127.0.0.1 - - [2019-07-08 12:00:18] "POST /webhooks/rest/webhook?stream=true&token= HTTP/1.1" 200 316 0.030916
2: No (/out_of_scope)
Your input ->  1
127.0.0.1 - - [2019-07-08 12:00:23] "POST /webhooks/rest/webhook?stream=true&token= HTTP/1.1" 200 154 0.027924
Your input ->  I want to go from A to B
127.0.0.1 - - [2019-07-08 12:00:25] "POST /webhooks/rest/webhook?stream=true&token= HTTP/1.1" 200 154 0.025931
Your input ->  I want to go from A to B
127.0.0.1 - - [2019-07-08 12:00:30] "POST /webhooks/rest/webhook?stream=true&token= HTTP/1.1" 200 154 0.033909
Your input ->  I want to go from E to F
127.0.0.1 - - [2019-07-08 12:00:35] "POST /webhooks/rest/webhook?stream=true&token= HTTP/1.1" 200 154 0.031914
Your input ->  I want to go from A to B
127.0.0.1 - - [2019-07-08 12:00:43] "POST /webhooks/rest/webhook?stream=true&token= HTTP/1.1" 200 154 0.032912

Am I missing something here? I have also tried with both “Yes”, and “No” replies to the button prompts in console. But the behavior is same.

2

@utkalsinha Did you also try typing in /joke instead of 1 or Yes? If not can you try and let me know if that works? Thanks.

3 
/joke did not worked, instead what happened was the "joke" intent was called. An on giving the exact same utterance, it is again asking for "Did you mean joke?" prompt. Also, currently, if I enter /<any intent name>, it just invokes the action corresponding to that intent. Please help.
4

Can you please start the bot in debug mode and share the logs? So something similar to rasa shell --debug. That might help us figure out what exactly is going on. Thanks.

5

Sure, will do that. But before that, just wanted to make sure one thing:

So according to the TwoStageFallback policy, when the user say, “yes”, does my Rasa bot gets retrained with this new user utterance for that intent so that next time Rasa bot detects the same utterance for the same intent and does not prompt back for "Did you mean …?"

6

@utkalsinha No, your model does not get retrained. If you want to avoid questions like “Did you mean …?” you maybe should think about adding more training data to your bot and retrain it. The TwoStageFallback does not retrain the model in any way.