Custom action not working with fallback policy

Rasa Open Source
1

I am following rasa weatherbot tutorial. I am able to get the results properly and everything works fine but since i have implemented fallback policy my custom action is not working when i set core_threshold & nlu_threshold >0.3. when both are set in 0.3 everything works fine but as soon as they are increases even when my confidence score is >.9 i am only getting fallback messages. Please help me.

Domain.yml

%YAML 1.1
---
actions:
- utter_greet
- utter_goodbye
- utter_ask_location
- action_weather
- utter_default
entities:
- purchase
- booking
- location
- goodbye
intents:
- inform
- greet
- goodbye
slots:
  location:
    type: text
templates:
  utter_ask_location:
  - text: In what location ?
  utter_default:
  - text: Sorry i cant understand
  utter_goodbye:
  - text: Talk to you later.
  - text: Bye Bye :(
  utter_greet:
  - text: Hello ! How can i help you

Stories.md

story 01

  • greet
    • utter_greet

story 02

  • goodbye
    • utter_goodbye

story 03

  • inform
    • utter_ask_location

story 04

  • inform
    • action_weather
  • greet
    • utter_greet
  • inform
    • utter_ask_location
  • inform
    • utter_ask_location
  • inform
    • utter_ask_location
  • inform
    • action_restart

Generated Story 8218453580895522855

  • greet
    • utter_greet
  • goodbye{“booking”: “book”}
    • action_default_fallback
    • rewind
  • goodbye{“purchase”: “buy”}
    • action_default_fallback
    • rewind
  • goodbye{“booking”: “train”}
    • action_default_fallback
    • rewind
  • inform
    • utter_ask_location
  • inform{“location”: “newyork”}
    • slot{“location”: “newyork”}
    • action_weather
  • inform{“location”: “london”}
    • slot{“location”: “london”}
    • action_weather
    • slot{“location”: “london”}
  • goodbye{“goodbye”: “cya”}
    • utter_goodbye

dialogue_management.py

from future import absolute_import from future import division from future import print_function from future import unicode_literals

import logging

import rasa_core

from rasa_core.agent import Agent

from rasa_core.policies.keras_policy import KerasPolicy

from rasa_core.policies.memoization import MemoizationPolicy

from rasa_core.interpreter import RasaNLUInterpreter

from rasa_core.utils import EndpointConfig

from rasa_core.run import serve_application

from rasa_core import config

from rasa_core.policies.fallback import FallbackPolicy

from rasa_core.policies.keras_policy import KerasPolicy

logger = logging.getLogger(name)

def train_dialogue(domain_file=‘C:\RASA_nlu\domain.yml’, model_path=‘C:\RASA_nlu\models\dialogue’, training_data_file=‘C:\RASA_nlu\data\stories.md’):

 fallback = FallbackPolicy(fallback_action_name="action_default_fallback",core_threshold=0.4,nlu_threshold=0.5)
 agent = Agent(domain_file, policies=[MemoizationPolicy(), KerasPolicy(max_history=3, epochs=200, batch_size=50),fallback])
 data = agent.load_data(training_data_file)

 agent.train(data)
 agent.persist(model_path)
 return agent

def run_weather_bot(serve_forever=True):

 interpreter = RasaNLUInterpreter('C:\\RASA_nlu\\models\\nlu\\default\\weathernlu')
 action_endpoint = EndpointConfig(url="http://localhost:5055/webhook")
 agent = Agent.load('C:\\RASA_nlu\\models\\dialogue', interpreter=interpreter, action_endpoint=action_endpoint)
 rasa_core.run.serve_application(agent, channel='cmdline')

 return agent

if name == ‘main’:

 train_dialogue()
 run_weather_bot()
2

Hi there @indranil180, It is probably the core fallback that is invoking your fallback action, not the NLU fallback – the confidence score you’re referring to (> 0.9) is the NLU confidence. I would suggest writing more stories with interactive learning so that your memoization policy can kick in and give you better core confidence for action prediction.

1 Like
3

Thanks for your suggestion. Surely i will do that. Is there any other approach i can perform other than adding more stories ?

4

This is really the way – unless something is going specifically wrong that you can track down, your bot is predicting fallback just because it doesn’t have enough data to know what to do next.