Rasa never send Fallback Intent on NLU response

Getting Started with Rasa
1

I was experimenting with the NLU parts of Rasa. In my config.yml I have a Fallback Policy with a 70% threshold.

When I either run a server or try the NLU only shell, I never get a Fallback Intent on the response even if the confidence is way below 70%. It does not make sense for the NLU only mode to use the Fallback policy, so I understand why this might happen.

But I am migrating from Dialogflow, so I would like the NLU response to return a Fallback Intent if the sent message does not fall under any defined intent. How may I do this?

2

Yes, using rasa shell nlu you won’t see any action predicitons, FallbackPolicy is only for dialogue management. What does your config look like? If you have a policies section with FallbackPolicy, then are you trying to also use dialogue management? Can you give an example of the bot behaviour you are trying to achieve?

3

Sure. In short, I’m trying to emulate Dialogflow behavior. I’ll send a raw text to Rasa. Rasa will parse it and give me the intents/entities and also return a response associated with the detected intent.

For, example, let’s say, Hello is trained to generate welcome_intent. I’ve also set welcome_intent to trigger utter_welcome_intent response. Let’s say this response contains two utterances,

Hi, good to meet you and I'm Rasa, how can I help you?

So, if I send a message saying Hello to Rasa, I want Rasa to send me the NLU parsing data and a random response from the two examples I’ve set. So far, I have to make two requests to get this data,

One to /webhooks/rest/webhook to get the response and one to /model/parse to get the NLU data. I was looking to do it in one request as the first request will also generate the NLU data.

Here’s my config.yml file contents. I’m not using MemoizationPolicy as I don’t need Rasa to remember what happened in the previous step to predict an action. Action will depend only on the current input.

# Configuration for Rasa NLU.
# https://rasa.com/docs/rasa/nlu/components/
language: en
pipeline:
  - name: WhitespaceTokenizer
  - name: RegexFeaturizer
  - name: LexicalSyntacticFeaturizer
  - name: CountVectorsFeaturizer
  - name: CountVectorsFeaturizer
    analyzer: "char_wb"
    min_ngram: 1
    max_ngram: 4
  - name: DIETClassifier
    epochs: 100
  - name: EntitySynonymMapper
    epochs: 100

# Configuration for Rasa Core.
# https://rasa.com/docs/rasa/core/policies/
policies:
  - name: MappingPolicy
  - name: "FallbackPolicy"
    nlu_threshold: 0.7
    core_threshold: 0.7
    fallback_action_name: "utter_Default Fallback Intent"

Here’s an excerpt from my domain.yml file:

intents:
  - welcome_intent:
      triggers: utter_welcome_intent
responses:
  utter_welcome_intent:
    - text: "Hi <username>! How are you doing?"
    - text: "Hello <username>! How can I help you?"
4

Thank you for the details! I understand what you’re going for now.

First, MemoizationPolicy is still appropriate for single-turn interactions, you would just set max_history to 1 so that only the previous turn is taken into account. You can also use triggers, as you are doing, but in this case if you later want to expand your dialogue (for example, to predict multiple actions after some intent), you would need to move it to a regular story, and use MemoizationPolicy.

Second, your fallback_action_name is in-valid; it would need to be the same as either the name defined in a custom action, or (what I think you’re going for), the same as a response template in your domain. You’d want something like:

policies:
  - name: MappingPolicy
  - name: "FallbackPolicy"
    nlu_threshold: 0.7
    core_threshold: 0.7
    fallback_action_name: "utter_default"

And then in your domain:

responses:
  utter_welcome_intent:
    - text: "Hi <username>! How are you doing?"
    - text: "Hello <username>! How can I help you?"
 utter_default:
    - text: "Sorry, I didn't understand that"

The reason you would never see this response now, is because the fallback_action_name refers to a non-existent action, so nothing happens.

Third, relating to your question about endpoints - Are you connecting your rasa server to an external channel, like Slack or Facebook? If not, the easiest way to get both response + NLU parse data is to run rasa shell --debug, which will run raw text through both NLU & Dialogue models, and the debug flag will show you predictions before returning the response.

Does that make sense?

5

Thank you for the detailed response. Your reply confirmed some of my assumptions.

Regarding, the fallback action name, I actually have utter_Default Fallback Intent as a response but forgot to include it in the domain.yml excerpt. I already get a Fallback response when I run the Rasa core.

Regarding your last question, the Rasa server will ultimately be used with a channel (messenger, google assistant and what not). But they won’t communicate directly will Rasa. I have an intermediate layer. So, it’ll look something like this:

Messenger/WhatsApp/Assistant <--> My Server <--> Rasa Server

That’s why I need the Rasa server to send me the NLU results and a response. Most of the dialogue management and logic happens in My Server.

For now, I’m actually using the Rasa SDK Agent class to load trained models and doing prediction on these models. If I find a way to do it in one call to a Rasa server, I’ll move to that solution.

–

I have another question unrelated to this topic. It would help me if you can respond to it when you have the time.

  1. How to manage multiple languages in Rasa. Currently, I’m they are being trained separately as separate projects. But all of my intents and entites are same across languages, only the examples differ. So, any way to manage it better?
6

Interesting - can I ask why you are separating out the dialogue management?

I think a misunderstanding here is that the response is not part of dialogue management; in Rasa it is predicted and returned by the core/dialogue management model, so if you added another dialogue management layer, you would be doubling up.

If you do for any reason want to call the NLU model only separately, that’s definitely possible; you can just use the parse endpoint.

At that point, if your intermediate layer is predicting the next utterance/action, it will not make sense to request it from Rasa; it should live in your intermediate layer.

I think you might still have a syntax issue though - re. your fallback intent, if utter_Default Fallback Intent is literally the utterance name, then it is not a legitimate utterance name since it contains whitespace. What does that section of your domain look like?

Also I am guessing you’ve seen Rasa as open source alternative to Google Dialogflow - Migration Guide and are basing your migration on that; were you trying to achieve the above using the NLU server approach described there?

7

Hi, I currently don’t need robust dialogue management as most of the interactions are one turn. I already have a service that takes decisions based on the detected intents and shows a response to the user. So, the dialogue management would not add anything to my service at its current state.

Regarding the syntax issue, I’m not sure about that. I checked with rasa data validate and it didn’t complain. I’m also getting a Fallback response from Rasa now. That’s how it looks,

utter_Default Fallback Intent:
    - text: "Sorry, can you repeat that?"
    - text: "Sorry, please repeat?"
    - text: "Can you repeat that please?"
    - text: "I am sorry, I did not get that. Can you rephrase?"

The Dialogflow migration guide was my starting point but now I’ve moved on to using the Agent class from rasa.core.agent package. Can you comment on whether it is okay to use this class for parsing data using Rasa? A small snippet of what I’m doing is

agent = Agent.load('/path/to/model.tar.gz')
text = 'This text will be processed by Rasa'
async def handleText(text):
    coreResponse = await agent.handle_text(text)
    nluResponse = await agent.parse_message_using_nlu_interpreter(text)
8

Sorry to get back to this so late, just noticed it again - Your agent set up looks fine. I’m going to check on the syntax of response names - I’m also getting a Fallback response from Rasa now - do you mean you get one of the responses listed under your utter_Default Fallback Intent?