Implementing simple fallback

Rasa Open Source
1

Hi everyone,

I’m having trouble implementing my fallback utter message. As of now, I already defined utter_default template on the domain.yaml and registered it as an action in that same file. Also, I created two stories:

## story_fallback
  * out_of_scope
   - action_default_fallback

## fallback
   - utter_fallback

The first one is working: when the intent is out_of_scope (I have some examples of this on my nlu_data.md) it activates the action_default_fallback and spits out my utter_fallback template.

However, since I’m training my model using:

python -m rasa_core.train -d domain.yml -s data/stories.md -o models/current/dialogue --nlu_threshold 0.1 --core_threshold 0.1 --fallback_action action_default_fallback

I was expecting the model to fallback to utter_fallback when the NLU confidence was bellow 0.1. But on reality this doesn’t happen:

2018-10-04 15:21:03 DEBUG    urllib3.connectionpool  - Starting new HTTP connection (1): localhost
2018-10-04 15:21:03 DEBUG    rasa_core.tracker_store  - Recreating tracker for id 'default'
2018-10-04 15:21:03 DEBUG    rasa_core.processor  - Received user message '' with intent '{'name': None, 'confidence': 0.0}' and entities '[]'
2018-10-04 15:21:03 DEBUG    rasa_core.processor  - Logged UserUtterance - tracker now has 19 events
2018-10-04 15:21:03 DEBUG    rasa_core.processor  - Current slot values: 
	address: None
	delivery_state: None
	name: paulo
2018-10-04 15:21:03 DEBUG    rasa_core.policies.memoization  - Current tracker state [{'slot_name_0': 1.0, 'intent_name': 1.0, 'prev_action_listen': 1.0}, {'slot_name_0': 1.0, 'intent_name': 1.0, 'prev_utter_greet': 1.0}, {'prev_action_listen': 1.0, 'slot_name_0': 1.0}]
2018-10-04 15:21:03 DEBUG    rasa_core.policies.memoization  - There is no memorised next action
2018-10-04 15:21:03 DEBUG    rasa_core.policies.ensemble  - Predicted next action using policy_2_KerasPolicy
2018-10-04 15:21:03 DEBUG    rasa_core.processor  - Predicted next action 'utter_thanks' with prob 0.66.

As you can see, it predicts another action (utter_thanks) when I type an empty string (that has an intent confidence of 0.0).

What am I doing wrong? I followed all the directions on the tutorial and I still can’t default when the confidence is low.

2

i have also same problem

3

if some one help according to rasa-core==0.9.4

rasa-nlu==0.12.3

4

Anyone has any idea of what is happening?

5

@vinaynagpal9 we only added the fallback policy in version 0.10 as far as i remember, please upgrade to the latest version 0.11

@prcastro that looks strange. which version of core are you using?

6

I’m using rasa_core version 0.11.1

7

@prcastro got this from the code

type: (…) -> bool

    """It should predict fallback action only if
    a. predicted NLU confidence is lower than ``nlu_threshold`` &&
    b. last action is NOT fallback action
    """

Was your last action a fallback action by any chance?

8

No, this happens even at the very start of the conversation

9

can you paste the contents of your metadata.json from your trained model here please? Also try upgrading to 0.11.11, that’s the latest version

10

Do you mean my NLU model? Here it is:

{
    "language": "pt",
    "pipeline": [
        {
            "model": "pt",
            "case_sensitive": false,
            "name": "nlp_spacy",
            "class": "rasa_nlu.utils.spacy_utils.SpacyNLP"
        },
        {
            "name": "tokenizer_spacy",
            "class": "rasa_nlu.tokenizers.spacy_tokenizer.SpacyTokenizer"
        },
        {
            "name": "intent_featurizer_spacy",
            "class": "rasa_nlu.featurizers.spacy_featurizer.SpacyFeaturizer"
        },
        {
            "name": "intent_entity_featurizer_regex",
            "regex_file": "regex_featurizer.json",
            "class": "rasa_nlu.featurizers.regex_featurizer.RegexFeaturizer"
        },
        {
            "BILOU_flag": true,
            "features": [
                [
                    "low",
                    "title",
                    "upper"
                ],
                [
                    "bias",
                    "low",
                    "prefix5",
                    "prefix2",
                    "suffix5",
                    "suffix3",
                    "suffix2",
                    "upper",
                    "title",
                    "digit",
                    "pattern"
                ],
                [
                    "low",
                    "title",
                    "upper"
                ]
            ],
            "max_iterations": 50,
            "L1_c": 0.1,
            "L2_c": 0.1,
            "name": "ner_crf",
            "classifier_file": "crf_model.pkl",
            "class": "rasa_nlu.extractors.crf_entity_extractor.CRFEntityExtractor"
        },
        {
            "name": "ner_synonyms",
            "synonyms_file": "entity_synonyms.json",
            "class": "rasa_nlu.extractors.entity_synonyms.EntitySynonymMapper"
        },
        {
            "C": [
                1,
                2,
                5,
                10,
                20,
                100
            ],
            "kernels": [
                "linear"
            ],
            "max_cross_validation_folds": 5,
            "scoring_function": "f1_weighted",
            "name": "intent_classifier_sklearn",
            "classifier_file": "intent_classifier_sklearn.pkl",
            "class": "rasa_nlu.classifiers.sklearn_intent_classifier.SklearnIntentClassifier"
        }
    ],
    "training_data": "training_data.json",
    "trained_at": "20181009-114048",
    "rasa_nlu_version": "0.13.5"
}

I’m retraining with the new version you suggested now. Let’s see.

11

Apparently it was a problem with the version, now it is working. Thanks for the tip.

1 Like