Rasa NLU predicting random intent

Rasa Open Source
1

I am using rasa_core==0.12.2 and rasa_nlu[tensorflow]==0.13.7 NLU config is as follows -

pipeline:
- name: tokenizer_whitespace
- name: intent_entity_featurizer_regex
- name: ner_crf
- name: intent_featurizer_count_vectors
  analyzer: 'word'
  min_ngram: 1  
  max_ngram: 1
  • name: intent_classifier_tensorflow_embedding epochs: 100 language: en

Dialogue config is as follows -

policies:
  - name: MemoizationPolicy
    max_history: 1
  - name: FallbackPolicy
    nlu_threshold: 0.60
    core_threshold: 0.3
  - name: FormPolicy

Bot gives random response even if that particular text never appeared in the nlu training data.

I tested following scenarios -

scenario 1 -

   User - What is hippa compliance?
   Bot - Hi I am a bot. How may I help you

scenario2 -

User - What is rasa nlu
Bot - I am fine

My expectation is that if bot is not trained on some conversation it should respond with fallback (Please try again I do not understand).

Why is bot giving random response?

2

Has anyone also faced the same issue?

3

hey @r4sn4, just check what is the confidence score you get for the above queries, is it above your nlu threshold that you have defined in the fallback policy? :slight_smile:

4

@JiteshGaikwad I have already checked threshold and it is coming to be above 0.7. Why threshold is coming out to be so high for words which never existed in my data? Even if it is resolving to random intent, it should resolve with low confidence score? On what basis it resolve to particular intent?

5

@r4sn4 might be it’s able to fine atleast some of the words in your training data, I mean as in the above example you had asked

What is hippa compliance?

What is rasa nlu

since you are using tensorflow_embedding as the classifier, it creates the bag of words from the training data and the words like what, is are getting matched with the bag of words.

can you check with sklearn classifer and see the nlu results

6

Check this post:

7

if this is the case. then it should resolve to intents where ‘what is’ is present in the conversation, but instead it is pointing to intents which does not contain such words at all. with SKlearn classifier, accuracy of model degrades even further.

8

@juste_petr Hi… Any solution for this problem?

9

@r4sn4 - Does it respond properly for things it should? And it is just random for things it doesn’t know about? or is it random for everything?

10

@jonathanpwheat - It responds properly for conversation it is trained on. Whereas for conversations it does not know it responds with random intent.

11

I just went through setting up the default fallback. I think you need to add a fallback_action_name to your config.yml file like this

  - name: "FallbackPolicy"
    nlu_threshold: 0.3
    core_threshold: 0.3
    fallback_action_name: "action_default_fallback"

Then add this to your actions.py file

class ActionDefaultFallback(Action):
    def name(self) -> Text:
        return "action_default_fallback"

    def run(
        self,
        dispatcher: CollectingDispatcher,
        tracker: Tracker,
        domain: Dict[Text, Any],
    ) -> List["Event"]:

        dispatcher.utter_template("utter_default", tracker)
        return [UserUtteranceReverted()]

in your domain.yml file, you create an utter_default template like this to handle your responses:

  utter_default:
    - text: "Sorry, I didn't get that. Could you please rephrase?"
    - text: "I didn't understand, could you rephrase that?"
    - text: "I'm sorry, but I didn't understand you. Could you please rephrase what you just said?"
    - text: "I'm afraid I didn't get what you just said. Could you rephrase that?"
    - text: "I didn't quite get that, could you rephrase your message?"
    - text: "Could you rephrase your message? I didn't get it, I'm sorry."

Also in your domain.yml file add the fallback action name to the bottom of your actions: section

action_default_fallback

I think that’s “all” I had to to do get a fallback response to work.

retrain it and type some garble in, and you should get one of the random responses back.

12

default action is already present in my code.

13

Hmm, ok. I don’t mean to talk down to you so please don’t take this as such.

Do you have it setup the way I do though?

I seem to remember getting randomish responses before I had it all right.

Just asking because I didn’t see fallback_action_name: "action_default_fallback" in your initial policies: block above, I think you need that, and the naming convention is important as well.

Again, sorry if you know this already :slight_smile: Maybe if you could post your config.yml, domain.yml and actions.py file to a gist so I (we) can check it out

14

I have upgraded rasa version to 1.3.6. Now not facing this issue.

2 Likes
15

Oh, that’s great news.