Improper intent recognition

rasa-nlu
rasa-core

(Sivaram) #1

i Am trying to experiment on executing fallback intents and failed to do so. I has given some random text as “hjsdhjsd” and i am getting answer of some other question as response.

I had fallback intent defined in dilogue management as below.

from rasa_core.agent import Agent fallback = FallbackPolicy(fallback_action_name=“utter_unclear”, core_threshold=0.2, nlu_threshold=0.1)

agent = Agent(‘domain.yml’, policies=[MemoizationPolicy(), KerasPolicy(), fallback]).

But still wrong action is performing. When i dig into reason, I found that entity is being recognized improper through rasa nlu.

When I am trying to identify the entity for any random text like ‘hkjdfkd’, the intent is wrongly identified as DepositEnquiryForAgency . This output is thrown when I run nlu_model.py. Could you please help me on how to correct this. i am surprised to see that rasa nlu is giving confidence as 0.907 for random text. COuld someone please let me know where i am doing wrong??

{‘intent’: {‘name’: ‘DepositEnquiryForAgency’, ‘confidence’: 0.9075979430989889}, ‘entities’: [], ‘intent_ranking’: [{‘name’: ‘DepositEnquiryForAgency’, ‘confidence’: 0.9075979430989889}, {‘name’: ‘goodbye’, ‘confidence’: 0.062111495788272415}, {‘name’: ‘greet’, ‘confidence’: 0.030290561112738677}], ‘text’: ‘jhfdkjsh’}

Data file :

{

“rasa_nlu_data”: {

"common_examples": [

  {

    "text": "Hello",

    "intent": "greet",

    "entities": []

  },

  {

    "text": "goodbye",

    "intent": "goodbye",

    "entities": []

  },

  {

    "text": "What's the pending deposit for federal",

    "intent": "DepositEnquiryForAgency",

    "entities": [

      {

        "start": 11,

        "end": 18,

        "value": "pending",

        "entity": "depositStatus"

      },

      {

        "start": 31,

        "end": 38,

        "value": "federal",

        "entity": "Jurisdiction"

      }

    ]

  },

  {

    "text": "hey",

    "intent": "greet",

    "entities": []

  },

  {

    "text": "hello",

    "intent": "greet",

    "entities": []

  },

  {

    "text": "hi",

    "intent": "greet",

    "entities": []

  },

  {

    "text": "heya",

    "intent": "greet",

    "entities": []

  },

  {

    "text": "howdy",

    "intent": "greet",

    "entities": []

  },

  {

    "text": "hey there",

    "intent": "greet",

    "entities": []

  },

  {

    "text": "bye",

    "intent": "goodbye",

    "entities": []

  },

  {

    "text": "goodbye",

   "intent": "goodbye",

    "entities": []

  },

  {

    "text": "bye bye",

    "intent": "goodbye",

    "entities": []

  },

  {

    "text": "see ya",

    "intent": "goodbye",

    "entities": []

  },

  {

    "text": "see you later",

    "intent": "goodbye",

    "entities": []

  },

  {

    "text": "Show me what's the pending deposit  for FLSUI",

    "intent": "DepositEnquiryForAgency",

    "entities": [

      {

        "start": 19,

        "end": 26,

        "value": "pending",

        "entity": "depositStatus"

      },

      {

        "start": 40,

        "end": 45,

        "value": "FLSUI",

        "entity": "Jurisdiction"

      }

    ]

  },

  {

    "text": "for Failed",

    "intent": "DepositEnquiryForAgency",

    "entities": [

      {

        "start": 4,

        "end": 10,

        "value": "Failed",

        "entity": "depositStatus"

      }

    ]

  },

  {

    "text": "Failed",

    "intent": "DepositEnquiryForAgency",

    "entities": [

      {

        "start": 1,

        "end": 6,

        "value": "Failed",

        "entity": "depositStatus"

      }

    ]

  },

  {

    "text": "Oh, sorry, in Held",

    "intent": "DepositEnquiryForAgency",

    "entities": [

      {

        "start": 14,

        "end": 18,

        "value": "Held",

        "entity": "depositStatus"

      }

    ]

  },

  {

    "text": "tell me failed deposit for flsui",

    "intent": "DepositEnquiryForAgency",

    "entities": [

      {

        "start": 8,

        "end": 14,

        "value": "failed",

        "entity": "depositStatus"

      },

      {

        "start": 27,

        "end": 32,

        "value": "flsui",

        "entity": "Jurisdiction"

      }

    ]

  },

  {

    "text": "How about federal",

    "intent": "DepositEnquiryForAgency",

    "entities": [

      {

        "start": 10,

        "end": 17,

        "value": "federal",

        "entity": "Jurisdiction"

      }

    ]

  },

  {

    "text": "How about flsui",

    "intent": "DepositEnquiryForAgency",

    "entities": [

      {

        "start": 10,

        "end": 15,

        "value": "flsui",

        "entity": "Jurisdiction"

      }

    ]

  },

  {

    "text": "How about failed",

    "intent": "DepositEnquiryForAgency",

    "entities": [

      {

        "start": 10,

        "end": 16,

        "value": "failed",

        "entity": "depositStatus"

      }

    ]

  }

]

}

}


(Vinay Verma) #2

Hi, try to increase the threshold value of RASA nlu and RASA core. It will work


(Sivaram) #3

But here my question is Rasa NLU is identifying confidence as 0.9 for my random text… So i need to have threshold more than 0.9 to go for fallback intent. Would it be a good idea to have threshold more than 0.9?


(Vladimir Vlasov) #4

What version of rasa_nlu do you use? Do you have oov turned on for count_vector_featurizer in your nlu pipeline?


(Sivaram) #5

i am using rasa-nlu 0.13.0. I am new bee to AI and bots. Could you please guide me where to turn on count_vector_featurizer??

Below is my config file

{ “pipeline”:“spacy_sklearn”, “path”:"./models/nlu", “data”:"./data/data.json" }


(Vinay Verma) #6

Try to add more examples and train it online and generate proper stories. The number of examples for each intent you have added are very less.


(Venkatesh Rathod) #7

If I turned on OOV_token for “intent_featurizer_count_vectors” the above problem mentioned will be solved?

pipeline:
- name: "intent_featurizer_count_vectors" #Creates bag-of-words representation of intent features
     OOV_token: oov

(Vladimir Vlasov) #8

Sorry, I thought you use different pipeline. Try using tensorflow_embedding pipeline (http://www.rasa.com/docs/nlu/choosing_pipeline/)