How can we improve confidence score of intents

Rasa Open Source
1

Hey guys i am using the pipeline of tensor-flow with rasa-nlu version 0.13.6. I want to know how i can get more confidence in my intents like ‘product_inform’,‘product_utility’ and ‘product_start_inform’. While creating the model, confidence score is very low around 0.01-0.03.

Please suggest how we can improve the confidence score.

Attached the files.

config.yml (245 Bytes) chatbot_domain.yml (1.7 KB) nlu_model.py (658 Bytes)

data.json (19.1 KB) stories.md (3.9 KB)

2

you have to increase your utterances in your training data. However tensorflow requires less as compared to spacy still try adding more utterances it can help in increasing confidence

3

It seems you’re in the beginning of your project and you don’t have enough data. A good tool that can help you is Chatito. You can generate synthetic datasets for any given intent.

2 Likes
4

Hi Nikhil, Please specify more utterance mean? DO we need to add more uttterence against a intent.

5

you will have to increase your training data for all of your intents. Bot will have higher confidence when they are being provided with large training data set .i.e in your data.json file

6

follow this –

8

Hi Rangaraj,

While building the model we are getting low confidence score. Are the above logs captured when we train our model?

please suggest how we can check the same at our end?

Below is the logs while we build the model.

C:\Users\e\Desktop\Chatbot>python nlu_model.py C:\Users\e\AppData\Local\Programs\Python\Python36\lib\site-packages\h5py_init_.py:34: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. from ._conv import register_converters as _register_converters 2018-10-15 10:29:15.736598: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 Epochs: 100%|█████████████████████████████████████████████████| 300/300 [00:04<00:00, 71.71it/s, loss=0.107, acc=1.000] C:\Users\e01410\AppData\Local\Programs\Python\Python36\lib\site-packages\rasa_nlu\extractors\entity_synonyms.py:85: UserWarning: Failed to load synonyms file from ‘./models/nlu/default/chatbotnlu\entity_synonyms.json’ “”.format(entity_synonyms_file)) {‘intent’: {‘name’: ‘greet’, ‘confidence’: 0.9288316965103149}, ‘entities’: [], ‘intent_ranking’: [{‘name’: ‘greet’, ‘confidence’: 0.9288316965103149}, {‘name’: ‘goodbye’, ‘confidence’: 0.20464512705802917}, {‘name’: ‘chatbot_name’, ‘confidence’: 0.08284890651702881}, {‘name’: ‘product_inform’, ‘confidence’: 0.06197477504611015}, {‘name’: ‘mood’, ‘confidence’: 0.05501033738255501}, {‘name’: ‘chatbot_info’, ‘confidence’: 0.01574290543794632}, {‘name’: ‘forget_password’, ‘confidence’: 0.0}, {‘name’: ‘product_upgrade’, ‘confidence’: 0.0}, {‘name’: ‘product_start_inform’, ‘confidence’: 0.0}, {‘name’: ‘product_utility’, ‘confidence’: 0.0}], ‘text’: ‘Hi’}

9

Your intent is properly identified in Rasa NLU

{"user_input": {"intent": {"name": "product_inform", "confidence": 0.9743123054504395}, "entities": [{"start": 8, "end": 27, "value": "about your products", "entity": "products", "confidence": 0.7121597839662546, "extractor": "ner_crf"}], "intent_ranking": [{"name": "product_inform", "confidence": 0.9743123054504395}, {"name": "product_utility", "confidence": 0.08250807225704193}, {"name": "chatbot_name", "confidence": 0.03798557445406914}, {"name": "product_issue", "confidence": 0.0035776719450950623}, {"name": "forget_password", "confidence": 0.0}, {"name": "account_access_issue", "confidence": 0.0}, {"name": "greet", "confidence": 0.0}, {"name": "mood", "confidence": 0.0}, {"name": "product_start_inform", "confidence": 0.0}, {"name": "product_upgrade", "confidence": 0.0}], "text": "tell me about your products", "project": "nlu", "model": "current"}, "project": "nlu", "model": "current", "log_logger": {"unpersistable": true}, "log_level": {"name": "info", "__class_uuid__": "02e59486-f24d-46ad-8224-3acdf2a5732a"}, "log_namespace": "query-logger", "log_source": null, "log_format": "", "log_time": 1539585185.2487378}

But your rasa core is unable to map the intent to action because of your story.

My suggestion is split each your story into one intent and one action and it should solve your issue.