Hi All, I am having an issue when running rasa core, the model is not predicting good, even for the greeting message, However, it is predicting almost perfect, like 90% accuracy or more when I am using the interactive train.
Any clue about why this could happen? I’m running both rasa core, “core.run” and “core.train” with the same config_policy.yml file.
Thanks for any help I can receive about it
Hi! When you say the model is not predicting well, do you mean the intent classification or the action prediction? To find out, you can run core with the --debug flag and you should see messages like this:
DEBUG rasa_core.processor - Received user message ‘hello’ with intent ‘{‘name’: ‘greet’, ‘confidence’: 0.9658612608909607}’ and entities ‘[]’
DEBUG rasa_core.processor - Predicted next action ‘utter_greet’ with prob 1.00.
If it’s the former, it’s probably an NLU issue, while the latter would be a core issue.
thank you @erohmensing, my issue it is in the prediction side The NLU classification works perfect, everything is classified over 85% accuracy ratio.
on the other hand, I can see the interactive train predicting mostly good.
However, when I am using a real life example like telegram or running on the command line, the bots answer could be mostly inaccurate 
after doing more training, it start to work a little better. however I am not understanding why, I am having this output when I am evaluating my model I am running the following:
python -m rasa_core.evaluate --nlu models/nlu/default/current --stories data/stories.md -o admin_reports/core_evaluate --endpoints endpoints.yml --core models/dialogue --verbose
the logs on the screen start to show some error stories
‘’‘WARNING:rasa_core.training.dsl:Found unknown intent ‘None’ on line 826. Please, make sure that all intents are listed in your domain yaml. WARNING:rasa_core.training.dsl:Found unknown intent ‘None’ on line 845. Please, make sure that all intents are listed in your domain yaml. WARNING:rasa_core.training.dsl:Found unknown intent ‘None’ on line 850. Please, make sure that all intents are listed in your domain yaml. WARNING:rasa_core.training.dsl:Found unknown intent ‘None’ on line 897. Please, make sure that all intents are listed in your domain yaml.’’’
the other issue I am seeing is related to the result of the evaluation, one important point here, since I am having a FormAction I train all the stories with the interactive-train:
INFO:main:Evaluating 36 stories
INFO:main:Evaluation Results on CONVERSATION level: INFO:main: Correct: 2 / 36 INFO:main: F1-Score: 0.105 INFO:main: Precision: 1.000 INFO:main: Accuracy: 0.056 INFO:main: In-data fraction: 0.0106 INFO:main:Evaluation Results on ACTION level: INFO:main: Correct: 375 / 663 INFO:main: F1-Score: 0.505 INFO:main: Precision: 0.491 INFO:main: Accuracy: 0.566 INFO:main: In-data fraction: 0.0106 INFO:main: Classification report: precision recall f1-score support
action_algo_mas 0.00 0.00 0.00 79
action_buscar_hora_disponible 0.00 0.00 0.00 8 action_cambiar_cita 0.00 0.00 0.00 15 action_cancelar_cita 0.00 0.00 0.00 17 action_chequear_cita 0.00 0.00 0.00 33 action_default_fallback 0.01 1.00 0.01 1 action_greet 0.00 0.00 0.00 38 action_listen 0.74 0.97 0.84 274 action_modificar_hora 0.00 0.00 0.00 8 action_reset 0.72 0.55 0.63 38 agenda_form 1.00 0.05 0.10 20 utter_ask_time 0.00 0.00 0.00 8 utter_esta_seguro 0.00 0.00 0.00 20 utter_goodbye 0.62 0.45 0.53 22 utter_que_mas 0.69 0.98 0.81 45 utter_thanks 0.84 0.86 0.85 37
micro avg 0.57 0.57 0.57 663
macro avg 0.29 0.30 0.24 663
weighted avg 0.49 0.57 0.51 663
INFO:rasa_nlu.evaluate:Confusion matrix, without normalization: [[ 0 0 0 0 0 0 0 75 0 0 0 0 0 0 0 4] [ 0 0 0 0 0 7 0 0 1 0 0 0 0 0 0 0] [ 0 0 0 0 0 14 0 1 0 0 0 0 0 0 0 0] [ 0 0 0 0 0 0 0 0 0 0 0 0 0 1 16 0] [ 0 0 0 0 0 33 0 0 0 0 0 0 0 0 0 0] [ 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0] [ 0 0 0 0 0 38 0 0 0 0 0 0 0 0 0 0] [ 0 0 0 0 0 0 0 266 0 8 0 0 0 0 0 0] [ 0 0 0 0 0 4 0 0 0 0 0 0 0 0 4 0] [ 0 0 0 0 0 0 0 17 0 21 0 0 0 0 0 0] [ 0 0 0 0 0 18 0 0 0 0 1 0 0 0 0 1] [ 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 0] [ 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0] [ 0 0 0 0 0 11 0 0 0 0 0 0 0 10 0 1] [ 0 0 0 0 0 0 0 1 0 0 0 0 0 0 44 0] [ 0 0 0 0 0 0 0 0 0 0 0 0 0 5 0 32]] INFO:main:Finished evaluation
I have seen I have only 2 correct of 36 on a conversation evaluation and the accuracy is about 0,05
I don’t know if I am doing something wrong when I am training the model or how to improve the accuracy in my training information.
Thanks for any help or information for me to understand how to improve all of this @Juste just copying you to know if there are any other example about this.
Thanks again
With respect to the evaluation issue (Found unknown intent ‘None’)… have you taken a look into your data/stories.md? Nowhere in your trianing data should there be stories where the intent is None.
What commands (with arguments) are you running to run core in regular mode and in interactive mode? Is is possible you’re pointing to the wrong core model for one of them?
@erohmensing thanks for your answer. I check the stories.md and look for any “None” and “none” in the file and there is nothing. The only thing I found was null but it was placed by the interactive train when a FormAction takes place in the story. This is a kind of mistery for me, did you try the evaluate function? and did not get the None message?
Below the commands I am using:
run-core: python -m rasa_core.run --nlu models/nlu/default/current --core models/dialogue --debug --credentials credentials.yml --endpoints endpoints.yml --enable_api --port 5002 -o admin_reports/logs/core_run.log
interactive-train: python -m rasa_core.train interactive -o models/dialogue -d domain.yml -s data/stories.md --nlu models/nlu/default/current --endpoints endpoints.yml
Hmm. Is there any case in your stories where there is a * marker and nothing after it? That would indicate the lack of an intent.
As for interactive working and run not, i’m still stumped. What does your config_policy.yml file look like?