Hi,
Thanks for the reply.
I will look into that.
Meanwhile , when i run evaluation on my testing data, i got the below results :
2019-06-22 16:40:43 INFO rasa.nlu.test - Intent evaluation results:
2019-06-22 16:40:43 INFO rasa.nlu.test - Intent Evaluation: Only considering those 20557 examples that have a defined intent out of 20557 examples
2019-06-22 16:40:43 INFO rasa.nlu.test - F1-Score: 0.9983095015409214
2019-06-22 16:40:43 INFO rasa.nlu.test - Precision: 0.9982888889376054
2019-06-22 16:40:43 INFO rasa.nlu.test - Accuracy: 0.9983460621686043
2019-06-22 16:40:43 INFO rasa.nlu.test - Classification report:
precision recall f1-score support
actor_search 0.99 0.99 0.99 1352
actress_search 1.00 1.00 1.00 1352
affirm 0.60 1.00 0.75 3
costar_search 1.00 1.00 1.00 1352
director_search 0.99 0.99 0.99 1352
goodbye 0.50 0.50 0.50 2
greet 0.00 0.00 0.00 2
movie_search 1.00 1.00 1.00 11897
producer_search 0.99 1.00 0.99 1352
rating_search 1.00 1.00 1.00 1893
micro avg 1.00 1.00 1.00 20557
macro avg 0.81 0.85 0.82 20557
weighted avg 1.00 1.00 1.00 20557
2019-06-22 16:40:43 INFO rasa.nlu.test - Model prediction errors saved to errors.json.
2019-06-22 16:40:44 INFO rasa.nlu.test - Confusion matrix, without normalization:
[[ 1339 0 0 0 4 0 0 0 9 0]
[ 1 1351 0 0 0 0 0 0 0 0]
[ 0 0 3 0 0 0 0 0 0 0]
[ 0 0 0 1352 0 0 0 0 0 0]
[ 6 0 0 1 1341 0 0 0 4 0]
[ 0 0 1 0 0 1 0 0 0 0]
[ 0 0 1 0 0 1 0 0 0 0]
[ 0 0 0 0 0 0 0 11897 0 0]
[ 3 0 0 0 3 0 0 0 1346 0]
[ 0 0 0 0 0 0 0 0 0 1893]]
2019-06-22 16:40:46 INFO rasa.nlu.test - Entity evaluation results:
2019-06-22 16:40:53 INFO rasa.nlu.test - Evaluation for entity extractor: CRFEntityExtractor
2019-06-22 16:40:58 INFO rasa.nlu.test - F1-Score: 0.9956100901680592
2019-06-22 16:40:58 INFO rasa.nlu.test - Precision: 0.9955535721798925
2019-06-22 16:40:58 INFO rasa.nlu.test - Accuracy: 0.995691860508227
2019-06-22 16:40:58 INFO rasa.nlu.test - Classification report:
precision recall f1-score support
acting_person 0.96 0.98 0.97 12889
aggmethod 1.00 1.00 1.00 2226
composer 0.33 0.28 0.30 512
condition 1.00 1.00 1.00 3793
contrib 1.00 1.00 1.00 249
count 1.00 1.00 1.00 6779
coworker_actor 1.00 1.00 1.00 1352
coworker_actress 1.00 1.00 1.00 1352
coworker_costar 1.00 1.00 1.00 4056
coworker_director 1.00 1.00 1.00 1352
coworker_producer 1.00 1.00 1.00 1352
director 0.99 0.99 0.99 21817
director_role 1.00 1.00 1.00 21109
frequency 1.00 1.00 1.00 6760
no_entity 1.00 1.00 1.00 121696
order 1.00 1.00 1.00 6492
producer 0.99 0.99 0.99 21934
producer_role 1.00 1.00 1.00 20918
selection_criteria 1.00 1.00 1.00 2172
time 1.00 1.00 1.00 8740
value_for_condition 1.00 1.00 1.00 2172
micro avg 1.00 1.00 1.00 269722
macro avg 0.97 0.96 0.96 269722
weighted avg 1.00 1.00 1.00 269722
In the above result, the accuracy obtained is for over all all intents and entities.
what i wanted is , for each intent and for each entity , i want to know the accuracy. Like how precision, f1 score and support is obtained for individual intent and entities, can i get the accuracy for individual values as well ?