Micro avg for an intent report is not supported

Rasa Open Source
1

Hi,

I tried to run “rasa test nlu” and found micro avg for an intent report is not supported.

Entity report section:

“micro avg”: { “precision”: 0.8003597122302158, “recall”: 0.82483781278962, “f1-score”: 0.8124144226380648, “support”: 1079 }, “macro avg”: { “precision”: 0.65809214482399, “recall”: 0.810636379480587, “f1-score”: 0.7176928001590804, “support”: 1079 }, “weighted avg”: { “precision”: 0.7214485084847554, “recall”: 0.82483781278962, “f1-score”: 0.7641163082269575, “support”: 1079 }

Intent report section

“accuracy”: 0.8175, “macro avg”: { “precision”: 0.8168217468721014, “recall”: 0.7874546771652321, “f1-score”: 0.7655503578083683, “support”: 800 }, “weighted avg”: { “precision”: 0.8727316398099432, “recall”: 0.8175, “f1-score”: 0.8210921219591292, “support”: 800 }

Why doesn’t RASA support micro avg for intent performance evaluation?

2

I cannot come up with a good reason, but I am curious why you’re interested in the "micro" variant for intents. What information does this give you that the weighted average doesn’t give? Why is it such a useful metric to you?

3

Heya Miner @miner that;s really a nice question why there is no Micro Avg for Intent Prediction Score and even for response selector too :slight_smile:

  1. Intent prediction scores are always available in a file intent_report.json . The relevant keys are - accuracy , macro avg and weighted avg . For macro avg and weighted avg we need to extract precision ,recall and f1-score .

  2. Entity prediction scores are available in either CRFEntityExtractor_report.json or DIETClassifier_report.json (whichever exists in the report folder). The relevant keys are - micro avg , macro avg and weighted avg . For all three, we need to extract precision ,recall and f1-score .

  3. Response Selection scores are always available in a file response_selection_report.json . The relevant keys are - accuracy , macro avg and weighted avg . For macro avg and weighted avg we need to extract precision ,recall and f1-score .

May be that hows they have designed ML Algos, but Major Logic can be this also:

Macro-average method can be used when you want to know how the system performs overall across the sets of data. You should not come up with any specific decision with this average. On the other hand, micro-average can be a useful measure when your dataset varies in size.

OR

The micro average is a global strategy that basically ignores that there is a distinction between classes. It is calculated by counting the total true positives, false negatives and false positives over all classes.

In classification tasks where the underlying problem is not a multilabel classification, the micro average actually equals the accuracy score.

In case of macro average, the precision and recall are calculated for each label separately and reported as their unweighted mean. Depending on how your classifier performs on each class, this can heavily influence the result.

I will get more resources on this, but I guess as for now this answer will burst your thought :thought_balloon: bubble :wink: Thanks for the great observation and question. Good Luck!

Note: This is just my own observation and suggestion, even I will also seek some information on this If I really required more information as for now I am good and happy :slight_smile:

4

I am just curious why rasa test doesn’t support it though supporting it for NER.

5

Hey @nik202, any update on why it doesn’t generate micro average?

6

@dhruv_bhimani I have not done further research on that, but maybe that how rasa wants to show.