High F1 score due to no-entity in NLU?

I always see very high F1 score in NLU and then i realized it’s because of no_entity.

image

For example, my model is doing very bad for many entities but avg scores are excellent.

Is it right way to calculate? or should no-entity be excluded from these results to get better picture of model?

Hi. Yes - that seems somewhat cheeky. Here they are treating non-entities as an entity named “no_entity”. Recall = True Positives / (True positives + False Negatives). Precision = True Positives / (True Positives + False Positives). F1-Score = 2x Precision x Recall / (Precision + Recall). Notice here that we never see any of the calculations using True Negatives. (this is the number of times the classifier correctly identifies a non-entity as not being any entity). This means that this doesn’t usually contribute to the F1-score. However, by creating a label called ‘no_entity’ , they can now count it toward the True Positives score - which does impact the F1-score. It isnt so bad if its just to work out the precision, recall and f1-score of “no_entity” on its own. However, to include it in the average - skews the results somewhat. I’d say - take it out and recalculate the averages. You’d get a more accurate representation of the NER’s performance.

1 Like