Rasa NLU crossvalidation result

Rasa Open Source
1

Hi guys,

I am sharing my rasa nlu model’s cross validation evaluation result. I think my model is over fitting. Can you please have a look and suggest, how can we avoid this.

  1. 2018-10-24 16:48:52 INFO rasa_nlu.classifiers.embedding_intent_classifier - Finished training

  2. embedding policy, loss=0.009, train accuracy=1.000

  3. 2018-10-24 16:48:52 INFO rasa_nlu.model - Finished training component.

  4. 2018-10-24 16:48:56 INFO main - CV evaluation (n=10)

  5. 2018-10-24 16:48:56 INFO main - Intent evaluation results

  6. 2018-10-24 16:48:56 INFO main - train Accuracy: 1.000 (0.000)

  7. 2018-10-24 16:48:56 INFO main - train Precision: 1.000 (0.000)

  8. 2018-10-24 16:48:56 INFO main - train F1-score: 1.000 (0.000)

  9. 2018-10-24 16:48:56 INFO main - test Accuracy: 0.940 (0.020)

  10. 2018-10-24 16:48:56 INFO main - test Precision: 0.978 (0.012)

  11. 2018-10-24 16:48:56 INFO main - test F1-score: 0.952 (0.016)

  12. 2018-10-24 16:48:56 INFO main - Entity evaluation results

  13. 2018-10-24 16:48:56 INFO main - Entity extractor: ner_crf

  14. 2018-10-24 16:48:56 INFO main - train Accuracy: 1.000 (0.000)

  15. 2018-10-24 16:48:56 INFO main - train Precision: 1.000 (0.000)

  16. 2018-10-24 16:48:56 INFO main - train F1-score: 1.000 (0.000)

  17. 2018-10-24 16:48:56 INFO main - Entity extractor: ner_crf

  18. 2018-10-24 16:48:56 INFO main - test Accuracy: 0.996 (0.004)

  19. 2018-10-24 16:48:56 INFO main - test Precision: 0.996 (0.004)

  20. 2018-10-24 16:48:56 INFO main - test F1-score: 0.996 (0.004)

  21. 2018-10-24 16:48:56 INFO main - Finished evaluation

  22. And my input details

  23. INFO:rasa_nlu.training_data.training_data:Training data stats:

  24. - intent examples: 796 (8 distinct intents)

  25. - Found intents: 'affirm', 'greet', 'enter_data', 'what_is_your_name', 'goodbye', 'order', 'are_you_a_robot', 'ask_howdoing'

  26. - entity examples: 644 (3 distinct entities)

  27. - found entities: 'phoneNumber', 'email', 'product'

Thanks

2

Can you format it? with ```

it is difficult to read

1 Like
3

HI,

I have formatted the logs. Please have a look and suggest.

Thanks

4

it doesn’t seem to overfit because the difference between train vs test F1 is not significantly higher however the accuracy of 1 for train does seem strange. Do you have your confusion matrix, do you see any confusion between intents?

is this the tensorflow pipeline?

It could also be that your train vs test split during crossvalidation creeps a bias due to imbalanced dataset. Do you have a validation dataset that the bot has never seen, try to evaluate on that without cross-validation

1 Like