# Rasa NLU crossvalidation result

**URL:** <https://forum.rasa.com/t/rasa-nlu-crossvalidation-result/2147>\
**Category:** Rasa Open Source\
**Created:** [October 24, 2018, 11:30am UTC](https://forum.rasa.com/t/rasa-nlu-crossvalidation-result/2147 "2018-10-24T11:30:37Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![rohitharitash](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/rohitharitash/32/844_2.png) [@rohitharitash](https://forum.rasa.com/u/rohitharitash)\
**Post date:** [October 24, 2018, 11:30am UTC](https://forum.rasa.com/t/rasa-nlu-crossvalidation-result/2147/1 "2018-10-24T11:30:37Z")

</div>

Hi guys,

I am sharing my rasa nlu model’s cross validation evaluation result. I think my model is over fitting. Can you please have a look and suggest, how can we avoid this.

1. 2018-10-24 16:48:52 INFO rasa\_nlu.classifiers.embedding\_intent\_classifier - Finished training

2. embedding policy, loss=0.009, train accuracy=1.000

3. 2018-10-24 16:48:52 INFO rasa\_nlu.model - Finished training component.

4. 2018-10-24 16:48:56 INFO **main** - CV evaluation (n=10)

5. 2018-10-24 16:48:56 INFO **main** - Intent evaluation results

6. 2018-10-24 16:48:56 INFO **main** - train Accuracy: 1.000 (0.000)

7. 2018-10-24 16:48:56 INFO **main** - train Precision: 1.000 (0.000)

8. 2018-10-24 16:48:56 INFO **main** - train F1-score: 1.000 (0.000)

9. 2018-10-24 16:48:56 INFO **main** - test Accuracy: 0.940 (0.020)

10. 2018-10-24 16:48:56 INFO **main** - test Precision: 0.978 (0.012)

11. 2018-10-24 16:48:56 INFO **main** - test F1-score: 0.952 (0.016)

12. 2018-10-24 16:48:56 INFO **main** - Entity evaluation results

13. 2018-10-24 16:48:56 INFO **main** - Entity extractor: ner\_crf

14. 2018-10-24 16:48:56 INFO **main** - train Accuracy: 1.000 (0.000)

15. 2018-10-24 16:48:56 INFO **main** - train Precision: 1.000 (0.000)

16. 2018-10-24 16:48:56 INFO **main** - train F1-score: 1.000 (0.000)

17. 2018-10-24 16:48:56 INFO **main** - Entity extractor: ner\_crf

18. 2018-10-24 16:48:56 INFO **main** - test Accuracy: 0.996 (0.004)

19. 2018-10-24 16:48:56 INFO **main** - test Precision: 0.996 (0.004)

20. 2018-10-24 16:48:56 INFO **main** - test F1-score: 0.996 (0.004)

21. 2018-10-24 16:48:56 INFO **main** - Finished evaluation

22. And my input details

23. INFO:rasa\_nlu.training\_data.training\_data:Training data stats:

24. 

```
- intent examples: 796 (8 distinct intents)

```

25. 

```
- Found intents: 'affirm', 'greet', 'enter_data', 'what_is_your_name', 'goodbye', 'order', 'are_you_a_robot', 'ask_howdoing'

```

26. 

```
- entity examples: 644 (3 distinct entities)

```

27. 

```
- found entities: 'phoneNumber', 'email', 'product'

```

Thanks

---

<div class="post-metadata">

**Author:** ![souvikg10](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/souvikg10/32/93_2.png) [@souvikg10](https://forum.rasa.com/u/souvikg10)\
**Post date:** [October 24, 2018, 12:09pm UTC](https://forum.rasa.com/t/rasa-nlu-crossvalidation-result/2147/2 "2018-10-24T12:09:53Z")

</div>

Can you format it? with ```

it is difficult to read

---

<div class="post-metadata">

**Author:** ![rohitharitash](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/rohitharitash/32/844_2.png) [@rohitharitash](https://forum.rasa.com/u/rohitharitash)\
**Post date:** [October 25, 2018, 11:03am UTC](https://forum.rasa.com/t/rasa-nlu-crossvalidation-result/2147/3 "2018-10-25T11:03:11Z")

</div>

HI,

I have formatted the logs. Please have a look and suggest.

Thanks

---

<div class="post-metadata">

**Author:** ![souvikg10](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/souvikg10/32/93_2.png) [@souvikg10](https://forum.rasa.com/u/souvikg10)\
**Post date:** [October 25, 2018, 12:06pm UTC](https://forum.rasa.com/t/rasa-nlu-crossvalidation-result/2147/4 "2018-10-25T12:06:20Z")

</div>

> [@rohitharitash](#):
>
> - 2018-10-24 16:48:56 INFO **main** - test F1-score: 0.952 (0.016)

it doesn’t seem to overfit because the difference between train vs test F1 is not significantly higher however the accuracy of 1 for train does seem strange. Do you have your confusion matrix, do you see any confusion between intents?

is this the tensorflow pipeline?

It could also be that your train vs test split during crossvalidation creeps a bias due to imbalanced dataset. Do you have a validation dataset that the bot has never seen, try to evaluate on that without cross-validation