# Error when testing Rasa NLU model

**URL:** <https://forum.rasa.com/t/error-when-testing-rasa-nlu-model/41650>\
**Category:** Rasa Open Source\
**Created:** [March 24, 2021, 12:40am UTC](https://forum.rasa.com/t/error-when-testing-rasa-nlu-model/41650 "2021-03-24T00:40:51Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![varton](https://avatars.discourse-cdn.com/v4/letter/v/4491bb/32.png) [@varton](https://forum.rasa.com/u/varton)\
**Post date:** [March 24, 2021, 12:40am UTC](https://forum.rasa.com/t/error-when-testing-rasa-nlu-model/41650/1 "2021-03-24T00:40:51Z")

</div>

Hi,

I am using Rasa 2.3.0 and I want to evaluate a train rasa nlu model on a test set. To do that, I execute the following command: `rasa test nlu -m models/nlu-20210323-152823 -u train_test_split/test_data.yml`

But the above command does not generate the intent\_report file. This is the stack trace:

`2021-03-23 20:37:34 INFO rasa.model - Loading model models/nlu-20210323-152823.tar.gz... /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/rasa/utils/train_utils.py:437: UserWarning: constrain_similarities is set to `False`. It is recommended to set it to `True`when using cross-entropy loss. It will be set to`True`by default, Rasa Open Source 3.0.0 onwards. category=UserWarning, /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/rasa/utils/train_utils.py:410: UserWarning: model_confidence is set to`softmax`. It is recommended to set it to `cosine`. It will be set to `cosine` by default, Rasa Open Source 3.0.0 onwards. category=UserWarning, 2021-03-23 20:37:41 INFO rasa.nlu.test - Running model for predictions: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 60.40it/s] 2021-03-23 20:37:41 INFO rasa.nlu.test - Entity evaluation results: 2021-03-23 20:37:41 INFO rasa.nlu.test - Evaluation for entity extractor: CRFEntityExtractor 2021-03-23 20:37:41 WARNING rasa.test - No labels to evaluate. Skip evaluation. 2021-03-23 20:37:41 INFO rasa.nlu.test - Classification report saved to results/CRFEntityExtractor_report.json. 2021-03-23 20:37:41 INFO rasa.nlu.test - Incorrect entity predictions saved to results/CRFEntityExtractor_errors.json. 2021-03-23 20:37:41 INFO rasa.utils.plotting - Confusion matrix, without normalization: [[0 0] [1 4]] /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/rasa/utils/plotting.py:193: UserWarning: Attempting to set identical left == right == 0 results in singular transformations; automatically expanding. axes[0].set(yticks=yticks, xlim=(0, max_xlims[0]), ylim=(min_ylim, max_ylim)) 2021-03-23 20:37:43 INFO rasa.nlu.test - Evaluation for entity extractor: DIETClassifier 2021-03-23 20:37:43 WARNING rasa.test - No labels to evaluate. Skip evaluation. 2021-03-23 20:37:43 INFO rasa.nlu.test - Classification report saved to results/DIETClassifier_report.json. 2021-03-23 20:37:43 INFO rasa.nlu.test - Every entity was predicted correctly by the model. 2021-03-23 20:37:43 INFO rasa.utils.plotting - Confusion matrix, without normalization: [[5]] /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/rasa/utils/plotting.py:193: UserWarning: Attempting to set identical left == right == 0 results in singular transformations; automatically expanding. axes[0].set(yticks=yticks, xlim=(0, max_xlims[0]), ylim=(min_ylim, max_ylim)) /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/rasa/utils/plotting.py:193: UserWarning: Attempting to set identical bottom == top == 0.0 results in singular transformations; automatically expanding. axes[0].set(yticks=yticks, xlim=(0, max_xlims[0]), ylim=(min_ylim, max_ylim))`

Any help to solve this issue is appreciated.

---

<div class="post-metadata">

**Author:** ![desmarchris](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/desmarchris/32/12277_2.png) [@desmarchris](https://forum.rasa.com/u/desmarchris)\
**Post date:** [March 25, 2021, 6:49pm UTC](https://forum.rasa.com/t/error-when-testing-rasa-nlu-model/41650/2 "2021-03-25T18:49:24Z")

</div>

hey @varton, can you share what your test\_data.yml looks like?

---

<div class="post-metadata">

**Author:** ![jonathanpwheat](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/jonathanpwheat/32/4880_2.png) [@jonathanpwheat](https://forum.rasa.com/u/jonathanpwheat)\
**Post date:** [May 24, 2021, 4:35pm UTC](https://forum.rasa.com/t/error-when-testing-rasa-nlu-model/41650/3 "2021-05-24T16:35:30Z")

</div>

I just wanted to jump in here, I realize this is many months old, but even a straight-out-of-the-box `rasa-init` with no changes does the same thing.

There is an issue filed in GitHub. Seems to be an issue with 2.x testing

> <https://github.com/RasaHQ/rasa/issues/8135>
>
> Rasa version:
> 2.3.4
> Rasa SDK version (if used & relevant):
> 2.3.1
> Rasa X version (if used & relevant):
> Python version:
> 3.8.8
> Operating system (windows, osx, ...):
> Linux-4.4.0-186-generic-x86\_64-with-glibc2.10
> Issue:
> This is annoying....

Here’s what it looks like with 0 changes

```auto
rasa test
2021-05-24 12:32:27 INFO rasa.model - Loading model models/20210524-122350.tar.gz...
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/utils/train_utils.py:565: UserWarning: model_confidence is set to `softmax`. It is recommended to try using `model_confidence=linear_norm` to make it easier to tune fallback thresholds.
  category=UserWarning,
2021-05-24 12:32:43 INFO rasa.shared.utils.validation - The 'version' key is missing in the training data file /mnt/d/Code/Playground/RASA2/rasa-init/tests/test_stories.yml. Rasa Open Source will read the file as a version '2.0' file. See https://rasa.com/docs/rasa/training-data-format.
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in './tests/test_stories.yml': 
Found intent 'greet' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in './tests/test_stories.yml':
Found intent 'mood_great' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in './tests/test_stories.yml':
Found intent 'goodbye' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in './tests/test_stories.yml':
Found intent 'mood_unhappy' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in './tests/test_stories.yml':
Found intent 'affirm' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in './tests/test_stories.yml':
Found intent 'deny' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in './tests/test_stories.yml':
Found intent 'bot_challenge' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
Processed story blocks: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:00<00:00, 1999.19it/s, # trackers=1]2021-05-24 12:32:43 INFO rasa.core.test - Evaluating 7 stories
Progress:
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:00<00:00, 12.98it/s]2021-05-24 12:32:44 INFO rasa.core.test - Finished collecting predictions.
2021-05-24 12:32:44 INFO rasa.core.test - Evaluation Results on END-TO-END level:
2021-05-24 12:32:44 INFO rasa.core.test - Correct: 7 / 7
2021-05-24 12:32:44 INFO rasa.core.test - Accuracy: 1.000
2021-05-24 12:32:44 INFO rasa.core.test - Stories report saved to results/story_report.json.
2021-05-24 12:32:44 INFO rasa.nlu.test - Evaluation for entity extractor: TEDPolicy 
2021-05-24 12:32:44 WARNING rasa.model_testing - No labels to evaluate. Skip evaluation.
2021-05-24 12:32:44 INFO rasa.nlu.test - Classification report saved to results/TEDPolicy_report.json.
2021-05-24 12:32:44 INFO rasa.nlu.test - Every entity was predicted correctly by the model.
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/utils/plotting.py:109: UserWarning: Attempting to set identical left == right == -0.5 results in 
singular transformations; automatically expanding.
  norm=LogNorm(vmin=zmin, vmax=zmax),
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/utils/plotting.py:109: UserWarning: Attempting to set identical bottom == top == -0.5 results in 
singular transformations; automatically expanding.
  norm=LogNorm(vmin=zmin, vmax=zmax),
2021-05-24 12:32:44 INFO rasa.utils.plotting - Confusion matrix, without normalization:
[]
2021-05-24 12:33:00 INFO rasa.core.test - Evaluation Results on ACTION level:
2021-05-24 12:33:00 INFO rasa.core.test - Correct: 35 / 35
2021-05-24 12:33:00 INFO rasa.core.test - F1-Score: 1.000
2021-05-24 12:33:00 INFO rasa.core.test - Precision: 1.000
2021-05-24 12:33:00 INFO rasa.core.test - Accuracy: 1.000
2021-05-24 12:33:00 INFO rasa.core.test - In-data fraction: 1
2021-05-24 12:33:01 INFO rasa.utils.plotting - Confusion matrix, without normalization: 
[[16 0 0 0 0 0 0]
 [0 3 0 0 0 0 0]
 [0 0 3 0 0 0 0]
 [0 0 0 4 0 0 0]
 [0 0 0 0 5 0 0]
 [0 0 0 0 0 3 0]
 [0 0 0 0 0 0 1]]
Failed stories written to 'results/failed_test_stories.yml'
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/rules.yml': 
Found intent 'goodbye' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/rules.yml':
Found intent 'bot_challenge' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/stories.yml': 
Found intent 'greet' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/stories.yml':
Found intent 'mood_great' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/stories.yml':
Found intent 'mood_unhappy' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/stories.yml':
Found intent 'affirm' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/stories.yml':
Found intent 'deny' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
2021-05-24 12:33:02 INFO rasa.model - Loading model models/20210524-122350.tar.gz...
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/utils/train_utils.py:565: UserWarning: model_confidence is set to `softmax`. It is recommended to try using `model_confidence=linear_norm` to make it easier to tune fallback thresholds.
  category=UserWarning,
2021-05-24 12:33:07 INFO rasa.nlu.test - Running model for predictions:
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 69/69 [00:00<00:00, 119.25it/s]2021-05-24 12:33:08 INFO rasa.nlu.test - Intent evaluation results:
2021-05-24 12:33:08 INFO rasa.nlu.test - Intent Evaluation: Only considering those 69 examples that have a defined intent out of 69 examples.
2021-05-24 12:33:08 INFO rasa.nlu.test - Classification report saved to results/intent_report.json.
2021-05-24 12:33:08 INFO rasa.nlu.test - Incorrect intent predictions saved to results/intent_errors.json.
2021-05-24 12:33:08 INFO rasa.utils.plotting - Confusion matrix, without normalization: 
[[6 0 0 0 0 0 0]
 [0 4 0 0 0 0 0]
 [0 0 7 0 0 0 0]
 [0 0 0 10 1 0 0]
 [0 0 0 0 13 0 0]
 [0 0 0 0 0 14 0]
 [0 0 0 0 0 0 14]]
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/utils/plotting.py:223: UserWarning: Attempting to set identical left == right == 0 results in singular transformations; automatically expanding.
  axes[1].set(yticks=yticks, xlim=(0, max_xlims[1]), ylim=(min_ylim, max_ylim))

```

---

<div class="post-metadata">

**Author:** ![AminaDerouiche](https://avatars.discourse-cdn.com/v4/letter/a/d07c76/32.png) [@AminaDerouiche](https://forum.rasa.com/u/AminaDerouiche)\
**Post date:** [June 4, 2021, 12:08pm UTC](https://forum.rasa.com/t/error-when-testing-rasa-nlu-model/41650/4 "2021-06-04T12:08:56Z")

</div>

Hello

Do you have an idea if this is fixed or not ? I saw that the issue has been closed but I’m having the same issue still. I’m at 2.5 right now, should I upgrade to my version in order for it to work?

Thanks

---

<div class="post-metadata">

**Author:** ![jonathanpwheat](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/jonathanpwheat/32/4880_2.png) [@jonathanpwheat](https://forum.rasa.com/u/jonathanpwheat)\
**Post date:** [June 4, 2021, 9:12pm UTC](https://forum.rasa.com/t/error-when-testing-rasa-nlu-model/41650/5 "2021-06-04T21:12:05Z")

</div>

It looks like it was closed with issue 8388 ([Fix for missing intent warnings when running rasa test by b-quachtran · Pull Request #8388 · RasaHQ/rasa · GitHub](https://github.com/RasaHQ/rasa/pull/8388)) but I don’t see reference to that issue in the change log yet ([Rasa Open Source Change Log](https://rasa.com/docs/rasa/changelog/))

It was merged into the main branch, so if you wanted to, I supposed you could clone the repo and give it a go from scratch 🙂 Or wait a couple weeks until they release 2.6.4

---

<div class="post-metadata">

**Author:** ![jonathanpwheat](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/jonathanpwheat/32/4880_2.png) [@jonathanpwheat](https://forum.rasa.com/u/jonathanpwheat)\
**Post date:** [June 21, 2021, 1:18pm UTC](https://forum.rasa.com/t/error-when-testing-rasa-nlu-model/41650/6 "2021-06-21T13:18:52Z")

</div>

@AminaDerouiche

Version 2.7.0 has been released and it fixes #8388 - [Rasa Open Source Change Log](https://rasa.com/docs/rasa/changelog/#bugfixes)

---

<div class="post-metadata">

**Author:** ![AminaDerouiche](https://avatars.discourse-cdn.com/v4/letter/a/d07c76/32.png) [@AminaDerouiche](https://forum.rasa.com/u/AminaDerouiche)\
**Post date:** [June 21, 2021, 1:19pm UTC](https://forum.rasa.com/t/error-when-testing-rasa-nlu-model/41650/7 "2021-06-21T13:19:51Z")

</div>

@jonathanpwheat thank you for your reactivity 🙂