Error when testing Rasa NLU model

Hi,

I am using Rasa 2.3.0 and I want to evaluate a train rasa nlu model on a test set. To do that, I execute the following command: rasa test nlu -m models/nlu-20210323-152823 -u train_test_split/test_data.yml

But the above command does not generate the intent_report file. This is the stack trace:

2021-03-23 20:37:34 INFO rasa.model - Loading model models/nlu-20210323-152823.tar.gz... /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/rasa/utils/train_utils.py:437: UserWarning: constrain_similarities is set to False. It is recommended to set it to Truewhen using cross-entropy loss. It will be set toTrueby default, Rasa Open Source 3.0.0 onwards. category=UserWarning, /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/rasa/utils/train_utils.py:410: UserWarning: model_confidence is set tosoftmax. It is recommended to set it to cosine. It will be set to cosine by default, Rasa Open Source 3.0.0 onwards. category=UserWarning, 2021-03-23 20:37:41 INFO rasa.nlu.test - Running model for predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 1/1 [00:00<00:00, 60.40it/s] 2021-03-23 20:37:41 INFO rasa.nlu.test - Entity evaluation results: 2021-03-23 20:37:41 INFO rasa.nlu.test - Evaluation for entity extractor: CRFEntityExtractor 2021-03-23 20:37:41 WARNING rasa.test - No labels to evaluate. Skip evaluation. 2021-03-23 20:37:41 INFO rasa.nlu.test - Classification report saved to results/CRFEntityExtractor_report.json. 2021-03-23 20:37:41 INFO rasa.nlu.test - Incorrect entity predictions saved to results/CRFEntityExtractor_errors.json. 2021-03-23 20:37:41 INFO rasa.utils.plotting - Confusion matrix, without normalization: [[0 0] [1 4]] /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/rasa/utils/plotting.py:193: UserWarning: Attempting to set identical left == right == 0 results in singular transformations; automatically expanding. axes[0].set(yticks=yticks, xlim=(0, max_xlims[0]), ylim=(min_ylim, max_ylim)) 2021-03-23 20:37:43 INFO rasa.nlu.test - Evaluation for entity extractor: DIETClassifier 2021-03-23 20:37:43 WARNING rasa.test - No labels to evaluate. Skip evaluation. 2021-03-23 20:37:43 INFO rasa.nlu.test - Classification report saved to results/DIETClassifier_report.json. 2021-03-23 20:37:43 INFO rasa.nlu.test - Every entity was predicted correctly by the model. 2021-03-23 20:37:43 INFO rasa.utils.plotting - Confusion matrix, without normalization: [[5]] /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/rasa/utils/plotting.py:193: UserWarning: Attempting to set identical left == right == 0 results in singular transformations; automatically expanding. axes[0].set(yticks=yticks, xlim=(0, max_xlims[0]), ylim=(min_ylim, max_ylim)) /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/rasa/utils/plotting.py:193: UserWarning: Attempting to set identical bottom == top == 0.0 results in singular transformations; automatically expanding. axes[0].set(yticks=yticks, xlim=(0, max_xlims[0]), ylim=(min_ylim, max_ylim))

Any help to solve this issue is appreciated.

hey @varton, can you share what your test_data.yml looks like?

I just wanted to jump in here, I realize this is many months old, but even a straight-out-of-the-box rasa-init with no changes does the same thing.

There is an issue filed in GitHub. Seems to be an issue with 2.x testing

Here’s what it looks like with 0 changes

rasa test
2021-05-24 12:32:27 INFO     rasa.model  - Loading model models/20210524-122350.tar.gz...
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/utils/train_utils.py:565: UserWarning: model_confidence is set to `softmax`. It is recommended to try using `model_confidence=linear_norm` to make it easier to tune fallback thresholds.
  category=UserWarning,
2021-05-24 12:32:43 INFO     rasa.shared.utils.validation  - The 'version' key is missing in the training data file /mnt/d/Code/Playground/RASA2/rasa-init/tests/test_stories.yml. Rasa Open Source will read the file as a version '2.0' file. See https://rasa.com/docs/rasa/training-data-format.
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in './tests/test_stories.yml': 
Found intent 'greet' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in './tests/test_stories.yml':
Found intent 'mood_great' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in './tests/test_stories.yml':
Found intent 'goodbye' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in './tests/test_stories.yml':
Found intent 'mood_unhappy' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in './tests/test_stories.yml':
Found intent 'affirm' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in './tests/test_stories.yml':
Found intent 'deny' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in './tests/test_stories.yml':
Found intent 'bot_challenge' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
Processed story blocks: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 7/7 [00:00<00:00, 1999.19it/s, # trackers=1]2021-05-24 12:32:43 INFO     rasa.core.test  - Evaluating 7 stories
Progress:
100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 7/7 [00:00<00:00, 12.98it/s]2021-05-24 12:32:44 INFO     rasa.core.test  - Finished collecting predictions.
2021-05-24 12:32:44 INFO     rasa.core.test  - Evaluation Results on END-TO-END level:
2021-05-24 12:32:44 INFO     rasa.core.test  -  Correct:          7 / 7
2021-05-24 12:32:44 INFO     rasa.core.test  -  Accuracy:         1.000
2021-05-24 12:32:44 INFO     rasa.core.test  - Stories report saved to results/story_report.json.
2021-05-24 12:32:44 INFO     rasa.nlu.test  - Evaluation for entity extractor: TEDPolicy 
2021-05-24 12:32:44 WARNING  rasa.model_testing  - No labels to evaluate. Skip evaluation.
2021-05-24 12:32:44 INFO     rasa.nlu.test  - Classification report saved to results/TEDPolicy_report.json.
2021-05-24 12:32:44 INFO     rasa.nlu.test  - Every entity was predicted correctly by the model.
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/utils/plotting.py:109: UserWarning: Attempting to set identical left == right == -0.5 results in 
singular transformations; automatically expanding.
  norm=LogNorm(vmin=zmin, vmax=zmax),
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/utils/plotting.py:109: UserWarning: Attempting to set identical bottom == top == -0.5 results in 
singular transformations; automatically expanding.
  norm=LogNorm(vmin=zmin, vmax=zmax),
2021-05-24 12:32:44 INFO     rasa.utils.plotting  - Confusion matrix, without normalization:
[]
2021-05-24 12:33:00 INFO     rasa.core.test  - Evaluation Results on ACTION level:
2021-05-24 12:33:00 INFO     rasa.core.test  -  Correct:          35 / 35
2021-05-24 12:33:00 INFO     rasa.core.test  -  F1-Score:         1.000
2021-05-24 12:33:00 INFO     rasa.core.test  -  Precision:        1.000
2021-05-24 12:33:00 INFO     rasa.core.test  -  Accuracy:         1.000
2021-05-24 12:33:00 INFO     rasa.core.test  -  In-data fraction: 1
2021-05-24 12:33:01 INFO     rasa.utils.plotting  - Confusion matrix, without normalization: 
[[16  0  0  0  0  0  0]
 [ 0  3  0  0  0  0  0]
 [ 0  0  3  0  0  0  0]
 [ 0  0  0  4  0  0  0]
 [ 0  0  0  0  5  0  0]
 [ 0  0  0  0  0  3  0]
 [ 0  0  0  0  0  0  1]]
Failed stories written to 'results/failed_test_stories.yml'
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/rules.yml': 
Found intent 'goodbye' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/rules.yml':
Found intent 'bot_challenge' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/stories.yml': 
Found intent 'greet' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/stories.yml':
Found intent 'mood_great' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/stories.yml':
Found intent 'mood_unhappy' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/stories.yml':
Found intent 'affirm' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/stories.yml':
Found intent 'deny' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
2021-05-24 12:33:02 INFO     rasa.model  - Loading model models/20210524-122350.tar.gz...
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/utils/train_utils.py:565: UserWarning: model_confidence is set to `softmax`. It is recommended to try using `model_confidence=linear_norm` to make it easier to tune fallback thresholds.
  category=UserWarning,
2021-05-24 12:33:07 INFO     rasa.nlu.test  - Running model for predictions:
100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 69/69 [00:00<00:00, 119.25it/s]2021-05-24 12:33:08 INFO     rasa.nlu.test  - Intent evaluation results:
2021-05-24 12:33:08 INFO     rasa.nlu.test  - Intent Evaluation: Only considering those 69 examples that have a defined intent out of 69 examples.
2021-05-24 12:33:08 INFO     rasa.nlu.test  - Classification report saved to results/intent_report.json.
2021-05-24 12:33:08 INFO     rasa.nlu.test  - Incorrect intent predictions saved to results/intent_errors.json.
2021-05-24 12:33:08 INFO     rasa.utils.plotting  - Confusion matrix, without normalization: 
[[ 6  0  0  0  0  0  0]
 [ 0  4  0  0  0  0  0]
 [ 0  0  7  0  0  0  0]
 [ 0  0  0 10  1  0  0]
 [ 0  0  0  0 13  0  0]
 [ 0  0  0  0  0 14  0]
 [ 0  0  0  0  0  0 14]]
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/utils/plotting.py:223: UserWarning: Attempting to set identical left == right == 0 results in singular transformations; automatically expanding.
  axes[1].set(yticks=yticks, xlim=(0, max_xlims[1]), ylim=(min_ylim, max_ylim))

Hello

Do you have an idea if this is fixed or not ? I saw that the issue has been closed but I’m having the same issue still. I’m at 2.5 right now, should I upgrade to my version in order for it to work?

Thanks

It looks like it was closed with issue 8388 (Fix for missing intent warnings when running rasa test by b-quachtran Β· Pull Request #8388 Β· RasaHQ/rasa Β· GitHub) but I don’t see reference to that issue in the change log yet (Rasa Open Source Change Log)

It was merged into the main branch, so if you wanted to, I supposed you could clone the repo and give it a go from scratch :slight_smile: Or wait a couple weeks until they release 2.6.4

1 Like

@AminaDerouiche

Version 2.7.0 has been released and it fixes #8388 - Rasa Open Source Change Log

1 Like

@jonathanpwheat thank you for your reactivity :slight_smile:

1 Like