I am using Rasa 2.3.0 and I want to evaluate a train rasa nlu model on a test set. To do that, I execute the following command:
rasa test nlu -m models/nlu-20210323-152823 -u train_test_split/test_data.yml
But the above command does not generate the intent_report file. This is the stack trace:
2021-03-23 20:37:34 INFO rasa.model - Loading model models/nlu-20210323-152823.tar.gz... /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/rasa/utils/train_utils.py:437: UserWarning: constrain_similarities is set to False. It is recommended to set it to Truewhen using cross-entropy loss. It will be set toTrueby default, Rasa Open Source 3.0.0 onwards. category=UserWarning, /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/rasa/utils/train_utils.py:410: UserWarning: model_confidence is set tosoftmax. It is recommended to set it to cosine. It will be set to cosine by default, Rasa Open Source 3.0.0 onwards. category=UserWarning, 2021-03-23 20:37:41 INFO rasa.nlu.test - Running model for predictions: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 60.40it/s] 2021-03-23 20:37:41 INFO rasa.nlu.test - Entity evaluation results: 2021-03-23 20:37:41 INFO rasa.nlu.test - Evaluation for entity extractor: CRFEntityExtractor 2021-03-23 20:37:41 WARNING rasa.test - No labels to evaluate. Skip evaluation. 2021-03-23 20:37:41 INFO rasa.nlu.test - Classification report saved to results/CRFEntityExtractor_report.json. 2021-03-23 20:37:41 INFO rasa.nlu.test - Incorrect entity predictions saved to results/CRFEntityExtractor_errors.json. 2021-03-23 20:37:41 INFO rasa.utils.plotting - Confusion matrix, without normalization: [[0 0] [1 4]] /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/rasa/utils/plotting.py:193: UserWarning: Attempting to set identical left == right == 0 results in singular transformations; automatically expanding. axes[0].set(yticks=yticks, xlim=(0, max_xlims[0]), ylim=(min_ylim, max_ylim)) 2021-03-23 20:37:43 INFO rasa.nlu.test - Evaluation for entity extractor: DIETClassifier 2021-03-23 20:37:43 WARNING rasa.test - No labels to evaluate. Skip evaluation. 2021-03-23 20:37:43 INFO rasa.nlu.test - Classification report saved to results/DIETClassifier_report.json. 2021-03-23 20:37:43 INFO rasa.nlu.test - Every entity was predicted correctly by the model. 2021-03-23 20:37:43 INFO rasa.utils.plotting - Confusion matrix, without normalization: [[5]] /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/rasa/utils/plotting.py:193: UserWarning: Attempting to set identical left == right == 0 results in singular transformations; automatically expanding. axes[0].set(yticks=yticks, xlim=(0, max_xlims[0]), ylim=(min_ylim, max_ylim)) /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/rasa/utils/plotting.py:193: UserWarning: Attempting to set identical bottom == top == 0.0 results in singular transformations; automatically expanding. axes[0].set(yticks=yticks, xlim=(0, max_xlims[0]), ylim=(min_ylim, max_ylim))
I just wanted to jump in here, I realize this is many months old, but even a straight-out-of-the-box rasa-init with no changes does the same thing.
There is an issue filed in GitHub. Seems to be an issue with 2.x testing
Here’s what it looks like with 0 changes
rasa test
2021-05-24 12:32:27 INFO rasa.model - Loading model models/20210524-122350.tar.gz...
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/utils/train_utils.py:565: UserWarning: model_confidence is set to `softmax`. It is recommended to try using `model_confidence=linear_norm` to make it easier to tune fallback thresholds.
category=UserWarning,
2021-05-24 12:32:43 INFO rasa.shared.utils.validation - The 'version' key is missing in the training data file /mnt/d/Code/Playground/RASA2/rasa-init/tests/test_stories.yml. Rasa Open Source will read the file as a version '2.0' file. See https://rasa.com/docs/rasa/training-data-format.
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in './tests/test_stories.yml':
Found intent 'greet' in stories which is not part of the domain.
More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in './tests/test_stories.yml':
Found intent 'mood_great' in stories which is not part of the domain.
More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in './tests/test_stories.yml':
Found intent 'goodbye' in stories which is not part of the domain.
More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in './tests/test_stories.yml':
Found intent 'mood_unhappy' in stories which is not part of the domain.
More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in './tests/test_stories.yml':
Found intent 'affirm' in stories which is not part of the domain.
More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in './tests/test_stories.yml':
Found intent 'deny' in stories which is not part of the domain.
More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in './tests/test_stories.yml':
Found intent 'bot_challenge' in stories which is not part of the domain.
More info at https://rasa.com/docs/rasa/stories
Processed story blocks: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:00<00:00, 1999.19it/s, # trackers=1]2021-05-24 12:32:43 INFO rasa.core.test - Evaluating 7 stories
Progress:
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:00<00:00, 12.98it/s]2021-05-24 12:32:44 INFO rasa.core.test - Finished collecting predictions.
2021-05-24 12:32:44 INFO rasa.core.test - Evaluation Results on END-TO-END level:
2021-05-24 12:32:44 INFO rasa.core.test - Correct: 7 / 7
2021-05-24 12:32:44 INFO rasa.core.test - Accuracy: 1.000
2021-05-24 12:32:44 INFO rasa.core.test - Stories report saved to results/story_report.json.
2021-05-24 12:32:44 INFO rasa.nlu.test - Evaluation for entity extractor: TEDPolicy
2021-05-24 12:32:44 WARNING rasa.model_testing - No labels to evaluate. Skip evaluation.
2021-05-24 12:32:44 INFO rasa.nlu.test - Classification report saved to results/TEDPolicy_report.json.
2021-05-24 12:32:44 INFO rasa.nlu.test - Every entity was predicted correctly by the model.
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/utils/plotting.py:109: UserWarning: Attempting to set identical left == right == -0.5 results in
singular transformations; automatically expanding.
norm=LogNorm(vmin=zmin, vmax=zmax),
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/utils/plotting.py:109: UserWarning: Attempting to set identical bottom == top == -0.5 results in
singular transformations; automatically expanding.
norm=LogNorm(vmin=zmin, vmax=zmax),
2021-05-24 12:32:44 INFO rasa.utils.plotting - Confusion matrix, without normalization:
[]
2021-05-24 12:33:00 INFO rasa.core.test - Evaluation Results on ACTION level:
2021-05-24 12:33:00 INFO rasa.core.test - Correct: 35 / 35
2021-05-24 12:33:00 INFO rasa.core.test - F1-Score: 1.000
2021-05-24 12:33:00 INFO rasa.core.test - Precision: 1.000
2021-05-24 12:33:00 INFO rasa.core.test - Accuracy: 1.000
2021-05-24 12:33:00 INFO rasa.core.test - In-data fraction: 1
2021-05-24 12:33:01 INFO rasa.utils.plotting - Confusion matrix, without normalization:
[[16 0 0 0 0 0 0]
[ 0 3 0 0 0 0 0]
[ 0 0 3 0 0 0 0]
[ 0 0 0 4 0 0 0]
[ 0 0 0 0 5 0 0]
[ 0 0 0 0 0 3 0]
[ 0 0 0 0 0 0 1]]
Failed stories written to 'results/failed_test_stories.yml'
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/rules.yml':
Found intent 'goodbye' in stories which is not part of the domain.
More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/rules.yml':
Found intent 'bot_challenge' in stories which is not part of the domain.
More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/stories.yml':
Found intent 'greet' in stories which is not part of the domain.
More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/stories.yml':
Found intent 'mood_great' in stories which is not part of the domain.
More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/stories.yml':
Found intent 'mood_unhappy' in stories which is not part of the domain.
More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/stories.yml':
Found intent 'affirm' in stories which is not part of the domain.
More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/stories.yml':
Found intent 'deny' in stories which is not part of the domain.
More info at https://rasa.com/docs/rasa/stories
2021-05-24 12:33:02 INFO rasa.model - Loading model models/20210524-122350.tar.gz...
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/utils/train_utils.py:565: UserWarning: model_confidence is set to `softmax`. It is recommended to try using `model_confidence=linear_norm` to make it easier to tune fallback thresholds.
category=UserWarning,
2021-05-24 12:33:07 INFO rasa.nlu.test - Running model for predictions:
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 69/69 [00:00<00:00, 119.25it/s]2021-05-24 12:33:08 INFO rasa.nlu.test - Intent evaluation results:
2021-05-24 12:33:08 INFO rasa.nlu.test - Intent Evaluation: Only considering those 69 examples that have a defined intent out of 69 examples.
2021-05-24 12:33:08 INFO rasa.nlu.test - Classification report saved to results/intent_report.json.
2021-05-24 12:33:08 INFO rasa.nlu.test - Incorrect intent predictions saved to results/intent_errors.json.
2021-05-24 12:33:08 INFO rasa.utils.plotting - Confusion matrix, without normalization:
[[ 6 0 0 0 0 0 0]
[ 0 4 0 0 0 0 0]
[ 0 0 7 0 0 0 0]
[ 0 0 0 10 1 0 0]
[ 0 0 0 0 13 0 0]
[ 0 0 0 0 0 14 0]
[ 0 0 0 0 0 0 14]]
/home/jwheat/anaconda2/envs/rasa260/lib/python3.7/site-packages/rasa/utils/plotting.py:223: UserWarning: Attempting to set identical left == right == 0 results in singular transformations; automatically expanding.
axes[1].set(yticks=yticks, xlim=(0, max_xlims[1]), ylim=(min_ylim, max_ylim))
Do you have an idea if this is fixed or not ?
I saw that the issue has been closed but I’m having the same issue still.
I’m at 2.5 right now, should I upgrade to my version in order for it to work?
It was merged into the main branch, so if you wanted to, I supposed you could clone the repo and give it a go from scratch Or wait a couple weeks until they release 2.6.4
Or wait a couple weeks until they release 2.6.4