Rasa test not loading all intents

Rasa Open Source
1

I have a Rasa 2.6.0 project that has 39 intents broken down into 4 different domain.yml files each of which is in their own directory under the main data directory along with associated stories.yml,rules.yml and nlu.yml

Running rasa data validate --domain data works and returns No story structure conflicts found.

But when running this statement

rasa test nlu -f 2 --cross-validation or simply rasa test

I get the following seemingly random list of intents that seem to be not included as part of my domain.

rasa test nlu -f 2 --cross-validation              
/home/jwheat/anaconda2/envs/rasa260/lib/python3.8/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/common-general/rules.yml': 
Found intent 'general_wait_estimate' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.8/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/common-general/rules.yml':
Found intent 'forget' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.8/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/common-helpdesk-triage/rules.yml': 
Found intent 'initial_contact' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.8/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/common-helpdesk-triage/stories.yml': 
Found intent 'triage_handoff_to_agent' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.8/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/common-helpdesk-triage/stories.yml':     
Found intent 'triage_submit_incident_request' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.8/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/account-reactivation/stories.yml':
Found intent 'abilities' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.8/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/account-reactivation/stories.yml':
Found intent 'account_reactivation_types_of_accounts' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.8/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/account-reactivation/stories.yml':
Found intent 'deny' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.8/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/account-reactivation/stories.yml':
Found intent 'affirm' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.8/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/account-reactivation/stories.yml':
Found intent 'account_reactivation' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.8/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/account-reactivation/stories.yml':
Found intent 'account_lapse_30' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.8/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/account-reactivation/stories.yml':
Found intent 'account_lapse_90' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories
/home/jwheat/anaconda2/envs/rasa260/lib/python3.8/site-packages/rasa/shared/utils/io.py:97: UserWarning: Issue found in 'data/account-reactivation/stories.yml':
Found intent 'account_lapse_90plus' in stories which is not part of the domain.
  More info at https://rasa.com/docs/rasa/stories

Rasa goes ahead and trains and runs the tests, and the results are pretty bad, I’m guessing due to these missing intents.

Any ideas if this is a bug, or some malformed training data? The intents all look ok to me.

Each of these indeed is included in the respective domain under the intents: key.

I’m not sure why it’s cherry picking these out as a problem.

2 Likes
2

Look at this, it’s unresolved at the moment :

1 Like
3

Thanks, and this is funny and a bit disturbing - looks like I already asked this - Testing with split domain files = UserWarnings I was even tagged at the end of that issue.

I must be losing it.

HOWEVER, this time it was with Rasa 2.6.0, so maybe I feel ok about double posting :confounded:

4

Hi. I am also having the same UserWarning, although I only have one domain.yml and nlu/stories files. My project directory is structured just as the inital directory (after rasa init) the only difference is that I removed rules.yml, other files, and folders named the same.

rasa data validate Does not show any problems with my training files.

1 Like
5

Haha, I never thought to try it with the basic rasa init bot. I just assumed it was my split files and directories.

6

Has this been resolved? I am running into this as well. Using -d domain_location gives the all-green when using data validate, but for test nlu --cross-validation it always throws these warnings regardless of what I try or how my directories and files are structured.