Problem with retrieval intents and ResponseSelector in with rasa 2.x

Hi all. there seems to be some problems with retrieval intents and the ResponseSelector responses in rasa open source 2.x

the problem can be reproduced using the responseselectorbot example provide in RasaHQ/rasa.

issue 1 - rasa 2.1.3 - Inconsistency between rasa documentation and rasa example in RasaHQ

Step-by-step Guide on Using Response Selector for FAQs and Chitchat indicates that ResponcesSelector responses should be added in domain.yml.

However, the responseselectorbot example rather place them outside of the domain file and more specifically in the nlu.yml.

Using the available responseselectorbot example (which defines responses in the nlu.yml), the command rasa data validate succeed but rasa train throws:

c:\programdata\anaconda3\envs\ihi\lib\site-packages\rasa\shared\utils\io.py:93: UserWarning: Action 'utter_chitchat' is listed as a response action in the domain
file, but there is no matching response defined. Please check your domain.
  More info at https://rasa.com/docs/rasa/responses

However, if one put the responses in the domain.yml (as suggested by the documentation) the command rasa data validate fails with:

2020-12-18 11:17:37 INFO     rasa.validator  - Validating intents...
2020-12-18 11:17:37 INFO     rasa.validator  - Validating uniqueness of intents and stories...
2020-12-18 11:17:37 INFO     rasa.validator  - Validating utterances...
c:\programdata\anaconda3\envs\ihi\lib\site-packages\rasa\shared\utils\io.py:93: UserWarning: The action 'utter_chitchat' is used in the stories, but is not a vali
d utterance action. Please make sure the action is listed in your domain and there is a template defined with its name.
  More info at https://rasa.com/docs/rasa/actions#utterance-actions
Project validation completed with errors.

Moreover, rasa train throws:

c:\programdata\anaconda3\envs\ihi\lib\site-packages\rasa\shared\utils\io.py:93: UserWarning: Action 'utter_chitchat' is listed as a response action in the domain
file, but there is no matching response defined. Please check your domain.
  More info at https://rasa.com/docs/rasa/responses

This is obtained with:

* platform: win7-64 
* rasa: 2.1.3
* rasa-sdk: 2.2.0
* rasa-x: 0.34.0
* tensorflow: 2.3.1 
* cudatoolkit: 10.1.243
* python: 3.7.9 

Several posts have reported related issues lately:

issue 2 - rasa 2.2.1 - rasa data validate crashes with responses defined in nlu.yml

This obtained with:

* platform: win7-64 
* rasa: 2.2.1
* rasa-sdk: 2.2.0
* rasa-x: 0.34.0
* tensorflow: 2.3.1 
* cudatoolkit: 10.1.243
* python: 3.7.9 
(ihi2) D:rasaHQ\rasa\examples\responseselectorbot>rasa data validate
2020-12-18 11:55:40.110817: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2020-12-18 11:55:44.278234: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library nvcuda.dll
2020-12-18 11:55:44.522258: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:02:00.0 name: GeForce GTX 1080 computeCapability: 6.1
coreClock: 1.7335GHz coreCount: 20 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 298.32GiB/s
2020-12-18 11:55:44.549261: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 1 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 980 computeCapability: 5.2
coreClock: 1.2155GHz coreCount: 16 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 208.91GiB/s
2020-12-18 11:55:44.556262: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2020-12-18 11:55:44.567263: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2020-12-18 11:55:44.577264: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2020-12-18 11:55:44.583264: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2020-12-18 11:55:44.595266: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2020-12-18 11:55:44.603266: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2020-12-18 11:55:44.613267: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2020-12-18 11:55:44.637270: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0, 1
2020-12-18 11:55:45 INFO     rasa.validator  - Validating intents...
2020-12-18 11:55:45 INFO     rasa.validator  - Validating uniqueness of intents and stories...
2020-12-18 11:55:45 INFO     rasa.validator  - Validating utterances...
2020-12-18 11:55:45 INFO     rasa.validator  - Story structure validation...
Processed story blocks: 100%|ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆ| 3/3 [00:00<00:00, 599.96it/s, # trackers=1]
2020-12-18 11:55:45 INFO     rasa.core.training.story_conflict  - Considering all preceding turns for conflict analysis.
Traceback (most recent call last):
  File "c:\programdata\anaconda3\envs\ihi2\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\programdata\anaconda3\envs\ihi2\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\ProgramData\Anaconda3\envs\ihi2\Scripts\rasa.exe\__main__.py", line 7, in <module>
  File "c:\programdata\anaconda3\envs\ihi2\lib\site-packages\rasa\__main__.py", line 116, in main
    cmdline_arguments.func(cmdline_arguments)
  File "c:\programdata\anaconda3\envs\ihi2\lib\site-packages\rasa\cli\data.py", line 262, in validate_files
    and _validate_story_structure(validator, args)
  File "c:\programdata\anaconda3\envs\ihi2\lib\site-packages\rasa\cli\data.py", line 297, in _validate_story_structure
    not args.fail_on_warnings, max_history=args.max_history
  File "c:\programdata\anaconda3\envs\ihi2\lib\site-packages\rasa\validator.py", line 249, in verify_story_structure
    trackers, self.domain, max_history, self.nlu_config
  File "c:\programdata\anaconda3\envs\ihi2\lib\site-packages\rasa\core\training\story_conflict.py", line 185, in find_story_conflicts
    trackers, domain, max_history, tokenizer
  File "c:\programdata\anaconda3\envs\ihi2\lib\site-packages\rasa\core\training\story_conflict.py", line 245, in _find_conflicting_states
    for element in _sliced_states_iterator(trackers, domain, max_history, tokenizer):
  File "c:\programdata\anaconda3\envs\ihi2\lib\site-packages\rasa\core\training\story_conflict.py", line 333, in _sliced_states_iterator
    _apply_tokenizer_to_states(tokenizer, sliced_states)
  File "c:\programdata\anaconda3\envs\ihi2\lib\site-packages\rasa\core\training\story_conflict.py", line 351, in _apply_tokenizer_to_states
    Message({TEXT: state[USER][TEXT]}), TEXT
KeyError: 'text'

rasa train produces:

c:\programdata\anaconda3\envs\ihi2\lib\site-packages\rasa\shared\utils\io.py:93: UserWarning: Action 'utter_chitchat' is listed as a response action in th
is no matching response defined. Please check your domain.
  More info at https://rasa.com/docs/rasa/responses

Thanks for considering

3 Likes

Hi @gdl1 thanks for bringing this up, the example needs to be updated to match the documentation.

Issue 2 is a known problem with Rasa 2.2 and, for now, Iā€™d recommend sticking with Rasa 2.1. until this issue is resolved.