How to handle receiving intents not listed in the domain?

Rasa (2.8.12) throws a ValueError when the user sends an intent message that’s not listed in the domain. I first came across this error as I connected rasa to telegram, and sent a ‘/start’ command which I didn’t have in my domain. This is exactly the same issue that was solved in this post. I fixed it by adding a ‘start’ intent to my domain with accompanying rules/stories.

That solution worked, but I am left wondering how to handle any cases where the user sends a message with a forward slash? Slash commands are quite common on slack and telegram, and the user might just send one by mistake. When this happens Rasa stops responding to follow up messages. The error can be replicated using rasa shell, so it seems it’s not confined to the telegram channel.

2022-03-21 12:21:41 INFO     root  - Rasa server is up and running.
Bot loaded. Type a message and press enter (use '/stop' to exit): 
Your input ->  /sayhi                                                                                                                                                        
/opt/venv/lib/python3.8/site-packages/rasa/shared/utils/io.py:97: UserWarning: Interpreter parsed an intent 'sayhi' which is not defined in the domain. Please make sure all intents are listed in the domain.
  More info at https://rasa.com/docs/rasa/domain
2022-03-21 12:22:24 ERROR    asyncio  - Task exception was never retrieved
future: <Task finished name='Task-2' coro=<configure_app.<locals>.run_cmdline_io() done, defined at /opt/venv/lib/python3.8/site-packages/rasa/core/run.py:131> exception=TimeoutError()>
Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/rasa/core/run.py", line 135, in run_cmdline_io
    await console.record_messages(
  File "/opt/venv/lib/python3.8/site-packages/rasa/core/channels/console.py", line 182, in record_messages
    async for response in bot_responses:
  File "/opt/venv/lib/python3.8/site-packages/rasa/core/channels/console.py", line 137, in _send_message_receive_stream
    async for line in resp.content:
  File "/opt/venv/lib/python3.8/site-packages/aiohttp/streams.py", line 39, in __anext__
    rv = await self.read_func()
  File "/opt/venv/lib/python3.8/site-packages/aiohttp/streams.py", line 338, in readline
    await self._wait("readline")
  File "/opt/venv/lib/python3.8/site-packages/aiohttp/streams.py", line 306, in _wait
    await waiter
  File "/opt/venv/lib/python3.8/site-packages/aiohttp/helpers.py", line 656, in __exit__
    raise asyncio.TimeoutError from None
asyncio.exceptions.TimeoutError
^C2022-03-21 12:46:25 ERROR    asyncio  - Task exception was never retrieved
future: <Task finished name='Task-8' coro=<RestInput.on_message_wrapper() done, defined at /opt/venv/lib/python3.8/site-packages/rasa/core/channels/rest.py:33> exception=ValueError("'sayhi' is not in list")>
Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/rasa/core/channels/rest.py", line 47, in on_message_wrapper
    await on_new_message(message)
  File "/opt/venv/lib/python3.8/site-packages/rasa/core/channels/channel.py", line 89, in handler
    await app.agent.handle_message(message)
  File "/opt/venv/lib/python3.8/site-packages/rasa/core/agent.py", line 577, in handle_message
    return await processor.handle_message(message)
  File "/opt/venv/lib/python3.8/site-packages/rasa/core/processor.py", line 108, in handle_message
    await self._predict_and_execute_next_action(message.output_channel, tracker)
  File "/opt/venv/lib/python3.8/site-packages/rasa/core/processor.py", line 657, in _predict_and_execute_next_action
    action, prediction = self.predict_next_action(tracker)
  File "/opt/venv/lib/python3.8/site-packages/rasa/core/processor.py", line 379, in predict_next_action
    prediction = self._get_next_action_probabilities(tracker)
  File "/opt/venv/lib/python3.8/site-packages/rasa/core/processor.py", line 921, in _get_next_action_probabilities
    prediction = self.policy_ensemble.probabilities_using_best_policy(
  File "/opt/venv/lib/python3.8/site-packages/rasa/core/policies/ensemble.py", line 817, in probabilities_using_best_policy
    winning_prediction = self._best_policy_prediction(tracker, domain, interpreter)
  File "/opt/venv/lib/python3.8/site-packages/rasa/core/policies/ensemble.py", line 694, in _best_policy_prediction
    predictions = {
  File "/opt/venv/lib/python3.8/site-packages/rasa/core/policies/ensemble.py", line 695, in <dictcomp>
    f"policy_{i}_{type(p).__name__}": self._get_prediction(
  File "/opt/venv/lib/python3.8/site-packages/rasa/core/policies/ensemble.py", line 729, in _get_prediction
    prediction = policy.predict_action_probabilities(
  File "/opt/venv/lib/python3.8/site-packages/rasa/core/policies/unexpected_intent_policy.py", line 576, in predict_action_probabilities
    is_unlikely_intent = self._check_unlikely_intent(
  File "/opt/venv/lib/python3.8/site-packages/rasa/core/policies/unexpected_intent_policy.py", line 671, in _check_unlikely_intent
    if not self._should_check_for_intent(query_intent, domain):
  File "/opt/venv/lib/python3.8/site-packages/rasa/core/policies/unexpected_intent_policy.py", line 633, in _should_check_for_intent
    if domain.intents.index(intent) not in self.label_thresholds:
ValueError: 'sayhi' is not in list

Is there something I can do to handle this? I would have thought in cases like these that Rasa would raise an ‘intent not listed’ warning, then ignore the forward slash and try to handle the message like an ordinary message.

I can see two issues saved on github describing the same or similar, no resolution though …

I would look at the channel code for slack and telegram and see about addressing the forward slash there.

Greg

Yes, @stephens, I think that will work. I’ll try it out.

Only thing I’m not keen on is maintaining a list of allowable intent messages so that the forward slash is kept in those cases.

EDIT: Just realised I can get a list of intents by hitting up the ‘domain’ API endpoint /domain?token=mysecret

Good idea, could enhance the channel config to specify the channel should do the API call to lookup the list of intents. But this should only be done at channel init (not every call)

slack:
  slack_channel: "CA003L0XZ"    # channel ID, not a channel name!
  slack_token: "xoxb-XXX"       # token obtained in the next step
  slack_signing_secret: "YYY"   # secret obtained in the next step
  allow_slash_all_intents: True. # default false
  allow_slash_token: mytoken

But allowing a list option is nice because you don’t need a token or api access

slack:
  slack_channel: "CA003L0XZ"    # channel ID, not a channel name!
  slack_token: "xoxb-XXX"       # token obtained in the next step
  slack_signing_secret: "YYY"   # secret obtained in the next step
  allow_slash_intents:
    - start
    - greet

You should open a GH enhancement issue with a proposal for Rasa to review.

Ah good points! I wasn’t thinking about putting the allow list in the channel config. Makes sense.

Coded up and added an enhancement and pull request. Not sure if I should have just done the one… if the pull is rejected I guess the enhancement stays.

In any case, I tried the allowed intent list in credentials route and it seems to be working a treat. My code is a tiny bit different from what’s in pull request as I have a custom connector.

image

‘/thanks’ in this case is not in my allow list. Previously this message would’ve broken the convo.