Http_timeout in slack

Rasa Open Source
1

Fetching the data from the backend is taking more than 3 seconds which is resulting in a timeout in slack bot. How do I resolve this? Is there a way I can send a ack to slack?

2020-10-04 20:58:24 ERROR    connectors.slack_connector  - Exception when trying to handle message.
2020-10-04 20:58:24 ERROR    connectors.slack_connector  - 
Traceback (most recent call last):
  File "/connectors/slack_connector.py", line 347, in process_message
    await on_new_message(user_msg)
  File "/env/lib/python3.7/site-packages/rasa/core/channels/channel.py", line 84, in handler
    await app.agent.handle_message(*args, **kwargs)
  File "/env/lib/python3.7/site-packages/rasa/core/agent.py", line 518, in handle_message
    return await processor.handle_message(message)
  File "/python3.7/site-packages/rasa/core/processor.py", line 97, in handle_message
    await self._predict_and_execute_next_action(message.output_channel, tracker)
  File "env/lib/python3.7/site-packages/rasa/core/processor.py", line 565, in _predict_and_execute_next_action
    action, tracker, output_channel, self.nlg, policy, confidence
  File "env/lib/python3.7/site-packages/rasa/core/processor.py", line 701, in _run_action
    await self.execute_side_effects(events, tracker, output_channel)
  File "env/lib/python3.7/site-packages/rasa/core/processor.py", line 604, in execute_side_effects
    await self._send_bot_messages(events, tracker, output_channel)
  File "/env/lib/python3.7/site-packages/rasa/core/processor.py", line 620, in _send_bot_messages
    await output_channel.send_response(tracker.sender_id, e.message())
  File "/env/lib/python3.7/site-packages/rasa/core/channels/channel.py", line 185, in send_response
    await self.send_text_message(recipient_id, message.pop("text"), **message)
  File "/connectors/slack_connector.py", line 60, in send_text_message
    blocks=block
  File "/connectors/slack_connector.py", line 47, in _post_message
    await self.client.chat_postMessage(channel=channel, **kwargs)
concurrent.futures._base.CancelledError
2020-10-04 20:58:25 WARNING  connectors.slack_connector  - Received retry #1 request from slack due to http_timeout.
2020-10-04 20:58:25 WARNING  connectors.slack_connector  - Received message on unsupported channel: D01CD75GVPS

@stephens

2

Hi Aditya,

If you have a long running custom action, you could return immediately, with a response that let’s the user know you are working on their request. Then, use an external event to send the response when the action is complete. There’s some example code here.

Greg

3

Thanks for the reply Stephens. I have a couple of questions:

I tried to send a response dispatcher.utter_message(“Please wait while we fetch the data…”) to the slack bot but then when it came to the part where I am fetching from the background, it gave me the same error.

Also, how external event will solve the problem of slack’s 3-second timeout issue?

4

The external event will solve the 3 second timeout because you would immediately reply to the user with a response that you are working on their request.

The default channel timeout is 3 seconds in Rasa. This is not a slack specific issue. There is an option to change this timeout using --response-timeout.

5

@stephens, I tried using external event, but it still gives me the concurrent.futures._base.CancelledError error. Am I missing something here?

Also, when I run the same conversation using Twilio channel, it does not give me any error. Shouldn’t I get the same timeout issue with twilio too if this was Rasa timeout?

My file structure looks like:

domain.yml

intents:
- EXTERNAL_get_user_details

entities:
- name
- city

slots:
  name:
    type: unfeaturized
  city:
    type: unfeaturized

actions:
  - action_valid_file

response:
utter_user_details:
- text: "*Name:*\n{name} \n*City:*\n{city}"

data/rules.yml

rules:   
- rule: Fetch user details.
    steps:
      - intent: EXTERNAL_get_user_details
      - action: action_valid_file

stories.md:

* greet_user

    - action_valid_file

    - utter_user_details

    - user_info_form

    - form{"name":"user_info_form"}

    - form{"name":null}

actions.py:

class ActionForValidFile(Action):

    def name(self) -> Text:
        return "action_valid_file"

    def run(self, dispatcher: CollectingDispatcher,
            tracker: Tracker,
            domain: Dict[Text, Any]) -> List[Dict[Text, Any]]:
        dispatcher.utter_message(**
            "Please wait while we fetch details from the user...")
        events = tracker.current_state()['events']
        base64_encode = convertPdfToBase64(base64_file_path) // API Call
        candidate_detail = get_details(base64_encode) // API Call
        slot_value = [
            SlotSet("name", candidate_detail['name']),
            SlotSet("city", candidate_detail['city'])
        ]
    return slot_value

Error:

Traceback (most recent call last):
  File "/Users/adity/env/lib/python3.7/site-packages/rasa/core/processor.py", line 680, in _run_action
    events = await action.run(output_channel, nlg, tracker, self.domain)
  File "/Users/adity/env/lib/python3.7/site-packages/rasa/core/actions/action.py", line 612, in run
    json=json_body, method="post", timeout=DEFAULT_REQUEST_TIMEOUT
  File "/Users/adity/env/lib/python3.7/site-packages/rasa/utils/endpoints.py", line 151, in request
    **kwargs,
  File "/Users/adity/env/lib/python3.7/site-packages/aiohttp/client.py", line 1012, in __aenter__
    self._resp = await self._coro
  File "/Users/adity/env/lib/python3.7/site-packages/aiohttp/client.py", line 504, in _request
    await resp.start(conn)
  File "/Users/adity/env/lib/python3.7/site-packages/aiohttp/client_reqrep.py", line 847, in start
    message, payload = await self._protocol.read()  # type: ignore  # noqa
  File "/Users/adity/env/lib/python3.7/site-packages/aiohttp/streams.py", line 591, in read
    await self._waiter
 concurrent.futures._base.CancelledError`
 connectors.slack_connector  - Received retry #1 request from slack due to http_timeout.
6

Take a look at the ReminderBot for a fully working example. Also note that you’ll need to use the CallbackChannel.

7

I did try implementing the external event, it still gives the same error. I also tried --response-timeout the argument with no luck.

If this is a channel timeout, why does it happen only for slack and not for Twilio? I am confused!

Is it by any chance related to this bug? got concurrent.futures._base.CancelledError · Issue #4047 · RasaHQ/rasa · GitHub

8

Can someone provide information on the exact timeout that is being triggered by this CancelledError failure? It does not seem that the --response-timeout relates to the 3s number.

9

Hi Greg

I have the same issue, but I’m afraid the example code wasn’t very helpful. As I understand, it’s possible to start an external event from inside a custom action, but are there more examples?

Additionally, I attempted to start the external event through a curl call in the terminal, with my slack conversation id. Although the custom action following the external_intent ran without issues, the slack conversation never received the message sent through the dispatcher.

The code for this custom action is:

class ActionGetAstronauts(Action):
    def name(self) -> Text:
        return 'action_get_astronauts'

    def run(
        self,
        dispatcher: CollectingDispatcher,
        tracker: Tracker,
        domain: Dict[Text, Any]
    ) -> List[Dict[Text, Any]]:

        response = requests.get(
            'http://api.open-notify.org/astros.json').json()

        names = ''
        for n in response['people']:
            names = names + n['name'] + ', '

        names = names[:-2]
        dispatcher.utter_message(names)
        print(names)

        return []
10

There’s an example with a curl call here. Can you provide the curl call you are making and the --debug Rasa log associated with the call?

11

Hi, I have the same problem with rasa slack integration. Do we have any updates on this issue?

1 Like
12

Hey Aditya, Have you resolved this issue now?

13

facing same issue…

14

returning ReminderScheduled in Action & doing async processing in ReactToReminder works as expected. Details in doc.

15

I had the same problem and it seems to be because Slack cancels the request after 3 seconds, which also cancels all the awaitables. For me, this meant that sometimes actions didn’t finish and the bot did not respond at all.

A workaround for the problem is to use asyncio.shield, which ensures that the inner function cannot be cancelled. My overall solution is a bit hacky, but works.

First, I created a helper method non_cancellable_shield that waits for the inner function even if the shield is canceled.

import asyncio
import logging
from typing import Any

logger = logging.getLogger(__name__)

async def non_cancellable_shield(func) -> Any:
  future = asyncio.ensure_future(func)
  try:
    # Shield task to prevent cancelling
    logger.info('Tries awaiting future')
    return await asyncio.shield(future)
  except asyncio.CancelledError:
    # Await original task
    logger.info('Shield cancelled')
    return await future

I then used this helper method in a custom Slack connector. I took the code from Github (https://github.com/RasaHQ/rasa/blob/2.7.x/rasa/core/channels/slack.py) and simply wrapped each Awaitable into the helper method. I.e. for example for _post_message:

    async def _post_message(self, channel: Text, **kwargs: Any) -> None:
        if self.thread_id:
            await non_cancellable_shield(self.client.chat_postMessage(
                channel=channel, **kwargs, thread_ts=self.thread_id
            ))
        else:
            await non_cancellable_shield(self.client.chat_postMessage(channel=channel, **kwargs))

To use the custom connector, I customized the Slack connector in credentials.yml as follows (the custom connector is in the file channels/custom_slack_connector.py and the class that inherits from the InputChannel is called CustomSlackConnector):

channels.custom_slack_connector.CustomSlackConnector:
  slack_token: ${SLACK_TOKEN}
  slack_channel: ${SLACK_CHANNEL}
  slack_signing_secret: ${SLACK_SIGNING_SECRET}

Since Slack can theoretically cancel the request at any point in the code, I also wrapped all other awaitables in the non_cancellable_shield helper method.

Note: I use Rasa 2.7.0 and Python 3.7.6

3 Likes
16

Any fix/updates on the slack timeout issue? @stephens

17

I’m not following or working on this issue. I don’t see an open github issue related to this. That would be the next step.

18

Okay @stephens , however are you aware of any workaround or any change to overcome this?

19

Hi @_adityaK did you solve your issue?