RasaServer Timeout Error

Rasa Open Source
1

Hi Team,

Facing timeout issue with Rasa Server. Rasa is not waiting for the custom action to complete by the Action server. Tried increasing the DEFAULT_STREAM_READING_TIMEOUT_IN_SECONDS in console.py file. Still I’m getting the same time out issue in rasa.

PFB Rasa details and Error logs from Rasa Server and Action Server

  • Rasa Version : 2.8.6
  • Minimum Compatible Version: 2.8.0
  • Rasa SDK Version : 2.8.6
  • Rasa X Version : None
  • Python Version : 3.8.10
  • Operating System : Windows-2012ServerR2-6.3.9600-SP0
  • Python Path : c:\chatbot\press_chatbot_backend\env\scripts\python.exe

Error

Rasa Server

2022-11-09 10:41:58 ←[1;30mDEBUG ←[0m ←[34mrasa.core.actions.action←[0m - Calling action endpoint to run action 'acti on_ask_amhpasswordreset'. 2022-11-09 10:42:58 ←[1;30mERROR ←[0m ←[34mrasa.core.processor←[0m - ←[31mEncountered an exception while running acti on 'action_ask_amhpasswordreset'.Bot will continue, but the actions events are lost. Please check the logs of your actio n server for more information.←[0m Traceback (most recent call last): File "c:\chatbot\press_chatbot_backend\env\lib\site-packages\rasa\core\actions\action.py", line 685, in run response = await self.action_endpoint.request( File "c:\chatbot\press_chatbot_backend\env\lib\site-packages\rasa\utils\endpoints.py", line 172, in request raise ClientResponseError( rasa.utils.endpoints.ClientResponseError: 503, Service Unavailable, body='b'<!DOCTYPE html><html lang=en><meta charset=U TF-8><title>\xe2\x9a\xa0\xef\xb8\x8f 503 \xe2\x80\x94 Service Unavailable</title>\n<style>\n html { font-family: sans-serif }\n h2 { color: #888; }\n .tb-wrapper p { margin: 0 }\n .frame-border { margin: 1rem }\n .frame-line > * { padding: 0.3rem 0.6rem }\n .frame-line { margin-bottom: 0.3rem }\n .frame-code { font-size: 16px; padding-left: 4ch }\n .tb-wrapper { border: 1px solid #eee }\n .tb-header { background: #eee; padding: 0.3rem; font-weight: bold }\n .frame-descriptor { background: #e2eafb; font-size: 14px }\n </st yle>\n<h1>\xe2\x9a\xa0\xef\xb8\x8f 503 \xe2\x80\x94 Service Unavailable</h1><p>Response Timeout\n''

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "c:\chatbot\press_chatbot_backend\env\lib\site-packages\rasa\core\processor.py", line 772, in _run_action events = await action.run( File "c:\chatbot\press_chatbot_backend\env\lib\site-packages\rasa\core\actions\action.py", line 709, in run raise RasaException("Failed to execute custom action.") from e rasa.shared.exceptions.RasaException: Failed to execute custom action. 2022-11-09 10:42:58 ←[1;30mDEBUG ←[0m ←[34mrasa.core.processor←[0m - Policy prediction ended with events '[<rasa.shar ed.core.events.DefinePrevUserUtteredFeaturization object at 0x000000FEB9A1CEB0>]'. 2022-11-09 10:42:58 ←[1;30mDEBUG ←[0m ←[34mrasa.core.processor←[0m - Action 'action_ask_amhpasswordreset' ended with events '[]'.

Action Server

2022-11-09 13:07:54 ←[1;30mDEBUG ←[0m ←[34mrasa_sdk.executor←[0m - Received request to run 'validate_amh_pass_reset_f orm' 2022-11-09 13:07:54 DEBUG rasa_sdk.executor - Received request to run 'validate_amh_pass_reset_form' 2022-11-09 13:07:54 ←[1;30mDEBUG ←[0m ←[34mrasa_sdk.executor←[0m - Finished running 'validate_amh_pass_reset_form' 2022-11-09 13:07:54 DEBUG rasa_sdk.executor - Finished running 'validate_amh_pass_reset_form' 2022-11-09 13:07:56 ←[1;30mDEBUG ←[0m ←[34mrasa_sdk.executor←[0m - Received request to run 'validate_amh_pass_reset_f orm' 2022-11-09 13:07:56 DEBUG rasa_sdk.executor - Received request to run 'validate_amh_pass_reset_form' 2022-11-09 13:07:56 ←[1;30mDEBUG ←[0m ←[34mrasa_sdk.executor←[0m - Finished running 'validate_amh_pass_reset_form' 2022-11-09 13:07:56 DEBUG rasa_sdk.executor - Finished running 'validate_amh_pass_reset_form' 2022-11-09 13:08:02 ←[1;30mDEBUG ←[0m ←[34mrasa_sdk.executor←[0m - Received request to run 'validate_amh_pass_reset_f orm' 2022-11-09 13:08:02 DEBUG rasa_sdk.executor - Received request to run 'validate_amh_pass_reset_form' 2022-11-09 13:08:02 ←[1;30mWARNING ←[0m ←[34mrasa_sdk.forms←[0m - ←[33mSkipping validation for amh_email: there is no validation method specified.←[0m 2022-11-09 13:08:02 WARNING rasa_sdk.forms - Skipping validation for amh_email: there is no validation method specif ied. 2022-11-09 13:08:02 ←[1;30mDEBUG ←[0m ←[34mrasa_sdk.executor←[0m - Finished running 'validate_amh_pass_reset_form' 2022-11-09 13:08:02 DEBUG rasa_sdk.executor - Finished running 'validate_amh_pass_reset_form' 2022-11-09 13:08:04 ←[1;30mDEBUG ←[0m ←[34mrasa_sdk.executor←[0m - Received request to run 'validate_amh_pass_reset_f orm' 2022-11-09 13:08:04 DEBUG rasa_sdk.executor - Received request to run 'validate_amh_pass_reset_form' 2022-11-09 13:08:04 ←[1;30mDEBUG ←[0m ←[34mrasa_sdk.executor←[0m - Finished running 'validate_amh_pass_reset_form' 2022-11-09 13:08:04 DEBUG rasa_sdk.executor - Finished running 'validate_amh_pass_reset_form' 2022-11-09 13:08:07 ←[1;30mDEBUG ←[0m ←[34mrasa_sdk.executor←[0m - Received request to run 'action_ask_amhpasswordres et' 2022-11-09 13:08:07 DEBUG rasa_sdk.executor - Received request to run 'action_ask_amhpasswordreset' before async function Response AMH password reset has been successfully completed for t in 123321 environment and Password is: abcd12333 after async function check reset response AMH password reset has been successfully completed for t in 123321 environment and Password is: abcd12333 2022-11-09 13:09:08 ←[1;30mDEBUG ←[0m ←[34mrasa_sdk.executor←[0m - Finished running 'action_ask_amhpasswordreset' 2022-11-09 13:09:08 DEBUG rasa_sdk.executor - Finished running 'action_ask_amhpasswordreset'

Could you please help me with fixing the timeout issue?

2

The action took just over a minute to run. Rasa doesn’t support more than 20 seconds so you’ll need to take a different approach for this action.

Maybe you could post an immediate response that a password reset email will be sent to you or something like that.

2022-11-09 13:08:07 DEBUG rasa_sdk.executor - Received request to run 'action_ask_amhpasswordreset' before async function Response AMH password reset has been successfully completed for t in 123321 environment and Password is: abcd12333 after async function check reset response AMH password reset has been successfully completed for t in 123321 environment and Password is: abcd12333
2022-11-09 13:09:08 ←[1;30mDEBUG ←[0m ←[34mrasa_sdk.executor←[0m - Finished running 'action_ask_amhpasswordreset' 2022-11-09 13:09:08 DEBUG rasa_sdk.executor - Finished running 'action_ask_amhpasswordreset'
3

Custom action which I’m running for the password reset is a sequence of steps which will take more than 2 mins to complete.

Synchronous - Action Server API

If i run the custom action API synchronously. I’m getting the response back to the chat window, even though it is taking 2 mins . But it is not allowing other chat users to concurrently chat.

Rasa Server Log:

2022-11-11 17:02:45 ←[1;30mDEBUG ←[0m ←[34mrasa.core.actions.action←[0m - Calling action endpoint to run action 'acti on_ask_amhpasswordreset'. 2022-11-11 17:05:45 ←[1;30mDEBUG ←[0m ←[34mrasa.core.processor←[0m - Policy prediction ended with events '[<rasa.shar ed.core.events.DefinePrevUserUtteredFeaturization object at 0x000000B98AF032B0>]'. 2022-11-11 17:05:45 ←[1;30mDEBUG ←[0m ←[34mrasa.core.processor←[0m - Action 'action_ask_amhpasswordreset' ended with events '[BotUttered('AMH password reset has been successfully completed for t in 123321 environment and Password is: abc d12333', {"elements": null, "quick_replies": null, "buttons": null, "attachment": null, "image": null, "custom": null}, {}, 1668146745.4029274)]'.

Asynchronous - Action Server API

Timeout issue occurs in the Rasa server if I run the API call asynchronously. But if it less than 59 seconds I’m getting the response back.

Rasa Server Log:

2022-11-09 10:41:58 ←[1;30mDEBUG ←[0m ←[34mrasa.core.actions.action←[0m - Calling action endpoint to run action 'acti on_ask_amhpasswordreset'. 2022-11-09 10:42:58 ←[1;30mERROR ←[0m ←[34mrasa.core.processor←[0m - ←[31mEncountered an exception while running acti on 'action_ask_amhpasswordreset'.Bot will continue, but the actions events are lost. Please check the logs of your actio n server for more information.←[0m Traceback (most recent call last): File "c:\chatbot\press_chatbot_backend\env\lib\site-packages\rasa\core\actions\action.py", line 685, in run response = await self.action_endpoint.request( File "c:\chatbot\press_chatbot_backend\env\lib\site-packages\rasa\utils\endpoints.py", line 172, in request raise ClientResponseError( rasa.utils.endpoints.ClientResponseError: 503, Service Unavailable, body='b'<!DOCTYPE html><html lang=en><meta charset=U TF-8><title>\xe2\x9a\xa0\xef\xb8\x8f 503 \xe2\x80\x94 Service Unavailable</title>\n<style>\n html { font-family: sans-serif }\n h2 { color: #888; }\n .tb-wrapper p { margin: 0 }\n .frame-border { margin: 1rem }\n .frame-line > * { padding: 0.3rem 0.6rem }\n .frame-line { margin-bottom: 0.3rem }\n .frame-code { font-size: 16px; padding-left: 4ch }\n .tb-wrapper { border: 1px solid #eee }\n .tb-header { background: #eee; padding: 0.3rem; font-weight: bold }\n .frame-descriptor { background: #e2eafb; font-size: 14px }\n </st yle>\n<h1>\xe2\x9a\xa0\xef\xb8\x8f 503 \xe2\x80\x94 Service Unavailable</h1><p>Response Timeout\n''

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "c:\chatbot\press_chatbot_backend\env\lib\site-packages\rasa\core\processor.py", line 772, in _run_action events = await action.run( File "c:\chatbot\press_chatbot_backend\env\lib\site-packages\rasa\core\actions\action.py", line 709, in run raise RasaException("Failed to execute custom action.") from e rasa.shared.exceptions.RasaException: Failed to execute custom action. 2022-11-09 10:42:58 ←[1;30mDEBUG ←[0m ←[34mrasa.core.processor←[0m - Policy prediction ended with events '[<rasa.shar ed.core.events.DefinePrevUserUtteredFeaturization object at 0x000000FEB9A1CEB0>]'. 2022-11-09 10:42:58 ←[1;30mDEBUG ←[0m ←[34mrasa.core.processor←[0m - Action 'action_ask_amhpasswordreset' ended with events '[]'.

4

You should configure your underlying Sanic(I guess) server and increase the value of SANIC_RESPONSE_TIMEOUT to more than 60 (default is 60 seconds), it will work fine then.