Rasa custom actions are failing with timeout

I am running rasa 2.0.2 on Kubernetes. While running custom action I am getting below error since today (03/29). Things were fine up until yesterday:-

-----------------------------Error-----------------------------------------------------------------

2021-03-30 06:04:55 ERROR rasa.core.processor - Encountered an exception while running action ‘action_submit’.Bot will continue, but the actions events are lost. Please check the logs of your action server for more information.

Traceback (most recent call last):

** File “/usr/local/lib/python3.6/dist-packages/rasa/core/processor.py”, line 681, in _run_action**

** events = await action.run(output_channel, nlg, tracker, self.domain)**

** File “/usr/local/lib/python3.6/dist-packages/rasa/core/actions/action.py”, line 612, in run**

** json=json_body, method=“post”, timeout=DEFAULT_REQUEST_TIMEOUT**

** File “/usr/local/lib/python3.6/dist-packages/rasa/utils/endpoints.py”, line 151, in request**

** kwargs,

** File “/usr/local/lib/python3.6/dist-packages/aiohttp/client.py”, line 1012, in aenter**

** self._resp = await self._coro**

** File “/usr/local/lib/python3.6/dist-packages/aiohttp/client.py”, line 504, in _request**

** await resp.start(conn)**

** File “/usr/local/lib/python3.6/dist-packages/aiohttp/client_reqrep.py”, line 847, in start**

** message, payload = await self._protocol.read() # type: ignore # noqa**

** File “/usr/local/lib/python3.6/dist-packages/aiohttp/streams.py”, line 591, in read**

** await self._waiter**

concurrent.futures._base.CancelledError

error: unexpected EOF

-------------------------------------------------end of error--------------------------------------------------

So as far as I think this error is coming due to time it is taking to get the response from action server. If that is the case, then how I can increase the read timeout value? Which parameter and which file I need to update?

Thanks Rahul

I appreciate any advice on above? Thanks

This has been resolved at our side. It was due to firewall issue.