ERROR pika.adapters.base_connection

alebeta90 · September 23, 2019, 2:31pm

Hello friends,

I have a docker setup of Rasa-X running in docker using postgresql + rabbitMQ. We run the latest images at this date. If more information about our setup is needit, please let me know.

The bot is working properly so far, but I encounter a problem, if the user give a late to the bot, it breaks and we need to restart the conversation /restart.

The error looks as follow and it seems to be related with RabbitMQ:



/usr/local/lib/python3.6/site-packages/rasa_core/__init__.py:12: UserWarning: The 'rasa_core' package has been renamed. You should change your imports to use 'rasa.core' instead.


  UserWarning,


2019-09-23 13:27:44 ERROR    pika.adapters.utils.io_services_utils  - _AsyncBaseTransport._produce() failed, aborting connection: error=ConnectionResetError(104, 'Connection reset by peer'); sock=<socket.socket fd=23, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('10.0.7.193', 39792)>; Caller's stack:


Traceback (most recent call last):


  File "/usr/local/lib/python3.6/site-packages/pika/adapters/utils/io_services_utils.py", line 1097, in _on_socket_writable


    self._produce()


  File "/usr/local/lib/python3.6/site-packages/pika/adapters/utils/io_services_utils.py", line 820, in _produce


    self._tx_buffers[0])


  File "/usr/local/lib/python3.6/site-packages/pika/adapters/utils/io_services_utils.py", line 79, in retry_sigint_wrap


    return func(*args, **kwargs)


  File "/usr/local/lib/python3.6/site-packages/pika/adapters/utils/io_services_utils.py", line 861, in _sigint_safe_send


    return sock.send(data)


ConnectionResetError: [Errno 104] Connection reset by peer


Traceback (most recent call last):


  File "/usr/local/lib/python3.6/site-packages/pika/adapters/utils/io_services_utils.py", line 1097, in _on_socket_writable


    self._produce()


  File "/usr/local/lib/python3.6/site-packages/pika/adapters/utils/io_services_utils.py", line 820, in _produce


    self._tx_buffers[0])


  File "/usr/local/lib/python3.6/site-packages/pika/adapters/utils/io_services_utils.py", line 79, in retry_sigint_wrap


    return func(*args, **kwargs)


  File "/usr/local/lib/python3.6/site-packages/pika/adapters/utils/io_services_utils.py", line 861, in _sigint_safe_send


    return sock.send(data)


ConnectionResetError: [Errno 104] Connection reset by peer


2019-09-23 13:27:44 ERROR    pika.adapters.base_connection  - connection_lost: StreamLostError: ("Stream connection lost: ConnectionResetError(104, 'Connection reset by peer')",)


2019-09-23 13:27:44 ERROR    pika.adapters.blocking_connection  - Unexpected connection close detected: StreamLostError: ("Stream connection lost: ConnectionResetError(104, 'Connection reset by peer')",)


2019-09-23 13:27:44 ERROR    rasa.core.brokers.pika  - Could not open Pika channel at host 'rabbit'. Failed with error: Stream connection lost: ConnectionResetError(104, 'Connection reset by peer') ````

I would like to know how to solve this issue, since it is killing bot performance.

Thanks in advance.

ricwo · September 26, 2019, 8:59am

Hi @alebeta90, thanks for your question. Would you mind clarifying what exactly leads to this error? You mention

if the user give a late to the bot

but I’m not 100 percent sure what that means.

Thanks!

alebeta90 · September 26, 2019, 11:59am

Hi @ricwo

sorry, I didn’t write my message properly. I meant if the user give a late message to the bot. For example you start the conversation, and for some reason you stop answering for 10 minutes and then you go back to the conversation and answer back to the bot, the bot is breaking with this message. We have couple of situations that the bot just start posting some messages in a loop.

All the best,

ricwo · September 26, 2019, 12:20pm

Hi again, thanks for clarifying that. I cannot reproduce this error. Are you having the conversation in the normal “Talk to your bot” tab? What version of Rasa X are you running? I see you posted this three days ago - would you mind trying to update to 0.21.0, along with Rasa 1.3.6?

alebeta90 · September 26, 2019, 12:31pm

Hi @ricwo

I just check the rasa version in our docker environment and it shows

root@c6a3bcc1dd2d:/app# rasa --version
Rasa 1.4.0a1

We are using rasa-x:latest and rasa:latest

Shall we downgrade to 1.3.6?

Thanks in advance

alebeta90 · September 26, 2019, 1:24pm

I forgot to answer.

We are having the conversation using Slack Channel

ricwo · September 30, 2019, 9:13am

Hi @alebeta90, thanks for confirming that. Would you mind trying talking to your bot from the Rasa X UI before starting connecting to it via slack?

alebeta90 · September 30, 2019, 12:57pm

Hi @ricwo,

We moved to version 1.3.6 and it seems to be working fine, but today I will perform more tests in order to confirm it good behavior.

All the best

alebeta90 · October 2, 2019, 2:23pm

Problem is persisting, also trying from different channels, like rocketchat.

RabbitMQ docker image tag is 3.7.17

ricwo · October 3, 2019, 8:05am

@alebeta90, have you tried talking to your bot from the Rasa X UI before connecting via other channels?

alebeta90 · October 24, 2019, 1:26pm

Hi @ricwo

So far we are testing with Rasa image: 1.3.6 Rasa X image: 0.21.5

And we have the same problem.

If the user is talking using Rasa X talk to your bot, slack or RocketChat the problem is happening. we get the output about connection problems with RabbitMQ.

Also if we are training the bot with the interactive learning and we switch to the stories section and we go back to the interactive learning, it shows the same message and the bot breaks.

Thanks for the help

ricwo · October 24, 2019, 5:44pm

Hi @alebeta90, would you mind upgrading to rasa X 0.22.0 and Rasa 1.4.0?

alebeta90 · October 25, 2019, 9:38am

Hi @ricwo,

and what about RabbitMQ ? should we keep the same version? RabbitMQ docker image tag is 3.7.17

all the best

Tobias_Wochinger · October 28, 2019, 3:28pm

We are also tackling this issue currently internally, so it’s hopefully gonna be fixed in the next release

alebeta90 · October 28, 2019, 4:08pm

Hi Tobias,

thanks for the message. Then the problem was also replicated internally in the Rasa team.

The root of the problem was already identified? the thing is that we can not move to production if the engine will stop working properly with this kind of errors.

thanks and all the best

alebeta90 · October 28, 2019, 6:15pm

here you can see the latest errors in using Rasa image 1.4.0

2019-10-28 18:10:30 ERROR    pika.adapters.utils.io_services_utils  - _AsyncBaseTransport._produce() failed, aborting connection: error=ConnectionResetError(104, 'Connection reset by peer'); sock=<socket.socket fd=25, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('10.0.1.216', 43892)>; Caller's stack:


Traceback (most recent call last):


  File "/build/lib/python3.6/site-packages/pika/adapters/utils/io_services_utils.py", line 1097, in _on_socket_writable


    self._produce()


  File "/build/lib/python3.6/site-packages/pika/adapters/utils/io_services_utils.py", line 820, in _produce


    self._tx_buffers[0])


  File "/build/lib/python3.6/site-packages/pika/adapters/utils/io_services_utils.py", line 79, in retry_sigint_wrap


    return func(*args, **kwargs)


  File "/build/lib/python3.6/site-packages/pika/adapters/utils/io_services_utils.py", line 861, in _sigint_safe_send


    return sock.send(data)


ConnectionResetError: [Errno 104] Connection reset by peer


Traceback (most recent call last):


  File "/build/lib/python3.6/site-packages/pika/adapters/utils/io_services_utils.py", line 1097, in _on_socket_writable


    self._produce()


  File "/build/lib/python3.6/site-packages/pika/adapters/utils/io_services_utils.py", line 820, in _produce


    self._tx_buffers[0])


  File "/build/lib/python3.6/site-packages/pika/adapters/utils/io_services_utils.py", line 79, in retry_sigint_wrap


    return func(*args, **kwargs)


  File "/build/lib/python3.6/site-packages/pika/adapters/utils/io_services_utils.py", line 861, in _sigint_safe_send


    return sock.send(data)


ConnectionResetError: [Errno 104] Connection reset by peer


2019-10-28 18:10:30 ERROR    pika.adapters.base_connection  - connection_lost: StreamLostError: ("Stream connection lost: ConnectionResetError(104, 'Connection reset by peer')",)


2019-10-28 18:10:30 ERROR    pika.adapters.blocking_connection  - Unexpected connection close detected: StreamLostError: ("Stream connection lost: ConnectionResetError(104, 'Connection reset by peer')",)


2019-10-28 18:10:30 ERROR    rasa.core.brokers.pika  - Could not open Pika channel at host 'rabbit'. Failed with error: Stream connection lost: ConnectionResetError(104, 'Connection reset by peer')

Let me know if we can provide with other logs, in order to help you debugging.

All the best

Tobias_Wochinger · October 28, 2019, 6:21pm

I created this pr ~~Log problematic sentences by havenS · Pull Request #468 · RasaHQ/rasa · GitHub~~ Pika timeout fix by wochinge · Pull Request #4689 · RasaHQ/rasa · GitHub to fix it. You can try out if this fixes your issues if you want.

alebeta90 · October 28, 2019, 6:23pm

Hi Tobias,

that PR since to be about something else. How it suppose to be related with our problem?

thanks

Tobias_Wochinger · October 28, 2019, 8:11pm

Sorry, I was on the phone This is the correct link.

alebeta90 · October 29, 2019, 11:59am

there is a docker image available to perform the test?

thanks a lot

Topic		Replies	Views
RabbitMQ connection refused [Deprecated] Rasa X Community Edition	3	1599	September 8, 2020
Connection Failed to Rabbitmq from Rasa to Rasa-x [Deprecated] Rasa X Community Edition	10	2059	March 25, 2020
Errors in rasa-production [Deprecated] Rasa X Community Edition	4	1587	January 23, 2020
Rasa-x_1 problem: Socket failed to connect, error=111 (Connection refused) [Deprecated] Rasa X Community Edition	7	2523	March 23, 2020
Rasa X - RabbitMQ Connection Closed Error [Deprecated] Rasa X Community Edition	5	2512	August 3, 2021

ERROR pika.adapters.base_connection

Related topics