Error when connect Rasa to Postgresql tracker db

Rasa Open Source
1

Hi All, newbie here, I setup rasa stack in docker follow the instruction in Deploy to a Server. rasa-production: I created new docker image base on rasa/rasa:1.1.9 to add Vietnamese spacy model rasa-x: rasa/rasa-x:stable host: Ubuntu 18. I used docker stack deploy instead of docker-compose everything are OK at begining I can use rasa-x to create data and train model. I have an issue with this stack:

  • it’s working fine at the beginning but after it’s idle for a while (few minutes) rasa return null [] to the request this is the request: curl -XPOST http://rasa-srv-01:5005/webhooks/rest/webhook -d ‘{“sender”: “Rasa”, “message”: “Chào bạn! Hu Hu”}’

if I restart rasa-production container then it works fine again until next timeout.

any recommend?

this is what I found in rasa-production logs:

2019-08-05 12:05:32 DEBUG rasa.core.agent - Deleted lock for conversation ‘Rasa’ (unused) 2019-08-05 12:05:32 ERROR rasa.core.channels.channel - An exception occured while handling user message ‘Chào bạn! Hu Hu’. Traceback (most recent call last): File “/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 1177, in _execute_context conn = self._revalidate_connection() File “/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 463, in _revalidate_connection "Can’t reconnect until invalid " sqlalchemy.exc.InvalidRequestError: Can’t reconnect until invalid transaction is rolled back

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File “/usr/local/lib/python3.6/site-packages/rasa/core/channels/channel.py”, line 459, in receive text, collector, sender_id, input_channel=input_channel File “/usr/local/lib/python3.6/asyncio/coroutines.py”, line 110, in next return self.gen.send(None) File “/usr/local/lib/python3.6/site-packages/rasa/core/channels/channel.py”, line 65, in handler await app.agent.handle_message(*args, **kwargs) File “/usr/local/lib/python3.6/asyncio/coroutines.py”, line 110, in next return self.gen.send(None) File “/usr/local/lib/python3.6/site-packages/rasa/core/agent.py”, line 488, in handle_message return await processor.handle_message(message) File “/usr/local/lib/python3.6/asyncio/coroutines.py”, line 110, in next return self.gen.send(None) File “/usr/local/lib/python3.6/site-packages/rasa/core/processor.py”, line 80, in handle_message tracker = await self.log_message(message) File “/usr/local/lib/python3.6/asyncio/coroutines.py”, line 110, in next return self.gen.send(None) File “/usr/local/lib/python3.6/site-packages/rasa/core/processor.py”, line 132, in log_message tracker = self._get_tracker(message.sender_id) File “/usr/local/lib/python3.6/site-packages/rasa/core/processor.py”, line 529, in _get_tracker return self.tracker_store.get_or_create_tracker(sender_id File “/usr/local/lib/python3.6/site-packages/rasa/core/tracker_store.py”, line 73, in get_or_create_tracker tracker = self.retrieve(sender_id) File “/usr/local/lib/python3.6/site-packages/rasa/core/tracker_store.py”, line 433, in retrieve result = query.filter_by(sender_id=sender_id).all() File “/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py”, line 3168, in all return list(self) File “/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py”, line 3324, in iter return self._execute_and_instances(context) File “/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py”, line 3349, in _execute_and_instances result = conn.execute(querycontext.statement, self._params) File “/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 988, in execute return meth(self, multiparams, params) File “/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py”, line 287, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File “/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 1107, in _execute_clauseelement distilled_params, File “/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 1182, in _execute_context e, util.text_type(statement), parameters, None, None File “/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 1466, in _handle_dbapi_exception util.raise_from_cause(sqlalchemy_exception, exc_info) File “/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py”, line 399, in raise_from_cause reraise(type(exception), exception, tb=exc_tb, cause=cause) File “/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py”, line 153, in reraise raise value.with_traceback(tb) File “/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 1177, in _execute_context conn = self._revalidate_connection() File “/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 463, in _revalidate_connection "Can’t reconnect until invalid " sqlalchemy.exc.StatementError: (sqlalchemy.exc.InvalidRequestError) Can’t reconnect until invalid transaction is rolled back *[SQL: SELECT events.id AS events_id, events.sender_id AS events_sender_id, events.type_name AS events_type_name, events.timestamp AS events_timestamp, events.intent_name AS events_intent_name, events.action_name AS events_action_name, events.data AS events_data * *FROM events * WHERE events.sender_id = %(sender_id_1)s] [parameters: [{}]]

2

yeah your guess is reasonable that the postgres connection is timing out. No idea why that would happen though. What operating system are you running on? You could also try switching to sqlite as the database just to check that’s really where the problem is.

3

I’m using Docker 1.19 on Ubuntu 18. btw when I switch to mongodb, it’s working fine but there is only tracker’s events are stored in mongodb, while in postgresql I saw 3 schemas: tracker (events), rasa, worker. rasa and worker are using to store model information like domain, domain-intents, domain-entities, domain-stories, domain-synomyms … so I wonder should I go ahead with mongodb which is working perfectly now or go back to postgresql and try to fix this issue. Thx

4

i am also facing the same issue with Mysql. "Can’t reconnect until invalid "

5

I have same issue. After some time I need to kill and restart my docker container with rasa/rasa-x. I don’t think it depends on the database: I tried bot postgresql and sqlite. I think that it depends on how the connection is managed by rasa/sqlalchemy orm when rasa is deployed on docker.

There is a similar bug on your tracking system: Issue in mysql connection · Issue #4284 · RasaHQ/rasa · GitHub.

This can be a very big issue in a production environment!

6

Hi Alex,

I’m also running in to the same issue, using rasa-x latest image and rasa-1.2.7-full.

I saw the work around from Github changing the session_timeout, but it is not the best case, I think it should be fix at Rasa side, how it handle the connection.

Someone had been lucky to solve this using postregresql? because the workaround in Github is using MySQL as a backend.

All the best, Alejandro B.

7

Do we still have this issue on connecting tracker DB in production environment? I am also looking for an option to connect to tracker DB instead of inMemory tracker. Do anyone have working example of Rasa in production using any tracker DB? Sharing will be helpful for others to use in their project.

8

can anyone please share any proper guide on how to connect rasa to posgresql to store responses ,i am facing lot of issue in connecting to it

9

can anyone please tell how to connect rasa 3.1 version to postgresql