MySQL server has gone away On RASA

ujjwaldeswal · April 16, 2021, 9:40am

Hi! I run rasa on the server and everything is fine, but when I not contact with bot for 1 - 2 days and then write a message - an error occurs. In first message case - rasa works but without slots values. When I send the next message - everything is fine. Can someone help me with this? Thanks!

One of the solutions may be like just increase the wait_timeout = 31536000 interactive_timeout = 31536000 at MySQL server-side. But I want to handle this error on the Rasa side.

MySQLdb._exceptions.OperationalError: (2006, 'MySQL server has gone away') The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 126, in send return self.gen.send(value) File "/usr/local/lib/python3.6/site-packages/socketio/asyncio_server.py", line 438, in _handle_event_internal r = await server._trigger_event(data[0], namespace, sid, *data[1:]) File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__ return self.gen.send(None) File "/usr/local/lib/python3.6/site-packages/socketio/asyncio_server.py", line 466, in _trigger_event ret = await self.handlers[namespace][event](*args) File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__ return self.gen.send(None) File "/usr/local/lib/python3.6/site-packages/rasa/core/channels/socketio.py", line 190, in handle_message await on_new_message(message) File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__ return self.gen.send(None) File "/usr/local/lib/python3.6/site-packages/rasa/core/channels/channel.py", line 81, in handler await app.agent.handle_message(*args, **kwargs) File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__ return self.gen.send(None) File "/usr/local/lib/python3.6/site-packages/rasa/core/agent.py", line 471, in handle_message return await processor.handle_message(message) File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__ return self.gen.send(None) File "/usr/local/lib/python3.6/site-packages/rasa/core/processor.py", line 80, in handle_message tracker = await self.log_message(message, should_save_tracker=False) File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__ return self.gen.send(None) File "/usr/local/lib/python3.6/site-packages/rasa/core/processor.py", line 150, in log_message tracker = self._get_tracker(message.sender_id) File "/usr/local/lib/python3.6/site-packages/rasa/core/processor.py", line 552, in _get_

Tobias_Wochinger · April 19, 2021, 11:18am

Can you please share the full stack trace? Which my sql driver + version are you using? This suggests it has to do with the driver

ujjwaldeswal · April 19, 2021, 1:18pm

Hey @Tobias_Wochinger , Below I have mentioned the full stack trace.

2021-04-16 09:30:23 DEBUG        rasa.core.tracker_store  - Creating a new tracker for id 'abbb261d304d472cacad7e42ca4c9dfb'.
        2021-04-16 09:30:23 DEBUG    rasa.core.lock_store  - Deleted lock for conversation 'abbb261d304d472cacad7e42ca4c9dfb'.
        2021-04-16 09:30:23 ERROR    asyncio  - Task exception was never retrieved
        future: <Task finished coro=<AsyncServer._handle_event_internal() done, defined at /usr/local/lib/python3.6/site-packages/socketio/asyncio_server.py:437> exception=OperationalError("(MySQLdb._exceptions.OperationalError) (2006, 'MySQL server has gone away')",) created at /usr/local/lib/python3.6/site-packages/engineio/asyncio_server.py:312>
        source_traceback: Object created at (most recent call last):
          File "/usr/local/bin/rasa", line 8, in <module>
            sys.exit(main())
          File "/usr/local/lib/python3.6/site-packages/rasa/__main__.py", line 76, in main
            cmdline_arguments.func(cmdline_arguments)
          File "/usr/local/lib/python3.6/site-packages/rasa/cli/run.py", line 119, in run
            rasa.run(**vars(args))
          File "/usr/local/lib/python3.6/site-packages/rasa/run.py", line 56, in run
            **kwargs
          File "/usr/local/lib/python3.6/site-packages/rasa/core/run.py", line 195, in serve_application
            app.run(host="0.0.0.0", port=port, ssl=ssl_context)
          File "/usr/local/lib/python3.6/site-packages/sanic/app.py", line 1096, in run
            serve(**server_settings)
          File "/usr/local/lib/python3.6/site-packages/sanic/server.py", line 770, in serve
            loop.run_forever()
          File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 126, in send
            return self.gen.send(value)
          File "/usr/local/lib/python3.6/site-packages/spf/framework.py", line 554, in _handle_request
            stream_callback)
          File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
            return self.gen.send(None)
          File "/usr/local/lib/python3.6/site-packages/sanic/app.py", line 917, in handle_request
            response = await response
          File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
            return self.gen.send(None)
          File "/usr/local/lib/python3.6/site-packages/engineio/asyncio_server.py", line 233, in handle_request
            b64, jsonp_index)
          File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
            return self.gen.send(None)
          File "/usr/local/lib/python3.6/site-packages/engineio/asyncio_server.py", line 383, in _handle_connect
            ret = await s.handle_get_request(environ)
          File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
            return self.gen.send(None)
          File "/usr/local/lib/python3.6/site-packages/engineio/asyncio_socket.py", line 89, in handle_get_request
            return await getattr(self, '_upgrade_' + transport)(environ)
          File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
            return self.gen.send(None)
          File "/usr/local/lib/python3.6/site-packages/engineio/asyncio_socket.py", line 128, in _upgrade_websocket
            return await ws(environ)
          File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
            return self.gen.send(None)
          File "/usr/local/lib/python3.6/site-packages/engineio/async_drivers/sanic.py", line 122, in __call__
            await self.handler(self)
          File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
            return self.gen.send(None)
          File "/usr/local/lib/python3.6/site-packages/engineio/asyncio_socket.py", line 222, in _websocket_handler
            await self.receive(pkt)
          File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
            return self.gen.send(None)
          File "/usr/local/lib/python3.6/site-packages/engineio/asyncio_socket.py", line 42, in receive
            run_async=self.server.async_handlers)
          File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
            return self.gen.send(None)
          File "/usr/local/lib/python3.6/site-packages/engineio/asyncio_server.py", line 410, in _trigger_event
            ret = await self.handlers[event](*args)
          File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
            return self.gen.send(None)
          File "/usr/local/lib/python3.6/site-packages/socketio/asyncio_server.py", line 505, in _handle_eio_message
            await self._handle_event(sid, pkt.namespace, pkt.id, pkt.data)
          File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
            return self.gen.send(None)
          File "/usr/local/lib/python3.6/site-packages/socketio/asyncio_server.py", line 433, in _handle_event
            data, namespace, id)
          File "/usr/local/lib/python3.6/site-packages/socketio/asyncio_server.py", line 343, in start_background_task
            return self.eio.start_background_task(target, *args, **kwargs)
          File "/usr/local/lib/python3.6/site-packages/engineio/asyncio_server.py", line 312, in start_background_task
            return asyncio.ensure_future(target(*args, **kwargs))
        Traceback (most recent call last):
          File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
            cursor, statement, parameters, context
          File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
            cursor.execute(statement, parameters)
          File "/usr/local/lib/python3.6/site-packages/MySQLdb/cursors.py", line 209, in execute
            res = self._query(query)
          File "/usr/local/lib/python3.6/site-packages/MySQLdb/cursors.py", line 315, in _query
            db.query(q)
          File "/usr/local/lib/python3.6/site-packages/MySQLdb/connections.py", line 239, in query
            _mysql.connection.query(self, query)
        MySQLdb._exceptions.OperationalError: (2006, 'MySQL server has gone away')

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 126, in send
        return self.gen.send(value)
      File "/usr/local/lib/python3.6/site-packages/socketio/asyncio_server.py", line 438, in _handle_event_internal
        r = await server._trigger_event(data[0], namespace, sid, *data[1:])
      File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
        return self.gen.send(None)
      File "/usr/local/lib/python3.6/site-packages/socketio/asyncio_server.py", line 466, in _trigger_event
        ret = await self.handlers[namespace][event](*args)
      File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
        return self.gen.send(None)
      File "/usr/local/lib/python3.6/site-packages/rasa/core/channels/socketio.py", line 190, in handle_message
        await on_new_message(message)
      File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
        return self.gen.send(None)
      File "/usr/local/lib/python3.6/site-packages/rasa/core/channels/channel.py", line 81, in handler
        await app.agent.handle_message(*args, **kwargs)
      File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
        return self.gen.send(None)
      File "/usr/local/lib/python3.6/site-packages/rasa/core/agent.py", line 471, in handle_message
        return await processor.handle_message(message)
      File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
        return self.gen.send(None)
      File "/usr/local/lib/python3.6/site-packages/rasa/core/processor.py", line 80, in handle_message
        tracker = await self.log_message(message, should_save_tracker=False)
      File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
        return self.gen.send(None)
      File "/usr/local/lib/python3.6/site-packages/rasa/core/processor.py", line 150, in log_message
        tracker = self._get_tracker(message.sender_id)
      File "/usr/local/lib/python3.6/site-packages/rasa/core/processor.py", line 552, in _get_tracker
        return self.tracker_store.get_or_create_tracker(sender_id)
      File "/usr/local/lib/python3.6/site-packages/rasa/core/tracker_store.py", line 78, in get_or_create_tracker
        tracker = self.create_tracker(sender_id)
      File "/usr/local/lib/python3.6/site-packages/rasa/core/tracker_store.py", line 97, in create_tracker
        self.save(tracker)
      File "/usr/local/lib/python3.6/site-packages/rasa/core/tracker_store.py", line 146, in save
        self.stream_events(tracker)
      File "/usr/local/lib/python3.6/site-packages/rasa/core/tracker_store.py", line 112, in stream_events
        self.event_broker.publish(body)
      File "/usr/local/lib/python3.6/site-packages/rasa/core/brokers/sql.py", line 74, in publish
        session.commit()
      File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1042, in commit
        self.transaction.commit()
      File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 504, in commit
        self._prepare_impl()
      File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 483, in _prepare_impl
        self.session.flush()
      File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2523, in flush
        self._flush(objects)
      File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2664, in _flush
        transaction.rollback(_capture_exception=True)
      File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
        exc_value, with_traceback=exc_tb,
      File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
        raise exception
      File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2624, in _flush
        flush_context.execute()
      File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
        rec.execute(self)
      File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 589, in execute
        uow,
      File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj
        insert,
      File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1136, in _emit_insert_statements
        statement, params
      File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1020, in execute
        return meth(self, multiparams, params)
      File "/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
        return connection._execute_clauseelement(self, multiparams, params)
      File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_clauseelement
        distilled_params,
      File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1324, in _execute_context
        e, statement, parameters, cursor, context
      File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1518, in _handle_dbapi_exception
        sqlalchemy_exception, with_traceback=exc_info[2], from_=e
      File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
        raise exception
      File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
        cursor, statement, parameters, context
      File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
        cursor.execute(statement, parameters)
      File "/usr/local/lib/python3.6/site-packages/MySQLdb/cursors.py", line 209, in execute
        res = self._query(query)
      File "/usr/local/lib/python3.6/site-packages/MySQLdb/cursors.py", line 315, in _query
        db.query(q)
      File "/usr/local/lib/python3.6/site-packages/MySQLdb/connections.py", line 239, in query
        _mysql.connection.query(self, query)
    sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (2006, 'MySQL server has gone away')
    [SQL: INSERT INTO events (sender_id, data) VALUES (%s, %s)]
    [parameters: ('abbb261d304d472cacad7e42ca4c9dfb', '{"sender_id": "abbb261d304d472cacad7e42ca4c9dfb", "event": "action", "timestamp": 1618565423.7079535, "name": "action_listen", "policy": null, "confidence": null}')]
    (Background on this error at: http://sqlalche.me/e/e3q8)

and the version is 8.0.23, Yes, I know how to resolve this issue by MySQL server-side, but I need to resolve this issue from rasa end because I have very limited access to MySQL server.

Tobias_Wochinger · April 20, 2021, 11:54am

Can you please format your code?

ujjwaldeswal · April 20, 2021, 1:06pm

Hey @Tobias_Wochinger , Format in the sense of indentation right? I believe formatting is correct. Because in maximum cases my chatbot is working fine, but when I try to chat with my chatbot after 12-14 hrs we got this error.

Tobias_Wochinger · April 21, 2021, 7:18am

Thanks that helped a lot - seems we should fix this as described here . I create an issue here. Happy to help you with the implementation if you want to tackle the issue yourself

ujjwaldeswal · April 21, 2021, 9:21am

Thanks a lot for sharing the information, Yes absolutely, it’s a great opportunity for me, @Tobias_Wochinger. and Be safe❤️

Tobias_Wochinger · April 27, 2021, 8:54am

Thank you! You, too, @ujjwaldeswal !

Topic		Replies	Views
Error happened when trying to save conversation tracker to 'SQLTrackerStore' Rasa Open Source	12	2809	September 22, 2021
RASA has problem with Oracle database Rasa Open Source	3	1008	December 11, 2019
Rasa with MySQL Database Rasa Open Source	8	3169	August 25, 2021
How to fix async.io error Rasa Open Source	2	1817	February 14, 2022
Rasa server ERROR from asyncio Rasa Open Source	1	499	September 22, 2022

MySQL server has gone away On RASA

Related topics