Document too large error with Mongo tracker

Rasa Open Source
1

Hi

I am getting error for storing documents in mongo tracker.

Here is the stack trace

 Caught an exception during respond.
 Traceback (most recent call last):
 
 File "/home/ubuntu/.local/lib/python3.6/site-packages/rasa_core/server.py", line 390, in respond
 sender_id=sender_id)

 File "/home/ubuntu/.local/lib/python3.6/site-packages/rasa_core/agent.py", line 401, in handle_text
 return self.handle_message(msg, message_preprocessor)

File "/home/ubuntu/.local/lib/python3.6/site-packages/rasa_core/agent.py", line 317, in 
handle_message
return processor.handle_message(message)

File "/home/ubuntu/.local/lib/python3.6/site-packages/rasa_core/processor.py", line 88, in 
handle_message
self._save_tracker(tracker)

File "/home/ubuntu/.local/lib/python3.6/site-packages/rasa_core/processor.py", line 445, in 
_save_tracker
self.tracker_store.save(tracker)


File "/home/ubuntu/.local/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1201, in 
_retry_with_session
return func(session, sock_info, retryable)

File "/home/ubuntu/.local/lib/python3.6/site-packages/pymongo/collection.py", line 847, in _update
retryable_write=retryable_write)

  File "/home/ubuntu/.local/lib/python3.6/site-packages/pymongo/pool.py", line 584, in command
self._raise_connection_failure(error)
 
 File "/home/ubuntu/.local/lib/python3.6/site-packages/pymongo/pool.py", line 745, in         
 _raise_connection_failure
  raise error

File "/home/ubuntu/.local/lib/python3.6/site-packages/pymongo/pool.py", line 579, in command
unacknowledged=unacknowledged)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/pymongo/network.py", line 128, in command
name, size, max_bson_size + message._COMMAND_OVERHEAD)
 
 File "/home/ubuntu/.local/lib/python3.6/site-packages/pymongo/message.py", line 965, in     
 _raise_document_too_large
raise DocumentTooLarge("%r command document too large" % (operation,))
pymongo.errors.DocumentTooLarge: 'update' command document too large
127.0.0.1 - - [2019-09-06 07:13:55] "POST /conversations/49/respond HTTP/1.0" 500 414 3.622841

I cleaned record for this user and it started working again. What are the limits with mongo tracker?

2

Seems like when searching for that error the limit is 16MB per document. How long was this tracker that it reached that big a size? :sweat_smile:

3

Over the period of time, tracker for particular user is completely exhausted. Is there any way to archive old conversation and free up space?

4

any solution for this problem?

If mongo has such limitation, then its difficult to use rasa with mongo tracker

5

For what length of time was that conversation? 16MB seems impossibly large. @Tobias_Wochinger is there anything we can do about that?

6

There is currently an issue for too big trackers: Tracker gets very big for long conversations · Issue #3011 · RasaHQ/rasa · GitHub @r4sn4 Can you tell us a bit more about your use case so that we can understand how the tracker gets that big? Do you know how many user messages / actions are in the tracker?

7

Hi

I reduced my tracker size now. Its working fine.

But there’s one query. What happens if the upper size limit of the Mongo object is reached when RASA appending events for particular user in the same object in the tracker?

Does rasa have auto archival policy to archive older events? or Do we need to create batch job to delete old events from Tracker?

8

Does rasa have auto archival policy to archive older events? or Do we need to create batch job to delete old events from Tracker?

Nope, that’s currently not possible. I hope to tackle the linked issue soon, so that we can solve the problems with the large trackers.

@r4sn4 Do you have some rough numbers

  • how many events are in your tracker
  • what are the time spans of the conversation events?