Error Custom Tracker with Postgres

I build custom Redis and InMemory trackers that change the max_event_history variable based off this article Using a custom tracker store to manage max event history in RASA. | by Simran Kaur Kahlon | Gray Matrix | Medium. They all worked just fine. So I want to do the same for the Postgres version.

When I attempted the Postgres version it did not work for me. Here are my endpoints:

tracker_store:
   type: myTracker.MyTrackerStore
   dialect: "postgresql"  # the dialect used to interact with the db
   url: "localhost"
   port: 5432
   db: "mydb"  # path to your db
   username: "post"
   password: "post" # password used for authentication

action_endpoint:
 url: "http://localhost:5055/webhook"

Here is my custom tracker. I basically copy the sqlTracker class definition (from https://github.com/RasaHQ/rasa/blob/a4ee09c41e3f173c3aa9e7df69d3f0f727eeb61d/rasa/core/tracker_store.py) and add in the max_event_history value.

class MyTrackerStore(SQLTrackerStore):
    """Store which can save and retrieve trackers from an SQL database."""
    def __init__(
        self,
        domain: Optional[Domain] = None,
        dialect: Text = "sqlite",
        host: Optional[Text] = None,
        port: Optional[int] = None,
        db: Text = "rasa.db",
        username: Text = None,
        password: Text = None,
        event_broker: Optional[EventBroker] = None,
        login_db: Optional[Text] = None,
        query: Optional[Dict] = None,
        **kwargs: Dict[Text, Any],
    ) -> None:
        self.max_event_history = 20
        import sqlalchemy.exc
        engine_url = self.get_db_url(
            dialect, host, port, db, username, password, login_db, query
        )
        self.engine = sa.create_engine(engine_url, **create_engine_kwargs(engine_url))
        logger.debug(
            f"Attempting to connect to database via '{repr(self.engine.url)}'."
        )
        # Database might take a while to come up
        while True:
            try:
                # if `login_db` has been provided, use current channel with
                # that database to create working database `db`
                if login_db:
                    self._create_database_and_update_engine(db, engine_url)
                try:
                    self.Base.metadata.create_all(self.engine)
                except (
                    sqlalchemy.exc.OperationalError,
                    sqlalchemy.exc.ProgrammingError,
                ) as e:
                    # Several Rasa services started in parallel may attempt to
                    # create tables at the same time. That is okay so long as
                    # the first services finishes the table creation.
                    logger.error(f"Could not create tables: {e}")
                self.sessionmaker = sa.orm.session.sessionmaker(bind=self.engine)
                break
            except (
                sqlalchemy.exc.OperationalError,
                sqlalchemy.exc.IntegrityError,
            ) as error:
                logger.warning(error)
                sleep(5)
        logger.debug(f"Connection to SQL database '{db}' successful.")
        # super().__init__(domain, event_broker, **kwargs)

You can see that super().init is commented out. When I had it in the code, it was giving me an error of:

TypeError: argument of type ‘NoneType’ is not iterable

Anyway, the error that I get when I initiate the rasa server locally, with the code from above is the following:

2021-08-04 20:14:35 DEBUG    rasa.nlu.classifiers.diet_classifier  - Failed to load model for 'ResponseSelector'. Maybe you did not provide enough training data and no model was trained or the path '/tmp/tmpsikts5ku/nlu' doesn't exist?
2021-08-04 20:14:35 DEBUG    myTracker  - Attempting to connect to database via 'postgresql://postgres:***@localhost:5432/mydb'.
2021-08-04 20:14:35 DEBUG    myTracker  - Connection to SQL database 'mydb' successful.
2021-08-04 20:14:35 DEBUG    rasa.core.tracker_store  - Connected to MyTrackerStore.
2021-08-04 20:14:35 DEBUG    rasa.core.lock_store  - Connected to lock store 'InMemoryLockStore'.
2021-08-04 20:14:36 DEBUG    rasa.model  - Extracted model to '/tmp/tmpbbh6eqan'.
2021-08-04 20:14:36 DEBUG    rasa.utils.tensorflow.models  - Loading the model from /tmp/tmpbbh6eqan/core/policy_1_TEDPolicy/ted_policy.tf_model with finetune_mode=False...
2021-08-04 20:14:43 DEBUG    rasa.utils.tensorflow.models  - Finished loading the model.
2021-08-04 20:14:43 DEBUG    rasa.core.nlg.generator  - Instantiated NLG to 'TemplatedNaturalLanguageGenerator'.
2021-08-04 20:14:43 ERROR    rasa.core.agent  - Could not load model due to 'MyTrackerStore' object has no attribute 'event_broker'.
/home/my_test/venv/lib/python3.6/site-packages/rasa/shared/utils/io.py:96: UserWarning: The model at 'models' could not be loaded. Error: <class 'AttributeError'>: 'MyTrackerStore' object has no attribute 'event_broker'
/home/my_test/venv/lib/python3.6/site-packages/rasa/shared/utils/io.py:96: UserWarning: Agent could not be loaded with the provided configuration. Load default agent without any model.
2021-08-04 20:14:43 DEBUG    rasa.core.nlg.generator  - Instantiated NLG to 'TemplatedNaturalLanguageGenerator'.
[2021-08-04 20:14:43 +0000] [27810] [ERROR] Experienced exception while trying to serve
Traceback (most recent call last):
  File "/home/my_test/venv/lib/python3.6/site-packages/sanic/app.py", line 1129, in run
    serve(**server_settings)
  File "/home/my_test/venv/lib/python3.6/site-packages/sanic/server.py", line 888, in serve
trigger_events(before_start, loop)
  File "/home/my_test/venv/lib/python3.6/site-packages/sanic/server.py", line 696, in trigger_events
    loop.run_until_complete(result)
  File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
  File "/home/my_test/venv/lib/python3.6/site-packages/rasa/core/run.py", line 289, in load_agent_on_start
    remote_storage=remote_storage,
  File "/home/my_test/venv/lib/python3.6/site-packages/rasa/core/agent.py", line 365, in __init__
    self.tracker_store = self.create_tracker_store(tracker_store, self.domain)
  File "/home/my_test/venv/lib/python3.6/site-packages/rasa/core/agent.py", line 869, in create_tracker_store
    return FailSafeTrackerStore(tracker_store)
  File "/home/my_test/venv/lib/python3.6/site-packages/rasa/core/tracker_store.py", line 1160, in __init__
    super().__init__(tracker_store.domain, tracker_store.event_broker)
AttributeError: 'MyTrackerStore' object has no attribute 'event_broker'
Traceback (most recent call last):
  File "/home/my_test/venv/bin/rasa", line 8, in <module>
    sys.exit(main())
  File "/home/my_test/venv/lib/python3.6/site-packages/rasa/__main__.py", line 116, in main
    cmdline_arguments.func(cmdline_arguments)
  File "/home/my_test/venv/lib/python3.6/site-packages/rasa/cli/run.py", line 90, in run
    rasa.run(**vars(args))
  File "/home/my_test/venv/lib/python3.6/site-packages/rasa/run.py", line 57, in run
    **kwargs,
  File "/home/my_test/venv/lib/python3.6/site-packages/rasa/core/run.py", line 226, in serve_application
    workers=number_of_workers,
  File "/home/my_test/venv/lib/python3.6/site-packages/sanic/app.py", line 1129, in run
    serve(**server_settings)
  File "/home/my_test/venv/lib/python3.6/site-packages/sanic/server.py", line 888, in serve
    trigger_events(before_start, loop)
  File "/home/my_test/venv/lib/python3.6/site-packages/sanic/server.py", line 696, in trigger_events
    loop.run_until_complete(result)
  File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
  File "/home/my_test/venv/lib/python3.6/site-packages/rasa/core/run.py", line 289, in load_agent_on_start
    remote_storage=remote_storage,
  File "/home/my_test/venv/lib/python3.6/site-packages/rasa/core/agent.py", line 365, in __init__
    self.tracker_store = self.create_tracker_store(tracker_store, self.domain)
  File "/home/my_test/venv/lib/python3.6/site-packages/rasa/core/agent.py", line 869, in create_tracker_store
    return FailSafeTrackerStore(tracker_store)
  File "/home/my_test/venv/lib/python3.6/site-packages/rasa/core/tracker_store.py", line 1160, in __init__
    super().__init__(tracker_store.domain, tracker_store.event_broker)
AttributeError: 'MyTrackerStore' object has no attribute 'event_broker'
2021-08-04 20:14:43 DEBUG    urllib3.connectionpool  - Starting new HTTPS connection (1): o251570.ingest.sentry.io:443
2021-08-04 20:14:43 DEBUG    urllib3.connectionpool  - https://o251570.ingest.sentry.io:443 "POST /api/2801673/store/ HTTP/1.1" 200 41

I have not done anything more in the other custom tracker versions regarding the ‘event_broker’. I am quite lost. Does anyone have any ideas? Thank you in advance.

Here is the full error I get when I include the super() in the code, in case anyone is wondering:

2021-08-04 20:48:49 DEBUG    rasa.model  - Extracted model to '/tmp/tmpm82m2g1v'.
2021-08-04 20:48:49 DEBUG    rasa.utils.tensorflow.models  - Loading the model from /tmp/tmpm82m2g1v/nlu/component_5_DIETClassifier.tf_model with finetune_mode=False...
2021-08-04 20:48:49.435169: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-08-04 20:48:49.442737: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2499990000 Hz
2021-08-04 20:48:49.442894: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5825fb0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-08-04 20:48:49.442972: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2021-08-04 20:48:49 DEBUG    rasa.nlu.classifiers.diet_classifier  - Following metrics will be logged during training: 
2021-08-04 20:48:49 DEBUG    rasa.nlu.classifiers.diet_classifier  -   t_loss (total loss)
2021-08-04 20:48:49 DEBUG    rasa.nlu.classifiers.diet_classifier  -   i_acc (intent acc)
2021-08-04 20:48:49 DEBUG    rasa.nlu.classifiers.diet_classifier  -   i_loss (intent loss)
2021-08-04 20:48:58 DEBUG    rasa.utils.tensorflow.models  - Finished loading the model.
/home/my_test/venv/lib/python3.6/site-packages/rasa/utils/train_utils.py:535: UserWarning: model_confidence is set to `softmax`. It is recommended to try using `model_confidence=linear_norm` to make it easier to tune fallback thresholds.
  category=UserWarning,
2021-08-04 20:48:58 DEBUG    rasa.nlu.classifiers.diet_classifier  - Failed to load model for 'ResponseSelector'. Maybe you did not provide enough training data and no model was trained or the path '/tmp/tmpm82m2g1v/nlu' doesn't exist?
2021-08-04 20:48:58 DEBUG    myTracker  - Attempting to connect to database via 'postgresql://postgres:***@localhost:5432/mydb'.
2021-08-04 20:48:58 DEBUG    myTracker  - Connection to SQL database 'mydb' successful.
[2021-08-04 20:48:58 +0000] [32273] [ERROR] Experienced exception while trying to serve
Traceback (most recent call last):
  File "/home/my_test/venv/lib/python3.6/site-packages/sanic/app.py", line 1129, in run
    serve(**server_settings)
  File "/home/my_test/venv/lib/python3.6/site-packages/sanic/server.py", line 888, in serve
    trigger_events(before_start, loop)
  File "/home/my_test/venv/lib/python3.6/site-packages/sanic/server.py", line 696, in trigger_events
    loop.run_until_complete(result)
  File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
  File "/home/my_test/venv/lib/python3.6/site-packages/rasa/core/run.py", line 255, in load_agent_on_start
    _tracker_store = TrackerStore.create(endpoints.tracker_store, event_broker=_broker)
  File "/home/my_test/venv/lib/python3.6/site-packages/rasa/core/tracker_store.py", line 123, in create
    return _create_from_endpoint_config(obj, domain, event_broker)
  File "/home/my_test/venv/lib/python3.6/site-packages/rasa/core/tracker_store.py", line 1254, in _create_from_endpoint_config
    domain, endpoint_config, event_broker
  File "/home/my_test/venv/lib/python3.6/site-packages/rasa/core/tracker_store.py", line 1284, in _load_from_module_name_in_endpoint_config
    host=store.url, domain=domain, event_broker=event_broker, **store.kwargs
  File "/home/my_test/rasa_test_apis/myTracker.py", line 113, in __init__
    super().__init__(domain, event_broker, **kwargs)
  File "/home/my_test/venv/lib/python3.6/site-packages/rasa/core/tracker_store.py", line 852, in __init__
    self.engine = sa.create_engine(engine_url, **create_engine_kwargs(engine_url))
  File "/home/my_test/venv/lib/python3.6/site-packages/sqlalchemy/engine/__init__.py", line 520, in create_engine
    return strategy.create(*args, **kwargs)
  File "/home/my_test/venv/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 61, in create
    entrypoint = u._get_entrypoint()
  File "/home/my_test/venv/lib/python3.6/site-packages/sqlalchemy/engine/url.py", line 168, in _get_entrypoint
    if "+" not in self.drivername:
TypeError: argument of type 'NoneType' is not iterable
Traceback (most recent call last):
  File "/home/my_test/venv/bin/rasa", line 8, in <module>
    sys.exit(main())
  File "/home/my_test/venv/lib/python3.6/site-packages/rasa/__main__.py", line 116, in main
    cmdline_arguments.func(cmdline_arguments)
  File "/home/my_test/venv/lib/python3.6/site-packages/rasa/cli/run.py", line 90, in run
    rasa.run(**vars(args))
  File "/home/my_test/venv/lib/python3.6/site-packages/rasa/run.py", line 57, in run
    **kwargs,
  File "/home/my_test/venv/lib/python3.6/site-packages/rasa/core/run.py", line 226, in serve_application
    workers=number_of_workers,
  File "/home/my_test/venv/lib/python3.6/site-packages/sanic/app.py", line 1129, in run
    serve(**server_settings)
  File "/home/my_test/venv/lib/python3.6/site-packages/sanic/server.py", line 888, in serve
    trigger_events(before_start, loop)
  File "/home/my_test/venv/lib/python3.6/site-packages/sanic/server.py", line 696, in trigger_events
    loop.run_until_complete(result)
  File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
  File "/home//my_test/venv/lib/python3.6/site-packages/rasa/core/run.py", line 255, in load_agent_on_start
    _tracker_store = TrackerStore.create(endpoints.tracker_store, event_broker=_broker)
  File "/home/my_test/venv/lib/python3.6/site-packages/rasa/core/tracker_store.py", line 123, in create
    return _create_from_endpoint_config(obj, domain, event_broker)
  File "/home/my_test/venv/lib/python3.6/site-packages/rasa/core/tracker_store.py", line 1254, in _create_from_endpoint_config
    domain, endpoint_config, event_broker
  File "/home/my_test/venv/lib/python3.6/site-packages/rasa/core/tracker_store.py", line 1284, in _load_from_module_name_in_endpoint_config
    host=store.url, domain=domain, event_broker=event_broker, **store.kwargs
  File "/home/my_test/rasa_test_apis/myTracker.py", line 113, in __init__
    super().__init__(domain, event_broker, **kwargs)
  File "/home/my_test/venv/lib/python3.6/site-packages/rasa/core/tracker_store.py", line 852, in __init__
    self.engine = sa.create_engine(engine_url, **create_engine_kwargs(engine_url))
  File "/home/my_test/venv/lib/python3.6/site-packages/sqlalchemy/engine/__init__.py", line 520, in create_engine
    return strategy.create(*args, **kwargs)
  File "/home/my_test/venv/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 61, in create
    entrypoint = u._get_entrypoint()
  File "/home/my_test/venv/lib/python3.6/site-packages/sqlalchemy/engine/url.py", line 168, in _get_entrypoint
    if "+" not in self.drivername:
TypeError: argument of type 'NoneType' is not iterable
2021-08-04 20:48:58 DEBUG    urllib3.connectionpool  - Starting new HTTPS connection (1): o251570.ingest.sentry.io:443
2021-08-04 20:48:58 DEBUG    urllib3.connectionpool  - https://o251570.ingest.sentry.io:443 "POST /api/2801673/store/ HTTP/1.1" 200 41

@anne576 Heya!

type: myTracker.MyTrackerStore why you are using this?

This is to call my custom tracker from the endpoints I believe?

@anne576

tracker_store:
   type: SQL
   dialect: "postgresql"  # the dialect used to interact with the db
   url: "localhost"  # (optional) host of the sql db, e.g. "localhost"
   db: "chatbot"  # path to your db
   username: "postgres"  # username used for authentication
   password: "123"  # password used for authentication
   # query: # optional dictionary to be added as a query string to the connection URL
   #   driver: my-driver

I guess you need to install psycopg2-binary and PyMySQL when you run rasa, do check the --debug

@anne576 I hope you have Postgres11 or 13 install on your localhost or machine.

Thank you for your answer. When I wasn’t building a custom tracker, I met the requirements for an SQL tracker_store and made it work just fine. So the issue isn’t connecting to a normal SQL tracker store. The configuration you suggested and dependencies do not quite relate to the custom tracker I am trying to build (or at least I cannot see how it is related to it). I checked and all these dependencies you mentioned are installed on my machine.

@anne576 Ohh Ok, I thought you just want to use Tracker store to store the conversation in Postgres.

Does anyone happen to know how I could fix this error?

@anne576 - you can pass None to the event broker. in your super().init you are passing an event broker as an argument, if you look at the SqlTrackerStore implementation, event_broker is optional.

pass None for event_broker variable perhaps

Unfortunately, this didn’t work either. Thank you for the suggestion though.

What was the error when you pass none. Infact don’t pass the event broker at all in the super()

1 Like

Okay, yes so not passing the event broker solved the error. Thank you. The only issue is that now the events are not stored at all in the tracker, outside of the max_events_history value. Am I wrong in thinking that the events should still be stored? And that the max_events_history only minimises the events for the prediction. I saw this explained here by “wochinge”: Tracker gets very big for long conversations · Issue #3011 · RasaHQ/rasa · GitHub.

@anne576 as far as i understand tracker store, it is supposed to storing active conversation session that impacts the conversation and thus would persist the max_events_history you would want, beyond that it will roll over unless you use an event broker and pass it along to RabbitMQ and then store it for long term. I believe this is slightly different than the max_history parameter that was mentioned in the issue you provided. Tobias can confirm this.

Okay thank you for that! Do you mind pointing me to an example/ showing me how to use an event broker and pass it along to RabbitMQ and then store it for long term?

This might help. Essentially the event brokers passes the tracker event but you need to add a listener on the other end to capture the event and store.

ELK is a nice option as explain here by greg.

So I have attempted this. My main issue is that there is a significant latency from the bot replying to a user as time passes & more users communicate with the bot. So I found that setting the max_event_history to let’s say 40, stops this from happening.

Now I have tried to do what you just mentioned, and yes the events can be stored and captured. But when the max_event_history parameter is in place, the events in the event_broker aren’t updated either. Unless, because I am not currently using Rasa X, I may have done something wrong :sweat_smile: . Is there a way to store these events and still have the max_event_history in there? Do you happen to have a different solution for the latency issue I’ve experienced? Thank you.

are you capturing the events from the event broker into another more static database like ES or perhaps just another table in postgres.

Did you try to query the tracker_store in the database of postgres. I wish i could draw the data flow here but here goes

Your tracker object which essentially hold all event history might be limited to accessing 40 events since that’s the parameter you set BUT you can still query the tracker table in your SQL database for events beyond 40. I think. I dont remember.

If this is not the case, then you can simply use the event broker and capture the events and write them to another table in postgres which looks like your tracker logs.

I think the max_event_history disables rasa to pull events beyond 40 but that doesn’t mean they don’t exist within the tracker store. I don’t think there is a mechanism to flush out older events

I tried an initial experiment, to see if the events are stored by the event_broker. So first configuration I tried was the SQL tracker_store and SQL event_broker (using postgres). This worked. So the events stored in the tracker_store db were all present in the event_broker db as well. Then I tested to see how this works with the max_event_history. I used my own SQL tracker_store, which is an instance of the SQL tracker_store and sets the max_event_history = 40, and the SQL event_broker. I looked at the events and once the 40 threshold was met, the tracker_store and event_broker did not receive more events.

I cannot imagine that the RabbitMQ would behave any differently.

So this leaves me in the same exact place unfortunately. I cannot access those events. No idea what I can do.

How do you access the events from the tracker store?

Are you writing plain SQL queries?

Just command line queries directly into postgres.