Rasa in production pod (Kubernet cluster) restarted when loading model

Rasa Open Source
1

Dear All,

I got issue with my production pod in a kubernet cluster restarted when it loads models from rasa-x. My model about 600M (because I used bert as language model). Could you please help to advise this issue?

2020-11-21 22:14:41 DEBUG rasa.core.brokers.broker - Instantiated event broker to ‘PikaEventBroker’. 2020-11-21 22:14:41 DEBUG rasa.core.tracker_store - Attempting to connect to database via ‘postgresql://postgres:password@rasa-postgresql:5432/rasa’. 2020-11-21 22:14:42 DEBUG rasa.core.brokers.pika - RabbitMQ connection to ‘rasa-rabbit’ was established. 2020-11-21 22:14:42 DEBUG rasa.core.brokers.pika - RabbitMQ channel was opened. Declaring fanout exchange. 2020-11-21 22:14:42 DEBUG rasa.core.tracker_store - Connection to SQL database ‘tracker’ successful. 2020-11-21 22:14:42 DEBUG rasa.core.tracker_store - Connected to SQLTrackerStore. 2020-11-21 22:14:42 DEBUG rasa.core.lock_store - Connected to lock store ‘RedisLockStore’. 2020-11-21 22:14:42 DEBUG rasa.core.nlg.generator - Instantiated NLG to ‘TemplatedNaturalLanguageGenerator’. 2020-11-21 22:14:42 DEBUG rasa.core.agent - Requesting model from server http://rasa-rasa-x-rasa-x:5002/api/projects/default/models/tags/production… 2020-11-21 22:16:33 DEBUG rasa.core.agent - Unzipped model to ‘/tmp/tmpf0c3jtul’ 2020-11-21 22:16:33 DEBUG rasa.core.agent - Found new model with fingerprint 3188af5288db22e5ec714e70eaf52aa1. Loading…

2

I increased timeout of request, it seemed work. Let monitor it.

1 Like
3

Hi @nhha1602! Thanks for bringing this up! Feel free to post updates here :rocket: If this solves your issue you can also mark your comment as a solution