Rasa 3.0 OOM kills on K8s

Are there any changes in the Memory needs of Rasa 3.0 because i have this warning below. Taking usually a lot longer to load the model compared to Rasa 2.0 and on k8s with the same configuration as Rasa 2.0(no language models, etc) also only using NLU, my pods are crashing after the first prediction. There are no logs :frowning:

I fear there is a memory leak maybe. any thoughts?

/usr/local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:447: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/cond_grad/gradients/cond/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), 
values=Tensor("gradients/cond_grad/gradients/cond/GatherV2_grad/Reshape:0", shape=(None,), dtype=float32), dense_shape=Tensor("gradients/cond_grad/gradients/cond/GatherV2_grad/Cast:0", 
shape=(1,), dtype=int32))) to a dense Tensor of unknown shape. 
This may consume a large amount of memory
3 Likes