Unable to train sara/rasa-demo bot in rasa x GCP Docker

I have just setup rasa x in gcp vm (computer engine) via docker (not the manual install). The model spends some time trying to train and then fails.

Here are a few excerpts from the log. @erohmensing , do you have any thoughts?

Including @mloubser

Can you please provide the version you tried to install? And any custom changes you made in e.g. docker-compose.override.yaml?

No custom changes. Initially, I tried 0.38.x and then upgraded to 0.39.x. I’m wondering if the VM has to be configured in a particular way to use NGINX?

You should be good on the VM side if it has enough CPU and memory for your workload; how long does training carry on before you see this error?

Typically 1 hour.

Does the worker pod go down? Do you see any resource complaints? If you’re training on rasa-demo, it can take quite a time and quite a lot of memory to train successfully, so I suspect a resource constraint issue

Easiest way to check: switch to a different, small repo, and see if it can train succesfully using that dataset

It is unable to train on a small repo.

Does it fail faster then, or also only after. 1 hour?

Seems to still train for a comparable amount of time.