RASA NLU misbehaving when trained on server but works when trained on local

(Abir) #1

Following issues have been bugging me lately -_-

  1. NLU models trained on one system can’t be used on another .

  2. NLU model trained on server does wrong intent classification , the very model when trained on local system works perfectly.

Note : All dependencies of local and server are identical . Apart from the fact that local uses py 3.5.6 and server uses py 3.6.3

@akelad @Juste @souvikg10 anyone ???

(Souvik Ghosh) #2

Can you not update or downgrade the Python version to test it out. Something worth trying

(Abir) #3

yeah that’s what i am doing now . But my question is if i am using SVM for intent classification how can that change over different python version … rasa has some seriously strange issues

(Souvik Ghosh) #4

you are using sklearn which ultimately depends on python, python is an interpretable language meaning it relies for kernel operations on a C compiler(if i am not mistaken), so i guess between versions, there are some underlying APIs, that can have an impact. Best is to try the rule of elimination.

(Abir) #5

But should it force misclassification ? i mean it can throw errors of some kind or error flagging version mismatch or something . Doesn’t this sound a bit ridiculous

(Souvik Ghosh) #6

I would suggest to make the version aligned between your servers and local first and then try to investigate the model performance discrepancy. preferably use docker to ensure version consistency

(Abir) #7

So i did an exact replica of dependencies including python build of server on my local . Issue persists , local model works perfectly but server model fails horrendously :confused: … any idea why is this happening ? Now each and every depency is identical , i checked it thoroughly .

(Souvik Ghosh) #8

Intesresting, we never face such issues porting our models since we package everything in Docker

What is case here

  • Do you train a model in your machine and then run it on a server on the same test set?
  • Do you train two models, one on machine and one on server?
  • When you say fails horrendously, what is the F1 score?
(Abir) #9

Imagine model A being trained on server and model B trained on local both with identical data , config files and also the dependencies . Model A gives numerous wrong prediction for a set of inputs X but for the very input set X local model gives excellent predictions . When we found out this issue we tried training the model on our local and tried porting it to the server , this resulted in a failure too as the model trained on local can’t be run on server and vice versa.

(Abir) #10

btw what did you mean by “packaging everything in docker” ? are you implying that you use docker image of rasa ?

(Souvik Ghosh) #11

we build our own but all our dependencies, models and code are in the same image

(Souvik Ghosh) #12

It is indeed really strange. what was the error that the model can’t be ported to the server?