[SOLVED]RASA NLU misbehaving when trained on server but works when trained on local

Rasa Open Source
1

Following issues have been bugging me lately -_-

  1. NLU models trained on one system can’t be used on another .

  2. NLU model trained on server does wrong intent classification , the very model when trained on local system works perfectly.

Note : All dependencies of local and server are identical . Apart from the fact that local uses py 3.5.6 and server uses py 3.6.3

@akelad @Juste @souvikg10 anyone ???

2

Can you not update or downgrade the Python version to test it out. Something worth trying

3

yeah that’s what i am doing now . But my question is if i am using SVM for intent classification how can that change over different python version … rasa has some seriously strange issues

4

you are using sklearn which ultimately depends on python, python is an interpretable language meaning it relies for kernel operations on a C compiler(if i am not mistaken), so i guess between versions, there are some underlying APIs, that can have an impact. Best is to try the rule of elimination.

5

But should it force misclassification ? i mean it can throw errors of some kind or error flagging version mismatch or something . Doesn’t this sound a bit ridiculous

6

I would suggest to make the version aligned between your servers and local first and then try to investigate the model performance discrepancy. preferably use docker to ensure version consistency

7

So i did an exact replica of dependencies including python build of server on my local . Issue persists , local model works perfectly but server model fails horrendously :confused: … any idea why is this happening ? Now each and every depency is identical , i checked it thoroughly .

8

Intesresting, we never face such issues porting our models since we package everything in Docker

What is case here

  • Do you train a model in your machine and then run it on a server on the same test set?
  • Do you train two models, one on machine and one on server?
  • When you say fails horrendously, what is the F1 score?
9

Imagine model A being trained on server and model B trained on local both with identical data , config files and also the dependencies . Model A gives numerous wrong prediction for a set of inputs X but for the very input set X local model gives excellent predictions . When we found out this issue we tried training the model on our local and tried porting it to the server , this resulted in a failure too as the model trained on local can’t be run on server and vice versa.

10

btw what did you mean by “packaging everything in docker” ? are you implying that you use docker image of rasa ?

11

we build our own but all our dependencies, models and code are in the same image

12

It is indeed really strange. what was the error that the model can’t be ported to the server?

13

So it’s solved . As i was using SVM , i just kept the following modules similar and now the issue is solved.

  • Spacy and Spacy Model (be careful to use similar model i.e sm or md or lg) [Be very specific about it ]
  • sklearn
  • scipy
  • sklearn-crf suite
  • Numpy

Make sure you agree on to py3 or py2 . Minor version mismatch wont impact.

1 Like