How to use fallback policy with MITIE or any pipeline that doesn’t return confidence value?
- I used just rasa core threshold but it doesn’t give good results !
So the NLU threshold won’t work with MITIE or any pipelines that don’t return confidence values. Is there a particular reason you’re using that pipeline? We don’t really recommend it because training is super slow etc
But the Core threshold should work as that’s not related to NLU. Could you describe what the problems are?
- I have 4244 training instances and upon testing : ner_spacy < ner_crf < ner_mitie , for intent classification I use Starspace approach . I saw SNIPS have data generation tool and am trying to create one at the moment.
- Take this scenario of classification: it classifies entity and intent wrong and the rasa core model receives the wrongly classified intent and entity. If we have a similar story for the wrongly classified intent and entity then response will be wrong as well (this is my understanding, correct me please if am wrong)
ner_spacy isn’t designed to handle custom entities btw. That’s for pretrained ones. Can I ask what kind of entities they are and what the performance difference is? There’s also a data generation tool for Rasa called Chatito: GitHub - rodrigopivi/Chatito: 🎯🗯 Generate datasets for AI chatbots, NLP tasks, named entity recognition or text classification models using a simple DSL!
Yes your second point is true. Since there’s no confidence value for MITIE it will assume the intent was correct – there’s not much that can be done about that i’m afraid, apart from using a pipeline that does provide confidence values 
But hold on - if you’re using the starspace approach for intent classification I assume you mean the tensorflow_embedding pipeline? That does provide you with a confidence value for intent classification, so the Fallback Policy should work
I already know about Chatito tool .
The tensorflow pipeline is presently used to get fallback policy. I wish not to use for some use cases as it learns embedding and if the utterance is not seen the similarity value could be low. especially when user sends descriptive messages it is hard to get good results with tensor flow pipeline.