Within the conversation I am asking the user for his name. For this I am using the ‘ner_spacy’ extractor which extracts the name pretty good.
If the user answers: “My name is Bob” everything works fine. ner_spacy ectracts the name and tensorflow recognizes the intent “provide_name”
If however the users just answers with his name like: “Bob Miller” then ‘nerspacy’ still extracts the name as an entity but tensorflow can’t recognize the intent as “provide_name”
Is there a good solution for this?
I can think of two:
Spamming my nlu file with huge name lists which makes using ‘ner_spacy’ redundant, probbably gives worse results and needs maintaing on my site.
Using entities as features. If there was a featurizer which would add recognized entities to the feature vector tensorflow might be able to detect the right intent, if spacy recognizes a name.
Would the 2 solution be viable? Is there a featurizer that already does this? Or is there a different solution which works better?
I am curious to know about your second solution? would you create a feature vector for a recognized entity and feed this feature vector to the tensorflow model? instead of a count_vector featurizer? Can you explain your pipeline
since the embeddings for tensorflow is trained from scratch, if you don’t have the names in your training data, i think it will be treated as OOV instead. I am not sure exactly how to change that? what you can do is basically create a outofscope intent, and use a OOV token that replaces unknown words, if you have one word answers like just the name, it is classified as outofscope intent but since you have an entity available, you can define custom logic to override the outofscope intent with a custom intent in custom python code in the action server if you are using rasa core for example
This was just a quick Idea, I have nothing working yet. Yes the plan was converting the entity to a feature vector. But I would yet have to research the best way to do this. And then adding this vector to the vector created by the count_vector featurizer. If I understood the code right the ngram_featurizer already extends text-features in a similar way.
To create vectors during the training I would have to manually call the spacy extractor as the entity data is not available there.
Another Idea, that probably is easier to implement and more reliable is writing a custom classifier that works in a similar way as the keyword classifier but uses intents as ‘keywords’.