I am using the supervised_embeddings and had to search for the default settings this uses which are:
- Choosing a Pipeline
- pipeline:
- name: “WhitespaceTokenizer”
- name: “RegexFeaturizer”
- name: “CRFEntityExtractor”
- name: “EntitySynonymMapper”
- name: “CountVectorsFeaturizer”
- name: “EmbeddingIntentClassifier”
Which means the WhitespaceTokenizer is having some fun with those occurrences of “I m” in my training data. I will clean this up to begin with and investigate other Tokenizers afterwards.
I am still a bit in the dark about whether the bot will learn to understand “I’m” and “Im” as equal, unfortunately the augmentation I use (EDA) really messes quite a bit with the data, it even introduces noise such as reversing the meaning of some training data (and not sure if that’s good or bad, I mean it should be good to a degree I suppose). It has however drastically improved my results/performance given the small amount of training data I have, so I’m keeping it for now.