In my pipeline I use WhitespaceTokenizer followed by RegexFeaturizer. I have a regex expression that matches any North American phone numbers (which is my entity of interest) even if they are written wrongly. However, in the cases like (123) 456 7891 where there is a")" followed by a space, the WhitespaceTokenizer splits the entity before the regex is applied. This ends up in wrong entity recognition. How can I prevent this behaviour? Is there a way I can override this?
The expected behaviour of WhitespaceTokenizer according to the documentation is,
However, if regex is a requirement, you could set use_word_boundaries to False and give it a try like that.
Haven’t given that one a shot myself but sounds like a good thing to try.
If it won’t work, you can implement your own custom component that would extract regex from the original input text, not the processed tokens and.
What I did is to use both DIETClassifier and RegexEntityExtractor as RegexFeaturizer requires some sort of Tokenizer and might need to write exceptions which could down the lane conflict with other recognitions and hence the performance.
So, right now I am using set in actions to remove duplicate detection.
Haven’t tried Duckling. My case is a bit unique that the phone numbers may not always follow the standards, and even then I have to detect. So, Regex just fits my case for now. I will definitely give it a go, though.
Regarding the use of RegexEntityExtractor instead of a RegexFeaturizer, I’m not that clear on why you think REE is better than RF in terms of possible future conflicts. Truth be told, REE is more prone to popping with conflicts than RF (RF just propagates the signal). You can take a look at the note related to REE inside of the Components for better understanding.
Since you’re having such a use case and regex is mandatory, maybe you could consider writing a custom component for that extraction and then reference that entity later in your action code. Inside of the custom NLU component you will have access to the original text.
