Hello Rasa Team,
with the LanguageModelTokenizer being deprecated and the LanguageModelFeaturizer implementing its behavior, I am wondering which effect using any tokenizer in the pipeline has to the outcome.
To my understanding the the LanguageModelFeaturizer does the tokenization, so it should get the complete examples as input. Is that right? If so are the tokens from the arbitrary tokenizer component used in any step?