I will do so, I guess at some point today. It would be really nice to have that indeed. 
Ok, I see and it makes a lot of sense. We have observed indeed that the model was doing weird things when tokenizing (we are working with Swedish), so this is indeed a very good addition. Yesterday we tried to train an experimental model without roles and similar data and we have seen that we got predictions with split words as well, so it doesn’t seem like the cleanup method was working properly, at least not for Swedish (in Rasa 1.10.14) and/or with this tokenizer.
But actually, we tried Rasa 2.1.3 for the first time with the same data later on and the problem disappeared at least for entities without roles. The data is exactly the same and the only difference in the pipeline is that we add the WhitespaceTokenizer
language: "sv"
pipeline:
- name: WhitespaceTokenizer
- name: LanguageModelFeaturizer
model_name: "bert"
model_weights: "models/tokenization-models/bert-base-swedish-cased/"
- name: RegexFeaturizer
- name: LexicalSyntacticFeaturizer
- name: CountVectorsFeaturizer
analyzer: "word"
min_ngram: 1
max_ngram: 5
- name: CountVectorsFeaturizer
analyzer: "char_wb"
min_ngram: 1
max_ngram: 10
- name: DIETClassifier
epochs: 500
- name: EntitySynonymMapper
This makes me think that the fact that we were forced to only use the LanguageModelTokenizer instead of another one seemed to be causing the problem of words being splitted into two. Like the example in my last post: “Anna Kronlid” > “Anna Kron” | “lid”. As you say, it does seem like that was causing issues for our Swedish model, so it’s really good that now we can add another one.
Thanks!