I am having an issue in distinguishing questions from non-questions. For example, in english you can say “Is it good?” and “It is good.”. The same statements in Greek are “Είναι καλό;” & “Είναι καλό.”. They have the same word order, the only difference is the question mark (";" in greek).
So, I am wondering what is happening in the tokenizer. I am using the
LanguageModelTokenizer, is there a way (i.e. in
rasa shell) to see how the tokenizer tokenises the sentence, and if the questionmark is kept?