How can i add a persian tokenizer as a component?

Rasa Open Source
1

Hi, How can I add a hazm tokenizer to the rasa? hazm library is a text normalizer and tokenizer for the Persian language

2

Hi @a_vakily ,

I recommend you check out the rasa-nlu-examples repository, and create an issue to add the hazm tokenizer.

You can then either implement it yourself, using the other tokenizer implementations as a reference, or see if someone from the community is interested on working on this with you.

3

Thank you Mr Arjaan Ok, i will.