Use hugging face model tokenizer

Rasa Open Source
1

hi

how can i use the tokenizer of the hugging face models? like this one:

from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained(“bert-base-uncased”)

model = AutoModelForMaskedLM.from_pretrained(“bert-base-uncased”)

i want to use this special tokenizer. thanks:)