# Purpose of Tokenizer when using LanguageModelFeaturizer

**URL:** <https://forum.rasa.com/t/purpose-of-tokenizer-when-using-languagemodelfeaturizer/39677>\
**Category:** Rasa Open Source\
**Created:** [February 1, 2021, 9:17am UTC](https://forum.rasa.com/t/purpose-of-tokenizer-when-using-languagemodelfeaturizer/39677 "2021-02-01T09:17:09Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Berken](https://avatars.discourse-cdn.com/v4/letter/b/3be4f8/32.png) [@Berken](https://forum.rasa.com/u/Berken)\
**Post date:** [February 1, 2021, 9:17am UTC](https://forum.rasa.com/t/purpose-of-tokenizer-when-using-languagemodelfeaturizer/39677/1 "2021-02-01T09:17:09Z")

</div>

Hello Rasa Team,

with the LanguageModelTokenizer being deprecated and the LanguageModelFeaturizer implementing its behavior, I am wondering which effect using any tokenizer in the pipeline has to the outcome.

To my understanding the the LanguageModelFeaturizer does the tokenization, so it should get the complete examples as input. Is that right? If so are the tokens from the arbitrary tokenizer component used in any step?

Thank you

---

<div class="post-metadata">

**Author:** ![Ghostvv](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/ghostvv/32/211_2.png) [@Ghostvv](https://forum.rasa.com/u/Ghostvv)\
**Post date:** [February 1, 2021, 1:02pm UTC](https://forum.rasa.com/t/purpose-of-tokenizer-when-using-languagemodelfeaturizer/39677/2 "2021-02-01T13:02:20Z")

</div>

`LanguageModelFeaturizer` doesn’t use the text directly, it uses the texts of tokens that are returned by `Tokenizer`

---

<div class="post-metadata">

**Author:** ![Berken](https://avatars.discourse-cdn.com/v4/letter/b/3be4f8/32.png) [@Berken](https://forum.rasa.com/u/Berken)\
**Post date:** [February 1, 2021, 2:23pm UTC](https://forum.rasa.com/t/purpose-of-tokenizer-when-using-languagemodelfeaturizer/39677/3 "2021-02-01T14:23:05Z")

</div>

Then I don’t understand how it implements the behavior of the LanguageModelTokenizer and how when for example using a BERT model in the LMFeaturizer the BERT model gets the token in the trained format?

---

<div class="post-metadata">

**Author:** ![Ghostvv](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/ghostvv/32/211_2.png) [@Ghostvv](https://forum.rasa.com/u/Ghostvv)\
**Post date:** [February 1, 2021, 4:38pm UTC](https://forum.rasa.com/t/purpose-of-tokenizer-when-using-languagemodelfeaturizer/39677/4 "2021-02-01T16:38:01Z")

</div>

it takes tokens, then feed its text into lm tokenizer, it might or might not additionally tokenize this text, then whatever lm tokenizer is fed into lm featurizer

---

<div class="post-metadata">

**Author:** ![simonm3](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/simonm3/32/17865_2.png) [@simonm3](https://forum.rasa.com/u/simonm3)\
**Post date:** [November 12, 2021, 4:32pm UTC](https://forum.rasa.com/t/purpose-of-tokenizer-when-using-languagemodelfeaturizer/39677/5 "2021-11-12T16:32:57Z")

</div>

why does it require a tokenizer before the languagemodefeaturizer rather than just using the tokenizer built in to the language model? and if it does then what criteria should be used to select the tokenizer?

---

<div class="post-metadata">

**Author:** ![juan57](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/juan57/32/18161_2.png) [@juan57](https://forum.rasa.com/u/juan57)\
**Post date:** [November 19, 2021, 8:02pm UTC](https://forum.rasa.com/t/purpose-of-tokenizer-when-using-languagemodelfeaturizer/39677/6 "2021-11-19T20:02:14Z")

</div>

I’m also interested to know - LanguageModelFeaturizer requires a Tokenizer ([Components](https://rasa.com/docs/rasa/components#tokenizers)). If there is already a LM tokenizer in the LanguageModelFeaturizer, why do we need to specify a separate tokenizer?