# Which one is better for normalization,,,spacy tokenizer or whitespace tokenizer?or can we use them both in pipline

**URL:** <https://forum.rasa.com/t/which-one-is-better-for-normalization-spacy-tokenizer-or-whitespace-tokenizer-or-can-we-use-them-both-in-pipline/36377>\
**Category:** Rasa Open Source\
**Created:** [October 28, 2020, 4:46pm UTC](https://forum.rasa.com/t/which-one-is-better-for-normalization-spacy-tokenizer-or-whitespace-tokenizer-or-can-we-use-them-both-in-pipline/36377 "2020-10-28T16:46:06Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![faiza\_conte](https://avatars.discourse-cdn.com/v4/letter/f/74df32/32.png) [@faiza\_conte](https://forum.rasa.com/u/faiza_conte)\
**Post date:** [October 28, 2020, 4:46pm UTC](https://forum.rasa.com/t/which-one-is-better-for-normalization-spacy-tokenizer-or-whitespace-tokenizer-or-can-we-use-them-both-in-pipline/36377/1 "2020-10-28T16:46:06Z")

</div>

which one is better for normalization,spacy tokenizer or whitespace tokenizer???or can we use them both in pipline

---

<div class="post-metadata">

**Author:** ![rctatman](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/rctatman/32/6734_2.png) [@rctatman](https://forum.rasa.com/u/rctatman)\
**Post date:** [October 29, 2020, 8:38pm UTC](https://forum.rasa.com/t/which-one-is-better-for-normalization-spacy-tokenizer-or-whitespace-tokenizer-or-can-we-use-them-both-in-pipline/36377/2 "2020-10-29T20:38:08Z")

</div>

You only want to use one tokenizer per pipeline. The whitespace tokenizer creates a new token every time it runs into white space. The SpaCy tokenizer adds some additional, language-specific rule checking after the white space splitting ([https://spacy.io/usage/linguistic-features#tokenization](https://spacy.io/usage/linguistic-features#tokenization)). Which tokenizer you use will depend on the rest of your pipeline: different pipeline components rely on different tokenizers.

---

<div class="post-metadata">

**Author:** ![faiza\_conte](https://avatars.discourse-cdn.com/v4/letter/f/74df32/32.png) [@faiza\_conte](https://forum.rasa.com/u/faiza_conte)\
**Post date:** [October 29, 2020, 8:40pm UTC](https://forum.rasa.com/t/which-one-is-better-for-normalization-spacy-tokenizer-or-whitespace-tokenizer-or-can-we-use-them-both-in-pipline/36377/3 "2020-10-29T20:40:12Z")

</div>

@rctatman assume am creating a training data from scratch in english, then which one would you suggest??

---

<div class="post-metadata">

**Author:** ![rctatman](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/rctatman/32/6734_2.png) [@rctatman](https://forum.rasa.com/u/rctatman)\
**Post date:** [October 29, 2020, 8:52pm UTC](https://forum.rasa.com/t/which-one-is-better-for-normalization-spacy-tokenizer-or-whitespace-tokenizer-or-can-we-use-them-both-in-pipline/36377/4 "2020-10-29T20:52:16Z")

</div>

If you’re planning on using SpaCy at all (which I would, it’s a great library), use their tokenizer.

---

<div class="post-metadata">

**Author:** ![faiza\_conte](https://avatars.discourse-cdn.com/v4/letter/f/74df32/32.png) [@faiza\_conte](https://forum.rasa.com/u/faiza_conte)\
**Post date:** [October 29, 2020, 8:53pm UTC](https://forum.rasa.com/t/which-one-is-better-for-normalization-spacy-tokenizer-or-whitespace-tokenizer-or-can-we-use-them-both-in-pipline/36377/5 "2020-10-29T20:53:46Z")

</div>

so you are saying using spacy tokenizer is better than whitespace tokenizer??..well oky thanks i will try it out