# Adding new token patterns to Whitespace Tokenizer

**URL:** https://forum.rasa.com/t/adding-new-token-patterns-to-whitespace-tokenizer/48069
**Category:** Rasa Open Source
**Created:** [September 28, 2021, 2:05pm UTC](https://forum.rasa.com/t/adding-new-token-patterns-to-whitespace-tokenizer/48069 "2021-09-28T14:05:28Z")
**Posts on this page:** 1
**Page:** 1

<div class="post-metadata">

### Author: ![souvikg10](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/souvikg10/32/93_2.png) [@souvikg10](https://forum.rasa.com/u/souvikg10)
#### Post date: [September 28, 2021, 2:05pm UTC](https://forum.rasa.com/t/adding-new-token-patterns-to-whitespace-tokenizer/48069/1 "2021-09-28T14:05:28Z")

</div>

Hi everyone, I would like to know how does one add new token patterns. I added some regex to the whitespace tokenizer for “-” and “/” as i have certain entities like `off-road` where i would like the whitespace tokenizer to split the word so DIET can use the word such as off or road to still manage to match with the entity off-road since people can say the term in many ways like i like off road biking or i like off-road 4x4 for exampe.

how does one handle this. when I add the token pattern to Whitespace Tokenizer, i get such warnings

```auto
Misaligned entity annotation in message 'vegetarian/vegan' with intent 'specify2019722'. Make sure the start and end values of entities ([(0, 16, '167569490')]) in the training data match the token boundaries ([(10, 11, '/')]). Common causes: 
  1) entities include trailing whitespaces or punctuation
  2) the tokenizer gives an unexpected result, due to languages such as Chinese that don't use whitespace for word separation
  More info at https://rasa.com/docs/rasa/training-data-format#nlu-training-data
```