# SpacyTokenizer token\_pattern

**URL:** <https://forum.rasa.com/t/spacytokenizer-token-pattern/51952>\
**Category:** Rasa Open Source\
**Created:** [March 14, 2022, 9:29pm UTC](https://forum.rasa.com/t/spacytokenizer-token-pattern/51952 "2022-03-14T21:29:00Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![tomp](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/tomp/32/21142_2.png) [@tomp](https://forum.rasa.com/u/tomp)\
**Post date:** [March 14, 2022, 9:29pm UTC](https://forum.rasa.com/t/spacytokenizer-token-pattern/51952/1 "2022-03-14T21:29:00Z")

</div>

I’m looking for documentation explaining how to add a token\_pattern for SpacyTokenizer but not finding anything but the default “None”:

```auto
pipeline:
- name: "SpacyTokenizer"
  # Regular expression to detect tokens
  "token_pattern": None

```

In particular, I would like to modify SpacyTokenizer so that it will also split numbers from words even when there are no spaces between them, e.g. so I can label the two parts of size and shape in user utterances like `8x12` or `8.5x11`.

Advice?

---

<div class="post-metadata">

**Author:** ![tomp](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/tomp/32/21142_2.png) [@tomp](https://forum.rasa.com/u/tomp)\
**Post date:** [March 15, 2022, 7:51pm UTC](https://forum.rasa.com/t/spacytokenizer-token-pattern/51952/2 "2022-03-15T19:51:44Z")

</div>

Mohd Shukri Hasan found this page with a good example:

> <https://github.com/RasaHQ/rasa/issues/10744>
>
> \### Rasa Open Source version
> 
> 2.8.17
> 
> \### Python version
> 
> 3.8
> 
> \### What …happened?
> 
> Given:
> 1. A config with \`SpacyTokenizer\` with a \`token\_pattern\`, SpacyFeaturizer, and at least one non-spacy token based featurizer (CountVectorsFeaturizer, LexicalSyntacticFeaturizer):
> \`\`\`
> language: en
> pipeline:
> - name: SpacyNLP
> case\_sensitive: false
> model: en\_core\_web\_md
> - name: SpacyTokenizer
> token\_pattern: "(\\\\d+|\\\\D+)"
> - name: SpacyFeaturizer
> - name: CountVectorsFeaturizer
> - name: DIETClassifier
> epochs: 10
> constrain\_similarities: true
> \`\`\`
> 
> 2. Input with multiple digits adjoined to a word e.g.
> \`\`\`
> it's 12euro
> \`\`\`
>  
> At inference time, inputting the example sentence above causes DIETClassifier to complain of mismatched input dimensions (stacktrace below). In the absence of a second featurizer i.e. given only SpacyFeaturizer, this does not happen, implying that the final tokens iterated over by SpacyFeaturizer differs from those iterated over by the non-Spacy featurizer. It could be because SpacyFeaturizer uses the \_spacy\_ tokens \[here\](https://github.com/RasaHQ/rasa/blob/main/rasa/nlu/featurizers/dense\_featurizer/spacy\_featurizer.py#L68) not necessarily the ones Rasa uses.
>  
> The goal is that if the amount and the currency are stuck together without white space like this:
> \`\`\`
> it's 12euro
> \`\`\`
> This would be split into tokens \`it's\`, \`12\`, \`euro\`.
>  
> At inference time, inputting the example sentence above causes DIETClassifier to complain of mismatched input dimensions (stacktrace below). In the absence of a second featurizer i.e. given only SpacyFeaturizer, this does not happen, implying that the final tokens iterated over by SpacyFeaturizer differs from those iterated over by the non-Spacy featurizer. It could be because SpacyFeaturizer uses the \_spacy\_ tokens \[here\](https://github.com/RasaHQ/rasa/blob/main/rasa/nlu/featurizers/dense\_featurizer/spacy\_featurizer.py#L68) not necessarily the ones Rasa uses.
> 
>  
> 
> \### Command / Request
> 
> \`\`\`shell
> rasa train nlu
> rasa shell nlu
> \> it's 12euro
> \`\`\`
> 
> \### Relevant log output
> 
> \`\`\`shell
> Traceback (most recent call last):
> File "\<stdin\>", line 1, in \<module\>
> File "/Users/melinda/.pyenv/versions/rasa2/lib/python3.8/site-packages/rasa/nlu/model.py", line 470, in parse
> component.process(message, \*\*self.context)
> File "/Users/melinda/.pyenv/versions/rasa2/lib/python3.8/site-packages/rasa/nlu/classifiers/diet\_classifier.py", line 988, in process
> out = self.\_predict(message)
> File "/Users/melinda/.pyenv/versions/rasa2/lib/python3.8/site-packages/rasa/nlu/classifiers/diet\_classifier.py", line 904, in \_predict
> return self.model.run\_inference(model\_data)
> File "/Users/melinda/.pyenv/versions/rasa2/lib/python3.8/site-packages/rasa/utils/tensorflow/models.py", line 318, in run\_inference
> \] = self.\_rasa\_predict(batch\_in)
> File "/Users/melinda/.pyenv/versions/rasa2/lib/python3.8/site-packages/rasa/utils/tensorflow/models.py", line 280, in \_rasa\_predict
> outputs = tf\_utils.sync\_to\_numpy\_or\_python\_type(self.predict\_step(batch\_in))
> File "/Users/melinda/.pyenv/versions/rasa2/lib/python3.8/site-packages/rasa/utils/tensorflow/models.py", line 228, in predict\_step
> return self.batch\_predict(batch\_in)
> File "/Users/melinda/.pyenv/versions/rasa2/lib/python3.8/site-packages/rasa/nlu/classifiers/diet\_classifier.py", line 1707, in batch\_predict
> text\_transformed, \_, \_, \_, \_, attention\_weights = self.\_tf\_layers\[
> File "/Users/melinda/.pyenv/versions/rasa2/lib/python3.8/site-packages/keras/engine/base\_layer.py", line 1037, in \_\_call\_\_
> outputs = call\_fn(inputs, \*args, \*\*kwargs)
> File "/Users/melinda/.pyenv/versions/rasa2/lib/python3.8/site-packages/rasa/utils/tensorflow/rasa\_layers.py", line 992, in call
> seq\_sent\_features, mask\_combined\_sequence\_sentence = self.\_tf\_layers\[
> File "/Users/melinda/.pyenv/versions/rasa2/lib/python3.8/site-packages/keras/engine/base\_layer.py", line 1037, in \_\_call\_\_
> outputs = call\_fn(inputs, \*args, \*\*kwargs)
> File "/Users/melinda/.pyenv/versions/rasa2/lib/python3.8/site-packages/rasa/utils/tensorflow/rasa\_layers.py", line 645, in call
> sequence\_features\_combined = self.\_combine\_sequence\_level\_features(
> File "/Users/melinda/.pyenv/versions/rasa2/lib/python3.8/site-packages/rasa/utils/tensorflow/rasa\_layers.py", line 570, in \_combine\_sequence\_level\_features
> sequence\_features\_combined = self.\_tf\_layers\[f"sparse\_dense.{SEQUENCE}"\](
> File "/Users/melinda/.pyenv/versions/rasa2/lib/python3.8/site-packages/keras/engine/base\_layer.py", line 1037, in \_\_call\_\_
> outputs = call\_fn(inputs, \*args, \*\*kwargs)
> File "/Users/melinda/.pyenv/versions/rasa2/lib/python3.8/site-packages/rasa/utils/tensorflow/rasa\_layers.py", line 338, in call
> return tf.concat(dense\_features, axis=-1)
> File "/Users/melinda/.pyenv/versions/rasa2/lib/python3.8/site-packages/tensorflow/python/util/dispatch.py", line 206, in wrapper
> return target(\*args, \*\*kwargs)
> File "/Users/melinda/.pyenv/versions/rasa2/lib/python3.8/site-packages/tensorflow/python/ops/array\_ops.py", line 1769, in concat
> return gen\_array\_ops.concat\_v2(values=values, axis=axis, name=name)
> File "/Users/melinda/.pyenv/versions/rasa2/lib/python3.8/site-packages/tensorflow/python/ops/gen\_array\_ops.py", line 1213, in concat\_v2
> \_ops.raise\_from\_not\_ok\_status(e, name)
> File "/Users/melinda/.pyenv/versions/rasa2/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 6941, in raise\_from\_not\_ok\_status
> six.raise\_from(core.\_status\_to\_exception(e.code, message), None)
> File "\<string\>", line 3, in raise\_from
> tensorflow.python.framework.errors\_impl.InvalidArgumentError: ConcatOp : Dimensions of inputs should match: shape\[0\] = \[1,3,128\] vs. shape\[1\] = \[1,2,300\] \[Op:ConcatV2\] name: concat
> \`\`\`
> 
> 
> \### Defintion of Done
> \- \[\] Facilitate discussion to scope fix
> \- \[\] Create new issue for refinement for fix

```auto
token_pattern: "(\\d+|\\D+)"

```

---

<div class="post-metadata">

**Author:** ![tomp](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/tomp/32/21142_2.png) [@tomp](https://forum.rasa.com/u/tomp)\
**Post date:** [March 18, 2022, 4:14pm UTC](https://forum.rasa.com/t/spacytokenizer-token-pattern/51952/3 "2022-03-18T16:14:05Z")

</div>

The final tokenization regex I settled with is this:

```auto
token_pattern: "(\\d+|[^\\s\\d\\W]+|[^\\w\\s]+)"

```

---

<div class="post-metadata">

**Author:** ![kassem404](https://avatars.discourse-cdn.com/v4/letter/k/71c47a/32.png) [@kassem404](https://forum.rasa.com/u/kassem404)\
**Post date:** [July 27, 2022, 11:11am UTC](https://forum.rasa.com/t/spacytokenizer-token-pattern/51952/4 "2022-07-27T11:11:13Z")

</div>

Hi @tom, did you find any documentation on how to write patterns ?

---

<div class="post-metadata">

**Author:** ![tomp](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/tomp/32/21142_2.png) [@tomp](https://forum.rasa.com/u/tomp)\
**Post date:** [July 27, 2022, 12:18pm UTC](https://forum.rasa.com/t/spacytokenizer-token-pattern/51952/5 "2022-07-27T12:18:13Z")

</div>

@kassem404 I did not find documentation, but the examples above were enough to get me going with the expected syntax. They answered questions like how to escape special characters (use double back slash instead of single back slash) and what character class symbols I could use (lots of options like \d, \W, etc.), and whether I needed to surround my regex with parentheses (for capturing groups) and/or double quotes. Beyond that, it was just a matter of experimenting to find the right regex semantics, and maybe some other existing regex documentation online can help with that. Let me know if you have specific questions.

---

<div class="post-metadata">

**Author:** ![kassem404](https://avatars.discourse-cdn.com/v4/letter/k/71c47a/32.png) [@kassem404](https://forum.rasa.com/u/kassem404)\
**Post date:** [July 27, 2022, 1:20pm UTC](https://forum.rasa.com/t/spacytokenizer-token-pattern/51952/6 "2022-07-27T13:20:30Z")

</div>

Thank you @tomp for replying, that’s really helpful !
