I’m looking for documentation explaining how to add a token_pattern for SpacyTokenizer but not finding anything but the default “None”: pipeline: - name: "SpacyTokenizer" # Regular expression to detect tokens "token_pattern": None In particular, I would like to modify SpacyTokenizer so that it will also split numbers from words even when there are no spaces between them, e.g. so I can label the two parts of size and shape in user utterances like 8x12 or 8.5x11. Advice?

The final tokenization regex I settled with is this: token_pattern: "(\\d+|[^\\s\\d\\W]+|[^\\w\\s]+)"

SpacyTokenizer token_pattern

kassem404 (yassine) July 27, 2022, 1:20pm 6

Thank you @tomp for replying, that’s really helpful !

Topic		Replies	Views
How to config token_pattern for CountVectorsFeaturizer in config.yml? Rasa Open Source	0	1091	July 22, 2019
RASA issue with SpacyTokenizer Rasa Open Source	3	1082	February 13, 2022
ValueError: Sequence dimensions for sparse and dense features don't coincide Rasa Open Source	23	2018	February 11, 2020
WhitespaceTokenizer ignored from pipeline Rasa Open Source	0	357	April 17, 2022
Adding new token patterns to Whitespace Tokenizer Rasa Open Source	0	569	September 28, 2021