Hi All, I’m reading about the related subject here https://rasa.com/docs/rasa/nlu/language-support/ Is there any sample about this one? Cheers

The id at the top indicates the languages setting on a pipeline leven and in general I think it is supposed to be the same as the lang id in the bytepair settings. I don’t know though if the two letter abbreviations in BytePair are the same what Rasa uses. I don’t know if there’s an international standard for so it’s good to manually check the BytePair website.

thanks. crystal clear. confirm after seeing the documentation about this regarding the country code.

so the next step should be changing the data/nlu.md into Indonesian language dataset? Then run rasa train again? Am i in the right direction?

just seen your reply at github @koaning posting that here. so everyone can see the end to end if they need. thanks a lot. marking this thread solved. [image]

Hi! Thank you for this clarification. I am currently working on making a pipeline for Estonian. I chose the first option and kept the suggested pipeline ( here ): ‘’’ # Configuration for Rasa NLU. # Components language: et_model pipeline: - name: SpacyNLP - name: SpacyTokenizer - name: SpacyFeaturizer - name: RegexFeaturizer - name: LexicalSyntacticFeaturizer - name: CountVectorsFeaturizer - name: CountVectorsFeaturizer analyzer: "char_wb" min_ngram: 1 max_ngram: 4 - name: DIETClassifier epochs: 100 - name: EntitySynonymMapper - name: ResponseSelector ‘’’ I understand now that the CountVectorsFeaturizer is written twice on purpose. But going through the pipeline another question arose. In LexicalSyntacticFeaturizer one of the features it generates is pos Take the Part-of-Speech tag of the token (``SpacyTokenizer`` required). I can see that the only benefit from Fasttext wrapped into Spacy is the word vectors. Loading Fasttext model into Spacy doesn’t give me lemmatization nor POS-es. [image] [image] Might this influence the performance somehow? I mean if Spacy doesn’t give POS-es, then I expect the LexicalSyntacticFeaturizer just gives NaN for every POS value…

The LexicalSyntacticFeaturizer adds things like “does this word start with a capital letter”. This is very different from the POS features that spaCy generates. The spaCy featurizer only adds the word vector features to my knowledge. It does not add POS information. I’d argue that it’s plausible though that POS features could make entity detection easier down the line. That is why there’s an open ticket on the Rasa NLU examples repository . Just to be clear, the LexicalSyntacticFeaturizer is not related to spaCy in our implementation.

Thank you for your quick answer, @koaning ! In that case the docs are a bit wrong? Why else is there written that SpacyTokenizer (I am not talking about spaCy featurizer) is needed for POS values. When I look at the code in the lexical_syntactic_featurizer.py it imports SpacyTokenizer: line 8 from rasa.nlu.tokenizers.spacy_tokenizer import POS_TAG_KEY and it also takes POS tags from the token: line 65 "pos": lambda token: token.data.get(POS_TAG_KEY) line 66 if POS_TAG_KEY in token.data line 67 else None, line 68 "pos2": lambda token: token.data.get(POS_TAG_KEY)[:2] line 69 if "pos" in token.data line 70 else None, In the spacy_tokenizer.py in line 40 it adds tag_ (which are the detailed part-of-speech tag according to spaCy docs) to the token. Therefore it seems to me that the LexicalSyntacticFeaturizer is related to spaCy (and gives None to the pos and pos2 features otherwise) and if I want to get the maximum out of Rasa I have to consider the fact that implementing Fasttext via spaCy lacks of linguistic data (gives only wordvectors) and I should give POS-es and lemmas via other resources. What do you think?

@Lindafr I wasn’t aware of the pos item in the docs. Interesting. I want to double check this now. Will report back in a few minutes with an extensive answer. Odds are that you’re totally correcting me on something now! (Well done!)

Yep. I totally learned something today. Thanks @Lindafr ! So here’s what I’ve been able to confirm: yes you can add spaCy POS features using the LexicalSyntacticFeaturizer. This was unknown to me before but I can confirm that indeed it adds features. My previous answer was wrong. Over on the rasa nlu examples project I’ve made a component called rasa_nlu_examples.meta.Printer. This component allows you to print some extra information from the component pipeline. Here’s two examples of pipelines with that component. Example 1: No POS language: en pipeline: - name: SpacyNLP model: "en_core_web_sm" - name: SpacyTokenizer - name: rasa_nlu_examples.meta.Printer alias: printer before - name: LexicalSyntacticFeaturizer "features": [ ["low", "title", "upper"], ["BOS", "EOS", "low", "upper", "title", "digit"], ["low", "title", "upper"], ] - name: rasa_nlu_examples.meta.Printer alias: printer after - name: CountVectorsFeaturizer analyzer: char_wb min_ngram: 1 max_ngram: 4 - name: DIETClassifier epochs: 1 policies: - name: MemoizationPolicy - name: KerasPolicy - name: MappingPolicy Example 2: With POS language: en pipeline: - name: SpacyNLP model: "en_core_web_sm" - name: SpacyTokenizer - name: rasa_nlu_examples.meta.Printer alias: printer before - name: LexicalSyntacticFeaturizer "features": [ ["low", "title", "upper", "pos"], ["BOS", "EOS", "low", "upper", "title", "digit", "pos"], ["low", "title", "upper", "pos"], ] - name: rasa_nlu_examples.meta.Printer alias: printer after - name: CountVectorsFeaturizer analyzer: char_wb min_ngram: 1 max_ngram: 4 - name: DIETClassifier epochs: 1 policies: - name: MemoizationPolicy - name: KerasPolicy - name: MappingPolicy Results When you train these pipelines you can see that indeed it adds features to the pipeline. Results without POS printer before text : yo intent: {'name': None, 'confidence': 0.0} entities: [] text_spacy_doc: yo tokens: ['yo', '__CLS__'] printer after text : yo intent: {'name': None, 'confidence': 0.0} entities: [] text_spacy_doc: yo tokens: ['yo', '__CLS__'] text_sparse_features: <2x18 sparse matrix of type '<class 'numpy.float64'>' with 12 stored elements in COOrdinate format> Results with POS printer before text : yo intent: {'name': None, 'confidence': 0.0} entities: [] text_spacy_doc: yo tokens: ['yo', '__CLS__'] printer after text : yo intent: {'name': None, 'confidence': 0.0} entities: [] text_spacy_doc: yo tokens: ['yo', '__CLS__'] text_sparse_features: <2x122 sparse matrix of type '<class 'numpy.float64'>' with 14 stored elements in COOrdinate format> You can see that the lexical feature with pos added really do add features. The sparse feature matrix is now large in shape but we’ve only added 1 non-zero component for each token (yo and __CLS__, the __CLS__ token represents the entire sentence). This implies that the POS features are added as sparse features to the pipeline.

The question now remains what is happening with FastText and spaCy for your use case. My pipeline was English and I was not using FastText vectors inside of spaCy. I’m not 100% sure how spaCy deals with the FastText vectors with regards to the POS tagging. Could you try running the same pipeline using the FastText component found in rasa-nlu-examples while also using the base spaCy model that does not have FastText added to it? You can use the printer inside of the examples project to also get a glimpse of what features get added during what phase of the pipeline.

Another option that’s worth mentioning, if you feel like FastText is perhaps just a bit too heavy. We also support BytePair embeddings that are a lot more lightweight. There’s a lot of options to pick from as shown here .

How to train Rasa for other language

Rasa Open Source

Lindafr (Lindafr) August 25, 2020, 4:55am 33

I think so too.

Topic		Replies	Views
Using fasttext pretrained word emmbeding for other language Rasa Open Source	3	1045	August 18, 2020
spaCy pretrained models break chatbot NLU capacities Rasa Open Source	6	844	October 16, 2019
Custom Featurizer for finetuned BERT features based on SpaCy Rasa Open Source	19	4763	February 12, 2020
SpacyNLP and supervised_embeddings in the same pipeline Rasa Open Source	0	462	May 21, 2019
Support for Language Models inside Rasa Release Announcements community , rasa	25	13145	November 25, 2021

How to train Rasa for other language

Related topics