Hello, I’m trying to use custom embeddings or pretrained embeddings with ner_crf for entity extraction, but can’t find a proper tutorial for it yet. I have tried using fasttext with spacy but I don’t think the embeddings are being used by ner_crf(as I’m not using POS tags feature with ner_crf). If I had to feed custom embeddings as an additional feature to ner_crf, how should I do it with/without spacy(Spacy doesn’t have support for Bert embeddings yet)?

Hello, @Juste Yes, I did and have followed this GitHub issue to use FastText with rasa. But going through the code I see that Spacy is only used when pos_features are used with the ner_crf(and pos features aren’t included in default params of ner_crf). I tried using the pos_features(in the config file) as well but did not find any improvements. So my questions are: assuming the embeddings I want are available in spacy and I create a package following the instructions, does rasa only use pos_tags as features for ner_crf? (Going through the code, I couldn’t find where the actual word embeddings are directly used. So maybe I’m missing something.) Let’s say Spacy doesn’t have embeddings I want to use(or I have custom embeddings for every word or subword). How do I pass them as features to ner_crf?

Hi @gowtham1997, did you manage to figure this out? I’m having the same problem and I also went through the code, without being able to figure out where and how the actual word embeddings are used by ner_crf.

I have a work-in-progress PR to discuss how to pass these kinds of features to ner_crf/CRFEntityExtractor. This would then pair with another new component like SpacyVectorEntityFeaturizer that would pass the features along. That way if any new components for custom NER came along, it would be reusable. custom NER features CRF RasaHQ:master ← jamesmf:custom_ner_features opened 04:35AM - 07 Aug 19 UTC (UTC) [jamesmf] jamesmf +213 -19 This is a work-in-progress implementation that allows custom entity-featurizers …to pass features along to `CRFEntityExtractor`. This would enable new featurizers that passed word/token vectors along to `CRFEntityExtractor`. Also following this `ner_features` pattern would make it easy to develop other components that could also do custom NER. The edit to the example is just to show that, while the `DummyNERFeaturizer` provides nothing but noise to the `CRFEntityExtractor`, it seems to be working. Just looking for feedback before I add tests or do much else. Addresses https://forum.rasa.com/t/feeding-custom-pretrained-embeddings-for-ner-crf/5406 **Proposed changes**: - Make `CRFEntityExtractor` look for `ner_features` on the message - Have `CRFEntityExtractor` convert the features from an array to `python-crfsuite` style dicts **Status (please check what you already did)**: - [X] made PR ready for code review - [x] added some tests for the functionality - [x] updated the documentation - [x] updated the changelog - [X] reformat files using `black` (please check [Readme](https://github.com/RasaHQ/rasa_nlu#code-style) for instructions)

Hi, is your solution now answer to the original question and best way to go if I have the same problem?

It’s not merged into master yet, but yes. If you have a SpacyFeaturizer whose component config specifies ner_feature_vectors: true, it should work. It will make token.vector available to CRFEntityExtractor for every token in the spacy.Doc

@jamesmf @souvikg10 - I read your tutorial too Just to check, this is how we did things: Downloaded the language I desire from here - Word vectors for 157 languages · fastText Started this code and saved the model - https://github.com/souvikg10/spacy-fasttext/blob/master/load_fastText.py Loaded the model with rasa, made our domain,stories, etc. files for that language and trained on it Started chatting and used config like this: language: "br" pipeline: "pretrained_embeddings_spacy" policies: - epochs: 45 max_history: 10 name: KerasPolicy - max_history: 10 name: AugmentedMemoizationPolicy - name: "FallbackPolicy" nlu_threshold: 0.2 core_threshold: 0.1 fallback_action_name: "action_default_fallback" This worked and we started to chat with the bot but I know we do not have any entity extraction so it is pretty lame, can you maybe help me with what should I do next, should I wait for your solution to come to master branch or are there things I need to do beforehand? Thanks.

You just need to replace the pipeline: "pretrained_embeddings_spacy" with individual components. You can pick and choose, but if you want mostly spacy based components, you could do: pipeline: - name: 'SpacyNLP' model: 'your_model_name_here' - name: 'SpacyTokenizer' - name: 'SpacyFeaturizer' ner_feature_vectors: true # this is the part that's new functionality - name: 'CRFEntityExtractor' - name: 'EmbeddingIntentClassifier' This would use spacy to tokenize, would create features for intents using the .vector attribute on the Doc, and would pass the .vector attribute on each token to the CRFEntityExtractor as (some of) the features to do custom entity extraction.

Hi @Juste , Can Rasa just use idea of this paper(" Massively Multilingual Sentence Embeddings for Zero-ShotCross-Lingual Transfer and Beyond ") in place of pretrained embeddings in DIET Classifier archiecture. Instead of using GLoVe and BERT or ConveRT. Early reply will be admired…

Feeding Custom/Pretrained embeddings for ner_crf

Rasa Open Source

Juste (Juste) March 7, 2019, 12:46pm 2

Hey @gowtham1997. Did you have a chance to look into the blogpost written by our contributor Souvik? Build a Rasa NLU Chatbot with spaCy and FastText – strai – Medium

Topic		Replies	Views
Using fasttext pretrained word emmbeding for other language Rasa Open Source	3	1047	August 18, 2020
Crf_entity_extractor with ner_features Rasa Open Source	1	639	February 25, 2020
Using NER as a Feature for CRFEntityExtractor Rasa Open Source	6	1803	June 28, 2021
Pass custom features to CRFEntityExtractor Rasa Open Source	4	1276	August 12, 2019
Word embeddings and RASA NLU Rasa Open Source	5	2099	August 10, 2020

Feeding Custom/Pretrained embeddings for ner_crf

Related topics