Hello, I am trying to implement an entity extraction model using the CRF extractor. To do so, I have made 2 configurations: one with only sparse features and another one that takes dense features (BERT) as well. Here are the configurations : With sparse features only: pipeline: name: WhitespaceTokenizer name: LexicalSyntacticFeaturizer name: CountVectorsFeaturizer analyzer: char_wb min_ngram: 1 max_ngram: 4 name: CRFEntityExtractor With pre-trained embedding: pipeline: name: HFTransformersNLP model_weights: “bert-base-multilingual-uncased” model_name: “bert” name: LanguageModelTokenizer name: LanguageModelFeaturizer name: CRFEntityExtractor The problem is that I have the exact same results as if the dense features are not being considered at all while I can see it training in the train phase. The same happens when I train it on a different dataset. The rasa version that I’m using is 2.5.1.

Hello @liaeh ! From what I understood (I am not sure if that is really the case), but CRF takes into account only sequence features while the languageModelFeaturizer return both sentence and sequence features. According to the documentation, the classifier decides wich kind of features to use. As the CRF is a sequence model, it should use sequence features only but yet as the results shows it fails to do that. What I did, and it seems to be working is that I used the entity extraction of the DIET classifier and put the transformer layer to 0. That way, the configuration is almost the same as the CRFentityExtractor except the forward neural networks between the features and the CRF. As the DIET architecture takes into consideration the language model features, it’s a way to trick the CRF into using them as well. Hope that this answers your problem.

Hi @imene_tar, thanks for the answer! Your workaround is helpful. Still, as you say it seems like the documentation is incorrect: CRFEntityExtractor is supposed to include all dense features of len(tokens), which is is not :confused: I have opened a GitHub issue about this and linked our forum posts - hopefully it helps with a fix :slight_smile:

Looking forward to read their replies.

I think I’ve found the issue. It’s explained in more detail on the GitHub issue here.

If the problem is the tokenizer then why is it working well when using DIET? Did you try your configuration without using DIET at all?

The DIET classifier in my example is set to only predicts intents. It doesn’t predict entities.

Yes, but when using DIET as en entity extactor with the exact same tokenizer, the language model is taken into consideration while it’s not the case when using CRF only. (As I stated previously, I think that DIET knows which features to choose that’s what the CRF is working in your case)

The difference is caused by the WhitespaceTokenizer though. I just ran the same configurations but without DIET and there I also seen two difference confusion matrices.

What I meant was that when using the LanguageModelTokenizer with DIET, the model is perfectly working so I don’t understand why it should cause any problem in CRF. Anyway, I have replaced it with the whitespacetokenizer as you suggested and I still have the exact same result with and without the language model.

That’s … odd. Could you share your full nlu data?

CRF with dense features

Rasa Open Source

imene_tar (Imene Tarakli) June 28, 2021, 6:09am 16

Hi Vincent! Were you able to test your model on my datasat? I am curious to know the nature of the problem.

Topic		Replies	Views
No Difference in Performance when Using or Changing Language Model Featurizers Rasa Open Source	3	1309	January 17, 2022
Difference between CRF_entity_extractor and DIET Tutorials, Resources & Videos entity	0	696	June 17, 2021
Using NER as a Feature for CRFEntityExtractor Rasa Open Source	6	1803	June 28, 2021
ValueError: Sequence dimensions for sparse and dense features don't coincide Rasa Open Source	23	2198	February 11, 2020
How does the CRF for entity extraction work? Best practices? Rasa Open Source	13	2853	June 3, 2022

CRF with dense features

Related topics