Hi, I have created a ‘sentence-similarity’ model using ‘sentence_transformers’.
I want to use this model in my pipeline as a custom
Featurizer and create embeddings for each sentence, then combine those vectors with vectors produced using
CountVectorsFeaturizer(I read somewhere it’s good to use special words of our domain using
DIETClassifier in my pipeline and I want to use my custom component between them but I read that,
DIETClassifier gets embeddings for each word and doesn’t get embedding of the whole sentence.
I’m stuck and I have no idea what to do. How can I implement my
Featurizer (I am working with rasa 2.3.1)? Should I change my tokenizer so that it tokenizes sentences? And can I concatenate sentence embedding from sentence transformer and word embedding from
CountVectorFeaturizer and feed them to
DIETClassifier? If not, which intent classifier and entity extractor I can use instead?
I appreciate any help