How can I use CLIP model(by Open AI) in my rasa project

Rasa Open Source
1

Hi, I am trying to build a clothing recommender system using CLIP transformer model(by Open AI) as CLIP works with multimodal data(text and image) and tells us how close the ‘embedding of a text’ is to the ‘embedding of an image’. Therefore I want to take the responses of my users from my rasa bot and get the closest image recommendation from CLIP model. My question: How I do integrate CLIP model with my rasa bot. Please help me, thank you!

2

I’m not familiar with CLIP but you can create a custom component. I posted about it recently here.

3

Thank you so much Greg, I am looking into it, if I run into any issues ill let you know sir.