Positional Encoding of _CLS_ Vector

Rasa Open Source
1

Hi, i’ve watched this video and i’m confused with hows positional encoding work with CLS vector

Since there’s a CLS token in DIET that summarize the entire utterance, and the example of positional encoding on 08:17 is word index vector. So, how positional enc works with CLS vector ?

I’m writing a thesis on how DIET works in intent and entities prediction, any answer would be appriciated

thanks :slight_smile:

2

Hi @setopaisen That’s a good question. The CLS token is always added to end of the utterance, so the position index for it is basically - length of utterance + 1 (Assuming indices start at 1)

1 Like
3

thanks, i also got my question answered in youtube comment at the same time. Sorry for writing the same question in two different places