DeepPavlov(rubert-base-cased) doesn't work after conversion from Pytorch

Rasa Open Source
1

Hello! Can you help me please, I’m trying to use DeepPavlov/rubert-base-cased model in a pipeline. But since the model checkpoint from Huggingface is only available for the pytorch backend I’ve converted it into a Tensorflow by using:

from transformers import TFBertModel
model = TFBertModel.from_pretrained(path, from_pt=True)
model.save_pretrained(path)

But when I’m trying to run Rasa I catch this:

ValueError: Layer #0 (named "bert"), weight <tf.Variable 'tf_bert_model/bert/embeddings/position_embeddings/embeddings:0' shape=(512, 768) dtype=float32, numpy=
array([[-0.02024085,  0.00553767, -0.00605057, ..., -0.02668325,
         0.01404192,  0.02237324],
       [-0.00611183,  0.00401095, -0.01944031, ...,  0.02741791,
        -0.00725622, -0.01654125],
       [ 0.00467447, -0.02064347, -0.00395851, ..., -0.00871686,
        -0.01898053, -0.01849902],
       ...,
       [ 0.00361648, -0.02359357, -0.02002623, ..., -0.01165487,
        -0.02084902, -0.00746869],
       [-0.02776472, -0.03941905, -0.00374909, ..., -0.01134684,
        -0.00692669,  0.0157611 ],
       [-0.02328027,  0.01691188, -0.01107913, ...,  0.0218234 ,
        -0.01890264, -0.02625411]], dtype=float32)> has shape (512, 768), but the saved weight has shape (2, 768).
2

This issue is also mentioned on our github repo. There’s a longer thread there.

1 Like