Hi there,
I’m trying to use AraBERTv2 in huggingface. When I directly give the model name, rasa can’t find it, so I downloaded the model files and I’m currently using the model from cache. The model name “bert” has this pre-defined shape in it’s weights, which is inconsistent with AraBERT’s shape, which causes the following error when I try to train NLU:
ValueError: Layer #0 (named "bert"), weight <tf.Variable 'tf_bert_model/bert/embeddings/word_embeddings/weight:0' shape=(64000, 768) dtype=float32, numpy=
array([[ 0.01264049, 0.01955802, -0.00695472, ..., 0.02972408,
0.02346177, 0.02438015],
[-0.01304033, 0.02966203, -0.00747581, ..., -0.01203315,
-0.00375808, 0.01051331],
[-0.00813525, -0.00043884, 0.01635659, ..., 0.00520323,
0.03812745, -0.02500577],
...,
[ 0.00136639, -0.01157611, -0.00390291, ..., -0.00182731,
-0.00883772, 0.00762005],
[-0.00144503, 0.00418852, -0.00485313, ..., 0.00467847,
0.00246883, 0.03807047],
[-0.01269101, -0.00266637, 0.0104845 , ..., 0.01443424,
0.00744028, 0.0108838 ]], dtype=float32)> has shape (64000, 768), but the saved weight has shape (28996, 768).
I came across this in the huggingface forum but since I'm not loading the model just like I do from huggingface for any other NLP task (with .from_pretrained() which seems the solution) I don't know if there's a solution for this in Rasa.
Here’s the link to that issue: load tf2 roberta model meet error #2598 In the meanwhile my plan is to try bert-base-arabic instead of this model.
Any help is appreciated.