The data model is English Words Only–that may be a source of issue.
I also was going through the paper last night on ConveRT (I can never spell it right) and I noticed it uses sub word tokens but not character level embeddings so i bet spelling has a huge impact on performance. I was going to experiment with that myself this am.