Viable alternative to ConveRTFeaturizer for DietClassifier

Does anybody else have any feedback as to a viable dense featurizer for DietClassifier? I’m interested, not only because ConveRT is no longer publicly available, but also because of this suggestion that ConveRTFeaturizer may be the culprit in preventing successful training on M1.