DIET Architecture - Individual Token Pathway

Rasa Open Source Feedback on Rasa Open Source
1

Hi,

I was reading about DIET Architecture and I got below doubt. The DIET Architecture has 2 individual token pathway

  1. Pre-trained Embedding
  2. Sparse Features + FFNN

But I don’t see any hyperparameters in DIETClassifier in the pipeline [Related to these 2]. Please help me to understand if these are controlled by the Pipeline components that come before DIETClassifier?

Pipeline
Pipeline1302×656 35.6 KB

2

You can see the DIET hyperparameters here

1 Like
3

@stephens Hi, Thank you for the reply! I checked the parameters in Rasa Docs as well. But still I couldn’t get below point.

DIET
DIET1515×618 118 KB

Please help.

4

All these on the left are INSIDE the DIET Classifier.

The LanguageModelFeaturizer and CountVectorsFeaturizer do not appear on that diagram.

An alternative link to the one proposed by Greg is this one. There you can find DIET’s components/hyperparameters. Here’s an example usage:

- name: DIETClassifier
  epochs: 141
  model_confidence: linear_norm
  loss_type: cross_entropy
  constrain_similarities: true
  number_of_transformer_layers: 2
  number_of_attention_heads: 4
  batch_size:
  - 64
  - 128
  evaluate_on_number_of_examples: 200
  evaluate_every_number_of_epochs: 5
  regularization_constant: 0.002
  random_seed: 1
  tensorboard_log_directory: ./.tensorboard/DIET
  tensorboard_log_level: epoch
  checkpoint_model: True
1 Like
5

@ChrisRahme Sorry for late reply. I still couldn’t understand this. The DIET Architecture has 2 token pathways…1) Sparse Feature 2) Pretrained Embedding …As per the youtube learning series. We can decide if we need to include both or one…Even in pretrained we can decide which language model to use. But I do not see these as hyperparameters in DIET Classifier :frowning: …So I thought maybe the previous pipeline decide

Could you please let me know how to control or change the language model in token pathway for DIET?