Tokens None from previous component(RASA custom component)

Getting Started with Rasa Tutorials, Resources & Videos
1

I created a custom component. I want to use tokens from the previous component, but I am getting "tokens none ". Please help me to solve this. Below is my “custom_component.py” code

from rasa.nlu.components import Component
from rasa.shared.nlu.training_data.message import Message
from fuzzywuzzy import process

class FuzzyExtractor(Component):
    name = "FuzzyExtractor"
    provides = ["entities"]
    requires = ["tokens"]
    defaults = {}
    language_list = ["en"]
    threshold = 90

    def __init__(self, component_config=None, *args):
        super(FuzzyExtractor, self).__init__(component_config)

    def train(self, training_data, cfg, **kwargs):
        pass

    def process(self, message:Message, **kwargs):

        print('message', message)

        entities = list(message.get('entities'))
        print('entities', entities)

       
        # lookup_data = ["shower cap", "body lotion", "moisturizer", "comb", "nail cutter", "newspapers", "hair drier", "magazines", "mouth wash", "iron box", "hair serum", "hair straightener", "comforters", "slippers", "blanket", "tooth brush", "room freshner", "face wash", "hand sanitiser", "body wash", "pillow covers", "Medical kit", "Gloves", "socks", "bath gloves", "ash tray", "face mask"]
        # print('lookup_data', lookup_data)

        tokens = message.get("tokens")
        print('tokens', tokens)

              # for token in tokens:
        #     print('token', token)
        #     # STOP_WORDS is just a dictionary of stop words from NLTK
        #     # if token.text not in STOP_WORDS:
            

        #     fuzzy_results = process.extract(
        #                                 'blankets', 
        #                                 lookup_data, 
        #                                 processor=lambda a: a['value']
        #                                     if isinstance(a, dict) else a, 
        #                                 limit=10)

        #     print('fuzzy_results', fuzzy_results)

        #     for result, confidence in fuzzy_results:
        #         if confidence >= self.threshold:
        #             entities.append({
        #                 "start": token.offset,
        #                 "end": token.end,
        #                 "value": token.text,
        #                 "fuzzy_value": result["value"],
        #                 "confidence": confidence,
        #                 "entity": result["entity"]
        #             })
        # print('entities', entities)

        message.set("entities", entities, add_to_output=True)

config.yml code

# Configuration for Rasa NLU.
# https://rasa.com/docs/rasa/nlu/components/
language: en

pipeline:
# # No configuration for the NLU pipeline was provided. The following default pipeline was used to train your model.
# # If you'd like to customize it, uncomment and adjust the pipeline.
# # See https://rasa.com/docs/rasa/tuning-your-model for more information.
  - name: WhitespaceTokenizer
  - name: custom_component.FuzzyExtractor
  - name: RegexFeaturizer
    "case_sensitive": false
  - name: LexicalSyntacticFeaturizer
  - name: CountVectorsFeaturizer
  - name: CountVectorsFeaturizer
    analyzer: char_wb
    min_ngram: 1
    max_ngram: 4
  - name: DIETClassifier
    entity_recognition: False
    epochs: 100
  - name: "CRFEntityExtractor"
  - name: EntitySynonymMapper
  - name: ResponseSelector
    epochs: 100
  - name: DucklingEntityExtractor
    url: http://localhost:8000
    dimensions:
    - time
    - number
    timezone: "Asia/Mumbai"
  - name: FallbackClassifier
    threshold: 0.5

# Configuration for Rasa Core.
# https://rasa.com/docs/rasa/core/policies/
policies:
# # No configuration for policies was provided. The following default policies were used to train your model.
# # If you'd like to customize them, uncomment and adjust the policies.
# # See https://rasa.com/docs/rasa/policies for more information.
#   - name: MemoizationPolicy
#   - name: TEDPolicy
#     max_history: 5
#     epochs: 100
#   - name: RulePolicy

Rasa shell --debug logs

Next message:
hi
message <rasa.shared.nlu.training_data.message.Message object at 0x0000019811BE4DC8>
entities []
tokens None
1 Like
2

Try to replace it with tokens = [t.text for t in message.get(“tokens”)]

3

Have you ever solved this? I occured the same problem when I was trying to create my custom component.

4

in rasa 2.6, I access tokens like this

tokens = [t.text for t in ex.get("text_tokens")]

where is ex is message object btw.

this will give you a list of tokens for every example

2 Likes