# Training rasa custom keras policy using rasa Agent

**URL:** <https://forum.rasa.com/t/training-rasa-custom-keras-policy-using-rasa-agent/21839>\
**Category:** Rasa Open Source\
**Created:** [November 28, 2019, 2:20pm UTC](https://forum.rasa.com/t/training-rasa-custom-keras-policy-using-rasa-agent/21839 "2019-11-28T14:20:29Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![sarankarthik](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@sarankarthik](https://forum.rasa.com/u/sarankarthik)\
**Post date:** [November 28, 2019, 2:20pm UTC](https://forum.rasa.com/t/training-rasa-custom-keras-policy-using-rasa-agent/21839/1 "2019-11-28T14:20:30Z")

</div>

I have developed my custom keras policy using model functional API of keras. As rasa ‘keras policy’ has developed for Sequential model of keras, I have customized every functions and its arguments in my custom keras policy.

When I have imported my custom keras policy and tried to train using the command ‘rasa train core’, it throws me an error.

```
> async def train_core(
> domain_file: Text = "domain.yml",
> model_directory: Text = "models",
> model_name: Text = "current",
> training_data_file: Text = "data/sample_stories_1.md",
> ):
> agent = Agent(
> domain_file,
> policies=[
> MemoizationPolicy(max_history=3),
> MappingPolicy(),
> Custom_KerasPolicy(batch_size=64, epochs=100, validation_split=0.2),
> ],
> )

```

I get the following error,

```
TypeError: Style() takes no arguments

```

Could I get help to solve this issue?

---

<div class="post-metadata">

**Author:** ![akelad](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/akelad/32/69_2.png) [@akelad](https://forum.rasa.com/u/akelad)\
**Post date:** [December 2, 2019, 1:11pm UTC](https://forum.rasa.com/t/training-rasa-custom-keras-policy-using-rasa-agent/21839/2 "2019-12-02T13:11:06Z")

</div>

@sarankarthik could you share the full stack trace? And also the code of your custom keras policy, as well as the configuration you used in the config file?

---

<div class="post-metadata">

**Author:** ![sarankarthik](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@sarankarthik](https://forum.rasa.com/u/sarankarthik)\
**Post date:** [December 2, 2019, 2:00pm UTC](https://forum.rasa.com/t/training-rasa-custom-keras-policy-using-rasa-agent/21839/3 "2019-12-02T14:00:47Z")

</div>

@akelad Thanks for the Response! The above-mentioned Problem is due to a dependency prompt-toolkit, which I have installed later. But now, I get a new error as follows,

> Traceback (most recent call last): File “c:\ica\_trial\rasa\rasa\core\registry.py”, line 20, in policy\_from\_module\_path return class\_from\_module\_path(module\_path, lookup\_path=“rasa.core.policies”) File “c:\ica\_trial\rasa\rasa\utils\common.py”, line 174, in class\_from\_module\_path m = importlib.import\_module(module\_name) File “c:\apps\sa2446\lib\importlib\__init_\_.py”, line 127, in import\_module return \_bootstrap.\_gcd\_import(name[level:], package, level) File “”, line 1006, in \_gcd\_import File “”, line 983, in \_find\_and\_load File “”, line 967, in \_find\_and\_load\_unlocked File “”, line 677, in \_load\_unlocked File “”, line 728, in exec\_module File “”, line 219, in \_call\_with\_frames\_removed File “C:\ICA\restaurantbot\custom\_keras\_policy.py”, line 47, in from Lstm import CUSTOM\_LSTM File “C:\ICA\restaurantbot\ProgLstm.py”, line 16, in from keras.engine.base\_layer import Layer, disable\_tracking, InputSpec ImportError: cannot import name ‘disable\_tracking’ from ‘keras.engine.base\_layer’ (c:\apps\sa2446\lib\site-packages\keras\engine\base\_layer.py)
> 
> During handling of the above exception, another exception occurred:
> 
> Traceback (most recent call last): File “c:\ica\_trial\rasa\rasa\core\policies\ensemble.py”, line 316, in from\_dict constr\_func = registry.policy\_from\_module\_path(policy\_name) File “c:\ica\_trial\rasa\rasa\core\registry.py”, line 22, in policy\_from\_module\_path raise ImportError(“Cannot retrieve policy from path ‘{}’”.format(module\_path)) ImportError: Cannot retrieve policy from path ‘custom\_keras\_policy.Custom\_KerasPolicy’
> 
> During handling of the above exception, another exception occurred:
> 
> Traceback (most recent call last): File “C:\Apps\sa2446\Scripts\rasa-script.py”, line 11, in load\_entry\_point(‘rasa’, ‘console\_scripts’, ‘rasa’)() File “c:\ica\_trial\rasa\rasa\__main_\_.py”, line 76, in main cmdline\_arguments.func(cmdline\_arguments) File “c:\ica\_trial\rasa\rasa\cli\train.py”, line 111, in train\_core kwargs=kwargs, File “c:\ica\_trial\rasa\rasa\train.py”, line 252, in train\_core kwargs=kwargs, File “c:\apps\sa2446\lib\asyncio\base\_events.py”, line 579, in run\_until\_complete return future.result() File “c:\ica\_trial\rasa\rasa\train.py”, line 308, in train\_core\_async kwargs=kwargs, File “c:\ica\_trial\rasa\rasa\train.py”, line 340, in \_train\_core\_with\_validated\_data kwargs=kwargs, File “c:\ica\_trial\rasa\rasa\core\train.py”, line 42, in train policies = config.load(policy\_config) File “c:\ica\_trial\rasa\rasa\core\config.py”, line 28, in load return PolicyEnsemble.from\_dict(config\_data) File “c:\ica\_trial\rasa\rasa\core\policies\ensemble.py”, line 324, in from\_dict “”.format(policy\_name) rasa.core.policies.ensemble.InvalidPolicyConfig: Module for policy ‘custom\_keras\_policy.Custom\_KerasPolicy’ could not be loaded. Please make sure the name is a valid policy.

This is my config file,

> language: “en”
> 
> pipeline:
> 
> - name: “WhitespaceTokenizer”
> 
> - name: “RegexFeaturizer”
> 
> - name: “CountVectorsFeaturizer”
> 
> - name: “EmbeddingIntentClassifier”
> 
> - name: “CRFEntityExtractor”
> 
> - name: “EntitySynonymMapper”
> 
> policies:
> 
> - name: “custom\_keras\_policy.Custom\_KerasPolicy” batch\_size: 100 epochs: 100 validation\_split: 0.2
> - name: MemoizationPolicy
> - name: MappingPolicy

My custom model\_architecture has shown below,

```
def model_architecture(self, data_path) -> keras.models.Models:        

     def train(self, data_path):

            #Get encoder inputs, decoder inputs and decoder targets from each splitted data to train in progressive encoder
            en_in_data1, en_ti = EncoderData.encoder_data(self, data_path, column)
            de_in_data1, de_out_data1, de_ti = DecoderData.decoder_data(self, data_path, column)
            
            _, _,num_encoder_tokens, _ = EncoderData.data_prep(self, data_path, column)
            _, _,num_decoder_tokens, _, _ = DecoderData.data_prep(self, data_path, column)

            encoder_inputs = Input(shape=(None, num_encoder_tokens))
            encoder = LSTM(latent_dim, return_sequences = True, return_state=True)
            encoder1 = LSTM(latent_dim, return_state=True)
            
            en_out, e_h, e_c = encoder(encoder_inputs)
            e_s = [e_h, e_c]

            encoder_outputs, state_h, state_c = encoder1(en_out, initial_state=e_s)

            encoder_states = [state_h, state_c]
            
            logger.info(f"encoder_input_shape:{encoder_inputs}")
            logger.info(f"encoder output shape:{encoder_outputs}")

            
            logger.info(f"-----------------------------------------------------------------")
         
            decoder_inputs = Input(shape=(None, num_decoder_tokens))

            decoder_lstm = LSTM(latent_dim, return_sequences=True, return_state=True)
            decoder_lstm1 = LSTM(latent_dim, return_sequences=True, return_state=True)
            decoder_outputs1, d_h, d_c = decoder_lstm(decoder_inputs,
                                                 initial_state=encoder_states)
            
            d_s = [d_h, d_c]
            decoder_outputs, de_h, de_c = decoder_lstm1(decoder_outputs1, initial_state=d_s)

            decoder_dense = Dense(num_decoder_tokens, activation='softmax')

            decoder_outputs = decoder_dense(decoder_outputs)
            
           
            
                    
            #Define Model input and output shapes for initial column 
            model1 = Model([encoder_inputs, decoder_inputs], decoder_outputs)

            model1.compile(optimizer='rmsprop', loss='categorical_crossentropy',
                            metrics=['accuracy'])

            print(model1.summary())

            return model1

      model1 = train(self, data_path)

```

---

<div class="post-metadata">

**Author:** ![sarankarthik](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@sarankarthik](https://forum.rasa.com/u/sarankarthik)\
**Post date:** [December 3, 2019, 10:17am UTC](https://forum.rasa.com/t/training-rasa-custom-keras-policy-using-rasa-agent/21839/4 "2019-12-03T10:17:29Z")

</div>

@akelad could you tell me how to integrate my custom keras policy into rasa?

It seems that Rasa agent starts to train the network only for the listed policies offered by Rasa.([rasa/agent.py at master · RasaHQ/rasa · GitHub](https://github.com/RasaHQ/rasa/blob/master/rasa/core/agent.py), [rasa/ensemble.py at master · RasaHQ/rasa · GitHub](https://github.com/RasaHQ/rasa/blob/master/rasa/core/policies/ensemble.py))

Please not the function ‘train’ in both the scripts. Could you help me in this situation?

---

<div class="post-metadata">

**Author:** ![akelad](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/akelad/32/69_2.png) [@akelad](https://forum.rasa.com/u/akelad)\
**Post date:** [December 6, 2019, 10:44am UTC](https://forum.rasa.com/t/training-rasa-custom-keras-policy-using-rasa-agent/21839/5 "2019-12-06T10:44:56Z")

</div>

@sarankarthik you just need to add it to your `config.yml` file and then it will also train the custom policy

---

<div class="post-metadata">

**Author:** ![sarankarthik](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@sarankarthik](https://forum.rasa.com/u/sarankarthik)\
**Post date:** [December 6, 2019, 3:25pm UTC](https://forum.rasa.com/t/training-rasa-custom-keras-policy-using-rasa-agent/21839/6 "2019-12-06T15:25:02Z")

</div>

@akelad, Thanks for the information. I have added it in my config.yml file and I am able to train the model. But when I chat, it throws me the following error,

```
    2019-12-06 15:14:05 INFO root - Starting Rasa server on http://localhost:5005
Using TensorFlow backend.
Bot loaded. Type a message and press enter (use '/stop' to exit): 
Your input -> hello                                                                                                                                                                                              
2019-12-06 15:14:22 ERROR asyncio - Task exception was never retrieved
future: <Task finished coro=<configure_app.<locals>.run_cmdline_io() done, defined at C:\Users\SA2446\AppData\Roaming\Python\Python37\site-packages\rasa\core\run.py:115> exception=TimeoutError()>
Traceback (most recent call last):
  File "C:\Users\SA2446\AppData\Roaming\Python\Python37\site-packages\rasa\core\run.py", line 119, in run_cmdline_io
    server_url=constants.DEFAULT_SERVER_FORMAT.format(port)
  File "C:\Users\SA2446\AppData\Roaming\Python\Python37\site-packages\rasa\core\channels\console.py", line 137, in record_messages
    async for response in bot_responses:
  File "c:\apps\sa2446\lib\site-packages\async_generator\_impl.py", line 366, in step
    return await ANextIter(self._it, start_fn, *args)
  File "c:\apps\sa2446\lib\site-packages\async_generator\_impl.py", line 205, in throw
    return self._invoke(self._it.throw, type, value, traceback)
  File "c:\apps\sa2446\lib\site-packages\async_generator\_impl.py", line 209, in _invoke
    result = fn(*args)
  File "C:\Users\SA2446\AppData\Roaming\Python\Python37\site-packages\rasa\core\channels\console.py", line 102, in send_message_receive_stream
    async for line in resp.content:
  File "c:\apps\sa2446\lib\site-packages\aiohttp\streams.py", line 40, in __anext__
    rv = await self.read_func()
  File "c:\apps\sa2446\lib\site-packages\aiohttp\streams.py", line 329, in readline
    await self._wait('readline')
  File "c:\apps\sa2446\lib\site-packages\aiohttp\streams.py", line 297, in _wait
    await waiter
  File "c:\apps\sa2446\lib\site-packages\aiohttp\helpers.py", line 585, in __exit__
    raise asyncio.TimeoutError from None
concurrent.futures._base.TimeoutError
2019-12-06 15:14:22 ERROR asyncio - Task exception was never retrieved
future: <Task finished coro=<RestInput.on_message_wrapper() done, defined at C:\Users\SA2446\AppData\Roaming\Python\Python37\site-packages\rasa\core\channels\channel.py:376> exception=TypeError("'>' not supported between instances of 'NoneType' and 'int'")>
Traceback (most recent call last):
  File "C:\Users\SA2446\AppData\Roaming\Python\Python37\site-packages\rasa\core\channels\channel.py", line 387, in on_message_wrapper
    await on_new_message(message)
  File "C:\Users\SA2446\AppData\Roaming\Python\Python37\site-packages\rasa\core\channels\channel.py", line 65, in handler
    await app.agent.handle_message(*args, **kwargs)
  File "C:\Users\SA2446\AppData\Roaming\Python\Python37\site-packages\rasa\core\agent.py", line 488, in handle_message
    return await processor.handle_message(message)
  File "C:\Users\SA2446\AppData\Roaming\Python\Python37\site-packages\rasa\core\processor.py", line 90, in handle_message
    await self._predict_and_execute_next_action(message, tracker)
  File "C:\Users\SA2446\AppData\Roaming\Python\Python37\site-packages\rasa\core\processor.py", line 353, in _predict_and_execute_next_action
    action, policy, confidence = self.predict_next_action(tracker)
  File "C:\Users\SA2446\AppData\Roaming\Python\Python37\site-packages\rasa\core\processor.py", line 180, in predict_next_action
    action_confidences, policy = self._get_next_action_probabilities(tracker)
  File "C:\Users\SA2446\AppData\Roaming\Python\Python37\site-packages\rasa\core\processor.py", line 565, in _get_next_action_probabilities
    tracker, self.domain
  File "C:\Users\SA2446\AppData\Roaming\Python\Python37\site-packages\rasa\core\policies\ensemble.py", line 353, in probabilities_using_best_policy
    if (confidence, p.priority) > (max_confidence, best_policy_priority):
TypeError: '>' not supported between instances of 'NoneType' and 'int'

```

I have also modified the Prediction\_action\_probabilities() method according to my requirements, as I am using seq2seq model. I have acquired the current state of dialogue using rasa dialogue state tracker and I ffed it as inputs to get the states values of encoder, which I then feed it to decoder model to predict the output as follows,

```
states_value = encoder_model.predict(input_seq) [equivalent to 'X = self.featurizer.create_X([tracker], domain)']

```

I feed this states value to decoder model,

`output_tokens, h, c = decoder_model.predict([target_seq] + states_value) [equivalent to 'y_pred = self.model.predict(X, batch_size=1)']`

I use the one hot encoded information of 3D vector of

`np.zeros((len(texts), max_seq_length, num_of_features), dtype='float32')`

Could I get help to solve this issue?

---

<div class="post-metadata">

**Author:** ![akelad](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/akelad/32/69_2.png) [@akelad](https://forum.rasa.com/u/akelad)\
**Post date:** [December 11, 2019, 8:36am UTC](https://forum.rasa.com/t/training-rasa-custom-keras-policy-using-rasa-agent/21839/7 "2019-12-11T08:36:12Z")

</div>

I mean it seems like your policy doesn’t have a priority from that error message? Did you make sure to define the priority property?

---

<div class="post-metadata">

**Author:** ![sarankarthik](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@sarankarthik](https://forum.rasa.com/u/sarankarthik)\
**Post date:** [December 12, 2019, 4:26pm UTC](https://forum.rasa.com/t/training-rasa-custom-keras-policy-using-rasa-agent/21839/8 "2019-12-12T16:26:09Z")

</div>

Thanks for the information. It works properly now.
