Until a few days ago I worked with the version 0.13.4 of Rasa Core and had no problems in training. The configuration of my Policy was the following:
policies:
- name: KerasPolicy
epochs: 90
batch_size: 2048
rnn_size: 100
featurizer:
- name: MaxHistoryTrackerFeaturizer
max_history: 6
state_featurizer:
- name: BinarySingleStateFeaturizer
- name: MemoizationPolicy
max_history: 6
- name: FallbackPolicy
nlu_threshold: 0.6
core_threshold: 0.3
At the time of training, this is my log:
(rasa1) PS C:\Byte\rasa1\Byte\bytebot-fcr-mig\bytebot-fcr-core> rasa train core --config .\policy.yml -d .\domain.yml --debug
2019-06-25 12:33:41 DEBUG rasa.skill - Selected skills:
Training Core model...
2019-06-25 12:33:43 DEBUG rasa.core.training.generator - Generated trackers will be deduplicated based on their unique last 6 states.
2019-06-25 12:33:43 DEBUG rasa.core.training.generator - Number of augmentation rounds is 3
2019-06-25 12:33:43 DEBUG rasa.core.training.generator - Starting data generation round 0 ... (with 1 trackers)
Processed Story Blocks: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 360/360 [00:00<00:00, 683.37it/s, # trackers=1]
2019-06-25 12:33:43 DEBUG rasa.core.training.generator - Finished phase (353 training samples found).
2019-06-25 12:33:43 DEBUG rasa.core.training.generator - Data generation rounds finished.
2019-06-25 12:33:43 DEBUG rasa.core.training.generator - Found 0 unused checkpoints
2019-06-25 12:33:43 DEBUG rasa.core.training.generator - Starting augmentation round 0 ... (with 50 trackers)
Processed Story Blocks: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 360/360 [00:15<00:00, 21.07it/s, # trackers=27]
2019-06-25 12:33:58 DEBUG rasa.core.training.generator - Finished phase (11601 training samples found).
2019-06-25 12:33:59 DEBUG rasa.core.training.generator - Starting augmentation round 1 ... (with 50 trackers)
Processed Story Blocks: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 360/360 [00:19<00:00, 18.73it/s, # trackers=44]
2019-06-25 12:34:18 DEBUG rasa.core.training.generator - Finished phase (24777 training samples found).
2019-06-25 12:34:18 DEBUG rasa.core.training.generator - Starting augmentation round 2 ... (with 50 trackers)
Processed Story Blocks: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 360/360 [00:19<00:00, 18.49it/s, # trackers=42]
2019-06-25 12:34:37 DEBUG rasa.core.training.generator - Finished phase (37790 training samples found).
2019-06-25 12:34:37 DEBUG rasa.core.training.generator - Found 37790 training trackers.
2019-06-25 12:34:37 DEBUG rasa.core.training.generator - Subsampled to 500 augmented training trackers.
2019-06-25 12:34:37 DEBUG rasa.core.training.generator - There are 353 original trackers.
2019-06-25 12:34:38 DEBUG rasa.core.agent - Agent trainer got kwargs: {'dump_stories': False}
2019-06-25 12:34:38 DEBUG rasa.core.featurizers - Creating states and action examples from collected trackers (by MaxHistoryTrackerFeaturizer(BinarySingleStateFeaturizer))...
Processed trackers: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 853/853 [00:03<00:00, 233.28it/s, # actions=6650]
2019-06-25 12:34:42 DEBUG rasa.core.featurizers - Created 6650 action examples.
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
masking (Masking) (None, 6, 451) 0
_________________________________________________________________
lstm (LSTM) (None, 100) 220800
_________________________________________________________________
dense (Dense) (None, 186) 18786
_________________________________________________________________
activation (Activation) (None, 186) 0
=================================================================
Total params: 239,586
Trainable params: 239,586
Non-trainable params: 0
_________________________________________________________________
2019-06-25 12:34:42 INFO rasa.core.policies.keras_policy - Fitting model with 6650 total samples and a validation split of 0.1
2019-06-25 12:34:42 DEBUG rasa.core.policies.policy - Parameters ignored by `model.fit(...)`: {}
Epoch 1/90
- 2s - loss: 5.1224 - acc: 0.1150
Epoch 2/90
- 1s - loss: 4.4750 - acc: 0.3254
Epoch 3/90
- 1s - loss: 3.6577 - acc: 0.3280
Epoch 4/90
- 1s - loss: 3.4156 - acc: 0.3280
Epoch 5/90
- 1s - loss: 3.3060 - acc: 0.3280
Epoch 6/90
- 1s - loss: 3.2438 - acc: 0.3280
Epoch 7/90
- 1s - loss: 3.2002 - acc: 0.3280
Epoch 8/90
- 1s - loss: 3.1624 - acc: 0.3280
Epoch 9/90
- 1s - loss: 3.1265 - acc: 0.3280
Epoch 10/90
Traceback (most recent call last):
...
Why do you subsample my training trackers to 500?
Going deeper into the source rasa_core/generator.py at master · RasaHQ/rasa_core · GitHub, I realized that commenting from line 372 to line 386 made the training went correctly, I did not do the sub-sample and the predictions were correct.
if self.config.augmentation_factor > 0:
augmented_trackers, original_trackers = [], []
for t in finished_trackers:
if t.is_augmented:
augmented_trackers.append(t)
else:
original_trackers.append(t)
augmented_trackers = self._subsample_trackers(
augmented_trackers, self.config.max_number_of_augmented_trackers
)
logger.debug(
"Subsampled to {} augmented training trackers."
"".format(len(augmented_trackers))
)
logger.debug(
"There are {} original trackers.".format(len(original_trackers))
)
finished_trackers = original_trackers + augmented_trackers
Could it be considered a bug? Are there any parameters that you may be missing? Please help.