Hi! Suppose the following diagram represents GraphSchema nodes representing NLU pipeline components that need to be trained constructed according to the “needs” relation: [image] Relations legend: D - domain, DCT - domain for core training, TD - training data, TT - training tracker, and SG - story graph. The motivation for this diagram is to find out which components can be run in parallel using multi-threaded or multi-node clusters (either Dask or Ray clusters). Testing with different number of threads for training revealed that there is no significant improvement in decreasing training time past 4 threads. My assumption for why is this the case is because of the 4 policies that are trained on the right side of the diagram. However, is there any room for improvement for more than 4 threads? Is my assumption why 4 threads are most optimal logical and/or correct? The diagram is constructed using this output of GraphSchema: output_graph_schema.txt (7.6 KB) Note: I have disabled cache to test full training optimization. Btw: What are “Resource” relations in this file above? Hi @koaning ! :slight_smile:

Parallelizing Training Mechanism in Rasa 3.0.x

koaning (Vincent D. Warmerdam) February 16, 2022, 6:25pm 4

That seems interesting. But how large is your training set? I might imagine that this effect is more pronounced on larger datasets.

1 Like

Topic		Replies	Views
Dask and Rasa Rasa Open Source	4	711	February 3, 2022
Rasa train taking lot of time Rasa Open Source	22	4868	July 6, 2021
Multi-thread training Rasa Open Source	2	987	June 2, 2020
How to only train Rasa core on CPU and only Rasa NLU on GPU Rasa Open Source	0	1754	April 23, 2020
DIETClassifier: Slow training (like 10 hours) Rasa Open Source	2	1187	May 14, 2020