Training aborts with internal server error "Error: 'NoneType' object is not iterable""

Martinbe1io · September 28, 2019, 3:34pm

hi,

I’m using rasa as docker image in an OpenShift environment.

the problem: training via http api aborts with the following message:

2019-09-28 17:28:23,438]-[pool-13-thread-1] ERROR io.be1.circular.messageprocessing.MessageProcessorRasaNLU - trainthread RASANLU exception=io.be1.circular.messageprocessing.rasaclient.ApiException: Internal Server Error 

Api Debugging shows 
INFORMATION: {"version":"1.3.6","status":"failure","message":"An unexpected error occurred during training. Error: 'NoneType' object is not iterable","reason":"TrainingError","details":{},"help":null,"code":500}

the version:

{"fingerprint":{"config":"99914b932bd37a50b983c5e7c90ae93b","core-config":"99914b932bd37a50b983c5e7c90ae93b","nlu-config":"82591095c3a322532cc63fa712f8d75b","domain":9.5870988696262259E17,"messages":6.9125350545795738E17,"stories":7.5109881865239424E17,"trained_at":1.5696807663729153E9,"version":"1.3.7"},"modelFile":"/tmp/tmp_9l86sxj","numActiveTrainingJobs":0}

initial training (via rasa init) worked well.

created java classes based on the open api spec (thanks for that) started server and try to do my first training (only nlu):

TrainingRequest trainingRequest=new TrainingRequest();
							trainingRequest.setNlu(rasadata);
							trainingRequest.config(rasaconfig);
							trainingRequest.domain(rasadomain);
							
							trainingRequest.saveToDefaultModelDirectory(true);
							trainingRequest.force(false);
							
							
							ObjectMapper m=new ObjectMapper();
							log.info("start training: {}",m.writeValueAsString(trainingRequest));
						
							File f=trainapi.trainModel(trainingRequest);

serialized trainingRequest:

start training:
 {"domain":"intents:\n","config":"#Configuration for Rasa NLU.\r\n#https://rasa.com/docs/rasa/nlu/components/\r\nlanguage: en\r\npipeline: supervised_embeddings\r\n\r\n# Configuration for Rasa Core.\r\n# https://rasa.com/docs/rasa/core/policies/\r\npolicies:\r\n  - name: MemoizationPolicy\r\n  - name: KerasPolicy\r\n  - name: MappingPolicy","nlu":"## intent:goodbye\n- bye\n- goodbye\n- see you around\n- see you later\n- talk to you later\n\n## intent:ask_identity\n- who are you\n- what is your name\n- how should i address you\n- may i know your name\n- are you a bot\n\n## intent:ask_shop_open\n- does the shop open on [monday](weekday)\n- does the shop open on [wednesday](weekday)\n- does the shop open on [friday](weekday)\n\n## intent:ask_eaten\n- what did you have for [breakfast](meal)\n- what did you have for [break fast](meal:breakfast)\n- what did you have for [breakfat](meal:breakfast)\n\n## intent:inform_zipcode\n- my zipcode is [12345](zipcode)\n- my zipcode is [33456](zipcode)\n- my zipcode is [94056](zipcode)\n\n## synonym:breakfast\n- brekfast\n- brokefast\n\n## lookup:weekday\n- monday\n- tuesday\n- wednesday\n- thursday\n- friday\n","stories":null,"force":false,"saveToDefaultModelDirectory":true}

any idea what’s wrong. I already tried to use an older version without any change

Martinbe1io · October 6, 2019, 6:09am

Hi, no answer until now, did i use the wrong Channel? Nobody who can Help me investigate the Problem ?

Martin

akelad · October 25, 2019, 1:01pm

Hi @Martinbe1io welcome to the community and sorry that we dropped this. Not sure if you have resolved this yet, but it would be great if you can post some more details about the issue you’re having. e.g. the full logs of your rasa container

Martinbe1io · November 3, 2019, 10:45am

Unfortunately not, find further information in this post

Container logs (I killed the container before, it restarted so everything is in initial state)

 oc logs rasalatestfull-5-824h8
2019-11-03 10:34:24 INFO     root  - Starting Rasa server on http://localhost:5005

Container Logs after the Training Process

[2019-11-03 10:35:50 +0000] [1] [ERROR] Exception occurred in one of response middleware handlers
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/sanic/app.py", line 958, in handle_request
    request, response
  File "/usr/local/lib/python3.6/site-packages/spf/framework.py", line 579, in _run_response_middleware
    _response = await _response
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/extension.py", line 266, in unapplied_cors_response_middleware
    set_cors_headers(req, resp, context, res_options)
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 251, in set_cors_headers
    headers_to_set = get_cors_headers(options, req.headers, req.method)
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 171, in get_cors_headers
    origins_to_set = get_cors_origins(options, request_headers.get('Origin'))
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 146, in get_cors_origins
    return sorted([o for o in origins if not probably_regex(o)])
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 146, in <listcomp>
    return sorted([o for o in origins if not probably_regex(o)])
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 275, in probably_regex
    return any((c in maybe_regex for c in common_regex_chars))
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 275, in <genexpr>
    return any((c in maybe_regex for c in common_regex_chars))
TypeError: argument of type 'NoneType' is not iterable
[2019-11-03 10:35:50 +0000] [1] [ERROR] Exception occurred in one of response middleware handlers
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/sanic/app.py", line 958, in handle_request
    request, response
  File "/usr/local/lib/python3.6/site-packages/spf/framework.py", line 579, in _run_response_middleware
    _response = await _response
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/extension.py", line 266, in unapplied_cors_response_middleware
    set_cors_headers(req, resp, context, res_options)
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 251, in set_cors_headers
    headers_to_set = get_cors_headers(options, req.headers, req.method)
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 171, in get_cors_headers
    origins_to_set = get_cors_origins(options, request_headers.get('Origin'))
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 146, in get_cors_origins
    return sorted([o for o in origins if not probably_regex(o)])
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 146, in <listcomp>
    return sorted([o for o in origins if not probably_regex(o)])
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 275, in probably_regex
    return any((c in maybe_regex for c in common_regex_chars))
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 275, in <genexpr>
    return any((c in maybe_regex for c in common_regex_chars))
TypeError: argument of type 'NoneType' is not iterable
[2019-11-03 10:35:50 +0000] [1] [ERROR] Exception occurred in one of response middleware handlers
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/sanic/app.py", line 958, in handle_request
    request, response
  File "/usr/local/lib/python3.6/site-packages/spf/framework.py", line 579, in _run_response_middleware
    _response = await _response
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/extension.py", line 266, in unapplied_cors_response_middleware
    set_cors_headers(req, resp, context, res_options)
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 251, in set_cors_headers
    headers_to_set = get_cors_headers(options, req.headers, req.method)
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 171, in get_cors_headers
    origins_to_set = get_cors_origins(options, request_headers.get('Origin'))
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 146, in get_cors_origins
    return sorted([o for o in origins if not probably_regex(o)])
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 146, in <listcomp>
    return sorted([o for o in origins if not probably_regex(o)])
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 275, in probably_regex
    return any((c in maybe_regex for c in common_regex_chars))
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 275, in <genexpr>
    return any((c in maybe_regex for c in common_regex_chars))
TypeError: argument of type 'NoneType' is not iterable
[2019-11-03 10:36:09 +0000] [1] [ERROR] Exception occurred in one of response middleware handlers
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/sanic/app.py", line 958, in handle_request
    request, response
  File "/usr/local/lib/python3.6/site-packages/spf/framework.py", line 579, in _run_response_middleware
    _response = await _response
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/extension.py", line 266, in unapplied_cors_response_middleware
    set_cors_headers(req, resp, context, res_options)
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 251, in set_cors_headers
    headers_to_set = get_cors_headers(options, req.headers, req.method)
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 171, in get_cors_headers
    origins_to_set = get_cors_origins(options, request_headers.get('Origin'))
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 146, in get_cors_origins
    return sorted([o for o in origins if not probably_regex(o)])
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 146, in <listcomp>
    return sorted([o for o in origins if not probably_regex(o)])
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 275, in probably_regex
    return any((c in maybe_regex for c in common_regex_chars))
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 275, in <genexpr>
    return any((c in maybe_regex for c in common_regex_chars))
TypeError: argument of type 'NoneType' is not iterable
[2019-11-03 10:36:09 +0000] [1] [ERROR] Exception occurred in one of response middleware handlers
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/sanic/app.py", line 958, in handle_request
    request, response
  File "/usr/local/lib/python3.6/site-packages/spf/framework.py", line 579, in _run_response_middleware
    _response = await _response
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/extension.py", line 266, in unapplied_cors_response_middleware
    set_cors_headers(req, resp, context, res_options)
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 251, in set_cors_headers
    headers_to_set = get_cors_headers(options, req.headers, req.method)
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 171, in get_cors_headers
    origins_to_set = get_cors_origins(options, request_headers.get('Origin'))
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 146, in get_cors_origins
    return sorted([o for o in origins if not probably_regex(o)])
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 146, in <listcomp>
    return sorted([o for o in origins if not probably_regex(o)])
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 275, in probably_regex
    return any((c in maybe_regex for c in common_regex_chars))
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 275, in <genexpr>
    return any((c in maybe_regex for c in common_regex_chars))
TypeError: argument of type 'NoneType' is not iterable
[2019-11-03 10:36:10 +0000] [1] [ERROR] Exception occurred in one of response middleware handlers
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/sanic/app.py", line 958, in handle_request
    request, response
  File "/usr/local/lib/python3.6/site-packages/spf/framework.py", line 579, in _run_response_middleware
    _response = await _response
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/extension.py", line 266, in unapplied_cors_response_middleware
    set_cors_headers(req, resp, context, res_options)
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 251, in set_cors_headers
    headers_to_set = get_cors_headers(options, req.headers, req.method)
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 171, in get_cors_headers
    origins_to_set = get_cors_origins(options, request_headers.get('Origin'))
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 146, in get_cors_origins
    return sorted([o for o in origins if not probably_regex(o)])
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 146, in <listcomp>
    return sorted([o for o in origins if not probably_regex(o)])
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 275, in probably_regex
    return any((c in maybe_regex for c in common_regex_chars))
  File "/usr/local/lib/python3.6/site-packages/sanic_cors/core.py", line 275, in <genexpr>
    return any((c in maybe_regex for c in common_regex_chars))
TypeError: argument of type 'NoneType' is not iterable

My Application logs of the training process with activated okhttp logging

[2019-11-03 10:36:09,568]-[netty task-4-10] INFO io.be1.circular.messageprocessing.MessageProcessorRasaNLU - train trainingdata=## intent:goodbye

- bye

- goodbye

- see you around

- see you later

- talk to you later

## intent:ask_identity

- who are you

- what is your name

- how should i address you

- may i know your name

- are you a bot

## intent:ask_shop_open

- does the shop open on [monday](weekday)

- does the shop open on [wednesday](weekday)

- does the shop open on [friday](weekday)

## intent:ask_eaten

- what did you have for [breakfast](meal)

- what did you have for [break fast](meal:breakfast)

- what did you have for [breakfat](meal:breakfast)

## intent:inform_zipcode

- my zipcode is [12345](zipcode)

- my zipcode is [33456](zipcode)

- my zipcode is [94056](zipcode)

## synonym:breakfast

- brekfast

- brokefast

## lookup:weekday

- monday

- tuesday

- wednesday

- thursday

- friday

trainreadtimeout=300000 trainwritetimeout=300000

[2019-11-03 10:36:09,569]-[netty task-4-10] INFO io.be1.circular.messageprocessing.MessageProcessorRasaNLU - trainthread result=STARTED

Nov. 03, 2019 10:36:09 VORM. okhttp3.internal.platform.Platform log

INFORMATION: --> GET http://rasalatestfull.circular-dev.svc.cluster.local:5005/version

Nov. 03, 2019 10:36:09 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Accept: application/json

Nov. 03, 2019 10:36:09 VORM. okhttp3.internal.platform.Platform log

INFORMATION: User-Agent: OpenAPI-Generator/1.0.0/java

Nov. 03, 2019 10:36:09 VORM. okhttp3.internal.platform.Platform log

INFORMATION: --> END GET

[2019-11-03 10:36:09,572]-[pool-127-thread-1] INFO io.be1.circular.messageprocessing.MessageProcessorRasaNLU - start training: {"domain":"intents:\n","config":"#Configuration for Rasa NLU.\r\n#https://rasa.com/docs/rasa/nlu/components/\r\nlanguage: en\r\npipeline: supervised_embeddings\r\n\r\n# Configuration for Rasa Core.\r\n# https://rasa.com/docs/rasa/core/policies/\r\npolicies:\r\n - name: MemoizationPolicy\r\n - name: KerasPolicy\r\n - name: MappingPolicy","nlu":"## intent:goodbye\n- bye\n- goodbye\n- see you around\n- see you later\n- talk to you later\n\n## intent:ask_identity\n- who are you\n- what is your name\n- how should i address you\n- may i know your name\n- are you a bot\n\n## intent:ask_shop_open\n- does the shop open on [monday](weekday)\n- does the shop open on [wednesday](weekday)\n- does the shop open on [friday](weekday)\n\n## intent:ask_eaten\n- what did you have for [breakfast](meal)\n- what did you have for [break fast](meal:breakfast)\n- what did you have for [breakfat](meal:breakfast)\n\n## intent:inform_zipcode\n- my zipcode is [12345](zipcode)\n- my zipcode is [33456](zipcode)\n- my zipcode is [94056](zipcode)\n\n## synonym:breakfast\n- brekfast\n- brokefast\n\n## lookup:weekday\n- monday\n- tuesday\n- wednesday\n- thursday\n- friday\n","stories":null,"force":false,"saveToDefaultModelDirectory":true}

Nov. 03, 2019 10:36:09 VORM. okhttp3.internal.platform.Platform log

INFORMATION: --> POST http://rasalatestfull.circular-dev.svc.cluster.local:5005/model/train

Nov. 03, 2019 10:36:09 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Content-Type: application/json; charset=utf-8

Nov. 03, 2019 10:36:09 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Content-Length: 1215

Nov. 03, 2019 10:36:09 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Accept: application/json

Nov. 03, 2019 10:36:09 VORM. okhttp3.internal.platform.Platform log

INFORMATION: User-Agent: OpenAPI-Generator/1.0.0/java

Nov. 03, 2019 10:36:09 VORM. okhttp3.internal.platform.Platform log

INFORMATION:

Nov. 03, 2019 10:36:09 VORM. okhttp3.internal.platform.Platform log

INFORMATION: {"domain":"intents:\n","config":"#Configuration for Rasa NLU.\r\n#https://rasa.com/docs/rasa/nlu/components/\r\nlanguage: en\r\npipeline: supervised_embeddings\r\n\r\n# Configuration for Rasa Core.\r\n# https://rasa.com/docs/rasa/core/policies/\r\npolicies:\r\n - name: MemoizationPolicy\r\n - name: KerasPolicy\r\n - name: MappingPolicy","nlu":"## intent:goodbye\n- bye\n- goodbye\n- see you around\n- see you later\n- talk to you later\n\n## intent:ask_identity\n- who are you\n- what is your name\n- how should i address you\n- may i know your name\n- are you a bot\n\n## intent:ask_shop_open\n- does the shop open on [monday](weekday)\n- does the shop open on [wednesday](weekday)\n- does the shop open on [friday](weekday)\n\n## intent:ask_eaten\n- what did you have for [breakfast](meal)\n- what did you have for [break fast](meal:breakfast)\n- what did you have for [breakfat](meal:breakfast)\n\n## intent:inform_zipcode\n- my zipcode is [12345](zipcode)\n- my zipcode is [33456](zipcode)\n- my zipcode is [94056](zipcode)\n\n## synonym:breakfast\n- brekfast\n- brokefast\n\n## lookup:weekday\n- monday\n- tuesday\n- wednesday\n- thursday\n- friday\n","force":false,"save_to_default_model_directory":true}

Nov. 03, 2019 10:36:09 VORM. okhttp3.internal.platform.Platform log

INFORMATION: --> END POST (1215-byte body)

Nov. 03, 2019 10:36:09 VORM. okhttp3.internal.platform.Platform log

INFORMATION: <-- 200 OK http://rasalatestfull.circular-dev.svc.cluster.local:5005/version (249ms)

Nov. 03, 2019 10:36:09 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Connection: keep-alive

Nov. 03, 2019 10:36:09 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Keep-Alive: 5

Nov. 03, 2019 10:36:09 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Content-Length: 58

Nov. 03, 2019 10:36:09 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Content-Type: application/json

Nov. 03, 2019 10:36:09 VORM. okhttp3.internal.platform.Platform log

INFORMATION:

Nov. 03, 2019 10:36:09 VORM. okhttp3.internal.platform.Platform log

INFORMATION: {"version":"1.3.6","minimum_compatible_version":"1.3.0a2"}

Nov. 03, 2019 10:36:09 VORM. okhttp3.internal.platform.Platform log

INFORMATION: <-- END HTTP (58-byte body)

Nov. 03, 2019 10:36:09 VORM. okhttp3.internal.platform.Platform log

INFORMATION: --> GET http://rasalatestfull.circular-dev.svc.cluster.local:5005/status

Nov. 03, 2019 10:36:09 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Accept: application/json

Nov. 03, 2019 10:36:09 VORM. okhttp3.internal.platform.Platform log

INFORMATION: User-Agent: OpenAPI-Generator/1.0.0/java

Nov. 03, 2019 10:36:09 VORM. okhttp3.internal.platform.Platform log

INFORMATION: --> END GET

Nov. 03, 2019 10:36:09 VORM. okhttp3.internal.platform.Platform log

INFORMATION: <-- 500 Internal Server Error http://rasalatestfull.circular-dev.svc.cluster.local:5005/model/train (423ms)

Nov. 03, 2019 10:36:09 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Connection: keep-alive

Nov. 03, 2019 10:36:09 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Keep-Alive: 5

Nov. 03, 2019 10:36:10 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Content-Length: 198

Nov. 03, 2019 10:36:10 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Content-Type: application/json

Nov. 03, 2019 10:36:10 VORM. okhttp3.internal.platform.Platform log

INFORMATION:

Nov. 03, 2019 10:36:10 VORM. okhttp3.internal.platform.Platform log

INFORMATION: {"version":"1.3.6","status":"failure","message":"An unexpected error occurred during training. Error: 'NoneType' object is not iterable","reason":"TrainingError","details":{},"help":null,"code":500}

Nov. 03, 2019 10:36:10 VORM. okhttp3.internal.platform.Platform log

INFORMATION: <-- END HTTP (198-byte body)

[2019-11-03 10:36:10,002]-[pool-127-thread-1] ERROR io.be1.circular.messageprocessing.MessageProcessorRasaNLU - trainthread RASANLU exception=io.be1.circular.messageprocessing.rasaclient.ApiException: Internal Server Error stack=io.be1.circular.messageprocessing.rasaclient.ApiException: Internal Server Error

at io.be1.circular.my-app//io.be1.circular.messageprocessing.rasaclient.ApiClient.handleResponse(ApiClient.java:966)

at io.be1.circular.my-app//io.be1.circular.messageprocessing.rasaclient.ApiClient.execute(ApiClient.java:882)

at io.be1.circular.my-app//io.be1.circular.messageprocessing.rasaclient.api.ModelApi.trainModelWithHttpInfo(ModelApi.java:829)

at io.be1.circular.my-app//io.be1.circular.messageprocessing.rasaclient.api.ModelApi.trainModel(ModelApi.java:806)

at io.be1.circular.my-app//io.be1.circular.messageprocessing.MessageProcessorRasaNLU.lambda$starttrainthread$0(MessageProcessorRasaNLU.java:273)

at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)

at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)

at java.base/java.lang.Thread.run(Thread.java:844)

[2019-11-03 10:36:10,002]-[pool-127-thread-1] INFO io.be1.circular.messageprocessing.MessageProcessorRasaNLU - trainthread RASANLU stoped thread: pool-127-thread-1 duration=0sec

Nov. 03, 2019 10:36:10 VORM. okhttp3.internal.platform.Platform log

INFORMATION: <-- 200 OK http://rasalatestfull.circular-dev.svc.cluster.local:5005/status (211ms)

Nov. 03, 2019 10:36:10 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Connection: keep-alive

Nov. 03, 2019 10:36:10 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Keep-Alive: 5

Nov. 03, 2019 10:36:10 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Content-Length: 361

Nov. 03, 2019 10:36:10 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Content-Type: application/json

Nov. 03, 2019 10:36:10 VORM. okhttp3.internal.platform.Platform log

INFORMATION:

Nov. 03, 2019 10:36:10 VORM. okhttp3.internal.platform.Platform log

INFORMATION: {"model_file":"\/tmp\/tmps01g3set","fingerprint":{"config":"99914b932bd37a50b983c5e7c90ae93b","core-config":"95c1b632bd1c398582f24965d588acc2","nlu-config":"b7d7e4d209b4a466873bc2625b1b884b","domain":2179095163910361755,"messages":588123012288595556,"stories":1787327009150494885,"trained_at":1569683288.60572052,"version":"1.3.6"},"num_active_training_jobs":0}

Nov. 03, 2019 10:36:10 VORM. okhttp3.internal.platform.Platform log

INFORMATION: <-- END HTTP (361-byte body)

Nov. 03, 2019 10:36:10 VORM. okhttp3.internal.platform.Platform log

INFORMATION: --> GET http://rasalatestfull.circular-dev.svc.cluster.local:5005/version

Nov. 03, 2019 10:36:10 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Accept: application/json

Nov. 03, 2019 10:36:10 VORM. okhttp3.internal.platform.Platform log

INFORMATION: User-Agent: OpenAPI-Generator/1.0.0/java

Nov. 03, 2019 10:36:10 VORM. okhttp3.internal.platform.Platform log

INFORMATION: --> END GET

Nov. 03, 2019 10:36:10 VORM. okhttp3.internal.platform.Platform log

INFORMATION: <-- 200 OK http://rasalatestfull.circular-dev.svc.cluster.local:5005/version (349ms)

Nov. 03, 2019 10:36:10 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Connection: keep-alive

Nov. 03, 2019 10:36:10 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Keep-Alive: 5

Nov. 03, 2019 10:36:10 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Content-Length: 58

Nov. 03, 2019 10:36:10 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Content-Type: application/json

Nov. 03, 2019 10:36:10 VORM. okhttp3.internal.platform.Platform log

INFORMATION:

Nov. 03, 2019 10:36:10 VORM. okhttp3.internal.platform.Platform log

INFORMATION: {"version":"1.3.6","minimum_compatible_version":"1.3.0a2"}

Nov. 03, 2019 10:36:10 VORM. okhttp3.internal.platform.Platform log

INFORMATION: <-- END HTTP (58-byte body)

Please have a look at the “start training” line where I print out the Json serialized training object.

Martin

Martinbe1io · November 6, 2019, 8:24pm

Unfortunately not until now, see my other post. Would be great if you can help until weekend m

akelad · November 7, 2019, 7:41pm

@Martinbe1io can you try adding the --cors "*" flag to the run command for rasa?

Martinbe1io · November 8, 2019, 5:55am

same but with less output on the container console - see bold lines market below from api logging

container starts:

rasalatestfull
Image: rasa/rasa f886533 1.2 GiB
Command: rasa run --enable-api --cors "*"
Ports: 5005/TCP
Mount: rasalatestfull-1 → /app read-write
Mount: default-token-v8wkh → /var/run/secrets/kubernetes.io/serviceaccount read-only

apical and result

trainreadtimeout=300000 trainwritetimeout=300000

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: --> GET http://rasalatestfull.circular-dev.svc.cluster.local:5005/version

[2019-11-08 06:52:46,144]-[netty task-4-2] INFO io.be1.circular.messageprocessing.MessageProcessorRasaNLU - trainthread result=STARTED

[2019-11-08 06:52:46,146]-[pool-16-thread-1] INFO io.be1.circular.messageprocessing.MessageProcessorRasaNLU - start training: {"domain":"intents:\n","config":"#Configuration for Rasa NLU.\r\n#https://rasa.com/docs/rasa/nlu/components/\r\nlanguage: en\r\npipeline: supervised_embeddings\r\n\r\n# Configuration for Rasa Core.\r\n# https://rasa.com/docs/rasa/core/policies/\r\npolicies:\r\n - name: MemoizationPolicy\r\n - name: KerasPolicy\r\n - name: MappingPolicy","nlu":"## intent:goodbye\n- bye\n- goodbye\n- see you around\n- see you later\n- talk to you later\n\n## intent:ask_identity\n- who are you\n- what is your name\n- how should i address you\n- may i know your name\n- are you a bot\n\n## intent:ask_shop_open\n- does the shop open on [monday](weekday)\n- does the shop open on [wednesday](weekday)\n- does the shop open on [friday](weekday)\n\n## intent:ask_eaten\n- what did you have for [breakfast](meal)\n- what did you have for [break fast](meal:breakfast)\n- what did you have for [breakfat](meal:breakfast)\n\n## intent:inform_zipcode\n- my zipcode is [12345](zipcode)\n- my zipcode is [33456](zipcode)\n- my zipcode is [94056](zipcode)\n\n## synonym:breakfast\n- brekfast\n- brokefast\n\n## lookup:weekday\n- monday\n- tuesday\n- wednesday\n- thursday\n- friday\n","stories":null,"force":false,"saveToDefaultModelDirectory":true}

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Accept: application/json

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: User-Agent: OpenAPI-Generator/1.0.0/java

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: --> END GET

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: --> POST http://rasalatestfull.circular-dev.svc.cluster.local:5005/model/train

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Content-Type: application/json; charset=utf-8

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Content-Length: 1215

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Accept: application/json

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: User-Agent: OpenAPI-Generator/1.0.0/java

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION:

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: {"domain":"intents:\n","config":"#Configuration for Rasa NLU.\r\n#https://rasa.com/docs/rasa/nlu/components/\r\nlanguage: en\r\npipeline: supervised_embeddings\r\n\r\n# Configuration for Rasa Core.\r\n# https://rasa.com/docs/rasa/core/policies/\r\npolicies:\r\n - name: MemoizationPolicy\r\n - name: KerasPolicy\r\n - name: MappingPolicy","nlu":"## intent:goodbye\n- bye\n- goodbye\n- see you around\n- see you later\n- talk to you later\n\n## intent:ask_identity\n- who are you\n- what is your name\n- how should i address you\n- may i know your name\n- are you a bot\n\n## intent:ask_shop_open\n- does the shop open on [monday](weekday)\n- does the shop open on [wednesday](weekday)\n- does the shop open on [friday](weekday)\n\n## intent:ask_eaten\n- what did you have for [breakfast](meal)\n- what did you have for [break fast](meal:breakfast)\n- what did you have for [breakfat](meal:breakfast)\n\n## intent:inform_zipcode\n- my zipcode is [12345](zipcode)\n- my zipcode is [33456](zipcode)\n- my zipcode is [94056](zipcode)\n\n## synonym:breakfast\n- brekfast\n- brokefast\n\n## lookup:weekday\n- monday\n- tuesday\n- wednesday\n- thursday\n- friday\n","force":false,"save_to_default_model_directory":true}

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: --> END POST (1215-byte body)

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: <-- 200 OK http://rasalatestfull.circular-dev.svc.cluster.local:5005/version (235ms)

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: <-- 500 Internal Server Error http://rasalatestfull.circular-dev.svc.cluster.local:5005/model/train (232ms)

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Connection: keep-alive

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Connection: keep-alive

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Keep-Alive: 5

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Keep-Alive: 5

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Content-Length: 198

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Content-Type: application/json

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Content-Length: 58

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION:

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Content-Type: application/json

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: {“version”:“1.3.6”,“status”:“failure”,“message”:“An unexpected error occurred during training. Error: ‘NoneType’ object is not iterable”,“reason”:“TrainingError”,“details”:{},“help”:null,“code”:500}

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION:

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: <-- END HTTP (198-byte body)

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: {"version":"1.3.6","minimum_compatible_version":"1.3.0a2"}

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: <-- END HTTP (58-byte body)

[2019-11-08 06:52:46,388]-[pool-16-thread-1] ERROR io.be1.circular.messageprocessing.MessageProcessorRasaNLU - trainthread RASANLU exception=io.be1.circular.messageprocessing.rasaclient.ApiException: Internal Server Error stack=io.be1.circular.messageprocessing.rasaclient.ApiException: Internal Server Error

at io.be1.circular.my-app//io.be1.circular.messageprocessing.rasaclient.ApiClient.handleResponse(ApiClient.java:966)

at io.be1.circular.my-app//io.be1.circular.messageprocessing.rasaclient.ApiClient.execute(ApiClient.java:882)

at io.be1.circular.my-app//io.be1.circular.messageprocessing.rasaclient.api.ModelApi.trainModelWithHttpInfo(ModelApi.java:829)

at io.be1.circular.my-app//io.be1.circular.messageprocessing.rasaclient.api.ModelApi.trainModel(ModelApi.java:806)

at io.be1.circular.my-app//io.be1.circular.messageprocessing.MessageProcessorRasaNLU.lambda$starttrainthread$0(MessageProcessorRasaNLU.java:273)

at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)

at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)

at java.base/java.lang.Thread.run(Thread.java:844)

[2019-11-08 06:52:46,388]-[pool-16-thread-1] INFO io.be1.circular.messageprocessing.MessageProcessorRasaNLU - trainthread RASANLU stoped thread: pool-16-thread-1 duration=0sec

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: --> GET http://rasalatestfull.circular-dev.svc.cluster.local:5005/status

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Accept: application/json

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: User-Agent: OpenAPI-Generator/1.0.0/java

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: --> END GET

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: <-- 200 OK http://rasalatestfull.circular-dev.svc.cluster.local:5005/status (143ms)

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Connection: keep-alive

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Keep-Alive: 5

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Content-Length: 361

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: Content-Type: application/json

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION:

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: {"model_file":"\/tmp\/tmpiefpm4xk","fingerprint":{"config":"99914b932bd37a50b983c5e7c90ae93b","core-config":"95c1b632bd1c398582f24965d588acc2","nlu-config":"b7d7e4d209b4a466873bc2625b1b884b","domain":2179095163910361755,"messages":588123012288595556,"stories":1787327009150494885,"trained_at":1569683288.60572052,"version":"1.3.6"},"num_active_training_jobs":0}

Nov. 08, 2019 5:52:46 VORM. okhttp3.internal.platform.Platform log

INFORMATION: <-- END HTTP (361-byte body)

akelad · November 11, 2019, 10:27am

@Tobias_Wochinger any ideas about this? I think we’ve fixed it in newer versions, but the workaround of --cors "*" should work

Martinbe1io · November 17, 2019, 5:50am

Any news what i can do ?

akelad · November 27, 2019, 12:35pm

@Martinbe1io sorry for getting back to you so late - do you still have this issue? Have you tried using a newer version of Rasa?

Martinbe1io · November 30, 2019, 1:28pm

upgraded to latest.full docker image: ae946c8 with same behavior ;-(

Martinbe1io · November 30, 2019, 6:38pm

Managed it on my own, I’m rather unhappy about the cryptic error messages. the problem was the following part in the training request:

“domain”:“intents:\n” (see the traininrequest JSON above)

if there are no intents in the file this error occurs…

Martin

akelad · December 2, 2019, 1:07pm

hmm, could you submit an issue for this on our Github repo? Then we can look into making the error message less cryptic

Topic		Replies	Views
TypeError: 'NoneType' object is not iterable on training Rasa Open Source	2	957	September 11, 2020
Error while online training - TypeError: 'Response' object is not iterable Rasa Open Source	8	2975	October 3, 2018
TypeError: 'NoneType' object is not iterable Rasa Open Source	2	1507	October 9, 2019
500 error when training a model with version 0.23.0 [Deprecated] Rasa X Community Edition	6	2414	December 11, 2019
Rasa train gives a cryptic TypeError Rasa Open Source	3	659	December 11, 2020

Training aborts with internal server error "Error: 'NoneType' object is not iterable""

Related topics