Through sheer luck I found out that this has to do with tensorflow. The scripts in the container try to import the tensorflow module and that dumps the 132 exit code when the server CPU does not support AVX.
Is there any way to define the tensorflow version that should be used?