Unicode in `rasa shell`

Rasa Open Source
1

Hello. I run rasa shell and my bot answer to me in json and unicode. Like this:

Custom json:
{
  "text": "\u041d\u0435\u0438\u0437\u0432\u0435\u0441\u0442\u043d\u0430\u044f \u043e\u0448\u0438\u0431\u043a\u0430",
  "isEndDialog": "True"
}

But the rasa interactive command is working normaly. I checked parameters of output in file rasa\nlu\run.py . And it’s OK print(json_to_string(result)) I trained my model with rasa train. Why my bot print in unicode and json in rasa shell?

Rasa 2.8.21, Rasa SDK 2.8.4, Python 3.8.12, Linux Manjaro

2

Hello @faupotron are you fetching data using .csv ? can you share the code file please, regarding fetching data?

3

@nik202, no, I run my bot for check how it answer and build dialoges. I do it in terminal, command rasa shell I see the message: Your input -> and type the message Bot answer to me in json and unicode instead of text

4

@faupotron you are not using custom action? can you activate your env and type rasa --version please.

5

@nik202 I’ve understood. Of course, I use custom actions. It’s in .yml files (not .csv). rasa --version:

Rasa Version      :         2.8.21
Minimum Compatible Version: 2.8.9
Rasa SDK Version  :         2.8.4
Rasa X Version    :         None
Python Version    :         3.8.12
Operating System  :         Linux-5.4.181-1-MANJARO-x86_64-with-glibc2.17
6

@faupotron I am not doubting your understanding, it’s my way of providing the solution, as this error is regarding UTF-8, etc regarding that why I asked. @faupotron did you create the env for installation Rasa?

Thanks. Nik

7

@nik202 Yes, I use conda env and install rasa into conda env

8

@faupotron share some files, please?

9

what kind of file do you mean?

10

rasa\nlu\run.py

What is this file and what code you are using in this?

11

@nik202 rasa\nlu\run.py is file from rasa module. It’s located ~/miniconda3/envs/rasa2/lib/python3.8/site-packages/rasa/nlu/ I just finded the same issue and checked this file. But I don’t change it

I drop it to goggle disk: https://drive.google.com/file/d/1UcVLtrAxTkhhtv3y9RMQWPhylLaDc1Ws/view?usp=sharing

I think that’s not the point

12

You only shared the below piece of information but not with the complete path, for me it can be any custom code.

Anyways you are using Miniconda3 and I never used that, I always use anaconda3 for creating a virtual environment, and it’s giving me great results without any issue.

13

Hi @faupotron You are looking in the wrong place. the rasa/nlu/run is used in rasa shell nlu, and it is using json.dumps(..., ensure_ascii=False,...) by default.

The key here is ensure_ascii to be false, because otherwise all unicode, non-ascii, characters will be encoded as shown in your example.

rasa shell uses the commandline channel, which uses rasa.core.channels.console._print_bot_output(message,....) which uses json.dumps(message["custom"], indent=2), color=color in the last but one line. There is no ensure_ascii=False parameter here, so it defaults to True which means all non ASCII-7 characters will be escaped.

So you can’t do anything about it unless you change that line of Rasa code.

Sorry to say
BR
Raoul

14

Not entirely true, you can of course write your own interactive channel and use that.

15

@raoulvm, hello! Thank you for answer. You gave me the right direction. I found a function that is responsible for the output in the rasa shell in the file rasa/core/channels/console.py I added the strings:

str_to_console = message.get("custom")["text"]
rasa.shared.utils.cli.print_color(str_to_console.encode().decode('UTF-8'), color=color)

And now my bot answer in console correctly, in right encode