Trying to escape unicode characters and not language characters

Rasa Open Source
1

Hi, I am trying to export a dataframe with hindi sentences, and I have added following line to escape emojis in sentences or some illegal characters.

intent_df = intent_df.applymap(lambda x: x.encode('unicode_escape').
                                   decode('utf-8') if isinstance(x, str) else x)

but this also removes hindi characters. And I am sure this same thing will happen with other languages too. So, can anyone help me out on how to just escape emojis and some unwanted symbols?

2

Does the data 100% have to go into Excel? We talked a bit about this on slack and I’m wondering if this might be a Windows/Excel thing as opposed to a unicode thing.

3

Yeah I am exporting and importing data to and fro yml and excel since LSP’s don’t accept yml an. So, I want to make sure that data is properly exported and imported in the format that Rasa accepts

4

If you cast it to base csv files and you inspect the text, does the issue persist there as well?

Same for when you drop .(encode('unicode_escape')?

5

I did tried to export to csv but nlu and domain has commas and tabs so TSV is also not possible