File Based Import Encoding Problem

[HichamElk]

Hello,

We have a problem when we import users from a file based import.
Users with a first or last name with an accent are incorrectly displayed on Saviynt.
Example: José / Chloé

At first we had special characters instead of accents like this 

So we converted our file to UTF-8 and reimported it into Saviynt, but now a black diamond with a question mark appears instead of the accent.

Could you please help us correct this problem?

[rushikeshvartak]

• Use database functions to remove accent characters 

[HichamElk]

Hi, 

Could you detail how to do it please ? 

Use database functions from the Saviynt application ? What type of request to do ? 

Because I have 1800 users and I can't check manually what users are badly encoded 

Thanks

[SumathiSomala]

@HichamElk Refer below doc

https://docs.saviyntcloud.com/bundle/EIC-Admin-v24x/page/Content/Chapter02-Identity-Repository/Datab...

https://forums.saviynt.com/t5/identity-governance/use-of-custom-db-function-to-replace-the-ascend-ch...

[HichamElk]

Ok, 

So we tried using this : 

{
    "ADDITIONALTABLES": 
    {
        "USERS": "SELECT username, firstname, lastname, displayName FROM USERS"
    },
	
    "COMPUTEDCOLUMNS": 
    [
        "username",
        "firstname",
        "lastname",
        "displayName"
    ],
	
    "PREPROCESSQUERIES": 
    [
        "UPDATE NEWUSERDATA SET firstname = FN_EIC_REPLACE(NEWUSERDATA.firstname, null)",
        "UPDATE NEWUSERDATA SET lastname = FN_EIC_REPLACE(NEWUSERDATA.lastname, null)",
        "UPDATE NEWUSERDATA SET displayName = FN_EIC_REPLACE(NEWUSERDATA.displayName, null)"
    ]
}

We entered this json in the part User Pre-Processor Config JSON in the page User File Based Import. Also our CSV file that contains those columns :

username,firstname,lastname,DisplayName

Selected First Row as Heading YES and then clicked on Upload and Preview
The preview still shows the special caracters. 

Are we doing it good or is there any mistake ? 

Thanks

[rushikeshvartak]

Did you added preprocessor JSON during import? 

[HichamElk]

We did it this way, 

We completed every field then clicked on upload and preview. Does it answer the question ? 

We haven't Imported as we don't see changes in the preview

[rushikeshvartak]

Share logs

[HichamElk]

Could you help us finding the logs of the preview import 

We dont' find anything related to it

Do you have any keyword to search or anything ? 

Thanks

[rushikeshvartak]

Once you upload files share logs for last 3 minutes 

[HichamElk]

You can find the export of the last 3 minutes here 

[rushikeshvartak]

Try with below JSON

{"ADDITIONALTABLES":{"USERS":"SELECT username, firstname, lastname, displayName FROM USERS"},"COMPUTEDCOLUMNS":["username","firstname","lastname","displayName"],"PREPROCESSQUERIES":["UPDATE NEWUSERDATA SET firstname = FN_EIC_REPLACE(NEWUSERDATA.firstname, null)","UPDATE NEWUSERDATA SET lastname = FN_EIC_REPLACE(NEWUSERDATA.lastname, null)","UPDATE NEWUSERDATA SET displayName = FN_EIC_REPLACE(NEWUSERDATA.displayName, null)"]}

[HichamElk]

Hi, 

This JSON doesn't work too, Saviynt seems to find no difference between old import and new one. Check the screenshot below 

Thanks 

[HichamElk]

Is it possible trying on your side the function and if it works, to show us what you have done to make it 

Thanks 

[rushikeshvartak]

Are you changing data like firstname/lastname during import ? Please share logs

[HichamElk]

We import the same data, nothing changes. 

Maybe there is something in the global configs ? We also have special characters like this on the attributes of accounts (schema upload)

[rushikeshvartak]

• Change some data in import file and validate.
• Logs does not have any relevant data related to import file

[HichamElk · Accepted solution]

Yes you are right, 

I just imported a new file in UTF8 with the special characters corrected and it worked fine, no function needed. 

Thanks for the help 

[rushikeshvartak]

✅👍Please click the 'Accept As Solution' button on the reply (or replies) that best answered your original question.

[rushikeshvartak]

Please confirm if provided preprocessor worked ? If not provide working solution to help others

[HichamElk]

Hi, 

Preprocessor solution was not needed.

What we have done is : 

• Retrieve the import file containing special characters.
• Copy and paste values into an excel file encoded in UTF8
• The values will then appear unencoded and readable. (UTF8 allows you to view special characters)
• Simply infile the new UTF8 CSV import file into Saviynt.