and more in a single search tool across platforms. Read the announcement here. |
05/06/2023 03:54 AM - edited 05/06/2023 04:22 AM
Hi,
We have configured SAP SuccessFactors and are trying to import all the users from the target. There are around 50000 records in the target but only first 1000 records are reconciled whenever we run the job.
We have used the below pagination logic:
"pagination": {
"nextUrl": {
"nextUrlPath": "${response?.completeResponseMap.d.__next==null?null:response?.completeResponseMap.d.__next}"
}
}
We could see the below error in the logs:
2023-05-06T14:36:22+05:30-ecm-worker-2023-05-06T09:06:22.211118314Z stdout F 2023-05-06 09:06:22,211 [quartzScheduler_Worker-3] ERROR rest.RestProvisioningService - Inside token Expiry Exception block. connectionParamMap.refreshTryCount : 0
I am attaching the ConnectionJSON and the ImportUsersJSON.
Please guide as to what might be wrong. Have we configured the JSONs correctly?.
Thanks,
Atul Singh
05/08/2023 09:39 AM
This does not look like the correct log file as I do not see the job thread in it. Can you share the correct one.
Please update ConfigJSON param in the REST connection with value {"showLogs":true} before running the job if not added already.
05/09/2023 12:11 AM - edited 05/09/2023 12:45 AM
Hi Sahil,
I am attaching the logs below. Config showLogs is set to true.
Thanks
05/08/2023 10:45 AM
Please use below.
"pagination":{
"nextUrl":{
"nextUrlPath":"${response?.completeResponseMap?.d?.__next == null ?null : response?.completeResponseMap?.d?.__next}"
}
}
05/09/2023 12:21 AM - edited 05/09/2023 01:02 AM
We have tried with the exact same Paginations logic as well. By including the not null operator(.?) with each key. The user count is not increasing. Only the first 1000 users are getting imported. Even though in the logs we can see that the subsequent '__next' url's are being called..also the user data is getting processed. But somehow the user count is not increasing and users are not appearing in Identity Repo/Users.
Also, when we use pagination, the job last run is coming as blank(-). The job is showing as Success even though not all users are imported.
The job end date is also not appearing.
As we can see in the above picture that the next pages are being called. Also, I have verified that Saviynt is calling the second last page also. I verified it from looking at the 'skipToken' as the skipToken is unique and constant for each page. Also, I could see the data of users getting processed. These users are not in the first 1000 users and even though Saviynt is processing these users data through API calls, these users are not getting imported.
I felt it could be a timeout issue so I configured this in the ConfigJSON:
{
"showLogs": true,
"connectionTimeoutConfig": {
"connectionTimeout": 1000,
"readTimeout": 7200,
"writeTimeout": 1000,
"retryWait": 100,
"retryCount": 10
}
}
But this didn't work as well.
I feel the job is somehow not terminating properly as the job end date is blank. But according to the pagination logic we are using:
"pagination":{
"nextUrl":{
"nextUrlPath":"${response?.completeResponseMap?.d?.__next == null ?null : response?.completeResponseMap?.d?.__next}"
}
}
The job should actually terminate. Because the last page doesn't have the '__next' key.
Is there any property that needs to be configured in externalConfig.properties file that I am missing?.
05/09/2023 12:48 AM
What is user base (total users) you are trying to import from SFSF? Is there any jobs (account import) running during user import?
in my case 25K take about 2.15hrs to complete
05/09/2023 01:09 AM
There are total 51000 to 52000 users we are trying to import.
Currently we have not configured the account import JOSN and the SAP users import Job is the only job we are running.
The job runs for around 15-20 minutes in our case and in the logs I can see that the next pages are being called and their users data is also being processed as these users are currently not in Users table in Saviynt.
But, the job ends after 15-20 minutes without any job end date. The job is showing status as Success even though only the same first 1000 users are imported.
05/09/2023 01:12 AM
Do you have Success Factors already configured?. Can you please cross check our Connection and Import JSON with yours to see if we are missing something?.
Or can you please share the JSONs you have configured after removing the confidential information.
That might be helpful for me.
Thanks
05/09/2023 02:00 AM
It would be helpful if you could please share the Connection and ImportUsers JSON. We are stuck with the Import since several days and are not able to proceed and further.
Thanks
05/09/2023 02:58 AM
05/09/2023 03:29 AM
Thanks for the ImportUsersJSON. I couldn't see much difference when comparing it to our JSON. Can you please share the ConnectionJSON as well?. Also do you have the STATUS_THRESHOLD_CONFIG set ?.
Thanks
05/09/2023 03:26 AM
Can u please try this for pagination:
"pagination":
{
"nextUrl":
{
"nextUrlPath": "${(response.completeResponseMap.d.__next == null) || (response.completeResponseMap.d.__next.equals(\"\"))?null:response.completeResponseMap.d.__next}"
}
}
05/11/2023 01:24 AM
I tried with the above pagination logic as well. It didn't work.
Actually..in the last page the '__next' key is not present at all. In all other pages the '__next' key is present.
05/23/2023 07:11 AM
i strongly believe this should be something to do with the import json URL , cause in postman you can only see 1000 due to limitation .are u sure you have expanded all the entity that you are using in import user json ? and hope the user import job should be full import .
05/11/2023 01:34 AM
@SB Could you figure out what might be going wrong?. I can see the pages being called till the second last page in the logs. The users data is also being processed in the logs. But the users are not appearing in Saviynt and also the import job is ending in an unusual way. There is no Last Run and Job End Date when the job is run with pagination included.
@rushikeshvartak @sahajranajee
Can you please look into this?. We are stuck with this since weeks. We have also raised a ticket with Saviynt. But we are not getting any updates.
05/11/2023 04:01 PM
This will require additional analysis and may need to look at the code. Can you work with Support team over the FD created to further troubleshoot.
05/23/2023 07:07 AM
05/25/2023 03:52 AM
Hi,
We found out that whenever we try to use any CONST statement in the import user json, we are facing this issue. Once we remove all CONST statement all user records got imported.
Something as simple as below is also not working:
"displayname": "#CONST#${String fn=response.employmentNav.personNav.personalInfoNav.results[0].firstName; String ln=response.employmentNav.personNav.personalInfoNav.results[0].lastName; return fn+\" \"+ln}~#~char",
"customproperty14": "#CONST#${String s=\"HRIS\"; return s}~#~char",
But when we try without pagination(only first 1000 users) or when we try to import for a few users by providing the userId in url's filter, the import works fine, all the CONST statements work as expected.
I am not sure why we are facing this issue. We have used CONST in others REST connectors as well and there it works perfectly fine even with Pagination.
Could it be because we are trying to import 50000+ records in SAP and some sort of timeout is occurring?
05/25/2023 05:35 AM - edited 05/25/2023 05:35 AM
response.employmentNav.personNav.personalInfoNav.results[0].firstName;
does it giving data for all users
Maybe Index out of bound error
05/26/2023 05:51 AM
I removed all the CONST statements and tried with a basic CONST statement:
"customproperty14": "#CONST#${String s=\"HRIS\"; return s}~#~char",
Even this is not working when we use Pagination.
Without Pagination when we import even with all the complex CONST statements, the first 1000 records are imported. Also, when we try for specific users by specifying their UserId in url's filter, it works fine.
It is only when we use Pagination and CONST statements together that our import job is ending weirdly. No job end date/time, No Last run. But the job ends as their is nothing in Running Jobs.
05/25/2023 10:24 AM
this is one of the example that we use CONST in our code with pagination, and it works fine. the use case here is if its employee, pick the email that belong to 67453 email type else go for 67454 this type. try to change as per your requirement.
"SECONDARYEMAIL": "#CONST#${String r=response.empInfo.isContingentWorker;if(r.equalsIgnoreCase('true')) {List responseList = response.empInfo.personNav.emailNav.results;int count = 0; int size = responseList.size(); Iterator iterator = responseList.iterator(); while (iterator.hasNext()){count++; Map dataMap = iterator.next(); if('67453'.equals(dataMap.emailType)){return dataMap.emailAddress} else if(count == size && '67454'.equals(dataMap.emailType)){return dataMap.emailAddress}}}else {List responseList = response.empInfo.personNav.emailNav.results; int count = 0; int size = responseList.size(); Iterator iterator = responseList.iterator(); while (iterator.hasNext()){count++; Map dataMap = iterator.next(); if('67454'.equals(dataMap.emailType)){return dataMap.emailAddress} else if(count == size && '67453'.equals(dataMap.emailType)){return dataMap.emailAddress}}}}~#~char"
05/26/2023 04:08 AM
Hi,
Can you please share the ImportUserJson?. It would be quite helpful.
Thanks
05/26/2023 05:57 AM
Actually, in our case even the basic CONST statements are not working when we include Pagination and try to import all 50000+ recods.
Something as simple as: "customproperty14": "#CONST#${String s=\"HRIS\"; return s}~#~char",
with all other CONST statements removed is also not working.
When we try to import for few users or remove Pagination, the job runs as expected with all CONST statements.(Only 1000 records are imported).
Once, we remove all CONST statements and include Pagination, all records are imported successfully. It's really weird because we are using CONST in other REST connectors and there it is working fine. We are facing issues with only the SAP SuccessFactors REST connector.
Without the CONST statements our requirement of importing user's phone,fax,secondarPhone cannot be satisfied. As this data is dynamic. It's present inside a List and the List size can be varying. So, without using some sort of logic(code) while importing it's impossible to import these 3 attributes.
05/30/2023 01:31 PM
we are currently using saviynt 23.4 and the sap successfactor CONST should work in 23.1 too,
06/08/2023 08:36 AM
this appears to be an issue and is being worked by Support.
09/25/2023 09:18 AM
The #CONST# keyword in colstoProps mapping for REST Connector is for supporting hardcoded mapping to customproperty and smaller groovy based functions like replace, concatenate etc. For using code to iterate through objects and mapping fields, we recommend using the User Import Preprocessor. This ensure that we need not evaluate repeated code for similar attributes while mapping them to multiple custom properties