Announcing the SAVIYNT KNOWLEDGE EXCHANGE unifying the Saviynt forums, documentation, training, and more in a single search tool across platforms. Click HERE to read the Announcement.

SAP SuccessFactor Pagination Not working

SinghAtul
Regular Contributor
Regular Contributor

Hi,

We have configured SAP SuccessFactors and are trying to import all the users from the target. There are around 50000 records in the target but only first 1000 records are reconciled whenever we run the job.

We have  used the below pagination logic:

"pagination": {
"nextUrl": {
"nextUrlPath": "${response?.completeResponseMap.d.__next==null?null:response?.completeResponseMap.d.__next}"
}
}

We could see the below error in the logs:

2023-05-06T14:36:22+05:30-ecm-worker-2023-05-06T09:06:22.211118314Z stdout F 2023-05-06 09:06:22,211 [quartzScheduler_Worker-3] ERROR rest.RestProvisioningService - Inside token Expiry Exception block. connectionParamMap.refreshTryCount : 0

I am attaching the ConnectionJSON and the ImportUsersJSON.

Please guide as to what might be wrong. Have we configured the JSONs correctly?.

Thanks,

Atul Singh

 

25 REPLIES 25

SB
Saviynt Employee
Saviynt Employee

This does not look like the correct log file as I do not see the job thread in it. Can you share the correct one. 

Please update ConfigJSON param in the REST connection with value {"showLogs":true} before running the job if not added already.


Regards,
Sahil

SinghAtul
Regular Contributor
Regular Contributor

Hi Sahil,

I am attaching the logs below. Config showLogs is set to true.

Thanks

GSR
New Contributor III
New Contributor III

Please use below.

 

"pagination":{
"nextUrl":{
"nextUrlPath":"${response?.completeResponseMap?.d?.__next == null ?null : response?.completeResponseMap?.d?.__next}"
}
}

SinghAtul
Regular Contributor
Regular Contributor

@GSR @SB 

We have tried with the exact same Paginations logic as well.  By including the not null operator(.?) with each key. The user count is not increasing. Only the first 1000 users are getting imported. Even though in the logs we can see that the subsequent '__next' url's are being called..also the user data is getting processed. But somehow the user count is not increasing and users are not appearing in Identity Repo/Users. 

Also, when we use pagination, the job last run is coming as blank(-). The job is showing as Success even though not all users are imported.

SinghAtul_0-1683616787286.png

The job end date is also not appearing.

SinghAtul_1-1683616855706.png

SinghAtul_0-1683618286681.png

As we can see in the above picture that  the next pages are being called. Also, I have verified that Saviynt is calling the second last page also. I verified it from looking at the 'skipToken' as the skipToken is unique and constant for each page. Also, I could see the data of users getting processed. These users are not in the first 1000 users and even though Saviynt is processing these users data through API calls, these users are not getting imported. 

I felt it could be a timeout issue so I configured this in the ConfigJSON:

{
"showLogs": true,
"connectionTimeoutConfig": {
"connectionTimeout": 1000,
"readTimeout": 7200,
"writeTimeout": 1000,
"retryWait": 100,
"retryCount": 10
}

}

But this didn't work as well.

I feel the job is somehow not terminating properly as the job end date is blank. But according to the pagination logic we are using:

"pagination":{
"nextUrl":{
"nextUrlPath":"${response?.completeResponseMap?.d?.__next == null ?null : response?.completeResponseMap?.d?.__next}"
}
}

The job should actually terminate. Because the last page doesn't have the '__next' key.

Is there any property that needs to be configured in externalConfig.properties file that I am missing?.

 

 

 

 

 

GSR
New Contributor III
New Contributor III

What is user base (total users) you are trying to import from SFSF? Is there any jobs (account import) running during user import?

in my case 25K take about 2.15hrs to complete

SinghAtul
Regular Contributor
Regular Contributor

There are total 51000 to 52000 users we are trying to import. 

Currently we have not configured the account import JOSN and the SAP users import Job is the only job we are running. 

The job runs for around 15-20 minutes in our case and in the logs I can see that the next pages are being called and their users data is also being processed as these users are currently not in Users table in Saviynt. 

But, the job ends after 15-20 minutes without any job end date. The job is showing status as Success even though only the same first 1000 users are imported.

SinghAtul
Regular Contributor
Regular Contributor

Do you have Success Factors already configured?. Can you please cross check our Connection and Import JSON with yours to see if we are  missing something?. 

Or can you please share the JSONs you have configured after removing the confidential information.

That might be helpful for me.

Thanks

SinghAtul
Regular Contributor
Regular Contributor

@GSR 

It would be helpful if you could please share the Connection and ImportUsers JSON. We are stuck with the Import since several days and are not able to proceed and further.

Thanks

GSR
New Contributor III
New Contributor III

PFA

SinghAtul
Regular Contributor
Regular Contributor

@GSR 

Thanks for the ImportUsersJSON. I couldn't see much difference when comparing it to our JSON. Can you please share the ConnectionJSON as well?. Also do you have the STATUS_THRESHOLD_CONFIG set ?.

Thanks

Vinit556
New Contributor III
New Contributor III

Can u please try this for pagination:

 

"pagination":

"nextUrl":

"nextUrlPath": "${(response.completeResponseMap.d.__next == null) || (response.completeResponseMap.d.__next.equals(\"\"))?null:response.completeResponseMap.d.__next}" 


}

SinghAtul
Regular Contributor
Regular Contributor

@Vinit556 

I tried with the above pagination logic as well. It didn't work.

Actually..in the last page the '__next' key is not present at all. In all other pages the '__next' key is present. 

 

i strongly believe  this should be something to do with the import json URL , cause in postman you can only see 1000 due to limitation .are u sure you have expanded all the entity  that you are using in import user json ? and hope the user import job should be full import .

SinghAtul
Regular Contributor
Regular Contributor

@SB Could you figure out what might be going wrong?. I can see the pages being called till the second last page in the logs. The users data is also being processed  in the logs. But the users are not appearing in Saviynt and also the import job is ending in an unusual way. There is no Last Run and Job End Date when the job is run with pagination included. 

@rushikeshvartak @sahajranajee 

Can you please look into this?. We are stuck with this since weeks. We have also raised a ticket with Saviynt. But we are not getting any updates. 

SB
Saviynt Employee
Saviynt Employee

This will require additional analysis and may need to look at the code. Can you work with Support team over the FD created to further troubleshoot.


Regards,
Sahil

musthak_ahamad
Regular Contributor
Regular Contributor

your import user json is same as mine , no difference.

i used basic auth to connect , i have attached  the json here , 

 

SinghAtul
Regular Contributor
Regular Contributor

Hi,

We found out that whenever we try to use any CONST statement in the import user json, we are facing this issue. Once we remove all CONST statement all user records got imported.

Something as simple as below is also not working:

"displayname": "#CONST#${String fn=response.employmentNav.personNav.personalInfoNav.results[0].firstName; String ln=response.employmentNav.personNav.personalInfoNav.results[0].lastName; return fn+\" \"+ln}~#~char",

"customproperty14": "#CONST#${String s=\"HRIS\"; return s}~#~char",

But when we try without pagination(only first 1000 users) or when we try to import for a few users by providing the userId in url's filter, the import works fine, all the CONST statements work as expected.

I am not sure why we are facing this issue. We have used CONST in others REST connectors as well and there it works perfectly fine even with Pagination.

Could it be because we are trying to import 50000+ records in SAP and some sort of timeout is occurring?

 

response.employmentNav.personNav.personalInfoNav.results[0].firstName;

does it giving data for all users

Maybe Index out of bound error


Regards,
Rushikesh Vartak
If you find the response useful, kindly consider selecting Accept As Solution and clicking on the kudos button.

@rushikeshvartak 

I removed all the CONST statements and tried with a basic CONST statement:

"customproperty14": "#CONST#${String s=\"HRIS\"; return s}~#~char",

Even this is not working when we use Pagination.

Without Pagination when we import even with all the complex CONST statements, the first 1000 records are imported. Also, when we try for specific users by specifying their UserId in url's filter, it works fine.

It is only when we use Pagination and CONST statements together that our import job is ending weirdly. No job end date/time, No Last run. But the job ends as their is nothing in Running Jobs.

 

this is one of the example that we use CONST in our code with pagination, and it works fine. the use case here is  if its employee, pick the email that belong to 67453  email type else go for 67454 this type. try to change as per your requirement.

"SECONDARYEMAIL": "#CONST#${String r=response.empInfo.isContingentWorker;if(r.equalsIgnoreCase('true')) {List responseList = response.empInfo.personNav.emailNav.results;int count = 0; int size = responseList.size(); Iterator iterator = responseList.iterator(); while (iterator.hasNext()){count++; Map dataMap = iterator.next(); if('67453'.equals(dataMap.emailType)){return dataMap.emailAddress} else if(count == size && '67454'.equals(dataMap.emailType)){return dataMap.emailAddress}}}else {List responseList = response.empInfo.personNav.emailNav.results; int count = 0; int size = responseList.size(); Iterator iterator = responseList.iterator(); while (iterator.hasNext()){count++; Map dataMap = iterator.next(); if('67454'.equals(dataMap.emailType)){return dataMap.emailAddress} else if(count == size && '67453'.equals(dataMap.emailType)){return dataMap.emailAddress}}}}~#~char"

 

Hi,

Can you please share the ImportUserJson?. It would be quite helpful.

Thanks

Actually, in our case even the basic CONST statements are not working when we include Pagination and try to import all 50000+ recods.

Something as simple as: "customproperty14": "#CONST#${String s=\"HRIS\"; return s}~#~char",

with all other CONST statements removed is also not working.

When we try to import for few users or remove Pagination, the job runs as expected with all CONST statements.(Only 1000 records are imported).

Once, we remove all CONST statements and include Pagination, all records are imported successfully. It's really weird because we are using CONST in other REST connectors and there it is working fine. We are facing issues with only the SAP SuccessFactors REST connector.

Without the CONST statements our requirement of importing user's phone,fax,secondarPhone cannot be satisfied. As this data is dynamic. It's present inside a List and the List size can be varying. So, without using some sort of logic(code) while importing it's impossible to import these 3 attributes.

we are currently using saviynt  23.4 and the sap successfactor CONST should work in 23.1 too, 

SB
Saviynt Employee
Saviynt Employee

this appears to be an issue and is being worked by Support.


Regards,
Sahil

SB
Saviynt Employee
Saviynt Employee

The #CONST# keyword in colstoProps mapping for REST Connector is for supporting hardcoded mapping to customproperty and smaller groovy based functions like replace, concatenate etc. For using code to iterate through objects and mapping fields, we recommend using the User Import Preprocessor. This ensure that we need not evaluate repeated code for similar attributes while mapping them to multiple custom properties


Regards,
Sahil