SuccessFactors user import does not include all userID records that should've been returned

[yusufw]

In the implementation, every physical person has exactly 1 empID and may have 1 or more userID. The employee's employment details is tied to the userID. For example:
[empID:1234,userID:8888,firstname:John,lastname:Smith,title:Consultant,status=inactive]
[empID:1234,userID:9999,firstname:John,lastname:Smith,title:Manager,status=active]
[empID:7890,userID:1111,firstname:=Ken,lastname:Doll,title:Engineer,status=active]

According to customer's requirements, Saviynt should only process 1 userID per physical person, so I'm using the preprocessor to remove unwanted userID records for each employeeID. I am trying to do a full import of all users in SuccessFactors, but not all userIDs in the SuccessFactors, are returned to Saviynt. How can I get Saviynt to read all userIDs.

Configuration Details

ImportUserJSON:

...
"url": "https://api4.successfactors.com/odata/v2/User?%24expand=manager,empInfo/personNav/phoneNav,empInfo/personNav/emailNav,empInfo/jobInfoNav/positionNav/locationNav/addressNavDEFLT/stateNav/picklistLabels,empInfo/workOrderNav,empInfo/jobInfoNav/eventNav,empInfo/jobInfoNav/eventReasonNav&asOfDate=${java.time.LocalDate.now()}&%24select=empId,firstName,nickname,mi,lastName,userId,empId,empInfo/jobInfoNav/eventNav/externalCode,empInfo/jobInfoNav/eventReasonNav/externalCode,country,empInfo/jobInfoNav/positionNav/locationNav/addressNavDEFLT/zipCode,department,manager/empId,businessPhone,empInfo/startDate,empInfo/jobInfoNav/startDate,empInfo/jobInfoNav/endDate,empInfo/jobInfoNav/terminationDate,empInfo/personNav/phoneNav/phoneNumber,empInfo/personNav/emailNav/emailType,empInfo/personNav/emailNav/emailAddress,status,manager/empId,empInfo/jobInfoNav/costCenter,empInfo/jobInfoNav/contractEndDate,empInfo/jobInfoNav/positionNav/cust_employeeGroup,empInfo/jobInfoNav/positionNav/cust_employeeSubgroup,empInfo/jobInfoNav/positionNav/jobTitle,empInfo/jobInfoNav/positionNav/code,empInfo/jobInfoNav/positionNav/locationNav/externalCode,empInfo/jobInfoNav/positionNav/locationNav/name,empInfo/jobInfoNav/positionNav/cust_PersonnelSubArea,empInfo/jobInfoNav/positionNav/locationNav/addressNavDEFLT/address1,empInfo/jobInfoNav/positionNav/locationNav/addressNavDEFLT/city,empInfo/jobInfoNav/positionNav/locationNav/addressNavDEFLT/stateNav/picklistLabels/label,empInfo/jobInfoNav/positionNav/locationNav/addressNavDEFLT/country,lastModified&%24filter=status%20in%20%27t%27%2C%27f%27",
...
"userResponsePath": "d.results",
"pagination": {
"nextUrl": {
"nextUrlPath": "${response.completeResponseMap.d.__next==null?'':response.completeResponseMap.d.__next.replace(',','%2C')}"
}
}

Preprocessor logic:

...
"PREPROCESSQUERIES":[
"DELETE FROM NEWUSERDATA WHERE customproperty1 NOT IN ( SELECT userId from ( SELECT CASE WHEN SUM(u.customproperty23 = 't') > 0 THEN ( SELECT actives.customproperty1 as userId FROM NEWUSERDATA actives WHERE actives.username = u.username AND actives.customproperty23 = 't' ORDER BY actives.startdate DESC LIMIT 1 ) ELSE ( SELECT inactives.customproperty1 as userId FROM NEWUSERDATA inactives WHERE inactives.username = u.username ORDER BY inactives.termdate DESC LIMIT 1 ) END AS userId FROM NEWUSERDATA u GROUP BY u.username ) AS C )",
...

where customproperty1 stores the userID.

Questions:

1. Is there a way to log every userID in the import? I'll like to know what userIDs are present before the preprocessor updates NEWUSERDATA.
2. What changes could be made to the query above, so that it returns all userIds.

What I've tried:

1. I have modified the import query's filter many times. In addition to the $filter=status in 't','f': 
   a. userId in '8888','9999' - whenever this filter is used, NEWUSERDATA will include all userIDs specified - and the preprocessor logic works as expected
   b. lastModified gt '2023-07-01'. NEWUSERDATA will include all userIDs where lastModified is greater than July 1 - and the preprocessor logic works as expected
   c. lastModified gt '2023-01-01'. NEWUSERDATA did NOT include all userIDs when userIDs where lastModified is greater than Jan 1, so the preprocessor logic did not work as expected, the DELETE statement did not delete the right records.
   d. <nothing>. NEWUSERDATA did NOT include all userIDs, i.e. 'userId = '9999' is missing and the preprocessor logic DOES NOT work as expected. I know '9999' is missing because I replaced the DELETE statement in the preprocessor with something like 'DELETE from NEWUSERDATA where customproperty1 != '9999', and no Saviynt user record was updated.
2. Using Postman, I ran the queries above against SuccessFactors, and was able to see the missing userIDs.

[adriencosson]

Hi @yusufw ,

You can leverage the below JSON in the ConfigJSON of your connector to display more logs of each API call and their response, which will display the retrieved users :

{"showLogs":true}

[yusufw]

ConfigJSON already has:

"showLogs":true

[Darshanjain]

Hi @yusufw 

When you say some userid's are missing,  due you think is it because of the pagination issues where it is not bringing all the users in the import?

Are you using userid as the reconciliation field or the employee id as recon field. if userid is stored as CP1 are you expecting that in CP1 multiple entries of userid will be present after the import .

Also check in the logs to see if you are able to get the users from postman with same Api call but why not in saviynt, All Api logs will be captured in the saviynt so that you can check where it is failing or why are those records missing.

Thanks

Darshan