Unable to reconcile more than 2000 accounts in REST connector

[Jaya]

Hi All,

We are facing a issue in our REST connector, Issue is - we have approx. 2400 accounts at target application & our reconciliation Job is able to pick only 2000 account records.

We raised a Freshdesk ticket & below is their response - 

"Thanks for the meeting. Let me write down a summary of what we discussed. We have enabled logs for the rest connection. With this we were able to verify there are 2460 records present in the source. First page contains 2000 rows. They are loaded into the Saviynt. The rest is missing as the 2nd page is not loaded. This is a pagination setup issue which needs to be corrected. Please see Rest connector guide in our Docs portal, and/or check with experts in our Forum. In case of need you can reach out to your TAM too."

Can anyone suggest Importaccountjsoncode changed in for pagination.

{
"accountParams": {
"connection": "userAuth",
"processingType": "SequentialAndIterative",
"statusAndThresholdConfig": {
"statusColumn": "customproperty11",
"activeStatus": [
"Staff"
],
"deleteLinks": true,
"accountThresholdValue": 10000,
"accountsInImportAction": "Active",
"correlateInactiveAccounts": false,
"inactivateAccountsNotInFile": false,
"deleteAccEntForActiveAccounts": true
},
"call": {
"call1": {
"callOrder": 0,
"stageNumber": 0,
"http": {
"url": "https://xxx.com/services/data/v49.0/query/?q=select+id,name,status__c,isActive__c,Email__c,Business_Pillar__c,Salesforce_Record__c,Title__c,Line_Manager__c,Grade__c,Department__c+from+Employee__c",
"httpHeaders": {
"Authorization": "${access_token}",
"Accept": "application/json"
},
"httpContentType": "application/json",
"httpMethod": "GET"
},
"listField": "records",
"keyField": "accountID",
"statusConfig": {
"active": "true",
"inactive": "false"
},
"colsToPropsMap": {
"accountID": "Id~#~char",
"name": "Name~#~char",
"displayName": "Name~#~char",
"customproperty11": "Status__c~#~char",
"customproperty12": "Email__c~#~char",
"customproperty30": "isActive__c~#~bool",
"customproperty20": "Business_Pillar__c~#~char",
"customproperty21":"Salesforce_Record__c~#~char",
"customproperty31": "Title__c~#~char",
"customproperty32": "Line_Manager__c~#~char",
"customproperty33": "Grade__c~#~char",
"customproperty34": "Department__c~#~char"
},
"pagination": {
"nextUrl": {
"nextUrlPath": "${response?.completeResponseMap?.next_page==null?null:response.completeResponseMap.next_page}"
}
}
}
}
}
}

[rushikeshvartak]

share Postman screenshot

[Jaya]

For single user-

[rushikeshvartak]

Your json url & postman screenshot does not match

[Vedanth-BK]

Hi @Jaya 

Please provide information to the below points with few screenshots from postman, it will help us to narrow down on the type of pagination and the logic.

1. How to navigate to the Next Page in the postman.
2.Where do you get the pagination information? is it in the response body or response headers?

Thank you 

Vedanth

[Jaya]

There is nothing for pagination in the query response. Even when I am running query in postman(which we have provided in import) it is bringing only 2000 accounts.

I need to check this with application team.

I have one more query, Import job automatically set status of those account which aren't found during import to 'Suspended from import services'. Can we change this to 'In-active'/'Manually Provisioned'.

[Vedanth-BK]

Hi @Jaya 

Please add "inactivateAccountsNotInFile": true, in your statusAndThresholdConfig.
Check out this link for more info.

[Jaya]

Thanks Vedanth for this.

@rushikeshvartak @Vedanth-BK 

I have got the pagination info, it's in the starting of response. Please help me to write this in my importjson I have tried but recon job is failing.

Actually nextResponseUrl is not giving full url to get the response we have to add domain name to get the response like-

https://companyname.com/{nextResponseUrl's Value}

[rushikeshvartak]

"pagination": {
"nextUrl": {
"nextUrlPath": "${response.objectList[0].getAt('nextRecordUrl').get(1).getAt('href')==''?null:<specify URL>.concat(response.objectList[0].getAt('_links').get(1).getAt('href').toString().replace('..',''))}"
}
}

 Sample

[Jaya]

Hello,

1. Getting Null value with below code-

"nextUrlPath": "${response.completeResponseMap.nextRecordsUrl?null:'https://xxx.my.salesforce.com'.concat(response.completeResponseMap.nextRecordsUrl)}"

2.  And it's taking single quotes and + with below code-

"nextUrlPath": "{'https://xxx.my.salesforce.com'+${response.completeResponseMap.nextRecordsUrl}}"

error logs with 2-

2024-02-05 14:16:19,084 [quartzScheduler_Worker-1] DEBUG rest.RestProvisioningService - Error while getting Account Import response for url- {'https://xxx.my.salesforce.com'+/services/data/v49.0/query/0r8xx2ceBEjpmREAYY-2000} is: null

Please help.

[Vedanth-BK]

Hi @Jaya ,

To add the logic for pagination, please share the screenshot of the last page when there are no more iterations required.

[Jaya]

Hi Vedanth,

There is no further linkage at starting/ending in the next URL call see below-

first call-

second call with url received from first call(nextRecordsUrl)-

[Vedanth-BK]

Hi @Jaya 

Please try with the below pagination.

 "pagination": {
              "nextUrl": {
                "nextUrlPath": "${response?.completeResponseMap?.contains('nextRecordsUrl') == false ? null : 'https://xxx.my.salesforce.com' +response?.completeResponseMap?.nextRecordsUrl}"
              }
            }

[Jaya]

Getting below error-

2024-02-08 17:17:17,912 [quartzScheduler_Worker-3] DEBUG println.PrintlnToLogger - Println :: | Error java.lang.IllegalArgumentException: Illegal character in path at index 1: ${response?.completeResponseMap?.contains('nextRecordsUrl') == false ? null : 'https://xxx.my.salesforce.com' +response?.completeResponseMap?.nextRecordsUrl}

[SB]

Can you try with the below. In case this does not work, can you share the complete job log. Do ensure to enable additional logging for REST by updating ConfigJSON param in the REST connection with value {"showLogs":true} if not done already.

"pagination": {
"nextUrl": {
"nextUrlPath": "${response?.completeResponseMap?.contains('done')==false?'https://xxx.my.salesforce.com'+response?.completeResponseMap?.nextRecordsUrl:null}"
}
}

[Jaya]

It is giving below error message & I will share the logs separately-

2024-02-13 16:37:26,986 [quartzScheduler_Worker-2] DEBUG println.PrintlnToLogger - Println :: | Error java.lang.IllegalArgumentException: Illegal character in path at index 1: ${response?.completeResponseMap?.contains('done') == false ?'https://xxx.my.salesforce.com'+response?.completeResponseMap?.nextRecordsUrl:null}
2024-02-13 16:37:26,986 [quartzScheduler_Worker-2] DEBUG println.PrintlnToLogger - Println :: | Error  at java.net.URI.create(URI.java:852)

[SB]

can u try with the below 2

"pagination": {
"nextUrl": {
"nextUrlPath": "${(response?.completeResponseMap?.contains('done')==false)?'https://xxx.my.salesforce.com'+response?.completeResponseMap?.nextRecordsUrl:null}"
}
}

OR

"pagination": {
"nextUrl": {
"nextUrlPath": "${(response?.completeResponseMap?.contains('nextRecordsUrl'))?'https://xxx.my.salesforce.com'+response?.completeResponseMap?.nextRecordsUrl:null}"
}
}

[Jaya]

Hi Sahil,

Getting below errors while using given codes-

first code error - 

 Illegal character in path at index 1: ${response?.completeResponseMap?.contains('done') == false ?'https://xxx.my.salesforce.com'+response?.completeResponseMap?.nextRecordsUrl:null}

second code error - 

2024-02-14 10:53:37,424 [quartzScheduler_Worker-2] DEBUG rest.RestProvisioningService - Error while getting Account Import response for url- ${(response?.completeResponseMap?.contains('nextRecordsUrl'))?'https://xxx.my.salesforce.com'+response?.completeResponseMap?.nextRecordsUrl:null} is: null

[SB]

Can you share the log just for the 2nd one. With the 2nd one I see we are past the Illegal Character error so I want to see if its now an issue with the path value we have defined.

[Jaya · Accepted solution]

Thanks all for your time & responses.

It is working with below pagination code-

"pagination": {
"nextUrl": {
"nextUrlPath": "${response?.completeResponseMap?.done==true?null:'https://xxx.com'+response.completeResponseMap.nextRecordsUrl}"
}
}