Pagination in REST connector

[reshma_sonawane]

Hi Team,

We are trying to import accounts for one of our REST application. In response or headers, there are no details available about next page. Below is the pagination supported in url by application.

https://*******/user/search?$$FIRST=0&$$LIMIT=2000&filters={"isActive":"true"}&apiKey=****&fields=*****

With this call, it will fetch record 1 to record 2000. For next page url should be,

https://*******/user/search?$$FIRST=2000&$$LIMIT=2000&filters={"isActive":"true"}&apiKey=****&fields=*****

Maximum records that can be fetched in single call is 2000.

We are trying to do pagination for this case. We tried few ways as below but nothing worked:

Case 1:

 "pagination": { "nextUrl": { "nextUrlPath": "${response?.completeResponseMap?.itemsPerPage<2000?null:'****/search?$$FIRST='+Math.addExact(response.completeResponseMap.startIndex,2000)+'&$$LIMIT=2000&filters={"isActive":"true"}&apiKey=****&fields=****}" }}

Error: 

DEBUG rest.RestUtilService - Got null response statusCode with erroMsg - [error:Error Illegal character in path at index 1: ${response?.completeResponseMap?.itemsPerPage<2000?null:'https://****/search?$$FIRST='+Math.addExact(response.completeResponseMap.startIndex,2000)+'&$$LIMIT=2000&filters=%7B%22isActive%22:%22true%22%7D&apiKey=***&fields=****}] 

Case 2:

"pagination": { "nextUrl": { "nextUrlPath": "${response?.objectList?.size()>0?'https://***/search?$$FIRST=response?.objectList?.size()&$$LIMIT=2000&filters=***** : null}" } } 

Error:

error:Error Illegal character in path at index 1: ${response?.objectList?.size()>0?'https://****/search?$$FIRST=response?.objectList?.size()&$$LIMIT=2000&filters=%7B%22isActive%22:%22true%22%7D&apiKey=******&fields=***** : null}]

Case 3:

"pagination": { "offset": { "offsetParam": "cursor", "batchParam": "limit", "batchSize": 2000, "totalCountPath": 100000 }, "nextUrl": { "nextUrlPath": "${response?.completeResponseMap?.next_page==null?null:response.completeResponseMap.next_page}" } }

No account imported

Case 4:

"pagination": { "offset": { "batchSize": 2000, "totalCountPath": 100000 }, "nextUrl": { "nextUrlPath": "${response?.completeResponseMap?.next_page==null?null:response.completeResponseMap.next_page}" } } 

Case 5:

"pagination": {
"offset": {
"offsetParam": "offset",
"batchParam": "$$LIMIT",
"batchSize": 2000,
"totalCountPath": "100000"
},
"nextUrl": {
"nextUrlPath": "${response?.completeResponseMap?.next_page==null?null:response.completeResponseMap.next_page}"
}
}

Can anyone help with this?

Thanks,

Reshma

[Sivagami]

Can you try something like this?

"pagination": {
"offset": {
"offsetParam": "$$FIRST",
"batchParam": "$$LIMIT",
"batchSize": 2000,
"totalCountPath": "100000"
}
}

I believe all the illegal character exceptions are due to the $$ in the parameter name. You could try URL encoding of dollar - %24.

Also, can you share a sample response (Hide sensitive information)? Do you receive pagination attributes in response like items perpage, offset or totalcount?

-Siva

[reshma_sonawane]

Hi Siva,

I tried with above pagination given with offset but it imported only first page(2k records as per limit). Also, tried with encoding of $$ but got same error as before.

Got null response statusCode with erroMsg - [error:Error Illegal character in path at index 1: ${response?.objectList?.size()>0?'https://****?%24%24FIRST=response?.objectList?.size()&%24%24LIMIT=2000&filters=**** : null}]

In the response, we don't get any pagination data. Please find sample response below:

{
"data": [
{
"ID": "****",
"name": "****",
"objCode": "USER",
"isActive": true,
"username": "****",
"title": "****",
"myInfo": "****",
"phoneNumber": "****",
"phoneExtension": "****",
"mobilePhoneNumber": null,
"firstName": "****",
"lastName": "****",
"managerID": "****",
"homeGroupID": "****",
"homeTeamID": "****",
"company": {
"ID": "****",
"name": "****",
"objCode": "****"
},
"accessLevelID": "****",
"ssoUsername": "****",
"emailAddr": "****"
}
]
}

[reshma_sonawane]

hi Team,

Any update on this issue? any reference for such cases.

Thanks,

Reshma