Rest connector Offset Pagination

[ShubhamBabbar]

For the Rest connector, how do I configure the Offset Pagination totalCountPath attribute for the example response (see below) where the response contains the total number of records in the attribute "totalResults". Next Page URLs are not available for this endpoint, so have to use Offset based pagination.

From testing various permutations, the Java Script capability does not seem to be enabled for the totalCountPath config attribute, log indicates total count = null for the below tried values.

• "totalCountPath": "responseText.totalResults"
• "totalCountPath": "completeResponseMap.totalResults"
• "totalCountPath": "completeResponseMap.meta.totalResults"
  "totalCountPath": "response.responseText.get(totalResults)"
• "totalCountPath": "response.responseText.totalResults"
• "totalCountPath": "${completeResponseMap.get('totalResults')}"
• 

Running the account job and monitoring the logs shows that the first 100 records are retrieved but the log also shows "DEBUG rest.RestUtilService - totalCount: null". And the paging stops and does not continue.

If I hard code a value such in the pagination config the logs show and pagination continues to the next batch of accounts, logs:
Got Webservice API Response: [headers:[Date: Mon, 17 Jul 2023 10:55:06 GMT, Content-Type: application/json, Transfer-Encoding: chunked, Connection: keep-alive, Max-Forwards: 20, X-CorrelationID: Id-0a1eb564324b24xyzxyz79ee 0; Id-0a1eb564334bf97e79xyzxyz 0; Id-0a1eb564344b7a6bzyxzyx 0, Cache-Control: no-cache,no-store,must-revalidate, Content-Security-Policy: default-src 'none'; script-src 'none'; object-src 'none'; frame-ancestors 'self'; frame-src 'none'; require-trusted-types-for 'script', HC-TransactionId: Id-0a1eb564324b24e5a6df79ee, Referrer-Policy: origin-when-cross-origin, Via: HiltiCloud, X-Content-Type-Options: nosniff, X-Frame-Options: SAMEORIGIN], responseText:{\"totalResults\":15806,\"itemsPerPage\":100,\"startIndex\":14100,\"schemas\":[\"urn:ietf:params:scim:api:messages:2.0:ListResponse\"],\"Resources\":[{\"schemas\":[\"urn:ietf:params:scim:schemas:core:2.0:User\"],\"id\":\"xyzxyzxyxzyxzyxzy\",\"nickName\":\"DXYZXYZYXZ\",\"active\":true}

[rushikeshvartak]

• "totalCountPath": "totalResults"

[ShubhamBabbar]

This had failed as well, Rushikesh.

[sk]

Try below pagination block

"pagination": {
"offset": {
"offsetParam": "startIndex",
"batchParam": "count",
"batchSize": 100,
"totalCountPath": "completeResponseMap.totalResults"
}
}

[ShubhamBabbar]

Hi Saathvik, this has already been tried and is not working as expected. The list of values already tried are mentioned in the main query above. 

[rushikeshvartak]

What if you hardcode max number ? 100000

[sk]

Did you try exact same block? Why I am asking this because in the screenshot you shared I see two totalCountPath variables and highlighted block is in totalCountPath as well. 

[rushikeshvartak]

Please share full json

[ShubhamBabbar · Accepted solution]

Thank you Rushikesh and Saathvik for your responses.

The issue has now been resolved.

Issue: The API return error 400 in the first call, as it does not support startIndex=0, the offset value for startIndex should be 1 (startIndex=1). Hence, on specifying the the totalCountPath as a fixed value it moves to the next page but was erroring out on using totalCountPath from the response object.

Solution: Initial URL set with '?startIndex=1&count=100' and pagination logic as :-
"pagination": {
"nextUrl": {
"nextUrlPath": "${response?.completeResponseMap?.itemsPerPage<100?null:'https://sample-url.com/Groups?count=100&startIndex='+Math.addExact(response.completeResponseMap.star...)}"
}
}