Announcing the Saviynt Knowledge Exchange unifying the Saviynt forums, documentation, training,
and more in a single search tool across platforms. Read the announcement here.

Rest connector Offset Pagination

ShubhamBabbar
New Contributor III
New Contributor III

For the Rest connector, how do I configure the Offset Pagination totalCountPath attribute for the example response (see below) where the response contains the total number of records in the attribute "totalResults". Next Page URLs are not available for this endpoint, so have to use Offset based pagination.

shubabbar_0-1689598750815.png

From testing various permutations, the Java Script capability does not seem to be enabled for the totalCountPath config attribute, log indicates total count = null for the below tried values.

  • "totalCountPath": "responseText.totalResults"
  • "totalCountPath": "completeResponseMap.totalResults"
  • "totalCountPath": "completeResponseMap.meta.totalResults"
    "totalCountPath": "response.responseText.get(totalResults)"
  • "totalCountPath": "response.responseText.totalResults"
  • "totalCountPath": "${completeResponseMap.get('totalResults')}"
  • shubabbar_1-1689599070349.png

     

Running the account job and monitoring the logs shows that the first 100 records are retrieved but the log also shows "DEBUG rest.RestUtilService - totalCount: null". And the paging stops and does not continue.

If I hard code a value such in the pagination config the logs show and pagination continues to the next batch of accounts, logs:
Got Webservice API Response: [headers:[Date: Mon, 17 Jul 2023 10:55:06 GMT, Content-Type: application/json, Transfer-Encoding: chunked, Connection: keep-alive, Max-Forwards: 20, X-CorrelationID: Id-0a1eb564324b24xyzxyz79ee 0; Id-0a1eb564334bf97e79xyzxyz 0; Id-0a1eb564344b7a6bzyxzyx 0, Cache-Control: no-cache,no-store,must-revalidate, Content-Security-Policy: default-src 'none'; script-src 'none'; object-src 'none'; frame-ancestors 'self'; frame-src 'none'; require-trusted-types-for 'script', HC-TransactionId: Id-0a1eb564324b24e5a6df79ee, Referrer-Policy: origin-when-cross-origin, Via: HiltiCloud, X-Content-Type-Options: nosniff, X-Frame-Options: SAMEORIGIN], responseText:{\"totalResults\":15806,\"itemsPerPage\":100,\"startIndex\":14100,\"schemas\":[\"urn:ietf:params:scim:api:messages:2.0:ListResponse\"],\"Resources\":[{\"schemas\":[\"urn:ietf:params:scim:schemas:core:2.0:User\"],\"id\":\"xyzxyzxyxzyxzyxzy\",\"nickName\":\"DXYZXYZYXZ\",\"active\":true}

8 REPLIES 8

rushikeshvartak
All-Star
All-Star
  • "totalCountPath": "totalResults"

Regards,
Rushikesh Vartak
If you find the response useful, kindly consider selecting Accept As Solution and clicking on the kudos button.

This had failed as well, Rushikesh.

Try below pagination block

"pagination": {
"offset": {
"offsetParam": "startIndex",
"batchParam": "count",
"batchSize": 100,
"totalCountPath": "completeResponseMap.totalResults"
}
}


Regards,
Saathvik
If this reply answered your question, please Accept As Solution and give Kudos to help others facing similar issue.

ShubhamBabbar
New Contributor III
New Contributor III

Hi Saathvik, this has already been tried and is not working as expected. The list of values already tried are mentioned in the main query above. 

What if you hardcode max number ? 100000


Regards,
Rushikesh Vartak
If you find the response useful, kindly consider selecting Accept As Solution and clicking on the kudos button.

Did you try exact same block? Why I am asking this because in the screenshot you shared I see two totalCountPath variables and highlighted block is in totalCountPath as well. 

sk_0-1689877489172.png

 


Regards,
Saathvik
If this reply answered your question, please Accept As Solution and give Kudos to help others facing similar issue.

rushikeshvartak
All-Star
All-Star

Please share full json


Regards,
Rushikesh Vartak
If you find the response useful, kindly consider selecting Accept As Solution and clicking on the kudos button.

ShubhamBabbar
New Contributor III
New Contributor III

Thank you Rushikesh and Saathvik for your responses.

The issue has now been resolved.

 

Issue: The API return error 400 in the first call, as it does not support startIndex=0, the offset value for startIndex should be 1 (startIndex=1). Hence, on specifying the the totalCountPath as a fixed value it moves to the next page but was erroring out on using totalCountPath from the response object.

Solution: Initial URL set with '?startIndex=1&count=100' and pagination logic as :-
"pagination": {
"nextUrl": {
"nextUrlPath": "${response?.completeResponseMap?.itemsPerPage<100?null:'https://sample-url.com/Groups?count=100&startIndex='+Math.addExact(response.completeResponseMap.star...)}"
}
}