Saviynt unveils its cutting-edge Intelligence Suite products to revolutionize Identity Security!
Click HERE to see how Saviynt Intelligence is transforming the industry.
Saviynt Copilot Icon

Rest connector Offset Pagination

ShubhamBabbar
Regular Contributor
Regular Contributor

For the Rest connector, how do I configure the Offset Pagination totalCountPath attribute for the example response (see below) where the response contains the total number of records in the attribute "totalResults". Next Page URLs are not available for this endpoint, so have to use Offset based pagination.

shubabbar_0-1689598750815.png

From testing various permutations, the Java Script capability does not seem to be enabled for the totalCountPath config attribute, log indicates total count = null for the below tried values.

  • "totalCountPath": "responseText.totalResults"
  • "totalCountPath": "completeResponseMap.totalResults"
  • "totalCountPath": "completeResponseMap.meta.totalResults"
    "totalCountPath": "response.responseText.get(totalResults)"
  • "totalCountPath": "response.responseText.totalResults"
  • "totalCountPath": "${completeResponseMap.get('totalResults')}"
  • shubabbar_1-1689599070349.png

     

Running the account job and monitoring the logs shows that the first 100 records are retrieved but the log also shows "DEBUG rest.RestUtilService - totalCount: null". And the paging stops and does not continue.

If I hard code a value such in the pagination config the logs show and pagination continues to the next batch of accounts, logs:
Got Webservice API Response: [headers:[Date: Mon, 17 Jul 2023 10:55:06 GMT, Content-Type: application/json, Transfer-Encoding: chunked, Connection: keep-alive, Max-Forwards: 20, X-CorrelationID: Id-0a1eb564324b24xyzxyz79ee 0; Id-0a1eb564334bf97e79xyzxyz 0; Id-0a1eb564344b7a6bzyxzyx 0, Cache-Control: no-cache,no-store,must-revalidate, Content-Security-Policy: default-src 'none'; script-src 'none'; object-src 'none'; frame-ancestors 'self'; frame-src 'none'; require-trusted-types-for 'script', HC-TransactionId: Id-0a1eb564324b24e5a6df79ee, Referrer-Policy: origin-when-cross-origin, Via: HiltiCloud, X-Content-Type-Options: nosniff, X-Frame-Options: SAMEORIGIN], responseText:{\"totalResults\":15806,\"itemsPerPage\":100,\"startIndex\":14100,\"schemas\":[\"urn:ietf:params:scim:api:messages:2.0:ListResponse\"],\"Resources\":[{\"schemas\":[\"urn:ietf:params:scim:schemas:core:2.0:User\"],\"id\":\"xyzxyzxyxzyxzyxzy\",\"nickName\":\"DXYZXYZYXZ\",\"active\":true}

8 REPLIES 8

rushikeshvartak
All-Star
All-Star
  • "totalCountPath": "totalResults"

Regards,
Rushikesh Vartak
If this helped you move forward, click 'Kudos'. If it solved your query, select 'Accept As Solution'.

This had failed as well, Rushikesh.

Try below pagination block

"pagination": {
"offset": {
"offsetParam": "startIndex",
"batchParam": "count",
"batchSize": 100,
"totalCountPath": "completeResponseMap.totalResults"
}
}


Regards,
Saathvik
If this reply answered your question, please Accept As Solution and give Kudos to help others facing similar issue.

Hi Saathvik, this has already been tried and is not working as expected. The list of values already tried are mentioned in the main query above. 

What if you hardcode max number ? 100000


Regards,
Rushikesh Vartak
If this helped you move forward, click 'Kudos'. If it solved your query, select 'Accept As Solution'.

Did you try exact same block? Why I am asking this because in the screenshot you shared I see two totalCountPath variables and highlighted block is in totalCountPath as well. 

sk_0-1689877489172.png

 


Regards,
Saathvik
If this reply answered your question, please Accept As Solution and give Kudos to help others facing similar issue.

rushikeshvartak
All-Star
All-Star

Please share full json


Regards,
Rushikesh Vartak
If this helped you move forward, click 'Kudos'. If it solved your query, select 'Accept As Solution'.

ShubhamBabbar
Regular Contributor
Regular Contributor

Thank you Rushikesh and Saathvik for your responses.

The issue has now been resolved.

 

Issue: The API return error 400 in the first call, as it does not support startIndex=0, the offset value for startIndex should be 1 (startIndex=1). Hence, on specifying the the totalCountPath as a fixed value it moves to the next page but was erroring out on using totalCountPath from the response object.

Solution: Initial URL set with '?startIndex=1&count=100' and pagination logic as :-
"pagination": {
"nextUrl": {
"nextUrlPath": "${response?.completeResponseMap?.itemsPerPage<100?null:'https://sample-url.com/Groups?count=100&startIndex='+Math.addExact(response.completeResponseMap.star...)}"
}
}