We are delighted to share our new EIC Delivery Methodology for efficiently managing Saviynt Implementations and delivering quick time to value. CLICK HERE.

Pagination using offset Based resposne

piyushm
Regular Contributor II
Regular Contributor II

Hello,

I am trying to build pagination logic based on the offset and the response below.

 

"totalResults""3762",
    "offset""0",
    "count""200",
    "hasMore"true
 
Based on the connector guide, below are the attributes  to be used.
"pagination": {
    "offset": {
      "offsetParam": "offset",
      "batchParam": "limit",
      "batchSize": 50,
      "totalCountPath": "5000"
    }
  }

I want to know what will be the "offsetparam" and "batchsizeparam" in my case? 

17 REPLIES 17

Sivagami
Valued Contributor
Valued Contributor

How will you pass the pagination parameters in the URL to get the next page result?

Example: 

Page 1 (Pulls first 200 records): https://myhost.mydomain.com/api/users?offset=0&count=200

Page 2 (Pulls Next 200 records): https://myhost.mydomain.com/api/users?offset=200&count=200

If your offset param in the url is offset and count is for the batch size as per the example URL above, then your pagination block will be something like below.

"pagination": {
    "offset": {
      "offsetParam": "offset",
      "batchParam": "count",
      "batchSize": 200,
      "totalCountPath": "5000"
    }
  }

-Siva

piyushm
Regular Contributor II
Regular Contributor II

Hi Siva,

The url doesn’t have any parameter. The only parameters are the ones in the response. The example you provided does not work in my case. I am passing offset in the http params in the json. Other than that they have userlimit parameter which can be used to fetch max 1000 records. But both these work only when used in http params and not in url. 

Sivagami
Valued Contributor
Valued Contributor

Can you share the snip of your request or sample curl to pull 1000 records? Make sure you hide your app specific information / url before sharing the same.

piyushm
Regular Contributor II
Regular Contributor II

Please find below the details.

Postman Request

piyushm_0-1682195357994.png

 

Postman Response

piyushm_1-1682195434599.png

 

ImportAcctEntJson

piyushm_2-1682195802353.png

 

SB
Saviynt Employee
Saviynt Employee

Can you try with the below and see if it works. 

"pagination": {
"offset": {
"offsetParam": "offset",
"batchParam": "userlimit",
"batchSize": 1000,
"totalCountPath": "5000"
}
}

In case the above does work, can you try to mention the totalCountPath as "completeResponseMap.totalResults" and see if it works. If this does work, you will not need to define the total record count manually and the application will auto identify the total number of records.


Regards,
Sahil

piyushm
Regular Contributor II
Regular Contributor II

@SB - I see in the logs it is iterating 5 times until it reaches offset =6000 and then print paginationParam.doBreak: true.

But in each iteration it is printing "pullObjectsByRest - objectList.size : 200\n"".

I don't see any new accounts being imported.

SB
Saviynt Employee
Saviynt Employee

In that case can you define "batchSize": 200


Regards,
Sahil

piyushm
Regular Contributor II
Regular Contributor II

I tried but it is just taking that as the offset  and setting it to 200 and calling the api 25 times as opposed to earlier 5 times (batch size as 1000). But still no new account is getting imported.

SB
Saviynt Employee
Saviynt Employee

Can you check in the logs to see the total accounts the job is importing. You should find the value close to the below string.

isImportSuccess: true


Regards,
Sahil

piyushm
Regular Contributor II
Regular Contributor II

Its printing 5000 as the totalcount in each iteration. I don't see any other parameter as total accounts.

Inside importAccountsFull:persistAccounts, pptTypeSep : ~#~\n","stream":"stdout","time":"2023-04-24T18:05:43.422014833Z"}"
"ecm-worker","2023-04-24T18:05:43.934+00:00","{"log":"2023-04-24 18:05:43,930 [quartzScheduler_Worker-1] DEBUG rest.RestUtilService - totalCount: 5000\n","stream":"stdout","time":"2023-04-24T18:05:43.930696268Z"}"
"ecm-worker","2023-04-24T18:05:43.934+00:00","{"log":"2023-04-24 18:05:43,930 [quartzScheduler_Worker-1] DEBUG rest.RestUtilService - offset: 5200\n","stream":"stdout","time":"2023-04-24T18:05:43.930726512Z"}"
"ecm-worker","2023-04-24T18:05:43.934+00:00","{"log":"2023-04-24 18:05:43,930 [quartzScheduler_Worker-1] DEBUG rest.RestUtilService - paginationParam.doBreak: true

"ecm-worker","2023-04-24T18:05:43.934+00:00","{"log":"2023-04-24 18:05:43,934 [quartzScheduler_Worker-1] DEBUG services.ImportUtilityService - isImportSuccess: true

Sivagami
Valued Contributor
Valued Contributor

@SB - As per the postman snip Piyush shared, the pagination parameters are sent in body and not as a query param in URL. When the below block is updated, Saviynt will update the query param in URL and wouldn't update the attribute values in body of the request to pull next set of values. Is there a way to send body params for subsequent calls in importjson?

"pagination": {
"offset": {
"offsetParam": "offset",
"batchParam": "userlimit",
"batchSize": 1000,
"totalCountPath": "5000"
}
}

SB
Saviynt Employee
Saviynt Employee

I am not sure if there a way to achieve this but let me check this and will update you.


Regards,
Sahil

SB
Saviynt Employee
Saviynt Employee

I did verify and this is currently not supported, though it is already in our future road map and would be available in next suitable upgrade.


Regards,
Sahil

piyushm
Regular Contributor II
Regular Contributor II

Hi @SB @Sivagami  . In this case, is the pagination  possible at all?

SB
Saviynt Employee
Saviynt Employee

This pagination is currently not supported.


Regards,
Sahil

piyushm
Regular Contributor II
Regular Contributor II

@SB , Is there way to include a script in the json to check the "hasMore" response parameter?

The current tool that we are migrating from has below logic to check the pagination.

TERMINATE_IF $response.hasMore$ == false

$offset$ = $response.offset$ + $response.userlimit$

$request.offset$ = $offset$

SB
Saviynt Employee
Saviynt Employee

Since your pagination response is sent in the body, it will not work.


Regards,
Sahil