04/21/2023 01:51 PM
Hello,
I am trying to build pagination logic based on the offset and the response below.
"pagination": { "offset": { "offsetParam": "offset", "batchParam": "limit", "batchSize": 50, "totalCountPath": "5000" } }
I want to know what will be the "offsetparam" and "batchsizeparam" in my case?
04/21/2023 05:31 PM
How will you pass the pagination parameters in the URL to get the next page result?
Example:
Page 1 (Pulls first 200 records): https://myhost.mydomain.com/api/users?offset=0&count=200
Page 2 (Pulls Next 200 records): https://myhost.mydomain.com/api/users?offset=200&count=200
If your offset param in the url is offset and count is for the batch size as per the example URL above, then your pagination block will be something like below.
"pagination": { "offset": { "offsetParam": "offset", "batchParam": "count", "batchSize": 200, "totalCountPath": "5000" } }
-Siva
04/21/2023 08:02 PM
Hi Siva,
The url doesn’t have any parameter. The only parameters are the ones in the response. The example you provided does not work in my case. I am passing offset in the http params in the json. Other than that they have userlimit parameter which can be used to fetch max 1000 records. But both these work only when used in http params and not in url.
04/22/2023 04:59 AM
Can you share the snip of your request or sample curl to pull 1000 records? Make sure you hide your app specific information / url before sharing the same.
04/22/2023 01:37 PM - edited 04/22/2023 01:39 PM
Please find below the details.
Postman Request
Postman Response
ImportAcctEntJson
04/24/2023 08:23 AM
Can you try with the below and see if it works.
"pagination": {
"offset": {
"offsetParam": "offset",
"batchParam": "userlimit",
"batchSize": 1000,
"totalCountPath": "5000"
}
}
In case the above does work, can you try to mention the totalCountPath as "completeResponseMap.totalResults" and see if it works. If this does work, you will not need to define the total record count manually and the application will auto identify the total number of records.
04/24/2023 10:29 AM
@SB - I see in the logs it is iterating 5 times until it reaches offset =6000 and then print paginationParam.doBreak: true.
But in each iteration it is printing "pullObjectsByRest - objectList.size : 200\n"".
I don't see any new accounts being imported.
04/24/2023 10:40 AM
In that case can you define "batchSize": 200
04/24/2023 11:15 AM
I tried but it is just taking that as the offset and setting it to 200 and calling the api 25 times as opposed to earlier 5 times (batch size as 1000). But still no new account is getting imported.
04/24/2023 11:36 AM
Can you check in the logs to see the total accounts the job is importing. You should find the value close to the below string.
isImportSuccess: true
04/24/2023 12:51 PM
Its printing 5000 as the totalcount in each iteration. I don't see any other parameter as total accounts.
Inside importAccountsFull:persistAccounts, pptTypeSep : ~#~\n","stream":"stdout","time":"2023-04-24T18:05:43.422014833Z"}"
"ecm-worker","2023-04-24T18:05:43.934+00:00","{"log":"2023-04-24 18:05:43,930 [quartzScheduler_Worker-1] DEBUG rest.RestUtilService - totalCount: 5000\n","stream":"stdout","time":"2023-04-24T18:05:43.930696268Z"}"
"ecm-worker","2023-04-24T18:05:43.934+00:00","{"log":"2023-04-24 18:05:43,930 [quartzScheduler_Worker-1] DEBUG rest.RestUtilService - offset: 5200\n","stream":"stdout","time":"2023-04-24T18:05:43.930726512Z"}"
"ecm-worker","2023-04-24T18:05:43.934+00:00","{"log":"2023-04-24 18:05:43,930 [quartzScheduler_Worker-1] DEBUG rest.RestUtilService - paginationParam.doBreak: true
"ecm-worker","2023-04-24T18:05:43.934+00:00","{"log":"2023-04-24 18:05:43,934 [quartzScheduler_Worker-1] DEBUG services.ImportUtilityService - isImportSuccess: true
04/25/2023 03:17 AM
@SB - As per the postman snip Piyush shared, the pagination parameters are sent in body and not as a query param in URL. When the below block is updated, Saviynt will update the query param in URL and wouldn't update the attribute values in body of the request to pull next set of values. Is there a way to send body params for subsequent calls in importjson?
"pagination": {
"offset": {
"offsetParam": "offset",
"batchParam": "userlimit",
"batchSize": 1000,
"totalCountPath": "5000"
}
}
04/25/2023 02:20 PM
I am not sure if there a way to achieve this but let me check this and will update you.
04/26/2023 08:49 AM
I did verify and this is currently not supported, though it is already in our future road map and would be available in next suitable upgrade.
04/26/2023 11:56 AM
04/26/2023 12:02 PM
This pagination is currently not supported.
04/27/2023 09:28 AM
@SB , Is there way to include a script in the json to check the "hasMore" response parameter?
The current tool that we are migrating from has below logic to check the pagination.
TERMINATE_IF $response.hasMore$ == false
$offset$ = $response.offset$ + $response.userlimit$
$request.offset$ = $offset$
05/01/2023 10:33 AM - edited 05/01/2023 10:33 AM
Since your pagination response is sent in the body, it will not work.