Rest connector offset paging in acctEntParams

[Paul_Meyer]

Is offset paging supported in the acctEntParams configuration or is there an issue with the paging config?

I have a REST connector configured with the below paging config. During recon task execution the logs show that the "SecurityUserRoles" endpoint is not being called according to the paging config. The endpoint is only called once.

The record count ("@odata.count") shows 54k total entries, with the offset paging it is expected to make multiple calls in batches of 10k.

Example response:

{
    "@odata.context": "https://host/data/$metadata#SecurityUserRoles",
    "@odata.count": 54775,
    "value": [
        {
            "@odata.etag": "W/\"JzEsNTYzNzE0NDU3Nic=\"",
            "UserId": "User1",
            "SecurityRoleIdentifier": "SYSTEMUSER",
            "AssignmentStatus": "Enabled",
            "AssignmentMode": "Manual",
            "SecurityRoleName": "System user",
            "UserLicenseType": "Universal"
        },
...
    ]
  }
}

acctEntParams config::

"acctEntParams": {
     "Roles": {
        "acctKeyField": "accountID",
        "entKeyField": "entitlementID",
        "call": {
          "call1": {
            "connection": "userAuth",
            "callOrder": 0,
            "stageNumber": 0,
            "http": {
              "httpHeaders": {
                "Authorization": "${access_token}"
              },
              "url": "https://host/api/Data/SecurityUserRoles?%24count=true",
              "httpContentType": "application/json",
              "httpMethod": "GET"
            },
            "listField": "value",
            "acctIdPath": "UserId",
            "entIdPath": "SecurityRoleIdentifier",
            "pagination": {
              "offset": {
                "offsetParam": "$skip",
                "batchParam": "$top",
                "batchSize": 10000,
                "totalCountPath": "completeResponseMap.@odata~dot#count"
              }
            }
          }
        }
      }
    }

[sahil]

You need to specify ${id} to iterate the API calls for each entitlement value. You can refer to the below REST Connector guide (Examples for JSON Construction - page) for an example of using pagination in acctEntParams .

https://docs.saviyntcloud.com/bundle/REST-v23x/page/Content/Examples-for-JSON-Construction.htm

[Paul_Meyer]

@sahil 

Thank you for the response. Is it possible to use "processingType": "http" for acctEntParams, instead of iterating each entitlement (httpEntToAcct) or each account (httpAcctToEnt)?

[rushikeshvartak]

httpEntToAcct will only iterate 

[Paul_Meyer]

Managed to get httpAcctToEnt working. The access recon job takes a much longer time to complete as a result and the job ends with the following SQL exception:

2023-05-17/09:17:35.399 [{}] [quartzScheduler_Worker-1] DEBUG rest.RestProvisioningService - pullObjectsByRest - responseStatusCode ::200
2023-05-17/09:17:35.400 [{}] [quartzScheduler_Worker-1] DEBUG rest.RestProvisioningService - Entered getResponseHeaders method
2023-05-17/09:17:35.400 [{}] [quartzScheduler_Worker-1] DEBUG rest.RestProvisioningService - responseError : null
2023-05-17/09:17:35.400 [{}] [quartzScheduler_Worker-1] DEBUG rest.RestProvisioningService - isAuthError: false
2023-05-17/09:17:35.400 [{}] [quartzScheduler_Worker-1] DEBUG rest.RestProvisioningService - pullObjectsByRest - responseMap.size : 3
2023-05-17/09:17:35.400 [{}] [quartzScheduler_Worker-1] DEBUG rest.RestProvisioningService - pullObjectsByRest - objectList.size : 0
2023-05-17/09:17:35.401 [{}] [quartzScheduler_Worker-1] DEBUG rest.RestProvisioningService - Exception in importAccessFullBySequentialAndIterative :
java.sql.SQLException: Connection has already been closed.
at com.saviynt.provisoning.rest.RestProvisioningService.processAccountEntitlementParams(RestProvisioningService.groovy:5944)
at com.saviynt.provisoning.rest.RestProvisioningService.importAccessFullBySequentialAndIterative(RestProvisioningService.groovy:298)
at com.saviynt.provisoning.rest.RestProvisioningService.importAccessFull(RestProvisioningService.groovy:166)
at com.saviynt.provisoning.rest.RestProvisioningService.doImport(RestProvisioningService.groovy:140)
at com.saviynt.ecm.integration.ExternalConnectionCallService.invokeExternalMethod(ExternalConnectionCallService.groovy:210)
at SapImportJob.execute(SapImportJob.groovy:104)
at org.quartz.core.JobRunShell.run(JobRunShell.java:199)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:546)
2023-05-17/09:17:35.403 [{}] [quartzScheduler_Worker-1] DEBUG rest.RestUtilService - Writing job history to import logs.
2023-05-17/09:17:35.403 [{}] [quartzScheduler_Worker-1] DEBUG rest.RestUtilService - Number of log entries to be written : 4

Are there any SQL timeout settings that can configured to prevent this exception?

[sahil]

You can try the timeout configuration and see if it helps resolve the issue.

{
  "showLogs": true,
  "connectionTimeoutConfig": {
    "timeout": 300,
    "retryWait": 500
  }
}

https://docs.saviyntcloud.com/bundle/REST-v23x/page/Content/Configuring-Provisioning-Tasks-and-Conne...