Rest connector - pagination parameters in ImportAccountEntJson

[anandinguva]

I have provided the JSON and the response from Postman below. With the JSON configured, when the Pagination parameter is included, the API throws 401 error in the Saviynt logs. Without the Pagination section specified, the Saviynt job is able to fetch only 100 accounts. Please help us understand how to FRAME THE PAGINATION parameter based on the POSTMAN response shown below..

ImportAccountEntJSON

{

"accountParams": {

  "connection": "acctAuth",

  "processingType": "SequentialAndIterative",

    "call": {

      "call1": {

        "callOrder": 0,

        "stageNumber": 0,

            "http": {

            "url": "https://xxxx.duosecurity.com/admin/v1/users",

            "basicUrl":"xxxx.duosecurity.com",

              "hostUrl":"/admin/v1/users",

            "httpContentType": "application/json",

            "httpMethod": "GET"

            },

"listField": "response",

"keyField": "accountID",

          "colsToPropsMap": {

          "accountID": "user_id~#~char",

          "name": "username~#~char",

          "status": "status~#~char",

"customproperty1": "status~#~char"

          }

,

          "pagination": {

                    "offset": {

                        "offsetParam": "offset",

                        "batchParam": "limit",

                        "batchSize": 100,

                        "totalCountPath": "metadata.total_objects"    

                    }

                }

        }

    }

}

}

POSTMAN response for GET call

{

    "metadata": {

        "next_offset": 100,

        "total_objects": 10583

    },

    "response": [

        {

            "alias1": null,

            "alias2": null,

            "alias3": null,

            "alias4": null,

            "aliases": {},

            "created": 1610690177,

            "desktoptokens": [],

            "email": "",

            "firstname": null,

            "groups": [],

            "is_enrolled": true,

            "last_directory_sync": null,

            "last_login": 1611917024,

            "lastname": null,

            "notes": "",

            "phones": [

                {

                    "activated": false,

                    "capabilities": [

                        "sms"

                    ],

                    "extension": "",

                    "last_seen": "2070-10-19T08:43:17",

                    "model": "xxxx xxxx",

                    "name": "",

                    "number": "+xxxxxx",

                    "phone_id": "xxxxxx",

                    "platform": "xxxx",

                    "sms_passcodes_sent": true,

                    "type": "Mobile"

                }

            ],

            "realname": "",

            "status": "active",

            "tokens": [],

            "u2ftokens": [],

            "user_id": "xxxxxx",

            "username": "xxxx@abc.com",

            "webauthncredentials": []

        }

  ]

}

[khalidakhter]

Hi,

To read the response of previous page, The REST Connector provides a binding called completeResponseMap in the pagination parameter.

Please update the value for totalCountPath to completeResponseMap.metadata.total_objects and run the import. 

[anandinguva]

Khalid - It still gives me 401 error. I included totalcountpath as you suggested, but it fails with 401 error. Without the pagination parameter in the JSON, the job runs successfully but only brings back the default number of records (in Duo's case it's 100). Pasting the latest json here:

{
"accountParams": {
"connection": "acctAuth",
"processingType": "SequentialAndIterative",
"call": {
"call1": {
"callOrder": 0,
"stageNumber": 0,
"http": {
"url": "https://xxxx.duosecurity.com/admin/v1/users",
"basicUrl":"xxxx.duosecurity.com",
"hostUrl":"/admin/v1/users",
"httpContentType": "application/json",
"httpMethod": "GET"
},
"listField": "response",
"keyField": "accountID",
"colsToPropsMap": {
"accountID": "user_id~#~char",
"name": "username~#~char",
"status": "status~#~char",
"customproperty1": "status~#~char"
},
"pagination":
{
"offset":{
"offsetParam": "offset",
"batchParam": "limit",
"batchSize": 10,
"totalCountPath": "completeResponseMap.metadata.total_objects"
}
}
}
}
}
}

[khalidakhter]

Hi,

As this is for the DUO application, I do not think pagination will work because of the dynamic URL. I will recommend using the OOTB Duo connection type for import. Let me know if you have any concerns regarding the DUO Connector.

[anandinguva]

Khalid - I dont understand the dynamic URL. How are other rest implementations working which have similar response - where it responds back with an offset value and the total resultcount. How are those different? We cannot use OOTB Duo connector since it reconciles Admin and User accounts. We are only managing User accounts and dont want to reconcile Admin accounts, is that possible? 

[khalidakhter]

@anandinguva The issue is not with the API response but the authentication mechanism which is quite different from other applications.
In offset pagination-based import, the offset parameters get appended with the current API URL and it gets updated in the next API calls as well to traverse all the records.

In this case, DUO authentication evaluates the token only for the URL mentioned in the HTTP parameter of the import JSON. But while calling the actual URL, the final URL gets appended with pagination parameters. Hence, the API authorization will get failed due to a URL mismatch.

[anandinguva]

@khalidakhter  Are you saying this is an issue with "Saviynt's" REST connector in handling Duo REST API? If that's the case, I will raise a defect ticket. Authentication/authorization should not be an issue since it's anyways handled through ConnectionJSON and its working fine with the "actual URL". 

As you can see below from postman screenshot, I appended the offset and limit parameters, and it's not throwing a 401 error.