Rest connector Offset Pagination

[Paul_Meyer]

For the Rest connector, how do I configure the Offset Pagination totalCountPath attribute for the example response (see below) where the response contains the total number of records in the attribute "@odata.count". Next Page URLs are not available for this endpoint, so have to use Offset based pagination.

From testing various permutations, the Java Script capability does not seem to be enabled for the totalCountPath config attribute, so doing ${response.completeResponseMap.get('@odata.count')} does not function as the log indicates total count = null.

For example:

{

    "@odata.context": "URL/data/$metadata#SystemUsers",

    "@odata.count": 790,

    "value": [

        {

            "@odata.etag": "xyz",

            "UserID": "abc",

},

{

...

}

]

}

}

[sahajranajee]

Hello,

The REST connector guide has a few examples on similar use cases on Offset Pagination. You can check them out here:
https://saviynt.freshdesk.com/support/solutions/articles/43000521736-rest-connector-guide

[Paul_Meyer]

The guide does not provide an example of how to retrieve the value for an attribute such as "@odata.count".

[sahajranajee]

Hi Paul,

Can you try :
"totalCountPath": "${completeResponseMap.get('@odata.count')}"

[Paul_Meyer]

Thanx for the reply. I'll log a support call.

I've configured as such:

"pagination": {
   "offset": {
      "offsetParam": "$skip",
      "batchParam": "$top",
      "batchSize": 10,
      "totalCountPath": "${completeResponseMap.get('@odata.count')}"
   }
}

Running the account job and monitoring the logs shows that the first 10 records are retrieved but the log also shows "DEBUG rest.RestUtilService - totalCount: null". And the paging stops and does not continue.

If I hard code a value such as 20 in the pagination config the logs show

    DEBUG rest.RestUtilService - totalCount: 20

    DEBUG rest.RestUtilService - offset: 20

Pagination continues to the second batch of 10 accounts.

[sahajranajee]

Hello,

Try a couple of the following combinations as well :

"totalCountPath": "completeResponseMap.get('@odata.count')",

"totalCountPath": "completeResponseMap.@odata.count",

[Paul_Meyer]

Thank you for the suggestions.

I've tried those combinations and they all result in "DEBUG rest.RestUtilService - totalCount: null".

[Rajesh-R · Accepted solution]

Hi Paul,

I think the following piece of information will be useful to you.

Field name with a dot(.) in the API Response is not Mapped to EIC Attributes

Cause: This error might occur if the field name is specified with a dot(.).

Resolution: To resolve this issue, specify the field name with the separator ~dot#. For example, replace odata.error with odata~dot#error.

Could you try this out and check if this resolves your issue?

[Paul_Meyer]

Rajesh,

Thank you very much.

The following pagination config works:

"pagination": {
   "offset": {
     "offsetParam": "$skip",
     "batchParam": "$top",
     "batchSize": 10000,
     "totalCountPath": "completeResponseMap.@odata~dot#count"
   }
}

For the D365 F&O connector (REST_Dynamics365(REST)), I've replaced the default pagination configs in the ImportAccountEntJSON config setting with the above, as well as appended "?%24count=true" to all the "url" config attributes. The default totalCountPath attribute value is set to 999999999 for all calls.