pagination is not working as expected

[Kumat]

Below is the JSON i use for ImportaccountEntJSON, i added pagination still i don't see account from page 2

{
"accountParams": {
"connection": "userAuth",
"processingType": "SequentialAndIterative",
"call": {
"call1": {
"callOrder": 0,
"stageNumber": 0,
"http": {
"url": "https://api.cloudflare.com/client/v4/accounts/09fd3a16b2b05b606bff7d91b2b9a4d7/members",
"httpHeaders": {
"Authorization": "Bearer ${access_token}",
"Accept": "application/json"
},
"httpContentType": "application/json",
"httpMethod": "GET"
},
"listField": "result",
"keyField": "name",
"colsToPropsMap": {
"name": "user.email~#~char",
"accountID": "user.id~#~char",
"customproperty2": "user.id~#~char",
"customproperty7": "roles[0].id~#~char",
"customproperty31": "STORE#ACC#ENT#MAPPINGINFO~#~char"
}
}
},
"pagination": {
"nextUrl": {
"nextUrlPath": "${response.completeResponseMap.startIndex>response.completeResponseMap.page_page?'null':'https://api.cloudflare.com/client/v4/accounts/09fd3a16b2b05b606bff7d91b2b9a4d7/members?startIndex='+...)}"
}
},
"acctEntMappings": {
"roles": {
"listPath": "roles",
"idPath": "id",
"keyField": "entitlementID",
"importAsEntitlement": false
}
}
},
"entitlementParams": {
"connection": "userAuth",
"processingType": "SequentialAndIterative",
"entTypes": {
"roles": {
"entTypeOrder": 0,
"entTypeLabels": {},
"call": {
"call1": {
"callOrder": 0,
"stageNumber": 0,
"http": {
"url": "https://api.cloudflare.com/client/v4/accounts/09fd3a16b2b05b606bff7d91b2b9a4d7/members",
"httpHeaders": {
"Authorization": "Bearer ${access_token}",
"Accept": "application/json"
},
"httpContentType": "application/json",
"httpMethod": "GET"
},
"listField": "result",
"keyField": "entitlement_value",
"colsToPropsMap": {
"entitlementID": "roles.id~#~char",
"entitlement_value": "roles.id~#~char",
"displayname": "roles.name~#~char",
"customproperty1": "deleted~#~char",
"customproperty2": "created_at~#~char",
"customproperty3": "updated_at~#~char"
},
"disableDeletedEntitlements": true
}
}
}
}
},
"acctEntParams": {
"processingType": "acctToEntMapping"
}
}

There are total of 26 accounts but in saviynt i can see only 21 are being collected.

Here is the postman response.

Please help me find the resolution.

[DixshantValecha]

Hi,

Thank you for providing the JSON configuration and the details. Based on the provided information, I noticed a potential issue in the configuration of the pagination for retrieving accounts. It appears that the nextUrlPath is not defined correctly, which might be causing the missing accounts from page 2.

please modify the nextUrlPath in the JSON configuration for the pagination section as follows:

"nextUrlPath": "${response.completeResponseMap.startIndex > response.completeResponseMap.page_page ? 'null' : 'https://api.cloudflare.com/client/v4/accounts/09fd3a16b2b05b606bff7d91b2b9a4d7/members?startIndex=' + response.completeResponseMap.page_page}"

In the original configuration, it seems there is an issue with the variable substitution within the nextUrlPath. The ... in the original configuration should be replaced with response.completeResponseMap.page_page.

Additionally, you can refer to this forum post as well:-

Rest connector - pagination parameters in ImportAc... - Saviynt Forums - 37742

Please validate and let us know if further details are needed on this.

[Kumat]

I have updated the JSON and still no change in the accounts import count

{
"accountParams": {
"connection": "userAuth",
"processingType": "SequentialAndIterative",
"call": {
"call1": {
"callOrder": 0,
"stageNumber": 0,
"http": {
"url": "https://api.cloudflare.com/client/v4/accounts/09fd3a16b2b05b606bff7d91b2b9a4d7/members",
"httpHeaders": {
"Authorization": "Bearer ${access_token}",
"Accept": "application/json"
},
"httpContentType": "application/json",
"httpMethod": "GET"
},
"listField": "result",
"keyField": "name",
"colsToPropsMap": {
"name": "user.email~#~char",
"accountID": "user.id~#~char",
"customproperty2": "user.id~#~char",
"customproperty7": "roles[0].id~#~char",
"customproperty31": "STORE#ACC#ENT#MAPPINGINFO~#~char"
}
}
},
"pagination": {
"nextUrl": {
"nextUrlPath": "${response.completeResponseMap.startIndex > response.completeResponseMap.page_page ? 'null' : 'https://api.cloudflare.com/client/v4/accounts/09fd3a16b2b05b606bff7d91b2b9a4d7/members?startIndex=' + response.completeResponseMap.page_page}"
}
},
"acctEntMappings": {
"roles": {
"listPath": "roles",
"idPath": "id",
"keyField": "entitlementID",
"importAsEntitlement": false
}
}
},
"entitlementParams": {
"connection": "userAuth",
"processingType": "SequentialAndIterative",
"entTypes": {
"roles": {
"entTypeOrder": 0,
"entTypeLabels": {},
"call": {
"call1": {
"callOrder": 0,
"stageNumber": 0,
"http": {
"url": "https://api.cloudflare.com/client/v4/accounts/09fd3a16b2b05b606bff7d91b2b9a4d7/members",
"httpHeaders": {
"Authorization": "Bearer ${access_token}",
"Accept": "application/json"
},
"httpContentType": "application/json",
"httpMethod": "GET"
},
"listField": "result",
"keyField": "entitlement_value",
"colsToPropsMap": {
"entitlementID": "roles.id~#~char",
"entitlement_value": "roles.id~#~char",
"displayname": "roles.name~#~char",
"customproperty1": "deleted~#~char",
"customproperty2": "created_at~#~char",
"customproperty3": "updated_at~#~char"
},
"disableDeletedEntitlements": true
}
}
}
}
},
"acctEntParams": {
"processingType": "acctToEntMapping"
}
}

I still see 21 accounts.

[DixshantValecha]

Here's the updated JSON please validate.

{
"accountParams": {
"connection": "userAuth",
"processingType": "SequentialAndIterative",
"call": {
"call1": {
"callOrder": 0,
"stageNumber": 0,
"http": {
"url": "https://api.cloudflare.com/client/v4/accounts/09fd3a16b2b05b606bff7d91b2b9a4d7/members",
"httpHeaders": {
"Authorization": "Bearer ${access_token}",
"Accept": "application/json"
},
"httpContentType": "application/json",
"httpMethod": "GET"
},
"listField": "result",
"keyField": "name",
"colsToPropsMap": {
"name": "user.email~#~char",
"accountID": "user.id~#~char",
"customproperty2": "user.id~#~char",
"customproperty7": "roles[0].id~#~char",
"customproperty31": "STORE#ACC#ENT#MAPPINGINFO~#~char"
}
}
},
"pagination": {
"nextUrl": {
"nextUrlPath": "${response.completeResponseMap.page_page < response.completeResponseMap.total_count ? 'https://api.cloudflare.com/client/v4/accounts/09fd3a16b2b05b606bff7d91b2b9a4d7/members?startIndex=' + (response.completeResponseMap.page_page + 1) : 'null'}"
}
},
"acctEntMappings": {
"roles": {
"listPath": "roles",
"idPath": "id",
"keyField": "entitlementID",
"importAsEntitlement": false
}
}
},
"entitlementParams": {
"connection": "userAuth",
"processingType": "SequentialAndIterative",
"entTypes": {
"roles": {
"entTypeOrder": 0,
"entTypeLabels": {},
"call": {
"call1": {
"callOrder": 0,
"stageNumber": 0,
"http": {
"url": "https://api.cloudflare.com/client/v4/accounts/09fd3a16b2b05b606bff7d91b2b9a4d7/members",
"httpHeaders": {
"Authorization": "Bearer ${access_token}",
"Accept": "application/json"
},
"httpContentType": "application/json",
"httpMethod": "GET"
},
"listField": "result",
"keyField": "entitlement_value",
"colsToPropsMap": {
"entitlementID": "roles.id~#~char",
"entitlement_value": "roles.id~#~char",
"displayname": "roles.name~#~char",
"customproperty1": "deleted~#~char",
"customproperty2": "created_at~#~char",
"customproperty3": "updated_at~#~char"
},
"disableDeletedEntitlements": true
}
}
}
}
},
"acctEntParams": {
"processingType": "acctToEntMapping"
}
}

[Kumat]

I still see only 21 accounts no change in the accounts count, accounts from page 2 are still not being collected.

[DixshantValecha]

Could you please provide me with the latest logs/console logs for this?

[Kumat]

Please find the attached log file.

[Kumat]

Here is the full response

[Edited attached text file to mask PII information by a moderator]

[sk]

Can you share you API response for call: "https://api.cloudflare.com/client/v4/accounts/09fd3a16b2b05b606bff7d91b2b9a4d7/members",

I believe you may need to use below pagination logic as per screenshot shared. But I wanted to double check by looking at your complete response 

"pagination": {
"page": {
"pageSizeParam": "per_page",
"pageSize": 20,
"pageRecordCount": "completeResponseMap.<path to count>.count",
"pageNumberParam": "page",
"totalCountPath": "completeResponseMap.<path to total count>.total_count",
"firstPageNumber": 1
}
}

[rushikeshvartak]

Please remove client emails

[DixshantValecha]

Thanks for the details,we are checking on this.

[DixshantValecha]

Hi @Kumat,

We kindly request that you initiate the process of raising a Freshdesk ticket for this particular matter, as it will require further triage and investigation.