REST Connector - Account import failing during Pagination

[rohitkumarraj]

Team,

I am working on REST connector and facing an issue where account Import are failing for an application due to adding some filter on account import params.

The original URL working fine in Postman.

Detail description: 

The normal GET user URL is working absolutely fine BUT when I'm trying to filter and import based on employee type then the URL not getting passed and getting some Illegal character error in logs.

Working URL: https://xyz.com/api/v2/user/all?pagenumber=1

Failing URL: https://xyz.com/api/v2/user/all?pagenumber=1&filter=(type eq EMP or type eq PAR)

Above highlighted part is what been added to fetch user based on employee type. This syntax working fine in Postman but failing in Saviynt.

- Postman snapshot of account import

Error in Logs:

We fixed this issue by making some change in the URL such a way that all the whitespaces coming after filter is replaced with + Hence the updated working URL is:

Failing URL: https://xyz.com/api/v2/user/all?pagenumber=1&filter=(type eq EMP or type eq PAR)

(transformed)Working URL: https://xyz.com/api/v2/user/all?pagenumber=1&filter=(type+eq+'EMP'+or+type+eq+'PAR') 

BUT the problem now is , Since the updated url is hardcoded into connection hence its working fine for pagenumber=1 BUT its again failing for pagenumber 2 and so on going back to original url(coz of nextURl response). In nutshell the import URL is not working in case of Pagination.

2022-09-30/08:08:29.804 [{}] [quartzScheduler_Worker-26] DEBUG rest.RestProvisioningService - Got Webservice API Response: [headers:[Date: Fri, 30 Sep 2022 08:08:29 GMT, Server: Apache, Vary: Origin,Access-Control-Request-Method,Access-Control-Request-Headers, X-Content-Type-Options: nosniff, X-XSS-Protection: 1; mode=block, Cache-Control: no-cache, no-store, max-age=0, must-revalidate, Pragma: no-cache, Expires: 0, X-Frame-Options: DENY, Content-Type: application/json, Keep-Alive: timeout=5, max=100, Connection: Keep-Alive, Transfer-Encoding: chunked], responseText:{"itemsPerPage":500,"startIndex":1,"totalResults":86461,"firstUri":"/api/v2/user/all?pagenumber=1&filter=(type eq 'EMP' or type eq 'PAR')","selfUri":"/api/v2/user/all?pagenumber=1&filter=(type eq 'EMP' or type eq 'PAR')","nextUri":"/api/v2/user/all?pagenumber=2&filter=(type eq 'EMP' or type eq 'PAR')","lastUri":"/api/v2/user/all?pagenumber=0&filter=(type eq 'EMP' or type eq 'PAR')","resources":[{"id":"E78BEA55-5118-ED11-BD6E-0003FF173A4A","name":{"firstname":"Jonathan","lastname":"Iles"},"username":"ilesj1"

Import Account JSON used:

{
"accountParams": {
"connection": "acctAuth",
"processingType": "SequentialAndIterative",
"statusAndThresholdConfig": {
"statusColumn": "customproperty3",
"activeStatus": [
"true",
"active",
"Active",
1
],
"deleteLinks": true,
"accountThresholdValue": 1000,
"correlateInactiveAccounts": true,
"inactivateAccountsNotInFile": true,
"deleteAccEntForActiveAccounts": true
},
"includeExistingInActiveAccounts": true,
"call": {
"call1": {
"callOrder": 0,
"stageNumber": 0,
"http": {
"url": "https://xyz.com/api/v2/user/all?pagenumber=1&filter=(type+eq+'EMP'+or+type+eq+'PAR')",
"httpHeaders": {
"contentType": "application/json",
"Authorization": "${access_token}",
"Accept": "application/json"
},
"httpContentType": "application/json",
"httpMethod": "GET"
},
"successResponses": {
"statusCode": [
200,
201,
200,
204
]
},
"unsuccessResponses": {
"statusCode": [
400,
403,
401,
404,
409,
501,
500
]
},
"listField": "resources",
"keyField": "accountID",
"statusConfig": {
"active": "Active",
"inactive": "Inactive"
},
"colsToPropsMap": {
"accountID": "id~#~char",
"name": "username~#~char",
"displayName": "name.firstname~#~char",
"customproperty1": "name.firstname~#~char",
"customproperty2": "name.lastname~#~char",
"customproperty3": "status~#~char",
"status": "status~#~char",
"customproperty10": "type~#~char",
"customproperty31": "STORE#ACC#ENT#MAPPINGINFO~#~char"
},
"pagination": {
"nextUrl": {
"nextUrlPath": "${(response?.completeResponseMap?.nextUri==null)?null:'xyz.com'+ response.completeResponseMap.nextUri}"
}
}
}
}
}
}

Anyone has any input on this? How can we make this work in case of pagination?

Thanks,
Rohit

[rushikeshvartak]

Can you try url as encoded url

https://xyz.com/api/v2/user/all?pagenumber=1&filter=(type%20eq%20EMP%20or%20type%20eq%20PAR)

[rohitkumarraj]

Hi Rushikesh - Had tried with url encoding but that's not helpful here. Because one time we can encode and put that url in import JSON. How about next time? As Saviynt is reading nextURI for next page(pagination), anyways to automate this for all the pages? any functions or something which Saviynt supports for this?

Thanks,
Rohit

[rushikeshvartak]

Did you check example 8 from documentation 

https://saviynt.freshdesk.com/support/solutions/articles/43000521736-rest-connector-guide%C2%A0 

[Vedanth_BK]

Hi Rohit,

The mentioned URL contains spaces and must be encoded before being invoked. Rest connector allows us to use replace functions in the pagination parameters. I've attached a sample below. Hope it helps.

Sample:

"pagination": {
"nextUrl": {
"nextUrlPath": "${(response?.completeResponseMap?.nextUri==null)?null:'xyz.com'+ response.completeResponseMap.nextUri.replace(' ','%20')}"
}
}