Pagination not working in HR Import

[Caesrob]

We have made a custom pagination to check for next pages in our HR import. We are using a modified version of the headers to evaluate the next URL from this documentation page: 

https://docs.saviyntcloud.com/bundle/REST-v23x/page/Content/Developers-Handbook.htm

The header we are using to get the next page looks like this and is called X-Pagination:

First page:

{"CurrentPage":1,"PageSize":100,"CurrentPageCount":100,"TotalCount":11864,"TotalPages":119,"PreviousPageLink":null,"NextPageLink":"http://XXX/Studenten/Pagination?pageNumber=2&pageSize=100"}

Second page:

{"CurrentPage":2,"PageSize":100,"CurrentPageCount":100,"TotalCount":11864,"TotalPages":119,"PreviousPageLink":"http://XXX/Studenten/Pagination?pageNumber=1&pageSize=100","NextPageLink":"http://XXX/Studenten/Pagination?pageNumber=3&pageSize=100"}

Last page:

{"CurrentPage":119,"PageSize":100,"CurrentPageCount":64,"TotalCount":11864,"TotalPages":119,"PreviousPageLink":"http://XXX/Studenten/Pagination?pageNumber=118&pageSize=100","NextPageLink":null}

Our nextUrlPath code is this:

"nextUrlPath": "${headers?.X-Pagination?.split(',')[6]?.contains('http')?headers.X-Pagination.split(',')[6].replace('\"NextPageLink\":\"','').replace('\"}','').trim():null}"

This code checks if the 6th element of the X-Pagination header (NextPageLink) contains 'http' as the last page does not have this. Then we split on the 6th element again and replace the things we don't need for the URL. A sample pagination URL would be:

http://XXX/Studenten/Pagination?pageNumber=7&pageSize=100"

The error we are receiving:

It seems we get this error because it can't parse something in the received URL. We have no idea what this is however.

If anyone can help us out, it would be greatly appreciated.

[SB]

Can you try with the below

"pagination": {
"nextUrl": {
"nextUrlPath": "${headers?.NextPageLink==null?null:headers.NextPageLink}"
}
}

[Caesrob]

Hello,

The below pagination makes the job run successful, but the output is still only 100 users. I don't see a specific error in the logs for the pagination, but it is not going through every page.

[SB]

Can you share the complete job logs. Do ensure to update ConfigJSON param in the REST connection with value {"showLogs":true} before collecting the logs. This is to enable additional logging

[Caesrob]

Hello,

These are the only logs I can see relating to pagination when I use your implementation.

I added the full logs if you want to check them out.

[SB]

Can you share the complete JSON you are using.

[Caesrob]

Hello,

Sure. Here is the complete JSON:

[SB]

Can you share as text file instead.

[Caesrob]

{
"connection": "acctAuth",
"successResponses": {
"statusCode": [
200,
201,
202,
203,
204,
205
]
},
"showLogs": true,
"url": "https://XXX/Studenten/Pagination?pageNumber=1&pageSize=1000",
"httpMethod": "GET",
"httpHeaders": {
"Accept": "*/*",
"Authorization" : "${access_token}"
},
"colsToPropsMap": {
"username": "eMailMetDomein~#~char",
"displayname": "#CONST#${String attr1 = response.roepnaam; String attr2 = response.voornaam; String attr3 = response.achternaam; if(attr1 != null){return attr1+ ' ' +attr3;}else{return attr2+ ' ' +attr3;}}~#~char",
"customproperty1": "stamnummer~#~char",
"customproperty2": "taal~#~char",
"firstname": "voornaam~#~char",
"preferedfirstname": "roepnaam~#~char",
"lastname": "achternaam~#~char",
"customproperty6": "eMail~#~char",
"email": "eMailMetDomein~#~char",
"secondaryEmail": "eMailPrive~#~char",
"secondaryPhone": "gsmNummer~#~char",
"customproperty3": "pingPingTag~#~char",
"customproperty4": "nfcCardId~#~char",
"customproperty5": "activatieWachtwoord~#~char",
"customproperty60": "inschrijvingen~#~longtext"
},
"pagination": {
"nextUrl": {
"nextUrlPath": "${headers?.NextPageLink==null?null:headers.NextPageLink}"
}
}}

We have also tried:

"pagination": {
            "page": {
              "pageSizeParam": "pageSize",
              "pageSize": 10,
              "pageRecordCount": "10",
              "pageNumberParam": "pageNumber",
              "totalCountPath": "headers.X-Pagination.TotalCount",
              "firstPageNumber": 1
            }
          }

 This returns an error in the logs:

[rushikeshvartak]

Can you share postman response 

[Caesrob]

This is a sample response for one user.

These are the headers getting returned.