Click HERE to see how Saviynt Intelligence is transforming the industry. |
07/08/2024 07:30 AM - edited 07/08/2024 10:33 PM
Hi,
I am trying to import entitlements in REST connector. There are 2 calls to get the entitlement data including pagination.
I want to use the response of CALL1 in the URL of CALL2 for ex:
Call 1 response:
Call 2 url: https://x.apis.com/big/v2/projects/ric-dev/queries/job_KkVNnGaDRWoiq7my_fXLiHIZ?pageToken=BFA6URMHSAAQAA...................&maxResults=40&timeoutMs=10
I want to utilize the jobid and pageToken in the URL of next call
I tried below approach but its giving error
Error:
Json:
"entTypes": {
"business_area": {
"entTypeOrder": 0,
"call": {
"call1": {
"showJobHistory": true,
"callOrder": 0,
"stageNumber": 0,
"http": {
"url": "https://x.apis.com/big/v2/projects/ric-dev/queries",
"httpHeaders": {
"Authorization": "${access_token}",
"Accept": "application/json"
},
"httpParams": "{\"defaultDataset\": {\"datasetId\": \"aditya\", \"projectId\": \"ric-dev\"}, \"dryRun\": false, \"kind\": \"bigquery#queryRequest\", \"maxResults\": 0, \"preserveNulls\": false, \"query\": \"SELECT id,attribute_id,attribute_value,attribute_value_displaytext,attribute_value_description FROM scm_attribute_values where attribute_id in (1)\", \"timeoutMs\": 10, \"useLegacySql\": false, \"useQueryCache\": true}",
"httpContentType": "application/json",
"httpMethod": "POST"
},
"disableDeletedEntitlements": true
},
"call2": {
"showJobHistory": true,
"callOrder": 1,
"stageNumber": 1,
"http": {
"url": "https://x.apis.com/big/v2/projects/ric-dev/queries/${response?.call1?.jobReference?.jobId}?pageToken...",
"httpHeaders": {
"Authorization": "${access_token}",
"Accept": "application/json"
},
"httpContentType": "application/x-www-form-urlencoded",
"httpMethod": "GET"
},
"listField": "rows.f",
"keyField": "entitlementID",
"colsToPropsMap": {
"entitlementID": "v[0]~#~char",
"entitlement_value": "v[0]~#~char",
"DISPLAYNAME": "v[2]~#~char",
"DESCRIPTION": "v[3]~#~char",
"ENTITLEMENT_GLOSSARY": "v[4]~#~char",
"acctEntMappingInfoColumnFromEnt": "STORE#ACC#ENT#MAPPINGINFO~#~char"
},
"pagination": {
"nextUrl": {
"nextUrlPath": "${response?.completeResponseMap?.pageToken==null?null:'https://x.apis.com/big/v2/projects/ric-dev/queries/' + response?.completeResponseMap?.jobReference.jobId + '?pageToken=' + response?.completeResponseMap?.pageToken + '&maxResults=10&timeoutMs=10'}"
}
},
"disableDeletedEntitlements": true
}
}
}
}
Thanks,
Aditya
07/08/2024 11:43 AM
Call 1 response does not seems complete
Please share postman screenshot and curl command [Refer https://codingnconcepts.com/postman/how-to-generate-curl-command-from-postman/ ]
⚠️‼️‼️Do not upload any attachments that contain sensitive information, such as IP Addresses, URLs, Company/Employee Names, Email Addresses, etc.‼️‼️⚠️
07/09/2024 05:54 AM - edited 07/09/2024 05:55 AM
Hi Rushikesh,
The Json is working fine when I hardcode the Job ID and Page token values in URL like below:
It is not resolving the variables
Error:
Got null response statusCode with erroMsg - [error:Error Illegal character in path at index 85: https://dummy.com/queries/${response.call1.jobReference.jobId}?pageToken=${response.call1.pageToken}...]
Below is postman response of call1:
cURL for call1:
curl --location 'https://dummy.com/queries' \
--header 'Content-Type: application/json' \
--header 'Authorization: XXX' \
--data '{
"defaultDataset": {
"datasetId": "iam",
"projectId": "p1"
},
"dryRun": false,
"kind": "q#queryRequest",
"location": "",
"maxResults": 1,
"parameterMode": "",
"preserveNulls": false,
"query": "SELECT id FROM scm_attribute_values where attribute_id in (1)",
"timeoutMs": 10,
"useLegacySql": false,
"useQueryCache": true
}'
cURL for call2:
curl --location 'https://dummy.com/queries/job_M2DC?pageToken=BGHIA&maxResults=2&timeoutMs=10&location=europe-west3' \
--header 'Authorization: XXX' \
--data ''
I have attached the import Json for reference.
Thanks,
Aditya Verma
07/22/2024 07:06 AM
07/22/2024 09:07 AM
{
"accountParams": {},
"entitlementParams": {
"connection": "oauthConnection",
"processingType": "SequentialAndIterative",
"successResponses": {
"statusCode": [200]
},
"entTypes": {
"business_area": {
"entTypeOrder": 0,
"call": {
"call1": {
"showJobHistory": true,
"callOrder": 0,
"stageNumber": 0,
"http": {
"url": "https://dummy.com/queries",
"httpHeaders": {
"Authorization": "${access_token}",
"Accept": "application/json"
},
"httpParams": "{\"defaultDataset\": {\"datasetId\": \"iam\", \"projectId\": \"project\"}, \"dryRun\": false, \"kind\": \"q#queryRequest\", \"maxResults\": 20, \"preserveNulls\": false, \"query\": \"SELECT id,attribute_id,attribute_value,attribute_value_displaytext,attribute_value_description FROM scm_attribute_values where attribute_id in (1)\", \"timeoutMs\": 10, \"useLegacySql\": false, \"useQueryCache\": true}",
"httpContentType": "application/json",
"httpMethod": "POST"
},
"listField": "rows.f",
"keyField": "entitlementID",
"colsToPropsMap": {
"entitlementID": "v[0]~#~char",
"entitlement_value": "v[0]~#~char",
"DISPLAYNAME": "v[2]~#~char",
"DESCRIPTION": "v[3]~#~char",
"ENTITLEMENT_GLOSSARY": "v[4]~#~char"
},
"disableDeletedEntitlements": true
},
"call2": {
"showJobHistory": true,
"callOrder": 1,
"stageNumber": 2,
"http": {
"url": "https://dummy.com/queries/${response.call1.jobReference.jobId}?pageToken=${response.call1.pageToken}&maxResults=20&timeoutMs=10&location=europe-west3",
"httpHeaders": {
"Authorization": "${access_token}",
"Accept": "application/json"
},
"httpContentType": "application/x-www-form-urlencoded",
"httpMethod": "GET"
},
"inputParams": {
"entitlementname": "business_area"
},
"listField": "rows.f",
"keyField": "entitlementID",
"colsToPropsMap": {
"entitlementID": "v[0]~#~char",
"entitlement_value": "v[0]~#~char",
"DISPLAYNAME": "v[2]~#~char",
"DESCRIPTION": "v[3]~#~char",
"ENTITLEMENT_GLOSSARY": "v[4]~#~char"
},
"pagination": {
"nextUrl": {
"nextUrlPath": "${response.call1.pageToken == null ? null : 'https://dummy.com/queries/' + response.call1.jobReference.jobId + '?pageToken=' + response.call1.pageToken + '&maxResults=2&timeoutMs=10&location=europe-west3'}"
}
},
"disableDeletedEntitlements": true
}
}
}
}
},
"acctEntParams": {}
}
07/29/2024 12:04 PM
Hi @rushikeshvartak ,
The pagination block was working fine, I am facing issue in the URL of call2.
The response variable is not resolving in the URL:
"call2": {
"showJobHistory": true,
"callOrder": 1,
"stageNumber": 2,
"http": {
"url": "https://dummy.com/queries/${response.call1.jobReference.jobId}?pageToken=${response.call1.pageToken}&maxResults=20&timeoutMs=10&location=europe-west3",
"httpHeaders": {
"Authorization": "${access_token}",
"Accept": "application/json"
},
"httpContentType": "application/x-www-form-urlencoded",
"httpMethod": "GET"
}
Let me know how I can resolve this.
Thanks,
Aditya
07/29/2024 12:14 PM
${response.call1.message.jobReference.jobId}
07/31/2024 01:58 AM - edited 07/31/2024 01:59 AM
Hi @rushikeshvartak ,
I am getting below error in logs, I have put this variable in the URL ${response.call1.message.jobReference.jobId}
I think the URL is not resolving the dynamic variable "${}"
URL:
"url": "https://dummy.com/ric-dev4x1/queries/${response.call1.message.jobReference.jobId}?pageToken=${response.call1.message.pageToken}&maxResults=20&timeoutMs=10&location=europe-west3"
Error:
Got Webservice API Response: [error:Error Illegal character in path at index 85: https://dummy.com/ric-dev-n-auth-hub-4x1/queries/${response.call1.message.jobReference.jobId}?pageTo...]
Got null response statusCode with erroMsg - [error:Error Illegal character in path at index 85: https://dummy.com/ric-dev-n-auth-hub-4x1/queries/${response.call1.message.jobReference.jobId}?pageTo...]
Error while getting Access Import response for entType- business_area, url- https://dummy.com/ric-dev-4x1/queries/${response.call1.message.jobReference.jobId}?pageToken=${respo... is: null
I am attaching the import json for reference. (line 90)
Thanks,
Aditya
09/18/2024 09:19 PM
Hey Aditya,
Were you able to resolve the issue?
08/09/2024 02:29 AM
Dear @rushikeshvartak ,
Do you have any update on this? We are totally blocked.
Kind regards,
Albert.
08/09/2024 09:07 AM
Hi @travemole
Can you please try with only one variable in the url first and let me know-
"url": "https://dummy.com/ric-dev4x1/queries/${response.call1.message.jobReference.jobId}?pageToken={put hard coded value}
and try it out viceversa and see if its resolving the variable
Thanks
Darshan
08/12/2024 12:34 AM
Hi @Darshanjain ,
Same issue, I tried with one global ${} call and also with only one parameter forcing the jobid.
"2024-08-12T06:16:16.641+00:00","ecm-worker","rest.RestProvisioningService","quartzScheduler_Worker-7-dhdg8","DEBUG","Got Webservice API Response: [error:Error Illegal character in scheme name at index 0: ${'https://xxxx/queries/'+response.call1.message.jobReference.jobId+'?pageToken='+response.call1.messag...'}]"
"2024-08-12T06:16:16.642+00:00","ecm-worker","rest.RestUtilService","quartzScheduler_Worker-7-dhdg8","DEBUG","pullObjectsByRest - responseStatusCode ::null"
I've read somewhere that you have to add:
"inputParams": { "dependentCall": true },
But I'm not sure this applies here as the issue is that is not executing the code. I think that is expecting a real "variable" if you know what I mean.
Kind regards,
Albert.
08/12/2024 02:21 AM
Hi @travemole ,
Can you send me the url which you used, as per logs it looks like both were used as binding with one global variable.
Also you can try with dependentCall = true as the 1st call response is being used in the second call.
Thanks
Darshan
08/12/2024 03:01 AM - edited 08/14/2024 08:32 AM
Hi @Darshanjain ,
same error: I even tried to leave only one parameter and simplify the JSON but nothing works.
Error
Got Webservice API Response: [error:Error Illegal character in query at index 178: https://bigquery.googleapis.com/bigquery/v2/projects/ric-dev-n-auth-hub-4x1/queries/job_1MIQfbcX1BUh...
JSON ImportAccountEntJSON:
{
"accountParams": {},
"entitlementParams": {
"connection": "oauthConnection",
"processingType": "SequentialAndIterative",
"successResponses": {
"statusCode": [
200
]
},
"entTypes": {
"business_area": {
"entTypeOrder": 0,
"call": {
"call1": {
"showJobHistory": true,
"callOrder": 0,
"stageNumber": 0,
"http": {
"url": "https://bigquery.googleapis.com/bigquery/v2/projects/xxxxxxxxxx/queries",
"httpHeaders": {
"Authorization": "${access_token}",
"Accept": "application/json"
},
"httpParams": "{\"defaultDataset\": {\"datasetId\": \"xxxxxxxxxxx\", \"projectId\": \"ric-dev-n-auth-hub-4x1\"}, \"dryRun\": false, \"kind\": \"bigquery#queryRequest\", \"maxResults\": 100, \"preserveNulls\": false, \"query\": \"SELECT a.id, a.attribute_id, a.attribute_value, a.attribute_value_displaytext, a.attribute_value_description, b.scope_value FROM master_attributes_values a, master_attributes_scopes b where a.attribute_id in (1) and a.attribute_scope_id=b.id\", \"timeoutMs\": 40, \"useLegacySql\": false, \"useQueryCache\": true}",
"httpContentType": "application/json",
"httpMethod": "POST"
},
"listField": "rows.f",
"keyField": "entitlementID",
"colsToPropsMap": {
"entitlementID": "v[0]~#~char",
"entitlement_value": "v[0]~#~char",
"DISPLAYNAME": "v[2]~#~char",
"DESCRIPTION": "v[3]~#~char",
"ENTITLEMENT_GLOSSARY": "v[4]~#~char",
"customproperty1": "v[5]~#~char",
"acctEntMappingInfoColumnFromEnt": "STORE#ACC#ENT#MAPPINGINFO~#~char"
},
"disableDeletedEntitlements": true
},
"call2": {
"showJobHistory": true,
"callOrder": 1,
"stageNumber": 2,
"http": {
"url": "https://bigquery.googleapis.com/bigquery/v2/projects/xxxxxxxx/queries/job_1MIQfbcX1BUhXD5QMsXBqJ8uKr...}",
"httpHeaders": {
"Authorization": "${access_token}",
"Accept": "application/json"
},
"httpContentType": "application/x-www-form-urlencoded",
"httpMethod": "GET"
},
"inputParams": {
"entitlementname": "business_area",
"dependentCall":true
},
"listField": "rows.f",
"keyField": "entitlementID",
"colsToPropsMap": {
"entitlementID": "v[0]~#~char",
"entitlement_value": "v[0]~#~char",
"DISPLAYNAME": "v[2]~#~char",
"DESCRIPTION": "v[3]~#~char",
"ENTITLEMENT_GLOSSARY": "v[4]~#~char",
"customproperty1": "v[5]~#~char",
"acctEntMappingInfoColumnFromEnt": "STORE#ACC#ENT#MAPPINGINFO~#~char"
},
"pagination": {
"nextUrl": {
"nextUrlPath": "${response?.completeResponseMap?.pageToken==null?null:'https://bigquery.googleapis.com/bigquery/v2/projects/rxxxxxxx1/queries/'+response.call1.message.jobR...'}"
}
},
"disableDeletedEntitlements": false
}
},
"acctEntMappings": {
"listField": "rows.f",
"idPath": "v[0]",
"keyField": "accountID",
"importAsAccount": false
}
}
}
},
"acctEntParams": {}
}
Response from POSTMAN directly:
Kind regards,
Albert.
08/12/2024 07:10 AM - edited 08/14/2024 08:33 AM
Hi @Darshanjain
08/12/2024 07:22 AM
HI @travemole
"responseColsToPropsMap": {
"customproperty8": "call1.message.pageToken~#~char"
}Using the response directly in call2:
${response.call1.message.pageToken}
Thanks
Darshan
08/12/2024 11:31 AM
Unfortunately this does not work in URL param ${response.call1.message.pageToken} have you guys tried locally in your test environments?
08/12/2024 02:07 PM
{
"accountParams": {},
"entitlementParams": {
"connection": "oauthConnection",
"processingType": "SequentialAndIterative",
"successResponses": {
"statusCode": [200]
},
"entTypes": {
"business_area": {
"entTypeOrder": 0,
"call": {
"call1": {
"showJobHistory": true,
"callOrder": 0,
"stageNumber": 0,
"http": {
"url": "https://bigquery.googleapis.com/bigquery/v2/projects/ric-dev-n-auth-hub-4x1/queries",
"httpHeaders": {
"Authorization": "${access_token}",
"Accept": "application/json"
},
"httpParams": "{\"defaultDataset\": {\"datasetId\": \"adityaverma_iam\", \"projectId\": \"ric-dev-n-auth-hub-4x1\"}, \"dryRun\": false, \"kind\": \"bigquery#queryRequest\", \"maxResults\": 100, \"preserveNulls\": false, \"query\": \"SELECT a.id, a.attribute_id, a.attribute_value, a.attribute_value_displaytext, a.attribute_value_description, b.scope_value FROM master_attributes_values a, master_attributes_scopes b where a.attribute_id in (1) and a.attribute_scope_id=b.id\", \"timeoutMs\": 40, \"useLegacySql\": false, \"useQueryCache\": true}",
"httpContentType": "application/json",
"httpMethod": "POST"
},
"listField": "rows.f",
"keyField": "entitlementID",
"colsToPropsMap": {
"entitlementID": "v[0]~#~char",
"entitlement_value": "v[0]~#~char",
"DISPLAYNAME": "v[2]~#~char",
"DESCRIPTION": "v[3]~#~char",
"ENTITLEMENT_GLOSSARY": "v[4]~#~char",
"customproperty1": "v[5]~#~char",
"acctEntMappingInfoColumnFromEnt": "STORE#ACC#ENT#MAPPINGINFO~#~char"
},
"disableDeletedEntitlements": true
},
"call2": {
"showJobHistory": true,
"callOrder": 1,
"stageNumber": 2,
"http": {
"url": "https://bigquery.googleapis.com/bigquery/v2/projects/ric-dev-n-auth-hub-4x1/queries/job_1MIQfbcX1BUhXD5QMsXBqJ8uKrXu",
"httpHeaders": {
"Authorization": "${access_token}",
"Accept": "application/json"
},
"httpContentType": "application/x-www-form-urlencoded",
"httpMethod": "GET"
},
"inputParams": {
"entitlementname": "business_area",
"dependentCall": true
},
"listField": "rows.f",
"keyField": "entitlementID",
"colsToPropsMap": {
"entitlementID": "v[0]~#~char",
"entitlement_value": "v[0]~#~char",
"DISPLAYNAME": "v[2]~#~char",
"DESCRIPTION": "v[3]~#~char",
"ENTITLEMENT_GLOSSARY": "v[4]~#~char",
"customproperty1": "v[5]~#~char",
"acctEntMappingInfoColumnFromEnt": "STORE#ACC#ENT#MAPPINGINFO~#~char"
},
"pagination": {
"nextUrl": {
"nextUrlPath": "${response?.completeResponseMap?.pageToken == null ? '' : 'https://bigquery.googleapis.com/bigquery/v2/projects/ric-dev-n-auth-hub-4x1/queries/' + response.call1}"
}
},
"disableDeletedEntitlements": false
}
},
"acctEntMappings": {
"listField": "rows.f",
"idPath": "v[0]",
"keyField": "accountID",
"importAsAccount": false
}
}
}
},
"acctEntParams": {}
}
08/12/2024 10:19 PM - edited 08/14/2024 08:33 AM
This does not work because values cannot be hardcoded.
"url": "https://bigquery.googleapis.com/bigquery/v2/projects/xxxxxxxxxx/queries/job_1MIQfbcX1BUhXD5QMsXBqJ8uKrXu",
job id changes on every execution and unfortunately I cannot decide the value of the same.
08/15/2024 12:13 AM
Dear All,
I think in this case there's a bug in the product. I've tested yesterday to put the SAME code in create account JSON and the response.call1 ... was available and displayed on the logs while in the ImportAccountEntJSON even the ${response} or ${response?.call1} or ... returns null and complains about $ found in URL parameter.
Kind regards,
Albert.
08/15/2024 04:15 AM - edited 08/15/2024 04:35 AM
Hi @rushikeshvartak @Darshanjain ,
Professional Services Team said in the ticket below said what I just explained above. It's not possible and Saviynt is not prepared to handle response in the URL parameter for ImportAccountEntJSON:
[#2041837] REST Pagination does not work in ImportAccountEntJSON : Saviynt Inc
For me this is something that should be implemented (if not a bug) as some api's are expecting this value as parameter in the URL and not always can be done in the same call "pagination section". I let you guys discuss this internally but I think that Google cloud platform is going to be more and more demanded during the years and this cannot be achieved in any other way with their api's.
Official doc: Method: jobs.query | BigQuery | Google Cloud
Thanks once again for your help and support on this. You can close this ticket.
Kind regards,
Albert.
08/15/2024 04:42 AM
Raise idea ticket if saviynt confirmed on ticket if its not supported
08/15/2024 04:56 AM
Hi @rushikeshvartak ,
Yes I did just before with a more global idea:
REST connector - global variables that can be | Saviynt Ideas Portal
Feel free to vote it and share it around if you think it could be useful.
Kind regards,
Albert.