We have a REST connector which creates very large provisioning comments (1000+ lines) when running tasks. We need to extract useful information out of this. I have tried and failed with two possibilities:
1. use REGEXP_SUBSTR() in analytic
execute is denied for this function
2. retrieve using /fetchTasks API and manipulate in powershell
I can see the provisioningcomments field but I can only retrieve a maximum 10000 characters (by setting the taskmetadatalength parameter to 10000). There is significantly more data in this field.
How else can I achieve this please?
I tried a further way: Create an analytic that gather this data, then access THAT via the API. But now elastic search doesn't like the provisioning comments field, it's too big apparently.
message [ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Document contains at least one immense term in field="provisioningcomments" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[123, 34, 103, 101, 116, 119, 111, 114, 107, 101, 114, 34, 58, 123, 34, 104, 101, 97, 100, 101, 114, 115, 34, 58, 34, 109, 121, 100, 108, 97]...', original message: bytes can be at most 32766 in length; got 319976]];
Is there a solution here?
Edited further to add: I came up with a pretty unsatisfactory solution, but it works: chunk up the big field into smaller ones, assuming it's never going to get bigger than a certain size, then access via the API /fetchRuntimeControlsDataV2 and reconstruct and manipulate in powershell.
substring(t.provisioningcomments,1,32765) as 'comments1',
substring(t.provisioningcomments,32766,32765) as 'comments2',
substring(t.provisioningcomments,65531,32765) as 'comments3',
substring(t.provisioningcomments,98296,32765) as 'comments4',
substring(t.provisioningcomments,163826,32765) as 'comments5',
substring(t.provisioningcomments,196591,32765) as 'comments6',
substring(t.provisioningcomments,229356,32765) as 'comments7',
substring(t.provisioningcomments,262121,32765) as 'comments8',
substring(t.provisioningcomments,294886,32765) as 'comments9',
substring(t.provisioningcomments,327651,32765) as 'comments10',
substring(t.provisioningcomments,360416,32765) as 'comments11',
substring(t.provisioningcomments,393181,32765) as 'comments12',
substring(t.provisioningcomments,425946,32765) as 'comments13'
from arstasks t
