SuccessFactors (REST) Full Import causing ECM restart

[ChrisBellobuono]

We are experiencing the exact issue described in this now closed forum post: https://forums.saviynt.com/t5/identity-governance/indicator-of-ecm-restart-in-v23-x/m-p/33983

Unfortunately, there is no mention as to what the actual fix was as it was resolved via a private ticket.

In our case the job runs for about 30-40 minutes, stops (or is killed), and ECM is restarted.  In all cases, no other jobs were running.  This is an example of what we see in the logs.

"2023-08-11T19:07:04.642+00:00","ecm-worker","","",""," created: WEB-INF/classes/com/saviynt/webservice/"
"2023-08-11T19:07:04.642+00:00","ecm-worker","","",""," created: WEB-INF/classes/com/saviynt/workflowmgt/"
"2023-08-11T19:07:04.641+00:00","ecm-worker","","","","ecm-start: extracting ECM.war"
"2023-08-11T19:07:00.365+00:00","ecm","domain.JobcontrolController","http-nio-8080-exec-147","INFO","Auditing for list of JobControl"
"2023-08-11T19:07:00.366+00:00","ecm","domain.JobcontrolController","http-nio-8080-exec-147","INFO","Auditing process completed for JobControl"
"2023-08-11T19:07:00.366+00:00","ecm","domain.JobcontrolController","http-nio-8080-exec-147","DEBUG","order by FIRED_TIME desc"
"2023-08-11T19:06:46.779+00:00","ecm-worker","","","","/init/ecm-start.sh: line 171: 60 Killed /usr/local/tomcat/bin/catalina.sh run"
"2023-08-11T19:06:44.916+00:00","ecm-worker","rest.RestUtilService","quartzScheduler_Worker-2","DEBUG","Exception in RestUtil.getAt : java.lang.IndexOutOfBoundsException: Index: 0,

Additionally, the job's history log is left like this:

It says it was successful, but there is no end date, the stats link on the left is missing, and from what I can see, none of the users were updated.

Any help would be greatly appreciated.

[Dave]

@BrandonLucas_BF - Are you able to help with this post?  

[BrandonLucas_BF]

Thanks for the tag @Dave.

While I cannot say for certain how our issue was resolved, I'm happy to report the issue in the other thread was resolved. Our symptoms were exactly like yours but we were doing a massive user import using Workday SOAP connection.. something to the tune of 17,000 records being imported for the user table. About halfway through our services would crash and restart and the job would be left with a status similar to yours showing success but no details and it was clearly not successful.

Our ticket was escalated to engineering for a couple of months to diagnose the issue and then fixed. We experienced the issue in both of our instances running v23.6 and it persisted between version upgrades from 23.5. If you have a ticket open, you might reference our ticket number: 1623059.

We experienced the same issue in prod after we moved the job there and a second ticket was opened to address that issue - 1659536. That was resolved in less than 12 hours using the findings from the earlier ticket, with the same resolution, so if your issue is the same as ours, it should be a quick fix.

Good luck!

[Dave]

Thanks @BrandonLucas_BF !

Also, @ChrisBellobuono - If you do open a ticket you can just link to this post as well so they can see this dialogue. The more information they have to work from, the better! 

[rushikeshvartak]

Job End Date , usually is not shown on form when Job is execution mode in background and one of the Service / pod is restarted due to load or any other reason.

[ChrisBellobuono]

Per the ticket I opened, some of my scripts were generating enough IndexOutOfBoundsExceptions, due to the number user being imported, causing the application to eventually restart.  After fixing the scripts so the errors stopped we no longer experienced any restarts.  It is still unclear at this time how the two are related.