Indicator of ECM Restart in v23.x

[BrandonLucas_BF]

What is the indicator in the logs for v23.x when a restart occurs? The restart all services screen says to wait for 20 or so minutes for the restart to complete, but I'd like to monitor in the logs. I saw a few things that seem to point to an in-progress restart, but I'm not sure.

Is this an indicator of the beginning of a restart procedure?

{log":"/init/ecm-start.sh: line 146:    61 Killed                  /usr/local/tomcat/bin/catalina.sh run\n"

Does logging similar to this show re-init of the restart sequence?

{log":"  created: WEB-INF/classes/com/mrhaki/grails/plugin/xframeoptions/web/\n"

{log":"  created: WEB-INF/classes/com/saviynt/\n"

{log":"  created: WEB-INF/classes/com/saviynt/annotations/\n"

{log":"  created: WEB-INF/classes/com/saviynt/archive/\n"

{log":"  created: WEB-INF/classes/com/saviynt/auth/\n"

{log":"  created: WEB-INF/classes/com/saviynt/awslambda/\n"

{log":"  created: WEB-INF/classes/com/saviynt/cache/\n"

{log":"  created: WEB-INF/classes/com/saviynt/campaigns/\n"

{log":"  created: WEB-INF/classes/com/saviynt/connectorms/\n"

{log":"  created: WEB-INF/classes/com/saviynt/constants/\n"

{log":"  created: WEB-INF/classes/com/saviynt/content/\n"

{log":"  created: WEB-INF/classes/com/saviynt/content/parser/\n"

{log":"  created: WEB-INF/classes/com/saviynt/custom/\n"

{log":"  created: WEB-INF/classes/com/saviynt/custom/exceptions/\n"

{log":"  created: WEB-INF/classes/com/saviynt/customize/\n"

{log":"  created: WEB-INF/classes/com/saviynt/dataset/\n"

{log":"  created: WEB-INF/classes/com/saviynt/ecm/\n"

{log":"  created: WEB-INF/classes/com/saviynt/ecm/analytics/\n"

[sai_sp · Accepted solution]

Hi @BrandonLucas_BF 

Some of the lines you shared are the logs printed during the restart process. You will see the modules and services loading in the logs and they should be self explanatory. Are you looking for specific lines in the logs for this?

[BrandonLucas_BF]

Thank you for confirming. I asked for two reasons. For one, this is something I've wondered about since upgrading to v23.x from 2020.1. A lot of times during a restart event I'll watch the logs for a while to try and ensure the restart is taking place properly and also that the system has completely recovered when these logs stop.

The second reason I asked is because I have a ticket open currently on a Workday full user import job that won't ever complete (no error). I have struggled to capture logs on this because it runs for several hours before seemingly stalling. This morning I was able to watch the job more closely and noticed that right when it appears to stall that the log is flooded with the events I originally reported that look like a restart. It appears the full import may be causing some instability in services about 1/3 through the import (6900/19000 users) forcing a restart and for the job to end. I am already working through a FD ticket with Saviynt support on this issue.

Although not the original intent of this thread, any advice is welcome. I will mark your last response as the answer to my original question.

[sai_sp]

Are there any other jobs running in parallel along with the workday user import? One suggestion would be to run the import standalone when no other jobs are running and then see if it's still happening. 

If you are suspecting that the import is causing the services to shut down (which should not be the case), the issue would need a deeper triage and our support team would be able to help you with it. 

[BrandonLucas_BF]

This job in question is a Workday full user import. Due to the length of time it takes to run a full import, there is a separate Workday incremental import that will try to run while the full user import is running. However, from what I've seen in testing this incremental job will not run until after the original full user import stalls out.

I am currently testing with all of our Workday import jobs suspended and only running the full user import to see what happens. My ticket is 1623059.

[BrandonLucas_BF]

Can confirm after further testing that even having no other jobs running is resulting in the services to shut down during full import for some reason. Ticket linked in prior post. Waiting for assistance from support.