Did you know? We have the ability to run parallel and multi-threaded application data import jobs

[Community_User]

Originally posted on June 5 2020 at 23:42 UTC

1. When do we run parallel import jobs?

When customers want to reconcile multiple applications at the same time

2. Which connectors support parallel import jobs?

All out of the box connectors

3. Which connectors support multi-threaded import jobs?

Box, Office365, GoogleApps

4. What is the need to run multi-threaded import jobs?

When Saviynt was required to perform file imports for scanning from cloud providers(Box, Office365, GoogleApps) and number of files ~ 1 million, it took days to import these files. This caused delay in overall import and scan completion time

By introducing the multi-threaded import capability, we are able to successfully import multiple files simultaneously from target applications like Box, O365 and GoogleApps within few hours

5. Are there any pre-configurations necessary to set up parallel import jobs?

jobs.enableparallelimport=true

sap.usefilebasedapproach=true // This setting is only needed for SAP parallel import jobs

Work with your respective technical partners/ team to get this configured

6. How can the parallel import jobs be executed?

Application Data Import (Parallel import) jobs need to be scheduled to be executed successfully like below

7. How can the multithreaded import jobs be executed?

Application Data Import (Multi-Threaded import job can be scheduled to be executed successfully like below. Important note is to add the maximum number of threads that can run at a time per server at any given point in time

To avert risk of running more number of threads than allowed, we should configure max threads/server threshold in quartz.properties org.quartz.threadPool.threadCount parameter

[Community_User · Accepted solution]

Originally posted on June 8 2020 at 08:28 UTC

Thanks Aarthi.

[Community_User]

Originally posted on July 14 2021 at 07:32 UTC

HI @Aarthi / Ajay - I have few questions around this :

1. Is providing the number of threads mandatory for the job to be multi threaded?
2. Doesn't it automatically take the available threads?
3. What happens if i mention the thread count = 10 and other processes ( Analytics / provisioning / email jobs are stalled because the threads are consumed by application import job?
4. Are there any best practices about scheduling the Multi threaded account/ access import jobs?? WE have seen jobs getting stuck with the status - "This job was suspended as there is already an import running for the same endpoint"
5. Can the account import and access import jobs for the same endpoint overlap?

[Community_User]

Originally posted on February 1 2022 at 09:00 UTC

Hi Aarthi,

We are unable to locate "Application Data Import (Parallel import) job". We are currently on v5.5SP3.7.16. Is there any specific configuration that needs to be enabled?

Regards,

Naveen

[Community_User]

Originally posted on February 1 2022 at 12:59 UTC

Naveen, you should check 

Application Data Import (Multi Threaded) - this is the job meant for parallel imports. 

[Community_User]

Originally posted on February 1 2022 at 13:17 UTC

Thanks for the confirmation, Kirti. May I please ask you if this job can be used to run imports parallelly? For example, running AD and Salesforce import parallelly at the same time. 

Thank you very much for your help in advance!