Using Application Data Import (Multi Threaded) instead of (Single Threaded)

[ArW]

Hello,

I'm running into some issues regarding the Data Import jobs.

When they are Single Threaded, they cannot be run when another Single Threaded job is running, even if it is not for the same endpoint. We plan to have a lot of jobs triggered and some that will be triggered in custom clients application using the Saviynt API so this is very problematic. We noticed on the other hands that multiple Multi Threaded jobs can be run at the same time. 

Is there any downside (maybe in performance) to using only Multi Threaded import jobs ? If not, then we will probably only focus on them

[rushikeshvartak]

Downsides of Using Only Multi-Threaded Import Jobs:

1. Increased Resource Consumption:
• Multi-threaded jobs utilize more system resources (CPU, memory) because they handle multiple data processing threads at the same time. This may lead to high resource usage on the Saviynt server, which can affect the overall performance of other processes and jobs running on the same instance.
2. Database Locking & Deadlocks:
• With multiple threads accessing the same database or tables, there’s a higher likelihood of database locking or even deadlocks, especially if multiple jobs are trying to update the same records or tables concurrently.
3. Job Synchronization Issues:
• There’s a chance of inconsistent data or conflicts, especially when multiple jobs attempt to write or update records simultaneously. If two jobs affect related entities (even if they don’t directly interact), the data consistency may be affected.
4. Higher Complexity in Debugging and Monitoring:
• Multi-threaded jobs may be more complex to debug or troubleshoot if something goes wrong, as multiple operations happen simultaneously, making it harder to pinpoint the issue.
5. Potential for Race Conditions:
• When jobs are multi-threaded, you need to be careful about race conditions, where one thread might finish before another, leading to unpredictable results.
6. Endpoint Throttling and Rate Limits:
• Some endpoints you connect to may have API rate limits or restrictions on how many concurrent requests they can handle. Using multi-threaded jobs could exceed those limits and cause throttling or failed imports.

When Multi-Threaded Import Jobs Make Sense:

• Large data imports where the dataset can be divided and processed concurrently without overlapping records.
• Non-sensitive data imports where timing and record locking aren’t critical.
• When you have sufficient system resources to handle the increased load.