.svg){kind=small}
MDM Batch Processor is a multi-threaded J2SE client application used in most of the MDM implementations to load large volumes of enterprise data into MDM during initial and delta loads. Oftentimes, processing large volumes of data might cause performance issues during the Batch Processing stage thus bringing down the TPS (Transactions per Second).
Poor performance of the batch processor often disrupts the data load process and impacts the go-live plans. Unfortunately, there is no panacea available for this common problem. Let us help you by highlighting some of the potential root causes that influence the Batch Processor performance. We will be suggesting remedies for each of these bottlenecks in the later part of this blog.
Infrastructure Concerns
Any complex, business-critical Enterprise application needs careful planning, well ahead of time, to achieve optimal performance and MDM is no exception. During development phase it is perfectly fine to host MDM, DB Server and Batch Processor all in one physical server. But the world doesn’t stop at development. The sheer volume of data MDM will handle in production needs execution of a carefully thought-out infrastructure plan. Besides, when these applications are running in shared environments Profiling, Benchmarking and Debugging become a tedious affair.
CPU Consumption
Batch Processor can consume lot of precious CPU cycles in most trivial of operations when it is not configured properly. Keeping an eye for persistently high CPU consumption and sporadic surges is vital to ensure CPU is optimally used by Batch Processor.
Deadlock
Deadlock is one of the frequent issues encountered during the Batch Processing in multi-threaded mode. Increasing the submitter threads count beyond the recommended value might lead into deadlock issue.
Stale Threads
As discussed earlier, a poorly configured Batch Processor might open up Pandora’s Box. Stale threads can be a side-effect of thread count configuration in Batch Processor. Increasing the submitter threads, reader and writer threads beyond the recommended numbers may cause some of the threads to wait indefinitely thus wasting precious system resources.
100% CPU Utilization
“Cancel Thread” is one of the Batch Processor daemon threads, designed to gracefully shutdown Batch Processor when the user intends to. Being a daemon thread, this thread is alive during the natural lifecycle of the Batch Processor. But the catch here is it hogs up to nearly 90% of CPU cycles for a trivial operation thus bringing down the performance.
Let us have a quick look at the UserCancel thread in Batch Processor client. The thread waits for user interruption indefinitely and checks for the same every 2 seconds once while holding on the CPU all the time.
Thread thread = new Thread(r, "Cancel");
thread.setDaemon(true);
thread.start();
while (!controller.isShuttingDown()) {
try
{
int i = System.in.read();
if (i == -1)
{
try
{
Thread.sleep(2000L);
}
catch (InterruptedException e) {}
}
else
{
char ch = (char)i;
if ((ch == 'q') || (ch == 'Q')) {
controller.requestShutdown();
}
}
}
catch (IOException iox) {}
}
Batch Processor Performance Optimization Tips
We have so far discussed potential bottlenecks in running Batch Processor at optimal levels. Best laid plans often go awry. What is worst is not having a plan. A well thought out plan needs to be in place before going ahead with data load. Now, let us discuss some useful tips that could help to improve the performance during data load process.
Infrastructure topology
For better performance, run the MDM application, DB Server and Batch Processor client on different physical servers. This will help us to leverage the system resources better.
Follow the best thread count principle
If there are N number of physical CPUs available to IBM InfoSphere MDM Server that caters to Batch Processor, then the recommended number of submitter threads in Batch Processor should be configured between 2N and 3N.
For an example, assume the MDM server has 8 CPUs then start profiling the Batch Processor by varying its submitter threads count between 16 and 24. Do the number crunching, keep an eye on resource consumption (CPU, Memory and Disk I/Os) and settle on a thread count that yields optimal TPS in MDM.
You can modify the Submitter.number property in Batch.properties to change the Submitter thread count.
For example:
Submitter.number = 4
Running Multiple Batch Processor application instances
If MDM server is beefed up with enough resources to handle huge number of parallel transactions, we should consider parallelizing the load process by dividing the data into multiple chunks. This involves running two or more Batch Processor client instances in parallel, either in same or different physical servers depending on the resources available in that server. Each Batch Processor application instance here must work with a separate batch input and output; however they can share the same server-side application instance or operate against a dedicated instance(each Batch Processor instance pointing to a different Application Server in the MDM cluster). This exercise will increase the TPS and lower the time spent in data load.
Customizing the Batch Controller
Well, this one is a bit tricky. We are looking at modifying the OOTB behavior here. Let us go ahead and do it as it really helps.
- Comment out the following snippet in runBatch() method of java
//UserCancel.start();
- Recompile the Batch Processor class and keep it in the jar
- Replace the existing DWLBatchFramework.jar, present under <Batch Processor Home>/lib with this new one which contains modified BatchController class
- Bounce the Batch Processor instance and check the CPU consumption
Manage Heap memory
Memory consumption may not be a serious threat while dealing with Batch Processor but in servers that host multiple applications along with Batch Processor the effective memory that can be allocated to it could be very low. During the data load process if high memory consumption is observed then allocating more memory to Batch Processor helps to ensure a smooth run. In the Batch Processor invoking script (named as runbatch.bat in Windows environments and runbatch.sh in UNIX environments), there are couple of properties that control the memory allocated to the Batch Processor client.
set minMemory=256M
set maxMemory=512M
It is recommended to keep the minMemory and maxMemory at 256M & 512M respectively. If the infrastructure is of high-end, then minMemory and maxMemory can be increased accordingly. Again, remember to profile the data load process and settle for optimal numbers.
Reader and Writer Thread Count
It is recommended by IBM to keep the Reader and Writer Number thread counts as 1. Since, they are involved in lightweight tasks this Batch Processor configuration should suit most of the needs.
Shuffle the data in the Input File
By shuffling the data in the input file, the percentage of similar records (records with high probability of getting collapsed/merged in MDM) being processed at the same time can be brought down thus avoiding long waits and deadlocks.
Scale on the Server side
Well, well, well. We have really strived hard to make Batch Processor client to perform at optimal levels. Still, poor performance is observed resulting in very low TPS? It is time to look into the MDM application. Though optimizing MDM is beyond the scope of this blog let us provide a high-level action plan to work on.
You can either:
- Increase the physical resources (more CPUs, more RAM) for the given server instance
- Hosting MDM in a clustered environment
- Allocating more application server instances to the existing cluster which hosts MDM
- Having dedicated cluster with enough resources for MDM rather than sharing the cluster with other applications
- Logging only critical, fatal errors in MDM
- Enabling SAM and Performance logs in MDM and tweaking the application based on findings
Hope you find this blog useful. Try out these tips when you are working on a Batch Processor data load process next time and share how useful you find them. I bet you’ll have something to say!
