There are some scenarios in which a fast pace is required for data monitoring. This is where real time dashboards come in. A real time dashboard can display information with a high degree of frequency. This information is used by the audience to make quick decisions to correct the behavior of certain foreseeable trends. Real time dashboards can also set off alerts based on business thresholds allowing the audience to react quickly based on up-to-the-minute information.
Real time business intelligence is relevant to multiple industries and departments. Dashboards can be used to monitor call centre performance, IT system usage, data security, dynamic pricing, and inventory optimization and even support fraud detection and risk management efforts. Based on our experience building a real time dashboard to monitor call center data, this article will highlight tips and lessons learned for anyone who is undertaking a similar project.
ETL
Scheduling Frequency In reality the dashboard is not real time. It is a batch job with a very short refresh/update frequency. In our case it was 1 minute, which is also a limitation on Data Services batch jobs. It is important to know these limitations and also to know how fast your entire ETL batch job can run.
If batch job total duration > batch job run frequency then you will create overlapping processes which can immediately cause two issues:
Overload your server or generate refresh slow downs
Create inconsistencies in your data if your ETL does not correctly do blocks and queues write process to DB tables.
Run Optimization Given the need presented above: batch job total duration < batch job run frequency, you must search for the optimal setting to run your batch job. In Data Services there are some situations that can easily speed up your job execution but there is a delicate balance you must consider. One example is the memory method it uses to run the batch job, which can be selected from:
Pageable
In Memory
Pageable
Also whenever you have a table comparison process, the performance can be optimized by running the comparison as a separate process.
Table comparison
The In method runs faster when your server has enough RAM resources, however if your server does not have enough free RAM it will overload and will not be able to catch up with new spawning processes, running lower and lower in physical memory until it causes a complete server crash.
Tomcat
Memory management Tomcat, under determined circumstances, does not perform well with memory management and garbage collection. When executing several QAAWS every minute the memory it uses can build up very quickly. Any Tomcat service on a 32bit Windows environment will have a limitation of 1.2 GB of memory to allocate. Tomcat becomes unstable when it reaches that limit and new requests are still coming in at a constant rate.
There are several tweaks on Tomcat and JVM memory that can be done to optimize this.
One example of these tweaks is the memory limits that can be set when Tomcat is started; these can be set using windows registry or run time modificators.
Cache Optim flag The QAAWS application that comes bundled with Business Objects has a limitation set to cache data requests. When the same request is done in short periods of time the cache data is sent instead of running the query again. To avoid this and get new data every time, you need to disable this functionality on the dsws application properties:
dswsproperties
To disable it, you need to set the qaaws.cache_dpresult.optim flag to false.
qaaws cache
Restart Script In order to keep Tomcat service from memory overloads it is a good practice to schedule an overnight restart that will force garbage collection. The script can be very basic or contain additional cleanup tasks.
Script
HTML Container Placing the flash file on html container will allow you to execute some actions prior and during the flash file execution. You can run javascripts, pass mandatory flash variables (i.e. suppressing tomcat error messages when running qaaws), etc.
The most basic html container for an xcelsius flash file will look as follows:
Xcelsius flash container
Auto refresh Java script As mentioned before, an HTML container will allow you to run javascript programs on the Explorer window executing your xcelsius flash file. Many applications can be given to javascript but one of them could be the browser page auto-refresh function. With the refresh you can wipe out outdated errors from screen.
No Error flash Variable The no error flash variable is a new functionality on Xcelisus SP4. It allows you to set the pop-up error windows off in cases where the error is not directly related to Xcelsius logic, i.e. Tomcat outages, scheduled system restarts, etc.
To add this functionality to any Xcelsius dashboard you need to add the flash variable input to your xlf file first.
Flash Variables
Finally you also you need to pass the variable to the SWF as indicated in the HTML container code example above.
I hope these lessons learned are helpful to anyone working on a real time dashboard. Feel free to leave a comment or question below if you need more information on any of the tips provided in this post.