Latency, the silent killer of competitiveness

Credit: Image credit: wwarby/Flickr

Whether it’s the inability to “connect the dots” before the 9-11 attacks on the United States or the failure to anticipate the financial meltdown of 2008, analysts often have all the data at their fingertips, but are unable to respond in a timely manner. They fall victim to one or more of latency’s “hotspots” that inhibit decision-makers.

There’s some great research done by Dr. Richard Hackathorn that shows there are three levels of latency in data analysis on a time continuum from the moment a triggered event occurs until an action is taken. The first level he calls data latency, which is the time used to get raw data stored persistently on a disk drive in a repository after being processed through business rules and data quality processes and the like.

The next level Hackathorn identifies is analysis latency. This is the time it takes to retrieve the stored data, process it, and apply data models. This leads to a third level, decision latency. That’s the time it physically takes to generate the alert, deliver the alert to an end point, and then act on the information. The longer the time-to-decision through all those levels of latency, the more the value of the information is diminished, according to Hackathorn.

Naturally, the converse is also true. The faster information travels through the three latency hotspots, the more valuable it will be.

To mitigate our latency hotspots we can select from an array of technologies. Complex event processing is one such technology that speeds time to decision-making. Another is an advanced message bus system such as the Advanced Messaging Queuing Protocol, an open standard for passing business messages between applications or organizations.

One more thing: We need to start questioning technology conventions. For example, the convention of storing data for analysis in traditional rows. Nice. Neat. Tidy. But the reality is, if you want to speed up an application, there are better approaches such as columnar-stored data.

My point here is if your organization is riddled with latency hot spots, the value of the information you are using is less than it should be. As a result, the decisions being made will be degraded in some way. If that’s your situation, look at your infrastructure for ways technology can reduce latency and improve both your information’s value and the decisions made from it.


ITWorld DealPost: The best in tech deals and discounts.
Shop Tech Products at Amazon