Hype Turns to Hope: Big Data Finally Comes of Age

In the 15 years I have been handling initiatives around Big Data and the analytics around it, I have never seen the excitement around these topics so high as today. I find myself speaking in more conferences about both, moderating small-group CXO discussions, and reading a lot more about them in the press.

And the Big Data and analytics discussions are no longer limited to the IT trade only. Marketing folks, CFOs, business leaders and even product heads are eager to know more. This has led to IT teams reviewing some Big Data themes with solid curiosity.

A few, who I interact with directly, have some R&D activities ongoing to evaluate Big Data technologies. A small percentage of the organizations have actually started implementing Big Data solutions, to the point where they are getting meaningful business ROI.

Caution in the air

A review of the Big Data success stories, of which there are plenty in the US, yields an interesting observation. The common factor in these success stories is the advanced capabilities to collect, integrate and process the data, then to gain insights and business value from the data.

Basically it’s not so much about the volume of the data, but an organization’s ability to acquire, explore, understand, and leverage it.

But while there are success stories, including a growing number in Asia, there are still fewer than there ought to be. This is understandable. Any discussion around Big Data brings caution among IT decision-makers, which at some level, is well-founded.

The technology, approaches and best practices around Big Data have been evolving rapidly over the previous years. There are a large number of technologies and approaches that fall under the umbrella of Big Data, and the landscape continues to change rapidly.

This leads people to question what is real and not hype, if now really is the right time to start initiatives, and if they will really get value from Big Data projects.

Scary scenarios

As with all technology adoption cycles, there have been scary scenarios where some Big Data technology vendors, primarily funded by investors alone as opposed to actual sales, have vanished. Experienced CIOs therefore opine that Big Data is still in the hype cycle and not yet a mainstream solution.

Decision-makers also ask: “What’s different?” Some years back, Data Warehousing and Business Intelligence technologies and approaches were being pushed, which, at a level, sound similar as the Big Data storyline in terms of concepts and advertised business benefits.

Surveys of organizations that invested in Data Warehouse initiatives show that many (if not most) of the organizations felt they have not received the promised business value, even though they invested significantly to put these items in place.

What was promoted was, “Build it and the business will invent ways to get value out of the information.” Interestingly, this messaging is strikingly similar to how Big Data solutions are being positioned.

In practice it took much more effort than expected to get the data into the data stores. After the large amount of data finally made it into the data warehouse, organizations found that the business or their data analysts did not find the earth-shaking business insights that had been advertised.

This was due to several reasons. The time and effort to create reports against the data store was larger than expected, and the cycle times inhibited the discovery process. Sometimes the analysts of the data were isolated into small business units, and widespread use and access to the data was not possible.

The end result was that there was a large effort expended to get the data into the data store, and very little business value derived from the data. The industry learned that to get insights and discovery requires a very interactive manipulation and exploration of the data, and the techniques were just too cumbersome for those objectives to be achieved.

Dramatic improvements

The good news is that technologies to address these issues have evolved substantially in recent years, and are now lot more robust and interoperable with existing IT infrastructures. Capabilities of these solutions have kept pace with the increase in data volumes.

Application Integration has matured to the point where it is now a relatively routine task to integrate complex, high-volume data sources. Brilliant log management technologies that can collect information from many systems, across geographies, are also available.

An example is Procter & Gamble, which counts more than 4.8 billion customers in over 180 countries. The US multinational has been using advanced integration capabilities to collect and process the diverse information and incorporate it into active Big Data analysis for spotting opportunities.

Data cleansing, data translation, and master data management tools and techniques have also matured to the point that it is feasible to process complex, dirty data sets with much greater confidence than earlier years.

Using these technologies, companies in the US like department store chain Macy’s and supermarket retailer Kroger are managing and incorporating customer and product information in real-time Big Data inventory and customer management processes.

The concerns about getting good, clean data into the data store can now be comfortably addressed. These areas have even matured to the point that there are cloud-based offerings for data integration, data cleansing and master data management, significantly lowering the cost and long lead times.

For example, TIBCO now has cloud-based offerings for integration, data cleansing, master data management, grid computing and visual analytics.

Visual, predictive, interactive

Dramatic improvements in analytics have started building the group of converts. Data presented in a tedious manner automatically kills interest, along with productivity and the desire to go deeper.

Not only are the analytics tools and techniques much stronger now. They appear much more mainstream in the areas of interactive visual analytics and are able to provide powerful statistical modeling and predictive analytics.

Naturally, data users are attracted to spending less time and getting much more out of Big Data. With these tools, both data analysts and businesses can interactively explore and use data to gain new insights.

Data visualization, better display of Big Data and data discovery tools, for example, can help banks in determining the main cause of the dissatisfaction among a high percentage of customers. They can link and find relationships between a rich set of interaction points and customer satisfaction scores.

For example, a moderate sized telecommunications company in the European Union was able to use this information to create customer retention promotions, which resulted in increased revenue of US$40 million a year.

It’s now possible to publish focused data subsets for audiences throughout the enterprise, which gives users capabilities of interacting with the data, and gaining the insights and discovery tailored to their specific needs. This takes the Big Data out of a small Business Intelligence team and expands its use and benefits throughout the enterprise.

There are proven technologies and newer technologies available that can usher in business benefits. But at the end of the day, it is the leaders in the organization who will have to bring about the change required to adopt and adapt to get the Big Value from their data initiatives.

It’s your move, CXOs.

About the Author

Kevin Pool is Chief Technology Officer at TIBCO Asia, a unit of global infrastructure and business intelligence software company TIBCO Software.

Photo credit: Shutterstock

Read more on

Suggested Articles

Some of you might have already been aware of the news that Questex—with the aim to focus on event business—will shut down permanently all media brands in Asia…

Some advice for transitioning into an advisory role

Global risks are intensifying but the collective will to tackle them appears to be lacking. Check out this report for areas of concern