Read how to survive in the data jungle and become a data driven organization and make the best decisions fast.

Many companies have the ancient problem. They want to analyse their data but the data is spread around in disparate sources and it’s hard to integrate. There’s even a new generation of companies emerging which have a lot of easy to deploy SaaS solutions, where the data is stuck and the analysis is done using the analytical features of each solution or by manually building combined excel-sheets. Most of the SaaS solutions have good API’s which can be used to extract the data out of the solution, but getting the data into one place integrated is still hard. Add fast moving external data like social media to the sources and the things get even more complicated. Good data is crucial for good decision making. Not even the best algorithms or models could make up the lack of high quality dataset in the decision making process.

How do you build those datasets then? The good old data warehouse used to be the gold standard for any data integration and reporting, but today it is not enough to satisfy the demands of the advanced analytics. The business questions that are asked using business intelligence and analytics are totally different and that causes the techniques to be different as well.

The data warehouse is optimized for slicing the data, drilling down on details and getting answers fast to questions like what happened in the past. The use of the data is decided before it is loaded into the database and often only the data that is relevant to specific domain is extracted from the sources to avoid the data warehouse getting too big and expensive, which causes a lot of business value being left on the table. The data is also forced into optimal structure for the low latency queries. All this is done before the data is loaded into a data warehouse, which is why it usually takes a lot of time before the business users get results to support their decision making.

The data warehouse is optimized for slicing the data, drilling down on details and getting answers fast to questions like what happened in the past.

Analytics requires a different approach on data management. In the analytics the questions are like what will happen and what should I do to make it happen. As we are looking for best variables to predict the future we don’t actually know what we can leave out in advance. The analytics data model is different compared to the dimensional model of the BI. Analytics needs a flat data model where the analyst can easily analyse many observations about the problem at hand at the same time. And often business is eager to get some event like marketing campaign analysed fast and then there is no time to put all that data through the heavy process of data warehousing.

Today we are seeing that the latter approach has started to more or less take over the traditional data warehousing as well. This is caused by a few trends:

  • The amount of data used by the organizations is increasing
  • The number of data sources is increasing
  • The number of external data sources is getting bigger that the number of internal data sources
  • The number of data types and formats is increasing
  • The phase of business and decision making is getting faster
  • The digitalization of products and customers has forced the companies recognize that the BI is not enough. The value is in the analytics

Today the approach which supports fast decision making by utilising analytics is to bring all data into one place in what ever format the data is, and without deciding the use or the structure of the data at load time. The data can be from any source and in any format like image, text, video, json, xml, structured or unstructured etc. In this model it is important that the data is brought together fast and time is not spent in the transformation and optimization in the data pipelines coming from the sources. The use of the data is decided when the data is read and the deeper understanding about the data is studied in the process of analysing and modelling the data.

The role of the traditional data warehouse is changing, but it is still relevant especially in the use cases where better security, compliance and extreme accuracy is needed. But for analytics purposes data warehouse is becoming more like one operational data source among others than the place for running analytics.

The first step to becoming a data driven organization is to get all data into one place, integrate it and expose it to the organisation and do all that fast, cause in the digital world new data sources are born all the time and some even die every now and then. If the organization has just a few people who have access to the data and create reports for everybody else, they will be the bottleneck in the decision making process.

Speed and agility are the main drivers of todays business. To get ahead in the competition, the path from data to business value must be fast. The whole organization must be empowered to make insights from the data to drive their decision making. The ones who can make good decisions fast will win the game.


Author: Jouni Leskinen

A bit about me. I’m the director of Research and Development at Madtrix. Concentrating on growing our business as the solution for complicated, manual marketing reports and dashboards that make all our lives much more difficult than it needs to be.

Madtrix brings you easy to understand, omni-channel data integrated marketing dashboards with data platform to collect all your data for better marketing.

To speak directly to me email