What do we mean by macro data integration?

During the session on Multi-channel Metrics at Search Engine Strategies in New York a couple of weeks ago, one of the questions that came up was about integrating web analytics data with other forms of data. What was the best way to do this? As is often the case with web analytics, the answer is usually “it depends”.

It depends on what you are trying to do.

One of the challenges faced by the typical digital marketer is the number of different data sources they are dealing with at any one point in time. These might include for example data from:

  • Ad-serving systems
  • Pay per click bid management systems
  • A web analytics system
  • Affiliate management systems
  • Transactional systems
  • Customer management systems

The list goes on. How do you pull all this stuff together to get a sense of what is really going on?

I tend to think about data integration on two levels:

  • Macro integration
  • Micro integration

Macro integration is where you pull the outputs of the various data systems together in to one place so that you can see relationships between different data sources. Excel is often the ultimate macro data integration tool and legions of analysts all over the country spend hours and hours pulling together spreadsheets from multiple data sources to provide a summarised view of what is going on.

This is not necessarily a bad thing as the process of pulling the data together means that the analysts is looking at it and is likely to spot any data quality problems. However, the problem is that 90% of the analyst’s time is spent in data production and only 10% of it is spent in data analysis. The value equation here is all wrong.

One approach is to try and automate the process as much as possible and get the various data sources into a single database. The data is likely to be processed or summarised, rather than in its original format, but at least it’s in the same place and makes it easier for you to be able to report out across the range of KPIs and key metrics. Inevitably this requires a degree of investment upfront to set up the data feeds correct and the reporting system set up. Standard databases and Business Intelligence reporting tools could be used for this type of work.

At a company that I worked at in the past, we took this approach to reporting on our key metrics. We extracted summarised visit, unique visitor and page view data from our web analytic system and stored it along side other key company data in a datmart designed specifically for that purpose. The extracts were done every night so that the next morning the key site metrics was available and reported against some of our other key transactional data using a Business Intelligence tool (Business Objects in our case).

After the inevitable initial teething problems, the process worked quite well and the questions from management and colleagues changed from asking about when the reports were going to be ready, to what the reports actually telling them in terms of business impact. We were able to go from spending 90% of an analyst’s time on data production, to 90% of an analyst’s time on… Analysis!

Obviously, this doesn’t come without a cost. Managing multiple data sources is a messy business. Significant resources may be required from your IT team and they may or may not have the necessary experience in dealing with this particular type of data. An alternative route may be to outsource the process to companies such as Blackfoot Inc and others who specialise in this type of data integration activity.

Macro integration is the process of pulling the outputs of different data sources together, so that you can more easily and effectively report them side by side. Next time I will be taking a look at the challenges and opportunities in pulling together and integrating data at a much more granular level - the customer. Micro integration as I call it.

Till then⿦

More from Applied Insights

See more: Articles
See more: Analytics strategy
See more: Data integration

Learn about our web analytics consulting Discover our innovative marketing applications Get to know Applied Insights