The problem with reporting web analytical data

Imagine this- you’ve poured time and effort into an analytical report, peer reviewed it, and it’s ready to go to the client...then you spot an error.

Shock, horror, panic stations...along with a few expletives, leave you wondering if you can still palm it off to the client. You’ve just experienced one problem with analytical data...one of several that every web analyst faces.

Analytical data can be the basis of a compelling case to get anything web-related done, after all, its hard evidence. Butits not foolproof. It tells your client where the pain points are, and gives them an idea on how to fix it, but if the information provided is inaccurate, or incorrectly interpreted, all it becomes is an exercise on how not to report on site activity.

So what are the more common dangers with analytical data?

Skewed figures

Analytical data, especially in the process of converting, can be skewed. Some of the mitigating factors can be controlled (like incorrect formulas or incorrect date ranges), but there are some factors that are beyond your reach. Noticed that spike in web activity, when its a typically quiet day, and you hadn’t promoted anything? Chances are, that was the result of a bot or crawler making multiple impressions in the one time.

The tell tale sign of bot/crawler activity is when there is a noticeable spike in web browsing activity in a short timeframe, with a short average duration. This will inevitably happen to an extent, no matter how well you plan for it. You can investigate further, and filter it out (bots will usually leave an ID) or alternatively, be wary of it, query it, and even discount the dimension from your overall stats if need be.

As mentioned earlier, incorrect formulas and date ranges can be easily controlled, as they require human intervention. If in doubt (and time permitting), investigate, otherwise, if its not critical to the report, leave that section of data out- you can always conduct a mini retrospective, see to the issue then and implement the findings going forward.

Another factor that can contribute to skewed results is multiple entries of the same page being recorded. While analytical products are comprehensive, some won’t, and can’t, detect multiple entries of the same page. If the site you’re working on is prone to title and architecture changes, keep this in mind. Understand the limitations of the product you’re working with prior to starting a reporting task, so you know if you need to factor time in to report on multiple instances if need be.

Another issue that can lead to skewed data reporting is browsing activity from unknown sources. The three most common sources for this result are browsing activity that reaches a firewall before the journey is continued, scripting capabilities that are blocked (if the product requires a script to identify and record activity), or as mentioned earlier, bot-like activity. An unknown source isn’t necessarily a bad thing, but must be taken with a grain of salt if you must report on it, otherwise, leave this dimension out.

If it is the instance that a firewall is blocking web activity (such as China's Great Firewall), there are online tools that can test your site to ensure it can be accessible from mainland China, such as Website Pulse's web accessibility tool.

Misinterpretation

As big as skewed data is an issue, misinterpretation is an even bigger one, as it can cover a metric or dimension, through to the whole report.

It’s always in your best interests (and your client’s) to read the brief thoroughly. If you need to schedule a meeting to discuss then finer points, then do so, so that both sides of the report understand the requirements.

Misinterpreting metrics or dimensions is a more common issue than you think. As interfaces are updated and names of key metrics and dimensions change, you need to understand and adapt in order to remain successful, if not competent.

Its easy to mistake an exit for a bounce, or a visit as being the number of users visiting your site, but a few minutes of research into the metrics you’re reporting can save you a heap of heartache later on. Resources outlining definitions of metrics and dimensions such as the Open Web Analytics metrics and dimensions wiki will also help develop your understanding.

Changing data upon re-entry

Data can sometimes change upon re-entry. There are two main factors for this- human or product activity. In the case of human activity, this is a relatively simple fix. Make sure the timeframes you are reporting on are considered before you start data mining. Besides that, make sure that you have a clear idea of the audience you are reporting on. If reporting on an intranet site, chances are, you will want to report on internal users only, so make sure the corresponding filter for internal users only is used.

If the issue is product based, this will be harder to diagnose, so your respective IT/S department will need to get involved to pinpoint the problem.

Irrelevance

One of the simplest, yet most common problems with analytical data can be relevance. It is easy to receive the brief, do the research, and then get carried away, mining for anything and everything, regardless if it was asked for. The biggest problem in this instance is that you’re essentially trying to either get too clever or haven’t exactly converted the brief into digestible questions that have clear business goals. In both instances, when time is money, you’re burning through it.

For example, reports that are generated to search for poor performing pages should focus on issues such as bounce rates, and perhaps even the more common issues associated that lead to the bounce rates, rather than looking at how many hits a page gets. It also doesn’t make much sense to search for how many hits or leads a site gets if the measure of that site’s success is how much revenue it generates through online sales. The solution to this issue is simple- look at the brief unpack it, convert it into research questions, and answer those questions.

While Web Analytics can give reveal the four factors of your audience, it only gives you the numbers. If ever in doubt, always ask, because, as the saying goes, “He who asks a question may be a fool for five minutes; he who asks no questions stays a fool forever".