Is a significant result a meaningful result?

In analysis, market research on October 13, 2011 by sdobney Tagged: , , ,

In quantitative market research it’s common to take a grand population sample for instance of consumers, then to drill down to specific subgroups and look for differences between different profiles. And via cluster analysis and other grouping algorithms analysts can identify groups with similar views or needs or attitudes within the broader population. But sometimes you have to consider if statistically significant differences between groups are significant from a business perspective.

To give an example. Take a usage and attitude study into paper kitchen towels. A usage and attitude study is a standard method for capturing broad market information about a product category for slicing and dicing to try to spot opportunities, assess market share trends and changing views and needs. The research captures information about frequency of buying, brand purchase behaviour, use of the kitchen towels (mop up spills, but perhaps also cleaning glasses), use of alternatives such as anti-bacterial cloths, and attitudes to factors such as the environment, recyling and product purchase drivers such as absorbancy, wet strength and design factors like pattern or colour.

A good U&A will provide a huge databank of information about customer preferences, habits and behaviours and analysts will work through the data looking at headline figures, but also for statistically significant differences between groups.

For instance a headline figure might be that white label (shop’s-own brand) kitchen towels make up 30% of the market. But on drill down the analyst might find a statistically significant difference between single households (singles) and households with children (families). ‘Statistically significant’ has an important technical meaning to do with sampling and statistical noise. In all surveys there will be differences between groups. Percentages will bounce around by one or two percentage points (one percentage point difference is little different from rounding error). However because of the potential for sample variation, the differences ‘don’t count’  (and so the numbers should be treated as if they are the same) unless they are large enough to reach a statistical threshold. This threshold then says there is a xx% probability that the two numbers are significantly different, normally we use a 95% probability.

The size of the threshold – known as the standard error – is directly related to the size of the sample, or the sample size for the subgroups being compared. Bigger samples allow for a finer resolution when examining results. Bigger samples also allow for more subgroups to be examined. This is where the researchers rule of thumb of a minimum of 100 interviews per significant subgroup comes into play. If you want say to look at five age bands, you’d need to look for 500 in the sample. If you wanted to look at gender within the age bands as potential target groups you’d need a sample of 2×500=1000. If you want to look at geographic variations – eg the 10 r 12 standard regions the sample size will be bigger again.

So back to the kitchen towels. If the results show a significant difference in the use of white label so that 40% of families are using white label, but only 20% of singles, the brand manager will immediately want to know how to target and win more market share among the families. The analyst then looks for ‘targetable’ differences between the groups in terms of attitudes and other behaviour. Obviously presence of children is the big difference, but a family may also be using more, using the towels for different purposes (eg mopping up spills at the dinner table) and might have specific environmental concerns from the wider environmental awareness of the children. These can all be checked and confirmed looking for further significant differences between two groups – singles and families within the U&A data.

The brand manager then has options. She can look at how to take the existing brand and extend it so it meets or encapsulates more of the views of the families – for instance value packs, highlighting the sustainability of the paper source, using bright-coloured patterns that will appeal to children. It might mean advertising on children’s TV for instace.

The extent to which the brand can be pushed also needs to consider whether the change would alineate the current core customers. Would a core single buyer buy if the product had brighter colours? It might be that a radical change would be too different for existing brand users. Then the brand manager needs to consider other options, such as creating one or more product lines. Creating product lines is expensive, resource hungry and risky from a financial view point. New brands each needs promotional support, it takes shelf space from existing products and it might diminish sales of existing products (cannibalisation). The decision therefore needs lots of consideration and an assessment of the potential financial and sales impact and acceptability for the core and potential new customers. For this reason most U&As are followed up with qualitative research after a period of reflection to be better able to implement the U&A findings and as U&As don’t typically collect information about price-volume effects or choice trade-offs, there would also ideally be some for of financial forecasting research carried out (which could be prototype testing, test marketing, couponing, limited offers etc as alternatives to or in addition to more market research).

Because of it’s value, U&A research forms a bedrock for a company’s strategic market research plan and is carried out on a periodic basis every 2, 3 or 4 years, and is followed by a programme of supporting research and decision making cascading out of the U&A, topped off by monitoring to track the impact of the decisions taken and how the market moves and changes.

However, though finding significant differences and finding groups or clusters with differences that are significantly different is thus a key part of quantitative research, not every statistically significant result is business significant.

In the kitchen towel study we might find that 67% of 35-44 year olds agree that “the kitchen is the heart of the home.” Among 45-54 year olds it’s 75%. We find the result is statistically significant – that is it’s a real difference. But how does this help us? Both groups agree, just in one group a few more agree. (Note the temptation to make the group a single ‘person’ – the 45-54 year olds agree ‘more’ – they ‘think’/they ‘see’/they ‘believe’.) The problem is not whether the difference exists, but how we use it. We wouled expect a message like ‘towels for the heart of your house’ to fit better with the older group, but it would also fit with the younger group, as a majority still think this way, but just slightly fewer (8%) of them compared to the older group.

Finding statistically significant differences is one thing, but using the data there also have to be differences that would have a business significant impact when reporting and understanding the data. For instance, the ‘kitchen at the heart of the home’ group may be part of a hidden cluster or segment and as part of this cluster, the business can see an investable opportunity to deliver something new to the market. But to know this you need to take results and data patterns as a whole and not as just one-by-one findings. For more information about quantitative research and analysis see our main site


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: