Making good quality surveys and conjoint designs

In conjoint analysis, market research, questionnaire design on April 15, 2013 by sdobney

In the past few weeks we’ve run into a flurry of market research surveys and conjoint analysis projects that have clearly been written or created by people who haven’t been that experienced in designing surveys. With standard consumer market research it’s often thought not to be a problem. The relatively small size of the sample versus the total population means that a poorly written or poorly thought through questionnaire can be discarded when the results turn out to be relatively poor. However, for more technical areas like conjoint where the quality of the design is quite critical to the quality of the responses, it is very easy to get wrong results, but not realise, or have enough expertise to know that the results are misleading. And for niche areas, like business-to-business, where the available contacts are small, poor questionnaires can drive customers potty, so much so that it becomes impossible to collect further data.Even with consumer-level questionnaires there is a strong argument that as a questionnaire or a survey is a communication with a customer or potential customer it should be created and produced with the same design values that you would include on any piece of marketing. If you customers think your surveys look cheap, do they also think the same of your products. This is without even getting to whether customers believe you take their time and comments seriously – or whether their efforts just disappear in to a black-hole and nothing appears to change.

So in designing questionnaires, it’s important to keep the customer in mind at all times. Not just one customer, but the range of customers you have. You’re trying to get a response from a sample of all possible customers, not just those who are technically adept, or up with the news about your business. Unfortunately 2-3 of the lower quality surveys we’ve seen recently have been written from an internal view – using jargon from inside the business (a particular bug-bear are TLAs which may be absolutely clear to product managers, but most consumers need things spelt out), assuming respondents share the same interests as the business, or share the same objectives. For instance, a common assumption of business studies students and occasionally those in large corporations, is that small businesses are focused on the same growth and business metrics as large businesses. In fact many smaller businesses do not have growth as an objective – family businesses, those run as lifestyle choices, and local businesses often have other objectives than pure size. Respondents are also very often much less involved in the details of your product than you are. Factors like ingredients, environmental issues etc may only be important to a section of the customer base, so some care is necessary to write questions that can be answered both by those who know, and by those who don’t care or don’t understand.

In technical areas, the importance of good design doubles or triples because the answers you will get will be a set of choices from the options that the respondent was shown. You won’t see where the respondent didn’t understand or didn’t know what to do. All the choices they gave are valid for analysis, and as a result the analysis will produce answers even if the answers are not right. For this type of reason conjoint designers are strong on piloting and on having hold-out samples to make sure the final data reflects real choice making behaviour.

As a result, it’s very easy to get a conjoint study that is poor, but not really know that it is poor. Rarely do clients step into the details of the design, any prohibitions in play, how price will be handled etc. And there is a tendency to short cut the design using non-descriptive levels like ‘Good’ or ‘Bad’. While these work in scales (and we could actually debate that), in conjoint you would get a utility score for Good. But what does this mean? How would you improve from Good to Better? Does Good on a product tend to raise expectations of other levels in the conjoint?

As a technical area, it is easy to find that you don’t know what you don’t know. The worry is that some companies’ experiences of conjoint are poor not because of conjoint itself but because the conjoint design had unrecognised hidden problems.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: