The second of our series on Rethinking Questionnaires is now at LinkedIn. As mentioned, our focus is on re-imagining what and how a questionnaire can be. Many of the research tools we use today are online replicas of surveys that could have been carried out 70-80 years ago; in reality not that much has changed. With the availability of computer-based interviewing via web-surveys the lack of innovation is quite surprising.
Our second demonstration of fly-menus which are used for drill-down and non-linear approaches to questionnaires can be found here: http://www.dobneyresearch.com/DemoQs/index_flymenu.php
Fly-menus are a structural approach to a problem that the linearity of questions in a questionnaire (from A to Z) is determined by the researcher and not down to the respondent saying what they are interested in, or what they would give their time to answer. A fly-menu approach says to the respondent, here are the topics, you choose which topics you want to tell us about, and in what order. This is why we also refer to this as a non-linear questionnaire structure.
It actually breaks quite a big taboo. In surveys we are fully aware that there are order-effects. That is the order in which you ask questions can affect the responses that we get. For some areas like prompting where we prefer to find out what someone knows or thinks spontaneously or semi-spontaneously before introducing the ideas, do still rely on question order. But for approaches such as customer satisfaction research, it should be the respondent/customer who determines the order of questions and prioritises what they want to order.
More specifically, we know that more attention is applied to earlier items when answering questions like scale questions. Though this illustrates a weakness of scale-based approaches in that simple re-ordering changes the answers, it also suggests that we should work with the respondent on the areas that they want to give attention to because then we will have better answers.
The original LinkedIn article on fly-menus is here.
Over at LinkedIn we are have included some posts and demonstrations on Rethinking Questionnaires with new ideas for question design, questionnaire structure, blending surveys with websites and social media. A demonstration can be found here: http://www.dobneyresearch.com/DemoQs/index_hotcolddemo.php
The first looks at hot-cold scales. These are use buttons and images to allow respondents to say what they like or don’t like, but crucially, we don’t force a response. Because of our work looking at choices and choice making, we feel that forcing tends to mix weakly-held with strongly-held views. Choices on the other hand enable a respondent to review a large group of items rapidly and naturally, much as they might do with an on-line or actual paper catalogue, only focusing on the items that are of interest.
The original LinkedIn article is here.
Edward Appleton in his entertaining and provocative market research blog asks “why, given that Market Research and Marketing been preaching customer-focus for decades, so often one experiences lousy customer experience?”
Read More »
In a seminar on Big Data in Barcelona this week, a consultant from Ernst and Young (EY) Consulting suprised me by suggesting that 43% of Big Data projects involved less than 10,000 records running up to 71% of Big Data projects that used less than 1 million records. This was a surprise as my assumption of Big Data was that it was really N-million record type projects.
Read More »
Conjoint analysis (or discrete choice estimation/stated preference research) broadly has four main components. The attributes and levels that make up the product or service that we want to test, a statistical design to choose combinations of attributes and levels in order to convert them into product profiles that reflect the decision space, a choice method – usually direct choice but it could include an estimation of volume (eg number of prescriptions for medical subjects), a ranking, a multiple selection eg of items into a consideration set, or a rating in a more old fashioned conjoint, a method of analysis – normally Hierarchical Bayes or MLE and a modelling. Of these, the element that is most troublesome is the statistical design. With large numbers of attributes and levels, it is impossible to test all combinations, so we have to choose a subset (a fractional factorial design). Read More »
Charting is one of the main reporting methods for quantitative research, and most companies use Powerpoint. With Open Office and Libre Office now reaching version 4.0+ we’re discovering that Open Office is now getting good enough for productive chart production for market research. Read More »
Looking through some data for an academic survey into American citizen’s beliefs in conspiracy theories the other day, it was striking that the academic in question had failed to screen out some odd or strange answers including some deep inconsistencies and some very dubious key pressing patterns. With everything online it’s easy to forget humans provide the answers.
Read More »