Examining the transactional data in our marketplace has revealed that the case for limiting interview lengths is even stronger than we had imagined, with critical benefits for respondents, suppliers, and researchers.
In short, lower survey lengths are associated with lower costs, higher respondent completion rates, and higher response quality. Longer surveys are associated with the opposite.
The Benefits of Shorter Surveys
Shorter surveys get researchers and respondents from one point to another faster. They save people time and increase the focus that can be placed on each question. This process can also force the researcher to discover the core of the issue they must uncover.
For example, many business researchers often only need to ask one question — “Would you recommend our product or service to a friend or colleague?” The answer requires almost no thought on the part of the participant. If they like the product, they can honestly say yes. If they don’t like the product, it’s easy to say no.
Whatever your field, every questionnaire should have one question that sums up the point of answering the survey. When the participant answers this question, they’ll know their voice has been heard. They can leave with a sense of satisfaction, and you’ll gain the information that you need.
Long Surveys Cost More to Field
Cost per Interview (CPI) is the price buyers pay for a completed interview. Aggregate data shows that buyers pay a significant premium for longer surveys; surveys longer than 45 minutes are upwards of 4x more expensive than surveys under 5 minutes.
High LOI Means Lower Conversion Rates
One of the big drivers of increased cost at longer interview lengths is that conversion gets lower as survey length increases. Conversion measures the likelihood that a given respondent is going to complete their intended survey. Conversion takes into account both a respondent’s demographic suitability for a survey and their willingness to complete all of its requirements without giving up. Our 2018 data clearly shows that conversion of projects is consistently higher when surveys are shorter. As the length of a survey increases, a respondent’s ability and willingness to complete its requirements drops sharply.
That’s even more intuitive when you consider the growth in mobile survey takers, which resembles the growth of mobile device usage by the internet population overall. Mobile respondents now constitute over 54% of survey attempts on our platform, up from just 37% in 2017. As you might imagine, respondents are even more sensitive to a survey’s length when attempting to complete a survey on a mobile device.
Data Quality Concerns with Long Surveys
Buyers may also be paying a heavy cost in terms of data quality ,when they design longer surveys. As survey length increases, response quality may decrease as respondents become more inattentive to the survey’s questions and their own answers, or perhaps even get frustrated by the time requirements.
Buyers on the Lucid Marketplace, as is the norm throughout the market research industry, remove any interviews that they deem to be of poor quality. This process is called reconciliation. Taking the number of removed interviews (reconciliations) and dividing that by the number of total interviews completed (survey-completes)determines the reconciliation rate. We’re able to see clearly that buyers on our marketplace reconcile more frequently as LOIs increase.
There’s surely some covariance between interview length, cost, and reconciliation rate. The data doesn’t imply that longer interviews cause higher reconciliation rates, just that they’re clearly correlated.
What Is the Recommended Length for a Survey?
When determining survey length, you’ll need to find a balance between retrieving enough information and keeping participants engaged. Studies show that the ideal survey length is approximately 10 to 20 minutes, with a median of 10 minutes.
At the same time, another study shows that it’s often a balance of length and difficulty that ultimately drives survey engagement and a respondent’s “perceived” survey length. Perceived length is different from the actual length, as the survey simply feels longer and more tedious to complete. Some sample groups will even respond to survey length and difficulty combinations differently based on past experience, such as education.
From the perspective of perceived time, 10 easy questions completed in a few minutes can seem like a breeze. Meanwhile, answering 10 difficult questions might take 14 minutes but seem like a chore that took even longer. An easy 30-minute survey about pets was one of the most popular surveys found in the study, even though it was long. This result shows that a pleasurable topic where people have an invested interest can reduce perceived time.
How to Reduce Interview Length
It’s vital to reduce interview length without sacrificing key data or pushing participants away. One study shows that the researcher can often split a long survey into two separate surveys.
The most important questions for the research project should remain in both questionnaires, while each survey should have its own set of separate questions that can provide helpful but not critical information. This approach allows researchers to focus on high-priority questions while creating samples for other datasets.
In a simple example, a researcher might place core questions about a participant’s favorite flavor of ice cream in each survey. Then, in one survey, the researcher might ask what flavor the participant hates. The other survey might ask if the participant’s taste in flavors has changed over time from childhood to adulthood.
Making these cuts requires careful attention to ensure that the structure of the questions isn’t lost. It’s essential to keep a consistent beginning, middle, and end in both surveys.
Next, it’s important to ask if some questions are simply nice-to-know facts that are not crucial or if the answer truly impacts the project’s outcome. It’s often beneficial to ask yourself whether or not the participant skipping the question would impact the outcome of the project.
Higher reconciliation rates and higher prices are bad for researchers. Higher reconciliation rates and lower conversion rates exact a heavy toll on suppliers too. Most of all, longer surveys are harmful and frustrating for the survey respondents themselves.
We all have an interest in building smaller surveys, but there’s been relatively little progress in many segments of the research industry. Survey designers are clearly not internalizing the costs associated with survey length. The marginal cost of any given additional question might seem small, but the aggregate impacts are real for that researcher and the industry ecosystem.
The survey industry has also missed the fact that there should be a balance of easy and difficult questions for most surveys. Even a short survey can seem long to participants with tedious or difficult questions. That’s why it’s crucial to treat the survey respondent like a customer who needs to receive some satisfaction from the transaction. Even surveys that don’t offer financial awards can provide an engaging experience as a diversion from the mundane for a few minutes.
Some firms have taken notice. Our Marketplace has seen the number of interviews under 5 minutes more than quadruple over the last 2 years.
If you’re wondering how you can reduce your interview length, the Lucid team is happy to help advise on strategies to reduce the burden on respondents without making sacrifices to the research itself.