By: Eli Ackerman, Executive Director, Global Monetization and Jack Peterson, Manager, Global Monetization
We have long counseled our research partners to limit their interview lengths. Longer interview lengths are simply more challenging for respondents to complete, especially when a growing number of participants are attempting to take surveys from their mobile devices. Examining the transactional data in our marketplace has revealed that the case for limiting interview lengths is even stronger than we had imagined, with critical benefits for respondents, suppliers, and for researchers.
In short, lower survey lengths are associated with lower costs, higher respondent completion rates, and higher response quality. Longer surveys are associated with the opposite.
Cost per Interview (CPI) is the price buyers pay for a completed interview. Aggregate data from 2018 shows that buyers pay a significant premium for longer surveys; surveys longer than 45 minutes are upwards of 4x more expensive than surveys under 5 minutes.
One of the big drivers of increased cost at longer interview lengths is that conversion gets lower with survey length. Conversion measures the likelihood that a given respondent is going to complete their intended survey. Conversion takes into account both a respondent’s demographic suitability for a survey and their willingness to complete all of its requirements without giving up. Our 2018 data clearly shows that conversion of projects is consistently higher when surveys are shorter. As the length of surveys increase, a respondent’s ability and willingness to complete its requirements drops sharply.
That’s even more intuitive when you consider the growth in mobile survey takers, which resembles the growth of mobile device usage by the internet population overall. Mobile respondents now constitute over 54% of survey attempts on our platform, up from just 37% in 2017. As you might imagine, respondents are even more sensitive to a survey’s length when attempting to complete a survey on a mobile device.
Buyers may also be paying a heavy cost in terms of data quality, when they design longer surveys. Buyers on the Lucid Marketplace, as is the norm throughout the market research industry, remove any interviews that they deem to be of poor quality. Reconciliation rates are calculated by the number of interviews that are tossed out for poor quality over the total number of completes. We would expect that as survey length increases, response quality may decrease as respondents become more inattentive to the survey’s questions and their own answers, or perhaps even get frustrated by the time requirements. We’re able to see clearly that buyers on our marketplace reconcile more frequently as LOIs increase.
There’s surely some covariance between interview length, cost, and reconciliation rate. The data doesn’t imply that longer interviews cause higher reconciliation rates, just that they’re clearly correlated.
Higher reconciliation rates and higher prices are bad for buyers. Higher reconciliation rates and lower conversion rates exact a heavy toll on suppliers too. Most of all, longer surveys are especially harmful and frustrating to the survey respondents themselves.
We all have an interest in building smaller surveys, but there’s been relatively little progress in many segments of the research industry. Survey-designers are clearly not internalizing the costs associated with survey-length. The marginal cost of any given additional question might seem small, but the aggregate impacts are real for that research firm and the industry ecosystem.
Some firms have taken notice. Our marketplace has seen the number of interviews under 5 minutes more than quadruple over the last 2 years.
If you’re wondering how you can reduce your interview length, Lucid is happy to help advise on strategies to reduce the burden on respondents without making sacrifices to the research itself.