Survey Design Mistakes That Can Ruin Respondent Experience

Aug 19, 2019 | Marketplace, Quality

It’s fair to say that the goal of creating a survey is to have respondents successfully complete it – and to get a survey complete, respondents must be able (and willing) to answer your questions. Sample buyers have much more control over survey success than they realize. In fact, poor survey design is the most likely culprit attributed to high survey drop rates. Surprised? You shouldn’t be. Poor respondent experience is a big deal, and it can be detrimental to your research in a number of ways.

In this post, we will explore specific survey elements that lead to high drop rates and how you can improve survey design to reduce drops. If your survey has a high drop rate (above 20%), this guide can help to identify what may be causing respondents to drop out of your survey.

Poor Mobile Optimization

Ignoring mobile respondents could cut your survey’s feasibility by almost half – which is significant, to say the least. With more than 44% of all traffic on the Lucid Marketplace on mobile devices (smartphone or tablet), mobile optimization is key.

In markets like the U.S. and Australia, nearly 50% of respondents are using mobile devices.

That number is even higher in emerging markets, like India and Indonesia, where desktop computers are less pervasive.

However, we know that sometimes a survey absolutely cannot be optimized for mobile, and that’s alright. If that’s the case for you, just make sure your survey is only targeting desktop users, so mobile respondents won’t get stuck in your survey and inflate your drop rate.

Long Length of Interview (LOI)

It’s understandable to want to be thorough with the questions you’re asking in your survey, but it’s important to consider how it’s being received by respondents. Your survey is competing against games, videos, and all the other online options that respondents have when spending time on the internet. If a survey is too long, respondents will become exhausted and choose to spend their time elsewhere.

If you’re still not sure about the right LOI, take a look at our internal research which shows that the sweet spot for LOI is 10 minutes – short and sweet!

Too Many Open Ends (OEs)

Another factor that affects respondent experience is excessive open end questions (OEs), which lead to respondent fatigue, especially among mobile respondents. This has a big impact on response quality because, in some cases, respondents will type gibberish in order to get through the survey. In other cases, drop rates will increase because respondents may simply exit the survey if there are too many OEs. 

So, if you can, try to stick to less than five OEs! 

Excessive / Lengthy Grid Questions

Repetitive grids also cause respondent fatigue, which often leads to respondents straightlining, speeding, or abandoning the survey entirely. 

This is especially important to consider for mobile devices, as grids require manual scrolling and can result in display issues. So, lengthy grid questions will likely make respondents have to work too hard to complete the survey. 

Repetitive Questions

Respondents are impatient when it comes to questions being asked in different ways over and over.

Although some researchers intentionally include repeat questions as a quality check to measure the consistency of respondent’s answers, we recommend employing that approach very thoughtfully so respondents are not left with a poor impression.

Media Loading Issues (Videos, Images, and Audios)

If your media has a long loading time, it will take longer to download the data, which can actually cost mobile respondents more with their service providers than the survey incentive is paying them. 

Another problem with long loading times is that the respondent may think the link is broken – so they’re likely to leave the survey before the media loads.

Trouble with Translation Quality

Getting survey translations right is very important because respondents who cannot understand the survey will drop from the survey. Or, they’ll complete the survey without understanding it, resulting in lower quality responses. 

It’s important to note that Google Translate is not a sufficient survey translation – professional translation is always required.

When in Doubt, Test it Out!

While all of the survey elements we’ve addressed are challenging for desktop users, they can be deal-breakers for mobile users. So, what can you do to avoid harming the respondent experience? Test your survey!

Testing, from beginning to end, is essential before launching any survey. This ensures the mechanics of the survey (redirects, links, buttons, etc) are working correctly. 

Testing is also the best way to diagnose a high drop rate. A natural checkpoint to look at your drop rate is after a “soft launch” of 10% of the total needed completes. If drop rate is high (above 20%), test to see what survey elements might be leading to a poor respondent experience. If it’s hard to finish your own survey, respondents will have an even harder time completing. 

Contact our team if you have questions about improving your survey design!

Recommended Posts

Stay in the know with LUCID. Subscribe to our newsletter.