At Hanover, we draw on many different methodologies to best answer our clients’ research questions. Sometimes, the information clients are looking for doesn’t yet exist, and in many of those cases we recommend survey research to obtain the data ourselves. Surveys allow us to ask relevant questions to a targeted group of people and provide objective, reliable results to our clients.
Surveys are one of the most-requested types of projects at Hanover and one of the most commonly used market research approaches today. Surveys can measure two types of information: facts (like product usage) and opinions (like customer satisfaction). Surveys also work across many different industries. Our education partners, for example, use surveys to gather feedback from students, staff, and parents on a variety of initiatives like after school programs or financial aid packages. Our B2B and B2C corporate clients use surveys to understand how customers are using their product or service and measure levels of customer satisfaction, test new concepts, gauge brand awareness, and other priorities. Surveys can be administered on their own, or as part of a comprehensive research program that includes other methodologies like data analysis and in-depth interviews.
Survey design in particular is deceptively difficult to execute well. Poor instrument and sample design can create confusion, reduce response rates, and produce low-quality data. Acting on inaccurate results obtained through surveys can cost companies more than lost revenue: time and effort spent on developing products, entering new markets, or rebranding initiatives based on inaccurate insights can significantly affect the success of a company’s initiatives in the market.
Here are five common pitfalls of survey design that can skew the results of a survey.
Not having a representative sample
Understanding who the survey is intended for and ensuring a representative sample is an important first step in survey design. For example, if a company is testing a new concept to gain market share, it should obtain feedback from both current and prospective customers. However, if the company has a respondent breakdown where customers outweigh prospects dramatically, customer feedback will be overrepresented when looking at the overall results of the survey. Considering the company’s goal of market expansion, the skewed survey findings may result in not accurately capturing feedback from non-customers. Obtaining a representative sample can become especially challenging when there is not an existing database of contacts or when there is limited information on those contacts. Survey research experts can help by identifying potential survey respondents and vetting them to ensure they meet the required parameters.
Lacking a clear goal for your survey
With surveys, it can be easy to get carried away by trying to address every question you have in a single survey. However, we find that the longer a survey is, the more likely that people will drop from the survey.  It’s also harder to tailor your sample correctly if you’re trying to accomplish too many things in one survey. Before beginning a survey, we recommend evaluating what your goal is. Is it brand awareness? Concept testing? Customer satisfaction? Pick one, and only include questions that pertain to this goal. In general, it is better to administer multiple, shorter surveys to cover a topic than to try to accomplish too many goals with a single survey.
Not knowing your audience
Often, surveys are written assuming too much about the respondents’ knowledge about the subject matter and include phrases, concepts, or jargon that are difficult to understand and result in inaccurate responses or high dropout rates. For example, a healthcare survey to the general population should avoid acronyms such as HIPPA, PPO, POS, and HMO. When this type of terminology is unavoidable, concise definitions should be included throughout the survey. Companies can also add screening questions to remove unknowledgeable respondents. A clear understanding of the respondent pool is key to matching your questions to the audience’s knowledge level.
Being loose with survey language
Asking leading or biased questions can be a source of inaccurate survey data. One misplaced word or phrase can throw the entire response off. For example, a survey question asking, “Why wouldn’t you eat at a fast food restaurant?” immediately biases the respondent to think about not eating at a fast food restaurant, and could skew your survey results. Alternatively, it might put the respondent on the defensive if the respondent eats fast food on a regular basis. Instead, you should ask the question neutrally, such as: “What factors affect your decision to eat at a fast food restaurant?” It’s just a few words but it can make a big difference. It is very easy to bias a survey respondent’s input and it is very hard to spot your own biases as a survey designer. Survey experts need a lot of practice over time to see what questions work and what don’t.
Using quantitative methods when you should be using qualitative
Surveys, especially those measuring opinions, apply a quantitative lens to potentially subjective material. Sometimes a qualitative approach through in-depth interviews, focus groups, or ethnographic studies is more appropriate for the research question at hand. Qualitative work should always be used if there is no clear hypothesis to test, if there is a small sample size, or if you’re looking to find out not what someone did, but why they did it. Too many open-ended questions in a survey can increase the likelihood of gibberish responses or drop-outs.
With surveys, companies have only one shot at getting the answers they need, and a million opportunities to make a mistake. Many times clients ask us to perform follow-up research on a survey project designed by someone whose expertise is not market research, only to find the results – and the entire basis of the research strategy – are bogus due to poor survey design.
A professional eye can help avoid these pitfalls and give you the best chance of success. Work with Hanover’s survey experts to help you develop, deploy, and analyze your survey.Â
