It seems nowadays that any monkey can do a survey, but oh too often in reality they provide less value than a bag of peanuts. There are seemingly few barriers to entry with DIY tools, but user beware. 

Surveys can be a highly insightful way to understand the attitudes and behaviours of customers, staff and the general population — IF done with consideration of survey theory. However, surveys are often over used in the media, business and government to hinder more than help.

Here are five things to consider to do more good than wasteful surveys:

1. Design solid questionnaires:

A well constructed questionnaire is critical in a worthwhile survey. Critical considerations in designing a questionnaire include …

1. avoiding wording questions that bias the responses (e.g. push polling — wording to encourage a certain response);
2. ordering the questionnaire flow to avoid earlier questions influencing the answers to later questions;
3. when using prompted response lists, rotating the order to avoid respondents selecting options based on their position on the list (e.g. earlier options may be selected more often);
4. avoid asking unnecessary questions that simply increase the length of the survey, and introduce respondent fatigue; and
5. do not use poor or confusing question wording without clear intention which may collect unreliable responses. Care, pilot testing and professional advice is worthwhile to improve the flow, timing, overcome any areas of confusion or misprogramming and ensure respondents are efficiently piped through the survey to only ask questions of relevant.

2. Ensure privacy and confidentiality:

It is critical that feedback collected by surveys is honest and provided without fear of being identified for sales, harassment in relation to negative comments or otherwise having their privacy breached. Concerns such as a customer or staff member providing negative feedback and fearing being contacted to resolve the issue, or receiving other potentially unpleasant repercussion can discourage people from being open and honest. Promising and ensuring confidentiality is respected and adhered to is vital. The Privacy Act has provisions to protect consumer privacy, including large associated fines for breaches. The Privacy (Market and Social Research) Code has provisions specific for professional market researchers, incluing the storing of consumer data in de-identified form.

3. Understand sampling theory:

Applying an appropriate survey sampling methodology is critical to ensure that the subset of the population is representative of the actual population. A probability sample is a sample in which every unit in the population has an equal chance of being selected in the sample, and this probability can be accurately determined. This makes it possible to produce unbiased estimates of population totals, by weighting sampled units according to their probability of selection. Non-probability sampling is any sampling method where some elements of the population have no chance of selection, or where the probability of selection can’t be accurately determined. Hence, because the selection of elements is non-random, non-probability sampling does NOT allow the estimation of sampling errors. Non-probability sampling is not statistically representative of the actual population. More here.

4. Recognise statistical error margins:

Assuming the survey sample is selected as a probability sample, a survey error margin can be calculated. The confidence level tells you how sure you can be of that the survey finding matches the actual population. It is expressed as a percentage and represents how often the true percentage of the population who would pick an answer lies within the confidence interval. When a 5% confidence interval applies, this would infer that if 50% answered in the affirmative to a survey question, at the 95% confidence level, the true percentage of the population is between 45% and 55%. Smaller sub-samples of the full survey sample (e.g. age, gender etc) would have larger confidence levels applied (as the sample is smaller). A non-probability survey sample has a more volatile and a reliable error margin unable to be easily calculated. Accordingly the result should be used as indicative and with caution when using a non-probability survey sample. Calculate and read more about confidence intervals here.

5. Avoid bias:

Of utmost importance in any survey is the analysis and interpretation of the results in a manner to reveal key findings and priority improvements. Critical in this is avoiding confirmation bias and other tendencies to read results seeking to confirm existing views. The ability to ensure that an independent perspective is introduced when analysing the results, and even a devil’s advocate perspective, is valuable in gaining value from surveys. Conducting surveys to simply confirm the status-quo is wasteful. When using surveys to test ideas for example, merit is likely to exist to have the results analysed and interpreted by people without fear or favour in the outcome of the survey. More about confirmation bias here

We hope this is useful. Just remember, not all surveys are created equal, and “garbage in, garbage out.” While ignorance is bliss for some, it can be dangerous, misleading and confusing. Surveys continue to be of great value in providing consumer insight. At a basic level, and at a more advanced level surveys allow for simple to more complex analysis such as frequencies, comparison of population segments, regression analysis, segmentation, choice modelling and other mapping of consumer behaviour and drivers.

Ps. If you would like some Square Holes help with questionnaire design, surveys or cross methodology qualitative – quantitative studies (we design 100’s each year), we’re here to help – visit www.squareholes.com or 1800 038 257.