Yesterday at work we all received an email telling us to expect the annual Staff Survey next week, and encouraging us all to participate.
I both look forward to this, and I dread it. I look forward to it partly because I know that surely participating must be a good thing, but also partly because I expect it’ll be dreadful and I’ll enjoy the awfulness of it. A case of autoschadenfreude, apparently.
In particular, I expect this approaching survey to fail hard, due to having too many mandatory responses.
As the author of a survey, you get to choose what questions are asked, in what order; and various characteristics of each question, such as what types of responses are accepted (text, choose-one, choose-many, etc), and whether a response is mandatory.
Survey authors should think very carefully about making any responses mandatory. It’s almost always a terrible idea.
Why a respondent might not answer a question
Let’s look at the possible reasons that a user might press the next button before they answered some question or other.
They forgot to answer: Especially on a page containing several questions, it’s easy to accidentally skip over a question. You asked ten questions, you got nine answers. It happens. So what do you do? You highlight to them that they didn’t provide an answer to that question. Now that you’ve reminded them of this question, we can eliminate the forgot to answer possibility from our consideration. (In other words: they should be allowed to override the “you forgot!” nag).
However, even after being reminded of the question, they still might not answer it. Why not?
They don’t know the answer: For example, you’ve asked what the annual turnover of their company is. Some people just won’t know, and maybe have no interest in knowing.
They do know the answer, but none of the answers you allowed for is correct: This could be due to carelessness on your part when you designed the survey (age: under 18, 18–24, 26–40, … — oops. What about 25?), or maybe there’s just an option that you just never considered (Title: Ms, Mrs, Miss, Mr. What about Dr? Rev? Mx?).
It’s impossible to answer the question correctly: A variant on the above problem, but whereas the above is more to do with multiple choice, there are other ways that you might have not allowed for reality. For example, you’ve asked for their full employment history, but only allowed a limited number of slots. Or you made it a text box with only a limited number of characters. Or you accidentally made it a checkbox. Or maybe it’s nothing to do with the design of the answer — maybe the question itself is flawed.
They’d rather not say: So many reasons for this. People have a right not to answer — this should go without saying.
In each of those cases, when the respondent encounters such a question to which your survey requires an answer, but the respondent is unable or unwilling to provide one, what happens next?
Don’t make me lie to you
At this point the respondent has several options:
- If they didn’t provide an answer because they don’t know the answer, then they could, in theory, go and find the answer. (Go and ask what the annual turnover of your company is. Go on, someone must know. You’ll find the answer eventually. Come back and complete the survey once you’ve found it. We’ll wait.)
Seriously, is this what you want your respondents to do? Looks like a lot of work. I don’t think this is going to happen. - They could simply give up. Just, don’t complete the survey. They tried to fill it in, but the survey wouldn’t accept no-response for an answer, so what other option is there? Life’s too short. Move on.
- They could guess.
- They could lie.
- They could just type in garbage. Click multiple-choice options at random. Fill in whichever response requires the fewest keypresses and the least mouse movement. Basically they could fill in anything into your survey just to make that damn annoying “you didn’t answer this question” nag go away!
Woohoo, people filled in your survey!
Congratulations, people completed your survey, and now you’ve got their answers.
Now what? You’re going to analyse the responses, right?
The problem is, because your made the responses mandatory, you’ve skewed the results:
- Some people will have given up on the survey, so you haven’t got anything from them at all — not even the answers they could provide.
- The people who can, and were willing, to give accurate answers probably did so, and that’s what you’ve now got in your results.
- However, you’ve also got some lies, and guesses, and other incorrect results.
So you’ve both got partial results, and the results you have got contain an unknown amount of garbage.
Good luck analysing that.
The more limited responses your survey allows, the more wrong the results will be.
What just happened?
So where did it all go wrong? Why did you choose to create such an awful survey, leading to such meaningless results?
You might have just not thought about it. There’s a good chance that the web site you used to create and run the survey makes responses mandatory by default, and you didn’t think to check that this was sensible. That’s OK: you’ll know better next time.
Oh, this was the next time?
OK then. Well, you might have deliberately chosen to make the responses mandatory. Why would you do that? Because it makes the data analysis easier, you say: you can just say “57% said ‘agree’” rather than all that complicated “71% gave a response, and 80% of those people said ‘agree’”. That sounds hard, eww! Let’s go with the easy option.
Survey makers seem to believe that mandatory responses is the “easy option” — easier to construct the survey, easier to analyse the data. But this comes at the expense both of skewing the data, and placing additional burden on the respondents. In effect, the message you’re sending the respondents is: “my time is more important than yours”.
All rules have exceptions
That said, there are cases where sometimes, despite all those reasons, it’s still OK to make a response mandatory.
For example:
- You really do need an answer. Maybe you deliberately only want homeowners to complete the survey, so it would be perfectly reasonable to start with a mandatory question, “Are you a homeowner?”.
Maybe you’re required by law to ask for the respondent’s age, or name, or address. - The respondent has indicated —perhaps via an earlier question — that they want some sort of reply or feedback from you, therefore you need to have a way of contacting the respondent. (In this case, the respondent has effectively opted in to making the response mandatory).
- The respondent has indicated — again, via an earlier question — that they are willing to provide the answer. “Do you have any other comments to make [y/n]” — and if yes, then it’s reasonable to make the comments box mandatory.
Thanks for your time
Designing a good survey isn’t a trivial task, but you should think long and hard before making responses mandatory. And if you’ve made all the responses mandatory, you should really think long and hard.
If you’re putting together a survey, and you’re more busy and important than your respondents — and you want them to know that — then go right ahead, make all the responses mandatory.
If you really want to send a strong message that the respondent is stupid, and can’t be trusted to leave some responses out (I mean, they’re just so careless!), then please ensure that you don’t leave in any namby-pamby optional responses, lest the poor feckless idiot filling it in forgets to provide you wish your precious data.
But you don’t want to send that sort of message, do you?
Be nice to your respondents. In return, they’ll give you better quality data, and everyone wins.
Thanks for reading.
This content originally appeared on my Medium account