We can agree most social polls suck. Lots of other survey questions stink, too. Perhaps the most common reason: The answers they offer aren’t “MECE” — mutually exclusive and collectively exhaustive.
The poll above is absurd on its face. Even in the posturing-heavy world of LinkedIn, only 3% of respondents claim to wake up between 3 and 4 each morning.
But the primary reason this question doesn’t work isn’t its absurdity; it’s that the answer options aren’t MECE:
1. The response options aren’t collectively exhaustive. You might wake up between 3 and 4 each morning. (You don’t, but you might.) Or between 4 and 5, or 6 and 7. But what if you wake up at some other time?
My old colleague Andrew Stockwell pointed out there’s one obvious option missing: 5 to 6 am. This list also ignores the other 20 hours of the day. And while not many LinkedIn users likely wake up at 2pm, I’d guess more start their day sometime between 7am and 3am than between 3 and 4.
If your response options don’t cover the full range of possibilities, you can’t get good data. I typically wake up between 7 and 8; so I’d either have to select an incorrect response or not participate in the poll. Either way, the results will be inaccurate.
2. The response options aren’t mutually exclusive. The last option (“5 mins before I have to”) is the problem here. What if my first Zoom of the day starts at 7am, and I wake up at 6:55? Should I choose “6-7am” or “5 mins before I have to”? Both options are accurate, because the list of responses isn’t mutually exclusive.
This often happens when a question writer tests two ideas at once. In this case, the first three answers investigate what hour of the day people wake; the last answer tests whether people sleep in as late as possible. You’ll also see non-mutually exclusive answers when the author wants to add personality to the question — which I suspect is the case here.
Whatever the reason, the poll’s author would have been better off changing that final option to “some other time of day.” This would’ve fixed both problems by making the responses both mutually exclusive and collectively exhaustive.
This poll contains other problems as well. (For instance, prefacing the question by announcing when one of the world’s most successful people starts their day surely biases the response.) But the biggest problem in this poll — as with nearly every survey question I see — is that the responses aren’t MECE.
What’s the best or worst social poll or survey questions you’ve seen? Let me know in the comments below, or on LinkedIn or Twitter.
Want good data delivered to your inbox? Subscribe here.