• About
  • Blog
  • Contact
Menu

The Good Data Project

Street Address
City, State, Zip
Phone Number
Good data illuminates the world. Bad data confuses and obscures.

Your Custom Text Here

The Good Data Project

  • About
  • Blog
  • Contact

Don't test two ideas in one question

January 18, 2022 Nate Elliott

Twitter poll asking “Favorite Wes Anderson film?” Response options are: “The Grand Budapest Hotel,” “Rushmore,” “Love them all,” and “Hate them all.” Source: Twitter.

To get the best survey data, make sure you investigate one idea at a time. The Twitter poll above tests two questions, and as a result won't get accurate responses to either.

Brian in Fukuoka wants to know what you think about Wes Anderson films. But he's unlikely to learn much, because he's asking about two different variables and offering insufficient responses for each:

1. Half the responses test which Wes Anderson films people like best. If your favorite is Rushmore or The Grand Budapest Hotel, you're covered. But what if, like me, you prefer The Royal Tenenbaums? Or The Darjeeeling Limited or Isle of Dogs? This poll only lets you select from two of Anderson's 10 feature films, and gives you no option to select if your favorite isn't listed.

2. The other half test how much you like Wes Anderson's oeuvre. If you love or hate everything he's made, just click the right button. But what if you're lukewarm about him, or think (correctly) that his catalog is deeply uneven? You only have the binary of "love" or "hate" to choose from.

Think we can fix this poll by simply offering all reasonable responses for both "what's your favorite film" and "how much do you like him"? Sorry. Not only would that create a very long list of options (something you can't do on social polls), but some respondents would be answering one question while others would be answering another.

The author should have created two Twitter polls (one to test each of the two ideas) and provided respondents with better options for each. I suggest:

1. What's your favorite Wes Anderson film?

  • The Grand Budapest Hotel

  • Rushmore

  • The Royal Tenenbaums

  • Other (describe in comments)

2. What do you think of Wes Anderson's films?

  • Love them all

  • They're all okay

  • Hate them all

  • Some are good, some are bad

How would you have rewritten this question? Let me know in the comments below, or on LinkedIn or Twitter.


Want good data delivered to your inbox? Subscribe here.

In Collecting Data
← The Sun's data-free "data visualization" Erodes Confidence In Charts and GraphsStop Biasing Your Survey Respondents →

©2021-2025 Nineteen Insights

POWERED BY SQUARESPACE