Stale Popcorn and Bad Data
What does stale popcorn have to do with bad data?
If you have watched or read a news story about food in the past 10 or so years, odds are you’ve heard about one of these stories, or one of the others at the bottom of this article.
These are just a few research takeaways from Cornell University professor Brian Wansink.
Wansink’s Food & Brand Lab Research
When Wansink joined Cornell not long after I had graduated, I was proud to see such influential research being conducted at one of my alma maters. Wansink’s Food & Brand Lab, which occupied part of the Warren Hall office belonging to a retired former professor of mine, seemed a rather fitting—and perhaps unintentional—way to honor him. That lab was the setting of an infamous automatically refilling soup bowl experiment, which I found nefarious, insightful, and funny.
Undoubtedly, headlines like these have influenced people to think twice about how they eat. But Wansink’s research has largely been discredited, and Cornell has forced Wansink to retire.
Over the past year, many articles have questioned Wansink’s research and dissected the unquestionable errors in his techniques. The errors have been painstakingly researched by those close to academia and the analyses of the shortcomings are convincing and pervasive. (Buzzfeed broke some of the news and has comprehensive reporting.)
I think Wansink had noble intentions, creating marketable lessons to teach the general public how to eat better and be healthier. But he used sloppy techniques to draw out findings that didn’t necessarily have quantitative support, even though there were probably valuable lessons to be learned.
Wansink and his researchers had hunted through large data sets for findings to write about. They hadn’t pre-determined what those findings might be but rather crafted narratives around happenstance results.
Some refer to this as p-hacking (“p” for “probability”), or data dredging; that is, finding data that appears statistically significant, even without any underlying connection. Others call this HARKing, or hypothesizing after the results are known.
Academic Publications based on bad data
Each of these studies helps the authors accomplish their goal: academic publication. It’s the lifeblood of professors who hope to attain tenure.
But those of us outside the academic bubble who rely on those studies for guidance get mixed messages. If research is built upon shoddy foundations, how do we know which study to trust?
And there are many academic studies we hear about in the popular press, sometimes with contradictory storylines from one year to the next. I think many of them pass the simple muster of statistical significance, which allows them to be published. The authors check a box, but we don’t necessarily get good research. Wansink was a master of taking these narrow results and crafting news out of them.
Wansink’s lab sold the story without ensuring its backup was rigorous enough. You need both the story and the details.
I think it’s important to start with questions and hypotheses, and then let the data help understand the real story. And that data might tell a different story than the one we were hoping to find. It’s dangerous, in my view, to shortcut that process because otherwise we might convince ourselves that an untruth is true.
Here’s more Wansink studies that have great headlines.
- If you connect a secret tube to a bowl of soup that causes it to refill itself automatically, people will will eat more than people whose bowls are not connected to such a device.
- People serve 22 percent more on a bigger plate than on a smaller plate.
- A messy kitchen will lead to eating 44 percent more snacks, and those who leave cereal boxes on their kitchen counters are 21 pounds heavier than those who don’t.
- Eating with an overweight friend will make you more likely to eat more.
- “Serve yourself from the stovetop rather than family style on the table. People who did so ate 19 percent less.”
Getting kids to eat well:
- Kids will be more likely to eat apples during school lunches if they have Elmo stickers on them.
- Giving vegetables cool names like “X-Ray Vision Carrots,” “Super Salad,” or “Tiny Tasty Tree Tops” will make kids want to eat them.