Go Back to the Article Archive >

Stale Popcorn and Bad Data

Popcorn

 

People will happily eat fourteen-day old stale popcorn while watching an action movie, and they will eat more quite a bit more if it’s in a larger bucket.

In a restaurant, if you sit near a window, you’re 80 percent more likely to order salad. And if you sit in a dark corner booth, you’re 80 percent more likely to order dessert.

Avoid the cereal aisle when shopping with kids, because sugary cereals have cartoon characters with eyeballs designed to make eye contact with children.

If you have watched or read a news story about food in the past 10 or so years, odds are you’ve heard about one of these stories, or one of the others at the bottom of this article.

These are just a few research takeaways from Cornell University professor Brian Wansink.

When Wansink joined Cornell not long after I had graduated, I was proud to see such influential research being conducted at one of my alma maters. Wansink’s Food & Brand Lab, which occupied part of the Warren Hall office belonging to a retired former professor of mine, seemed a rather fitting—and perhaps unintentional—way to honor him. That lab was the setting of an infamous automatically refilling soup bowl experiment, which I found nefarious, insightful, and funny.

Undoubtedly, headlines like these have influenced people to think twice about how they eat. But Wansink’s research has largely been discredited, and Cornell has forced Wansink to retire.

Over the past year, many articles have questioned Wansink’s research and dissected the unquestionable errors in his techniques. The errors have been painstakingly researched by those close to academia and the analyses of the shortcomings are convincing and pervasive. (Buzzfeed broke some of the news and has comprehensive reporting.)

I think Wansink had noble intentions, creating marketable lessons to teach the general public how to eat better and be healthier. But he used sloppy techniques to draw out findings that didn’t necessarily have quantitative support, even though there were probably valuable lessons to be learned.

Wansink and his researchers had hunted through large data sets for findings to write about. They hadn’t pre-determined what those findings might be but rather crafted narratives around happenstance results.

Some refer to this as p-hacking (“p” for “probability”), or data dredging; that is, finding data that appears statistically significant, even without any underlying connection. Others call this HARKing, or hypothesizing after the results are known.

XKCD: Green Jelly Beans Linked to Acne!
The punchline from an XKCD comic: https://xkcd.com/882/

Each of these studies helps the authors accomplish their goal: academic publication. It’s the lifeblood of professors who hope to attain tenure.

But those of us outside the academic bubble who rely on those studies for guidance get mixed messages. If research is built upon shoddy foundations, how do we know which study to trust?

And there are many academic studies we hear about in the popular press, sometimes with contradictory storylines from one year to the next. I think many of them pass the simple muster of statistical significance, which allows them to be published. The authors check a box, but we don’t necessarily get good research. Wansink was a master of taking these narrow results and crafting news out of them.

Wansink’s lab sold the story without ensuring its backup was rigorous enough. You need both the story and the details.

I think it’s important to start with questions and hypotheses, and then let the data help understand the real story. And that data might tell a different story than the one we were hoping to find. It’s dangerous, in my view, to shortcut that process because otherwise we might convince ourselves that an untruth is true.


Here’s more Wansink studies that have great headlines.

Overeating:

Getting kids to eat well:

Eating out:

Magic Auto-Refilling Soup Bowl
Flickr/cal222. The magic auto-refilling soup bowl.

Subscribe to Shelf Talk's Food,

Beverage, & CPG Digest: