Advertisement

SKIP ADVERTISEMENT

The Public Editor

Awash in Data, Thirsting for Truth

Mark Twain complained about “lies, damned lies and statistics” more than a century ago. “Figures often beguile me,” he wrote, “particularly when I have the arranging of them myself.”

Now, in a numbers-soaked era when the word “algorithm” seems to be on everyone’s lips and entire news outlets, like Nate Silver’s FiveThirtyEight or The Times’s own Upshot, are based on interpreting statistics, some core journalistic questions arise: How important is data to reporting? And does it get readers closer to the truth or obscure it?

I’m thinking about these questions because of two recent pieces in The New York Times, both of which have generated considerable reader response.

The first was a major exposé of brutal working conditions at Amazon, based on six months of reporting and told largely through anecdotes from former and current employees. Though some, including Amazon brass, contested it, the story certainly communicated essential knowledge about this hard-driving company. I wrote in a blog post that while I admired much about the piece, its length and display might have been overdone, given the lack of hard proof that Amazon’s practices make it an outlier.

The executive editor, Dean Baquet, disagreed. “I reject the notion that you can report a story like this in any way other than with anecdotes,” he told me. “You talk to as many people as possible and you draw conclusions.”

The second piece was a magazine cover story, built on data, arguing that a feared digital-age “creative apocalypse” never happened. Rather, it said, the fortunes of the creative class, far from plunging in the post-Napster era, have improved. Many readers disputed the article’s conclusions and charged that the numbers it relied upon had been “cherry-picked” to make a flawed case.

One reader, James Margraf, wrote to accuse the story (with a nod to Stephen Colbert) of “truthiness.” Like others, he objected to a statistic that he said was an apples-to-oranges comparison of the number of working musicians years ago and more recently. Only the more recent figures included school music teachers. With them, the number of working musicians goes up; without them, it would have declined. “While there are other questionable claims in the story, this particular use of data seems hard to defend,” Mr. Margraf wrote.

The story’s thesis and details have been well worked over since it was published. The Future of Music Coalition, an advocacy group for musicians, posted a long response on its site. The article’s author, Steven Johnson, responded in detail, and the group then answered him. The Times’s standards department considered a request for correction and decided against it. “We looked at it thoroughly and did not find a correctable error,” Greg Brock, senior editor for standards, told me. In my view, the original article should have clearly explained how the category had changed.

I asked the magazine editor, Jake Silverstein, to address these issues. (You can read our full exchange on the Public Editor’s Journal blog.) On the specific point raised by Mr. Margraf and others, he wrote that Mr. Johnson and Times fact-checkers had concluded that the use of the statistic was appropriate, noting that some people have migrated from one category to another over the course of the last 15 years.

As for whether Mr. Johnson chose specific data to prove a point, Mr. Silverstein said: “I reject completely the notion that he cherry-picked any data. Cherry-picking implies that an unrepresentative fact was used, an outlier that doesn’t reflect the preponderance of data on some subject, but that makes an argument more dramatic and vivid. In fact, both Steven and our fact-checkers carefully reviewed all the data they could get their hands on and frequently chose the number that made his case more moderate and less dramatic.” The story didn’t set out to prove a preordained conclusion, he said, but took a dive into the available data and invited the reader along to explore it.

Still, Mr. Silverstein acknowledged that data-driven stories present challenges.

“I think the danger with stories based on numbers is that they can come out feeling too neat, as if the complexity and messiness of the real world can be reduced to mere data,” he said.This is why we publish a mix of stories that aim to capture the world in a variety of modes. Data-journalism is one of those modes, and it’s a useful one, so long as you’re aware that it can’t tell a complete story all on its own. There will always be a huge amount of complexity and heterogeneity within a data set.”

I also talked about this with Alex Howard, who, as a fellow at the Tow Center for Digital Journalism at Columbia University, wrote “The Art and Science of Data-Driven Journalism.” He says, “I think we’re all looking for the truth, and the defense is that it’s somewhere in the numbers.”

Mr. Howard admits his own bias toward that way of thinking, rather than toward traditional narrative-journalism style, but also admits that data isn’t always reliable — and that neither is the way journalists sometimes interpret it. (Entertaining examples of how it can go wrong, he noted, may be found on wtfviz.net.) “What can turn data into sound journalism,” he said, “is context, along with fact-checking and verification.”

Mr. Howard also believes in consulting experts to help interpret the data or point out its flaws. Stories on climate change and the importance of vaccines are places where data is being used, but not always to good effect, he says. He’s also a strong proponent of news organizations’ sharing the original data, so readers can dig into it and perhaps challenge conclusions. As usual, transparency goes a long way.

Narrative stories, powered by anecdote and shoe-leather reporting, aren’t going away, nor should they. And the data-driven approach is here to stay, too. Combining the two — when possible — may be the best of all worlds, bringing readers the closest approximation of the truth.

Correction: It’s appropriate that I include my own numbers-related misstep here. In my last column, I wrote that the $300 million budget for the Times newsroom was up 50 percent since 2008’s $200 million. While the approximate numbers are correct, the accounting method has changed. The percentage increase should not have been included.

Follow the public editor on Twitter at twitter.com/sulliview and read her blog at publiceditor.blogs.nytimes.com. The public editor can also be reached by e-mail: public@nytimes.com.

A version of this article appears in print on  , Section SR, Page 10 of the New York edition with the headline: Awash in Data, Thirsting for Truth. Order Reprints | Today’s Paper | Subscribe

Advertisement

SKIP ADVERTISEMENT