Measuring and explaining reader engagement

We need to know the “what” and the “why” to make sense of the data

UPDATE: Not claiming any credit here as Chartbeat often does explainer blog posts — but here are some details on the Chartbeat data from Su Hang, the lead data scientist on the climate coverage study: Taking the temperature of climate change engagement: Our analysis

That is an interesting and engaging headline. Great if true, and I don’t dispute it is. Read the full story here.

But, I would like to see the footnotes:

Can we see the data, how these stories were coded and how that “time spent” was distributed across stories and sites? “Doubling” time spent on a topic is an aggressive claim. @Chartbeat
— Damon Kiesow (@dkiesow) July 31, 2019

Josh Schwartz from Chartbeat quickly and helpfully responded that as the data is all owned by the respective publishers, they can’t share too many details.

We're not authorized to share individual publishers' deanonymized data, so we can't share stories and/or sites. But the study used a consistent methodology for tagging stories and ran it across a consistent set of sites, so we're highly confident that the stats are robust
— Josh Schwartz (@joshuadschwartz) July 31, 2019

Totally fair. But the finding that “time spent” almost doubled on climate stories between 2017–2019 is an extraordinary claim and requires further supporting detail if the industry is going to make sense of the findings. That is important not just for these specific climate stories but for the use of similar metrics in the future.

And the responsibility here is really not Chartbeat’s. They have access to a great mass of data but there are limits to how much they can share publicly.

I would really like to hear more from CJR about how they understood and contextualized the data in the story. I trust the measurements and math Chartbeat provided. But we don’t know enough to trust the interpretation.

A few immediate questions I had that I would like to know more about:

The “time spent” metric is total time, not an average per story? So that means, more coverage would result in more time spent even if engagement was still relatively low on individual stories.
The NYT and Guardian (referenced in the story) both increased their coverage. Did a few large properties skew the metric or was it evenly distributed? Josh (above) indicates they felt it was consistent.
How were stories coded as being “climate-related?” If news organizations are mentioning the climate crisis more in weather stories, sports stories, or political stories is that an increase in “time spent” on “climate” or an increase in media’s contextualizing the issue? (Which also would be great.)
Were climate-related stories promoted differently between 2017–2019 or were climate-related stories just bigger news recently — as in fires, heat, flooding as per question #3? Does that mean readers are engaging more on the topic, or that they are being exposed more to the topic by how media covers it?

That’s it. The story here is really interesting — people are paying more attention to an important topic. But I would just love to hear more that explains the “why” not just the “what.”

Also published on Medium.