Cover

Contents

Image

A Note to Our Readers

This is a book to help you decide which numbers and studies you probably can trust and which ones you surely should trash.

The rules of statistics are the rules of clear thinking, codified. This book explains the role, logic, and language of statistics, so that we can ask better questions and get better answers.

While the book’s largest audience has been health and other science writers, we believe that it can also be helpful to many other writers and editors, as well as to students of journalism. Health studies are emphasized in many of the chapters because they are so important and they illustrate many principles so well. But this book shows how statistical savvy can help in writing about business, education, environmental policy, sports, opinion polls, crime, and other topics.

News & Numbers is the brainchild of the late Victor Cohn, a former science editor of the Washington Post and sole author of the first edition. I’m glad I could help with later editions, but this is still “Vic’s book.” His inspiring spirit lives on with this edition.

I am particularly pleased that one of his daughters, Deborah Cohn Runkle, a science policy analyst at the American Association for the Advancement of Science, has provided her expertise to help update and expand this latest edition of News & Numbers.

We’ve added a chapter to delve deeper into writing about risks. With President Obama’s health system overhaul plan now law, we’ve added new things to think about in the chapter on health care costs and quality. There’s also a new section on “missing numbers” in the last chapter that we hope will stir your thinking. And we’ve added other new information that we hope you will enjoy along with the old.

Lewis Cope

A Tribute to Victor Cohn, 1919–2000

Victor was a pioneer science writer and a master of his craft. Often referred to as the “Dean of Science Writers,” he became the gold standard for others in his profession.

Beginning his career in the mid-1940s, following service as a naval officer in World War II, he quickly showed an uncanny ability to write about complex medical and other scientific topics in clear, easy-to-understand ways. He provided millions of readers with stories about the landing of the first humans on the moon, the development of the polio vaccine, the then-new field of transplant surgery, the latest trends in health care insurance and medical plans, and many, many other exciting developments over a career that lasted more than 50 years. Throughout, he remained diligent at explaining the cost and ethical issues that came with some of the advances, particularly in the medical sciences.

As part of all this, he showed his fellow journalists the importance of probing numbers to discover what they can reveal about virtually every aspect of our lives. He wrote News & Numbers to share his techniques for doing this in the most revealing and the most responsible way. His quest for excellence in reporting lives on in the Victor Cohn Prize for Excellence in Medical Science Writing, awarded yearly by the Council for the Advancement of Science Writing. With this new edition, Victor’s message lives on.

Lewis Cope, coauthor of this edition

Foreword

I’ve long thought that if journalists could be sued for malpractice, many of us would have been found guilty some time ago. We often err in ways that inevitably harm the public – for example, by distorting reality, or framing issues in deceptively false terms. Among the tools we sometimes wield dangerously as we commit our version of malpractice is the subject of this book: numbers. At one time or another, most of us have written a story that either cited as evidence some number of dubious provenance, or that used numbers or statistics in ways that suggested that the meaning of a medical study or other set of findings was entirely lost upon on us.

Fortunately for many of us, before we did any serious harm, someone handed us a copy of Vic Cohn’s marvelous News & Numbers, now released in a third edition co-authored by Vic and Lewis Cope, with the assistance of Vic’s talented daughter, Deborah Cohn Runkle. I was rescued in this fashion early in my journalistic career, and later had the honor of meeting Cohn and thanking him for his wonderful book. With the advent of this new edition, it is heartening that an entirely new generation of journalists will now have the chance to be saved similarly from their sins.

Much of the content of this book will be familiar to readers of previous editions, even as some of the examples have been updated to reflect recent events, such as the now-discredited vaccines-cause-autism controversy, or the 2010 BP oil spill in the Gulf of Mexico. Perhaps the most important lesson is that almost all stories of a scientific nature deal with an element of uncertainty. And with so much to study amid the rapidly changing sciences of medicine and health care, “truth” often looks more like the images of a constantly shifting kaleidoscope than a message carved on a stone tablet. Thus the book’s excellent advice: “Good reporters try to tell their readers and viewers the degree of uncertainty,” using words such as “may” or “evidence indicates” and seldom words like “proof.”

From the standpoint of the First Amendment, it’s a good thing for society that reporters don’t have to be licensed. But it’s not so good that one can become a reporter – even for an esteemed national publication or news channel – without even a rudimentary grasp of statistics. This book’s crash course on probability, statistical power, bias, and variability is the equivalent of educating a novice driver about the rules of the road. Readers will also be introduced to the wide array of types of medical and scientific studies, and the strengths and weaknesses of each.

Portions of several chapters are devoted to the all-important topic of writing about risk. Important concepts are defined and differentiated, such as relative risk and absolute risk – two different ways of measuring risk that should always be stated together, to give readers the broadest possible understanding of a particular harm. A useful discussion focuses not just on distortions that journalists may make, but common public perceptions and misperceptions that affect the way readers or viewers respond to various risks.

Among the new entries in this edition is a chapter on health costs, quality, and insurance, which wisely cautions careful observation of the effects of the 2010 Affordable Care Act. Because this chapter was written so far ahead of the implementation of most of the law in 2014, its main message is “wait and see” what happens. Perhaps equally important is to encourage journalists to consider and convey to our audiences the totality of the law’s effects, which inevitably will bring tradeoffs – for example, possibly more spending on health care because many more Americans have health insurance. As critical as verifying the “numbers” coming out of health reform will be understanding how the many different sets of numbers will relate to each other, and what values – and I don’t mean numerical ones – Americans will assign to the collective results.

Overall, this new edition upholds Cohn’s perspective that behind bad use of numbers is usually bad thinking, sometimes by the user and sometimes by the person who cooked up the numbers in the first place. And Cohn was a staunch believer in the notion that journalists had a duty to be good thinkers. This edition’s epilogue quotes a list Cohn once made of what constitutes a good reporter; one entry asserts, “A good reporter is privileged to contribute to the great fabric of news that democracy requires.” This edition powerfully evokes Cohn’s spirit, and his belief that, with that privilege, the responsibility also comes to get the facts – and the numbers – right.

Susan Dentzer
Editor-in-Chief, Health Affairs

Acknowledgments

Victor Cohn’s main mentor and guide in preparation of the first edition of this book was Dr. Frederick Mosteller of the Harvard School of Public Health. The project was supported by the Russell Sage Foundation and by the Council for the Advancement of Science Writing.

Cohn did much of the original work as a visiting fellow at the Harvard School of Public Health, where Dr. Jay Winsten, director of the Center for Health Communications, was another indispensable guide. Drs. John Bailar III, Thomas A. Louis, and Marvin Zelen were valuable helpers, as were Drs Gary D. Friedman and Thomas M. Vogt at the Kaiser organizations; Michael Greenberg at Rutgers University; and Peter Montague of Princeton University. (For those who aided Cohn with the first edition of this book, the references generally are to their universities or other affiliations at the time of that edition’s publication.)

For their assistance with later editions, special thanks go to: Dr. Michael Osterholm of the University of Minnesota, for his great help on epidemiology; Rob Daves, director of the Minnesota Poll at the Minneapolis-St. Paul Star Tribune, for sharing his great expertise on polling; and Dr. Margaret Wang of the University of Missouri-Columbia, for her great enthusiasm about all aspects of patient care.

Very special thanks go to Cohn’s daughter Deborah Cohn Runkle, a senior program associate at the American Association for the Advancement of Science. Without her encouragement and assistance, this edition would not have been possible.

Others who provided valuable counsel for the second edition include Dr. Phyllis Wingo of the American Cancer Society; Dr. Ching Wang at Stanford University; John Ullmann, executive director of the World Press Institute at Macalester College in St. Paul; and the great library staff at the Star Tribune in Minneapolis-St. Paul.

Many other people helped with the first edition of this book. Thanks go to Drs. Stuart A. Bessler, Syntex Corporation; H. Jack Geiger, City University of New York; Nicole Schupf Geiger, Manhattanville College; Arnold Relman, New England Journal of Medicine; Eugene Robin, Stanford University; and Sidney Wolfe, Public Citizen Health Research Group. Thanks also go to Katherine Wallman, Council of Professional Associations on Federal Statistics; Howard L. Lewis, American Heart Association; Philip Meyer, University of North Carolina; Lynn Ries, National Cancer Institute; Mildred Spencer Sanes; and Earl Ubell – and also Harvard’s Drs. Peter Braun, Harvey Fineberg, Howard Frazier, Howard Hiatt, William Hsaio, Herb Sherman, and William Stason.

This book has been aided in the past by the Robert Wood Johnson Foundation, the Ester A. and Joseph Klingenstein Fund, and the American Statistical Association, with additional help from the Commonwealth Fund and Georgetown University.

Despite all this great help, any misstatements remain the authors’ responsibility.

Notes on Sources

Book citations – The full citations for some frequently cited books are given in the bibliography.

Interviews and affiliations – Unless otherwise indicated, quotations from the following are from interviews: Drs. Michael Osterholm, University of Minnesota; John C. Bailar III, Peter Braun, Harvey Fineberg, Thomas A. Louis, Frederick Mosteller, and Marvin Zelen, at Harvard School of Public Health: H. Jack Geiger, City University of New York; and Arnold Relman, New England Journal of Medicine. In most cases, people cited throughout the book are listed with their academic affiliations at the time that they first were quoted in an edition of News & Numbers.

Quotations from seminars – Two other important sources for this manual were Drs. Peter Montague at Princeton University (director, Hazardous Waste Research Program) and Michael Greenberg at Rutgers University (director, Public Policy and Education Hazardous and Toxic Substances Research Center). Quotations are from their talks at symposiums titled “Public Health and the Environment: The Journalist’s Dilemma,” sponsored by the Council for the Advancement of Science Writing (CASW) at Syracuse University, April 1982; St. Louis, March 1983; and Ohio State University, April 1984.

Part I

Learning the Basics

A Guide to Part I of News & Numbers

In the first five chapters, we cover the basics:

1. Where We Can Do Better

Improving how stories with numbers are reported.

2. The Certainty of Uncertainty

Scientists are always changing their minds.

3. Testing the Evidence

Thinking clearly about scientific studies.

4. What Makes a Good Study?

Separating the wheat from the chaff.

5. Your Questions and Peer Review

What to ask the experts.

1

Where We Can Do Better

Almost everyone has heard that “figures don’t lie, but liars can figure.” We need statistics, but liars give them a bad name, so to be able to tell the liars from the statisticians is crucial.

Dr. Robert Hook

A team of medical researchers reports that it has developed a promising, even exciting, new treatment. Is the claim justified, or could there be some other explanation for their patients’ improvement? Are there too few patients to justify any claim?

An environmentalist says that a certain toxic waste will cause many cases of cancer. An industry spokesman denies it. What research has been done? What are the numbers? How valid are they?

We watch the numbers fly in debates ranging from pupil-testing to global warming to the cost of health insurance reforms, and from influenza threats to shocking events such as the catastrophic Deepwater Horizon oil spill in the Gulf of Mexico in April 2010.

Even when we journalists say that we are dealing in facts and ideas, much of what we report is based on numbers. Politics comes down to votes. Dollar-figures dominate business and government news – and stir hot-button issues such as sports stadium proposals. Numbers are at the heart of crime rates, nutritional advice, unemployment reports, weather forecasts, and much more.

But numbers offered by experts sometimes conflict, triggering confusion and controversies. Statistics are used or misused even by people who tell us, “I don’t believe in statistics,” then claim that all of us, or most people, or many do such and such. We should not merely repeat such numbers, but interpret them to deliver the best possible picture of reality.

And the really good news: We can do this without any heavy-lifting math. We do need to learn how the best statisticians – the best figurers – think. They can show us how to detect possible biases and bugaboos in numbers. And they can teach us how to consider alternate explanations, so that we won’t be stuck with the obvious when the obvious is wrong.

Clear thinking is more important than any figuring. There’s only one math equation in this book: 1 = 200,000. This is our light-hearted way of expressing how one person in a poll can represent the views of up to 200,000 Americans. That is, when the poll is done right. The chapter on polling tells you how to know when things go wrong.

Although News & Numbers is written primarily to aid journalists in ferreting meaning out of numbers, this book can help anyone answer three questions about all sorts of studies and statistical claims:

What can I believe? What does it mean? How can I explain it to others?

The Journalistic Challenges

The very way in which we journalists tell our readers and viewers about a medical, environmental, or other controversy can affect the outcome.

If we ignore a bad situation, the public may suffer. If we write “danger,” the public may quake. If we write “no danger,” the public may be falsely reassured.

If we paint an experimental medical treatment too brightly, the public is given false hope. If we are overly critical of some drug that lots of people take, people may avoid a treatment that could help them, maybe even save their lives.

Simply using your noggin when you view the numbers can help you travel the middle road.

And whether we journalists will it or not, we have in effect become part of the regulatory apparatus. Dr. Peter Montague at Princeton University tells us: “The environmental and toxic situation is so complex, we can’t possibly have enough officials to monitor it. Reporters help officials decide where to focus their activity.”

And when to kick some responses into high gear. During the early days after the Deepwater Horizon oil well explosion, both company (BP) and federal officials tended to soft-pedal and underestimate the extent of the problem. Aggressive reporters found experts who sharply hiked the estimates of how much oil was pouring into the Gulf. These journalists provided running tallies of the miles of shorelines where gooey globs were coming ashore, the numbers of imperiled birds and wildlife, and the number of cleanup workers feeling ill effects. Nightly news broadcasts showed graphic video of oil gushing out of the ground a mile under the Gulf. National attention was soon galvanized on the crisis; both government and industry action intensified.

Five Areas for Improvement

As we reporters seek to make page one or the six o’clock news:

1. We sometimes overstate and oversimplify. We may report, “A study showed that black is white,” when a study merely suggested there was some evidence that such might be the case. We may slight or omit the fact that a scientist calls a result “preliminary,” rather than saying that it offers strong and convincing evidence.

    Dr. Thomas Vogt, at the Kaiser Permanente Center for Health Research, tells of seeing the headline “Heart Attacks from Lack of ‘C’ ” and then, two months later, “People Who Take Vitamin C Increase Their Chances of a Heart Attack.” Both stories were based on limited, far-from-conclusive animal studies.

    Philip Meyer, veteran reporter and author of Precision Journalism, writes, “Journalists who misinterpret statistical data usually tend to err in the direction of over-interpretation … The reason for this professional bias is self-evident; you usually can’t write a snappy lead upholding [the negative]. A story purporting to show that apple pie makes you sterile is more interesting than one that says there is no evidence that apple pie changes your life.”

    We’ve joked that there are only two types of health news stories – New Hope and No Hope. In truth, we must remember that the truth usually lies somewhere in the middle.

2. We work fast, sometimes too fast, with severe limits on the space or airtime we may fill. We find it hard to tell editors or news directors, “I haven’t had enough time. I don’t have the story yet.” Even a long-term project or special may be hurriedly done. In a newsroom, “long term” may mean a few weeks.

    A major Southern newspaper had to print a front-page retraction after a series of stories alleged that people who worked at or lived near a plutonium plant suffered in excess numbers from a blood disease. “Our reporters obviously had confused statistics and scientific data,” the editor admitted. “We did not ask enough questions.”

3. We too often omit needed cautions and perspective. We tend to rely too much on “authorities” who are either most quotable or quickly available or both. They may get carried away with their own sketchy, unconfirmed but “exciting” data – or have big axes to grind, however lofty their motives. The cautious, unbiased scientist who says, “Our results are inconclusive” or “We don’t have enough data yet to make any strong statement” or “I don’t know” tends to be omitted or buried deep down in the story.

    Some scientists who overstate their results deserve part of the blame. But bad science is no excuse for bad journalism.

    We may write too glowingly about some experimental drug to treat a perilous disease, without needed perspective about what hurdles lie ahead. We may over-worry our readers about preliminary evidence of a possible new carcinogen – yet not write often enough about what the U.S. surgeon general calls the “now undisputable” evidence that secondhand tobacco smoke “is a serious health hazard.”

4. Seeking balance in our reporting on controversial issues, we sometimes forget to emphasize where the scientific evidence points.

    Study after study has found no evidence that childhood immunizations can cause autism – yet lay promoters (and some doctors) continue to garner ink and airtime on a popular daytime TV show (see Chapter 12).

    On the hot-button issue of “global warming,” we must not get carried away by occasional super-cold winters. Year-to-year temperatures vary by their very nature. Climate experts had to study decades of weather and interpret data going back thousands of years to detect the slow, yet potentially dangerous, warming of our planet. The bottom line: Most scientists now agree that global warming is real, and is linked to the burning of fossil fuels. The scientific and societal debate continues over details such as how urgent the threat might be – and precisely what to do about it. Another bottom line: Don’t write the nay-sayers off as kooks and tools of industry. Sometimes even very minority views turn out to be right (see Chapter 9).

5. We are influenced by intense competition and other pressures to tell the story first and tell it most dramatically. One reporter said, “The fact is, you are going for the strong [lead and story]. And, while not patently absurd, it may not be the lead you would go for a year later.”

Or even a few hours later. Witness the competitive rush to declare election-night winners, and the mistakes that sometimes result.

We are also subject to human hope and human fear. A new “cure” comes along, and we want to believe it. A new alarm is sounded, and we too tremble – and may overstate the risk. Dr. H. Jack Geiger, a respected former science writer who became a professor of medicine, says:

I know I wrote stories in which I explained or interpreted the results wrongly. I wrote stories that didn’t have the disclaimers I should have written. I wrote stories under competitive pressure, when it became clear later that I shouldn’t have written them. I wrote stories when I hadn’t asked – because I didn’t know enough to ask – “Was your study capable of getting the answers you wanted? Could it be interpreted to say something else? Did you take into account possible confounding factors?”

How can we learn to do better? How do we separate the wheat from the chaff in all sorts of statistical claims and controversies? That’s what the rest of this book is all about.

Notes

. Vogt, Making Health Decisions.

. Meyer, Precision Journalism.

. “SRP [Savannah River Plant] link to diseases not valid,” Atlanta Journal-Constitution, August 14, 1983.

. U.S. Surgeon General Richard Carmona, quoted by the Washington Post, June 28, 2006, in “U.S. Details Dangers of Secondhand Smoking.” Details of the second-hand risk are in the surgeon’s general report on smoking that was released at that time.

. Jay A. Winsten, “Science and the Media: The Boundaries of Truth,” Health Affairs (Spring 1985): 5–23.