EnglishDeutsch

Misleading communication of risk: Editors should enforce transparent reporting in abstracts

Many scientific publications use misleading numbers to communicate risk. This article examines publications in medical journals and demonstrates how missing transparency constrains the accurate understanding of risks.

In 1996 a review of mammography screening reported in its abstract a 24% reduction of breast cancer mortality (1) ; a review in 2002 claimed a 21% reduction (2). Accordingly, health pamphlets, websites, and invitations broadcast a 20% (or 25%) benefit (3). Did the public know that this impressive number corresponds to a reduction from about five to four in every 1000 women, that is, 0.1%? The answer is, no. In a representative quota sample in nine European countries, 92% of about 5000 women overestimated the benefit 10-fold, 100-fold, and more, or they did not know (4). For example, 27% of women in the United Kingdom believed that out of every 1000 women who were screened, 200 fewer would die of breast cancer. But it is not only patients who are misled. When asked what the "25% mortality reduction from breast cancer" means, 31% of 150 gynaecologists answered that for every 1000 women who were screened, 25 or 250 fewer would die (3).

In 1995, the UK Committee on Safety of Medicines issued a warning that third generation oral contraceptive pills increased the risk of potentially life threatening thrombosis twofold. The news provoked great anxiety, and many women stopped taking the pill, which led to unwanted pregnancies and abortions—some 13 000 additional abortions in the next year in England and Wales—and an extra £46m (€55m; $71m) in costs for the NHS.5 Yet how big was the twofold risk? The studies revealed that for every 7000 women who took the earlier, second generation pills, one had a thrombosis, and this number increased to two in women who took third generation pills. The problem of misleading reporting has not gone away. In 2009, the BMJ published two articles on oral contraceptives and thrombosis; one made the absolute numbers transparent in the abstract, (6) whereas the other reported that "oral contraceptives increased the risk of venous thrombosis fivefold." (7)

These two examples illustrate a general point. Absolute risks (reductions and increases), such as from one to two in 7000, are transparent, while relative risks such as "twofold" provide incomplete and misleading risk information (3, 8). Relative risks do not inform about the baseline risk—for example, whether twofold means from one to two or from 50 to 100 in 7000—and without this information, people overestimate benefits or harms (3, 9) In the case of the pill scare, the losers were women, particularly adolescent girls, taxpayers, and the drug industry. Reporting relative risks without baseline risk is practised not only by journalists because big numbers make better headlines or by health organisations because they increase screening participation rates. The source seems to be medical journals, from which figures spread to press releases, health pamphlets, and the media.

An analysis of the articles published in the Annals of Internal Medicine, BMJ, JAMA, Journal of the National Cancer Institute, Lancet, and the New England Journal of Medicine, 2003-4, showed that 68% (150/222) failed to report the underlying absolute risks in the abstract. Among those, about half did report the absolute risks elsewhere in the article, but the other half did not (10). Similarly, an analysis of 119 systematic reviews in BMJ, JAMA, and Lancet from 2004 to 2006 showed that every second article discussed only relative risks or odds ratios (11).

Conveying relative risks without baseline risk is the first "sin" against transparent reporting. The second is mismatched framing—reporting benefits, such as relative risk reductions, in big numbers and harms, such as absolute risk increases, in small numbers (3). If we use the example of a treatment that reduces the probability of getting disease A from 10 to five in 1000, whereas it increases the risk of disease B from five to 10 in 1000, authors who use mismatched framing would report the benefit as a 50% risk reduction and the harm as an increase of five in 1000; that is, 0.5%. Medical journals permit mismatched framing. One in three articles in the BMJ, JAMA, and Lancet from 2004 to 2006 used mismatched framing when both benefits and harms were reported (11).

Have editors since stopped non-transparent reporting? To check the current situation, we examined the abstracts of all free accessible research articles published in the BMJ in 2009 that reported drug interventions. Of the 37 articles identified, 16 failed to report the underlying absolute numbers for the reported relative risk measures in the abstract. Among these, 14 reported the absolute risks elsewhere in the article, but two did not report them anywhere. Moreover, absolute risks or the number needed to treat (NNT) were more often reported for harms (10/16) than for benefits (14/27).

How can those who are responsible for accurate communication of risk do better? And who should be monitoring them to ensure that they do? Steps can be taken to improve the transparency of risk communication (12). Firstly, editors should enforce transparent reporting in journal abstracts: no mismatched framing, no relative risks without baseline risks, and always give absolute numbers such as absolute risks and NNT.

Secondly, institutions that subscribe to medical journals could give journal publishers two years to implement the first measure and, if publishers do not comply, cancel their subscriptions.
Thirdly, writers of guidelines, such as the CONSORT statement, should stipulate transparent reporting of benefit and harms in abstracts.

Publication

Gigerenzer, G., Wegwarth, O., & Feufel, M. (2010). Misleading communication of risk: Editors should enforce transparent reporting in abstracts. British Medical Journal, 341, 791-792. doi:10.1136/bmj.c4830
Download the article

Teaser: Falling Dice
© Foto Andrea Damm / pixelio.de

References

1. Larsson LG, Nyström L, Wall S, Rutqvist L, Andersson I, Bjurstam N, et al. The Swedish randomised mammography screening trials: analysis of their effect on the breast cancer related excess mortality. J Med Screen1996;3:129-32.

2. Nyström L, Andersson I, Bjurstam N, Frisell J, Nordenskjöld B, Rutqvist LE. Long-term effects of mammography screening: Updated overview of the Swedish randomised trials. Lancet 2002;359:909-19.

3. Gigerenzer G, Gaissmaier W, Kurz-Milcke E, Schwartz LM, Woloshin S. Helping doctors and patients to make sense of health statistics. Psychol Sci Public Interest 2007;8:53-96.

4. Gigerenzer G, Mata J, Frank R. Public knowledge of benefits of breast and prostate cancer screening in Europe. J Natl Cancer Inst2009;101:1216-20.

5. Furedi A. The public health implications of the 1995 "pill scare." Hum Reprod Update1999;5:621-6.

6. Lidegaard Ø, Løkkegaard E, Svendsen AL, Agger C. Hormonal contraception and risk of venous thromboembolism: national follow-up study. BMJ2009;339:b2890.

7. Van Hylckama Vlieg A, Helmerhorst FM, Vandenbroucke JP, Doggen CJM, Rosendaal FR. The venous thrombotic risk of oral contraceptives, effects of oestrogen dose and progestogen type: results of the MEGA case-control study. BMJ2009;339:b2921.

8.
Gigerenzer G, Edwards A. Simple tools for understanding risks: from innumeracy to insight. BMJ 2003;327:741-4.

9. Covey J. A meta-analysis of the effects of presenting treatment benefits in different formats. Med Decis Making 2007;27:638-54.

10. Schwartz LM, Woloshin S, Dvorin EL, Welch HG. Ratio measures in leading medical journals: structured review of accessibility of underlying absolute risks. BMJ 2006;333:1248-52.

11. Sedrakyan A, Shih C. Improving depiction of benefits and harms: analyses of studies of well-known therapeutics and review of high-impact medical journals. Med Care 2007;45:523-8.

12. Gigerenzer G, Gray JAM. Launching the century of the patient. In: Gigerenzer G, Gray JAM, eds. Better doctors, better patients, better decisions: envisioning healthcare 2020. MIT Press.