Everyone makes claims about what works. Politicians claim that stop-and-search policing will reduce violent crime; friends might assert that vaccines cause autism; advertisers declare that natural food is healthy. A group of scientists describes giving all schoolchildren deworming pills in some areas as one of the most potent anti-poverty interventions of our time. Another group counters that it does not improve children’s health or performance at school.

Unfortunately, people often fail to think critically about the trustworthiness of claims, including policymakers who weigh up those made by scientists. Schools do not do enough to prepare young people to think critically1. So many people struggle to assess evidence. As a consequence, they might make poor choices.

To address this deficit, we present here a set of principles for assessing the trustworthiness of claims about what works, and for making informed choices (see ‘Key Concepts for Informed Choices’). We hope that scientists and professionals in all fields will evaluate, use and comment on it. The resources were adapted, drawing on the expertise of two dozen researchers, from a framework developed for health care2 (see ‘Randomized trial’).

Ideally, these concepts should be embedded in education for citizens of all ages. This should be done using learning resources and teaching strategies that have been evaluated and shown to be effective.

Trustworthy evidence

People are flooded with information. Simply giving them more is unlikely to be helpful, unless its value is understood. A 2016 survey in the United Kingdom showed that only about one-third of the public trusts evidence from medical research; about two-thirds trust the experiences of friends and family3.

Not all evidence is created equal. Yet people often don’t appreciate which claims are more trustworthy than others; what sort of comparisons are needed to evaluate different proposals fairly; or what other information needs to be considered to inform good choices.

For example, many people don’t grasp that two things can be associated without one necessarily causing the other. The media sometimes perpetuates this problem by using language suggesting that cause and effect has been established when it has not4 — for instance, statements such as ‘coffee can kill you’ or ‘drinking one glass of beer a day can make you live longer’. Worse, exaggerated causal claims often pepper press releases from universities and journals5.

RANDOMIZED TRIAL

Pupils at a school in Uganda.Credit: Mikkel Ostergaard/Panos

The Informed Health Choices (IHC) Project was initially developed between 2012 and 2017 by a collaboration including some of the co-authors of this article (A.D.O., A.D., I.C. and M.O.). The project includes its own set of key concepts2, learning resources and a database of multiple-choice questions to assess how well users can apply the concepts.

In 2016, a randomized trial involving 120 schools and more than 10,000 schoolchildren in Uganda showed that these resources improved the ability of 10–12-year-old children to apply 12 of the key concepts7. These concepts included, for example, recognizing that personal experiences alone are an insufficient basis for claims about effects, and that small studies can be misleading.

In this trial, 69% of schoolchildren who were taught the key concepts passed a multiple-choice test of their ability to think critically about health claims. By comparison, just 27% of children who were not told about the concepts passed the same test.

 

Studies that make fair comparisons are crucial, yet people often don’t know how to appraise the validity of research. Systematic reviews that synthesize well-designed studies that are relevant to clearly defined questions are more trustworthy than haphazard observations. This is because they are less susceptible to biases (systematic distortions) and the play of chance (random errors). Yet results from single studies are often reported in isolation, as facts. Hence the familiar flip-flopping headlines such as ‘chocolate is good for you’, followed the next week by ‘chocolate is bad for you’.

To make good choices, other types of information are needed too — for example, about costs and feasibility. Judgements must also be made about the relevance of information from research (how applicable or transferable it is), and about the balance between the likely desirable and undesirable effects of a drug, therapy or regulation.

When it comes to carbon taxes, for example, policymakers need to consider evidence about the environmental and economic effects of such taxes, judge how comparable their context is with that of the studies and weigh how onerous the administrative difficulties are. They also need to model how tax burdens will be distributed across socio-economic groups and think about whether the taxes will be accepted in their jurisdictions.

Critical thinking

Individuals and organizations across many fields are working to enable people to make informed decisions. These efforts include synthesizing the best available evidence in systematic reviews; making that information more accessible, such as through plain-language summaries or open access; and teaching people how to use such resources. Examples of such review organizations are Cochrane (previously called the Cochrane Collaboration), which focuses on health care; the Campbell Collaboration, which looks at the effects of social policies; the Collaboration for Environmental Evidence; and the International Society for Evidence-Based Health Care. Others include the Center for Evidence-Based Management, the Africa Centre for Evidence, the International Initiative for Impact Evaluation (known as 3ie) and Britain’s What Works Centres.

Unfortunately, academics tend to work in silos and can miss opportunities to learn from others. The expertise of the authors of this article spans 14 fields: agriculture, economics, education, environmental management, international development, health care, informal learning, management, nutrition, planetary health, policing, speech and language therapy, social welfare, and veterinary medicine.

We have identified many concepts that apply across these fields (see ‘Key Concepts for Informed Choices’ and ‘Key concepts in action’). Some further concepts are more relevant in some fields than in others. For example, it is often important to consider potential placebo effects when assessing claims about medical treatments and nutrition; these are rarely relevant to interventions in the environment.

KEY CONCEPTS IN ACTION

A maternity ward in Dar es Salaam, Tanzania.Credit: Gary Carlton/Alamy

Claims

Beliefs alone about how interventions work are not reliable predictors of the presence or size of effects.

Most people feel that it is hard to influence parents’ engagement with their children’s education. The assumption is therefore that more intensive (and more costly) interventions would be more likely to be effective. However, studies of intensive interventions have often failed to show effects on pupils’ attainment, as measured using standard tests (see go.nature.com/2gfy8io).

Meanwhile, a recent evaluation of the effects of simply text-messaging parents weekly with updates about their child’s schooling had positive effects on children’s attendance, homework submission and mathematics attainment (see go.nature.com/2t7ormy). These effects were small, but the cost was very low. This illustrates that — contrary to our hunches — inexpensive interventions can be helpful, and expensive ones can fail.

Comparisons

Conditions should be as similar as possible.

‘Scared Straight’ programmes take young offenders on prison visits on the assumption that this experience and listening to inmates’ descriptions of life inside will deter juvenile delinquency. Some studies have found that such prison visits were followed by large reductions in delinquent behaviour. But a lot can change in a group of youngsters over time, including their becoming older and more mature. How can anyone know that the prison visits caused the reduction?

Fairer experiments were done in which youths were randomly assigned to visit prison or not, creating groups that were more comparable. Comparisons between these groups showed more delinquency in the youngsters who had been exposed to prisons than among those who had not8,9.

Choices

When there are important uncertainties about the effects of interventions, those should be reduced by fair comparisons.

In the health sector, financing schemes in which funds are released only if a specific action is taken or performance target is met have become popular. Billions of dollars have been invested in promoting these schemes in low- and middle-income countries, with the aim of achieving international development goals10. For example, health providers have been offered cash rewards for increasing the percentage of births in clinics (rather than at home), with the intention of improving maternal and newborn health and survival.

But performance-based financing schemes can have unintended adverse effects, such as encouraging health-care workers to falsify records or to neglect other activities. In Tanzania, some health facilities threatened new mothers with fines or denial of vaccinations for their children10. For interventions in which there is much uncertainty about the pros and cons, further fair comparisons should be done before or while rolling out such schemes.

 

Our collaboration has already prompted many of us to develop frameworks for specific fields and to suggest improvements to the original Informed Health Choices framework2. There is power in identifying an issue that resonates across different domains; it provides momentum to align efforts.

The Key Concepts for Informed Choices is not a checklist. It is a starting point. Although we have organized the ideas into three groups (claims, comparisons and choices), they can be used to develop learning resources that include any combination of these, presented in any order. We hope that the concepts will prove useful to people who help others to think critically about what evidence to trust and what to do, including those who teach critical thinking and those responsible for communicating research findings.

Next steps

Evidence-informed practice is now taught to professionals in many different fields, and these efforts must grow. It is also crucial that schoolchildren learn these key concepts, rather than delaying acquisition of these skills until adulthood. Young people who have been explicitly taught critical thinking make better judgements than those who have not6. Educating people about such concepts at a young age sets an important foundation for future learning.

An important part of the work of encouraging critical thinking is learning and sharing strategies that promote healthy scepticism, but which avoid unintended adverse consequences. These include inducing nihilism (extreme scepticism); allowing for disingenuous claims that uncertainty is a defensible argument against action (on climate change, for example); or encouraging false beliefs — such as that all research is untrustworthy because of competing interests among those who promote particular interventions.

Competing interests take various forms in different fields, but the challenges and remedies are similar: recognition of potential conflicts, transparency and independent evaluations. Achieving these depends on improved public understanding of the need for independent evaluation, and public demand for investment in it, as well as unbiased communication of findings.

Further development and specialization of the Key Concepts for Informed Choices is needed, and we welcome suggestions. For example, more consideration needs to be given to how these concepts can be applied to actions to address system-wide changes, taking into account complex, dynamic interactions and feedback loops, such as in climate-change mitigation or adaptation strategies.

We have therefore created a website (www.thatsaclaim.org) on which our key concepts can be adapted to different fields and target users, translated into other languages and linked to learning resources.