When counting things that matter, it matters what we count

The old adage suggesting that “you can’t manage what you don’t measure” carries a simple message:  achieving complex goals requires the collection of data on inputs, on progress and on outcomes.[1] We need to know precisely what, if anything, is changing as a result of actions we take toward a defined objective. As a corollary, the choice of indicators used to represent the target itself and the efforts involved is critical. We need data, yes. But they have to be the right kinds of data to be of any use.

There’s a lot of talk, these days, about data collection, indicator selection and metrics. It comes against a backdrop of calls for more donor and government accountability and the framing of a post-2015 global development agenda. On the one hand, there is growing demand for a stronger evidence base to policymaking and programming.[2] On the other hand, the process leading to the new Sustainable Development Goals (SDGs) has provoked much debate concerning appropriate metrics. That 17 over-arching SDGs are associated with 169 targets underscores how complex it will be to measure the success of such a multifaceted agenda.[3]

This matters to everyone, not just to statisticians and metrologists.[4] Development agendas should reflect priority concerns, and those concerns can only be prioritised by governments if they are able to assess the scale of a given need, determine the relative cost-effectiveness of potential interventions, and measure the success of supporting chosen interventions over time. In other words, being able to measure what matters lies at the very heart of development.

But we have a problem. It was recently pointed out that the data used to monitor progress of the now-expired Millennium Development Goals was “patchy and impossible to corroborate”, especially since fewer than 100 countries collect reliable data even on the most basic metrics of poverty.[5] In that context, three major agendas are being promoted worldwide to address what the United Nations is calling a “data deprivation” problem: these involve calls for i) a global data revolution, ii) open access to data, and iii) data innovation.

Data Revolution

Where countries and their governments cannot (or do not) measure what matters to human wellbeing there is a danger that data deprivation leads to a “denial of basic rights”. The clarion call for a data revolution is fueled by recognition that government actions need to be better informed by evidence: that is, by rigorous empirical data of what works and what progress is being made. This is essential for improved prioritisation of resources and for enhancing public accountability.[6]

A new brief by the Global Panel on Agriculture and Food Systems for Nutrition argues that few national statistical systems collect the data needed across all domains of food systems from production through consumption, and that no countries use empirical data to accurately measure the impacts of various agriculture and food policies on nutrition outcomes.[7] The Panel therefore joins the call on policymakers worldwide to enhance national statistical capacity to generate disaggregated data needed to support domestic and global priority-setting agendas, as well as timely assessment of policy impacts on diets and nutrition outcomes.

Of course, data collection, analysis and dissemination are not without cost. The financing gap associated with resolving such data needs has been estimated at between $100 million and $200 million per year. More recently, a report from the Brookings Institute suggested that the global price tag associated with filling data gaps on just health using existing mechanisms, such as population censuses, living standards and health surveys, would be roughly US$23 million per year.

Since the generation of good data is fundamental to good governance, such costs simply have to be factored into national government and donor budgets. For that to happen, the right kinds of data must be collected, used, and seen to be used. Which brings us to data access.

Open Data

Open Data has quickly become a gold standard. In calling for a data revolution, the GNR (2014) made the data that it used openly accessible through its online portal. Many donors, such as the United States Agency for International Development (USAID) now formally require recipients of grants or research awards to adhere to federal requirements for the sharing and curation of all data.[8] There are also multi-stakeholder/multi-donor activities such as GODAN – the Global Open Data for Agriculture and Nutrition initiative – which seek to support standardised approaches to prioritising the collection, and open access to, high quality data that can support accelerated gains in agriculture and food systems for nutrition.

Another thrust is the growing use of Freedom of Information Acts by researchers who seek access to unshared government data. Almost 100 nations now have such Acts on the books; most are high income countries, like Denmark and the United Kingdom, but some are lower income, such as India.[9]

Data Innovation

In tandem with calls for more data and open data, comes a related call for better data. The kinds of data currently used to measure problems and solutions relevant to agriculture, food policies and nutrition focus on certain narrow parts of the food system, such as farm output, aggregate food supply, commodity prices and anthropometric outcomes. These provide only a partial and indirect assessment of people’s food intake, dietary quality, or the drivers of food choices. As a result, it remains difficult for governments to intervene effectively to enhance food systems when constraints are poorly understood and impacts of policies are inadequately measured. New kinds of metrics are needed to measure things like quality of diets and dietary sufficiency, as well as food system efficiency and sustainability, including improved measures of nutrition-sensitive agriculture, healthy value chains and sustainable diets.

There is a number of new activities seeking to address such gaps. For example, the Food and Agriculture Organization (FAO) is undertaking an extensive inventory and assessment of existing indicators for food and nutrition security, aiming to determine the value of combining such indicators to generate novel indices, but also to identify knowledge gaps and the scope for developing novel metrics.[10] Another initiative is IMMANA (Innovative Methods and Metrics for Agriculture and Nutrition Actions). IMMANA is a grants and fellowships program aimed at accelerating the development of novel and innovative interdisciplinary metrics to fill key knowledge gaps in the space between agriculture, health and nutrition.

Let Them Eat Numbers?

Evidence-based policymaking requires good data. This is not a luxury, it’s a genuine cost of doing efficient business. A continuing weakness in policy formulation relevant to diets, food systems and nutrition is the lack of key data on which to base important decisions and frame relevant actions.

It’s a shame that the framers of the post-2015 Sustainable Development Goals didn’t embrace the issue of data deprivation and identify targets relating to data collection and sharing in the context of a global data revolution. It may not be too late; but for this to happen, calls for more data, open data, and data innovation need to coalesce and be amplified from all corners of the development community. Pande and Blum (2015) argued that governments which allow their statistical agencies to serve as generators of, and open clearinghouses for, development data should be lauded; others that hide, censor or ignore key data relevant to human needs should be held accountable for standing in the way of human development.

Dr. Patrick Webb is Alexander McFarlane Professor of International Nutrition at the Friedman School of Nutrition Science and Policy, Tufts University. He serves as the Policy and Evidence Advisor for the Global Panel on Agriculture and Food Systems for Nutrition, and Co-Director for the Fellowships programme under Innovative Methods and Metrics for Agriculture and Nutrition Actions (IMMANA).

Learn more about the Tecnical Brief on Metrics and Data

Images: Climate Change, Agriculture and Food Security (header), European Commission DG ECHO.