Five lessons about the use of evidence in public policy in Peru
In an article first published in the On Think Tanks blog, Daniel Boyco of the Peruvian Alliance for the use of Evidence discusses promoting the use of research evidence in policy in Peru.
The Peruvian Alliance for the use of Evidence (in Spanish, Alianza Peruana para el Uso de Evidencia, will be referred to as ‘Alianza’ here) is conformed by a group of individuals and organisations interested in promoting a debate on the necessary methods, tools and capacities to generate a public culture that prioritises the critical use of evidence during the policy-making process.
The Alianza’s objective is to promote the use of evidence for public policy. Its been working since the beginning of 2014, organizing meetings and public events between its members to discuss mechanisms and experiences in Peru on the use of evidence for public policy. It is inspired by the Alliance for Useful Evidence, from the UK.
Since 2014, the Alianza has organized several meetings and events with members of organizations from the public and private sector, civil society and the international community, in order to discuss how to improve the use of evidence in policy making in Peru. So far, there have been 10 internal meetings – for members of the Alliance (which is open and anyone to join) – and three public events.
Currently, the Alianza is being funded by INASP under the VakaYiko initiative, which supports interventions to stimulate demand for use of evidence.
What have we accomplished?
The meetings provide a safe and dynamic space for the exchange of ideas in which organizations can present themselves and their work, while members of the Alliance ask questions and offer advice. Because the Alliance has members from all sectors, including Ministries and other public institutions, internal meetings create an opportunity where those who generate evidence can sit and discuss with those who produce it.
Public events also provide this common space and are enriched with interventions and discussions with academic researchers, university students, private consultants and other professionals. These public events, above all, are designed to encourage a public discussion.
The need for this debate has been widely stressed during these meetings and events.
What have we have learned?
[Note: this section draws from presentations and discussions held during the meetings and public events only.]
In this process, we have had the opportunity to learn a few things about how evidence is understood in the policy making process in Peru:
- Different definition – different hierarchies
- Different producers – public and private; international and local
- Not all evidence is used – for different reasons
- Not all sectors are sufficiently studied
- What the Alliance brings to the table
Below you can find a few of our reflections.
- Not everyone understands evidence in the same way
The use of evidence in the public policy making process is important because it can improve the quality of public policy and their outcomes. But, how do we define evidence?
According to Aníbal Velásquez, former Peruvian Health Minister and a former expert in evaluations for the Ministry of Development and for USAID, for evidence to be useful to policy making, it needs to be scientific. This implies defining evidence is as rigorous and methodically obtained information. In that sense, evidence is opposed to prejudices, assumptions and personal interests. This is a definition often preferred by most think tanks and researchers.
But are rigorous, scientifically understood research methods the only valid form of producing evidence that is useful to public policy?
We’ve learned from our discussions that there is not one single type of evidence used in public policy. On the contrary, evidence can range from hard quantitative data to subjective opinions and perceptions captured by qualitative methods – or even by hearsay. And there are various grey areas in between.
There are different hierarchies of evidence. But how (and by whom) are they determined?
This depends on who is producing the evidence, who is demanding or using it and what will it be used for. The evidence in itself is not objective, as it is subject to values, assumptions and prejudices on what constitutes “real” evidence for this or that process.
For example, the Ministry of Economy and Finance (MEF), in its Budgeting Based on Results scheme, states that evidence for programmatic design and management needs to be produced through a monitoring and evaluation process. And for evidence to be considered useful in this process, it needs to be quantitative. Because this process serves to determine budgeting for public policing, the ability of a social programme to quantify its results is of great importance for its sustainability.
This, should be noted, does not mean that ministers, advisors or anyone working at MEF considers qualitative evidence as inappropriate for policy making. But the policy making instrument, results-based budgeting, demands quantitative evidence to be given priority.
What about that which cannot be measured? Can the success of a social programme be based only on what can be quantified? Some things need to be explained or described rather than measured.
The United Nations Office for Project Services (UNOPS) works to produce and feed social programmes with real-time evidence during their implementation. Their reasoning goes as follows: social programmes work to alter and improve the livelihood of people, which means altering unwanted behaviours that perpetuate poverty (such as not taking children to school or to the health centre). Because altering people’s behaviour is rarely an easy task, if a programme does not feel the need to re-visit its original design after implementation, chances are that they are not looking properly on how things are working out.
This not only requires the programme be flexible, but also conditions the way evidence is collected. In their evaluation and monitoring framework, UNOPS uses a representative sample to collect quantitative data, but fieldwork researchers are encouraged to collect qualitative evidence as well, which is then prioritized in the analysis. Perceptions and opinions from the beneficiaries on the quality and usefulness of the programme’s results are considered more important than raw statistics.
Of course, the power to prioritize quantitative over qualitative evidence is affected by the institutional nature of the organisztions. UNOPS, as a UN agency, may be able to engage in a more flexible discussion on the pros and cons of different kinds of evidence. MEF, as a ministry, has to operate within strictly monitored budgetary lines.
Ipsos Perú, a market and social research company that regularly produces evidence for the public sector, stated the importance of combining quantitative data with qualitative information. This is because quantitative data is useful for being indicative, but qualitative tends to be more explanatory.
And other types of information can prove to be as powerful. Testimonies of violence perpetrated by the State during conflictive times captured by truth commissions may not be statistically representative (and may not intend to), but can certainly impact public opinion as well as public policy. Also, testimonies of rural schoolteachers, whose work is affected by lack of technical support, can also be a powerful source of information that may serve to contrast statistics and quantifiable evidence.
In this tension to determine the hierarchy of evidence there is, of course, a question of power. Although it may be true that different actors value evidence in different manners, these actors will surely have different levels of influence over the policy making process. UNOPS, when monitoring social programmes may provide recommendations, but their work also depends on convincing public officials to follow their advice [Note: in On Think Tanks, we have written a great deal on the importance of communications]. MEF, on the other side, determines the public budget, and its institutional appreciation of quantitative data automatically creates an impact on how public policy is evaluated.
The means by which evidence can be incorporated into the policy making process will therefore be conditioned to the values and opinions on what constitutes useful evidence, primarily by the policymaker, but also of by those who produce it, as well as, especially in certain cases, of public opinion.
- Many different actors produce evidence for policy making
Different sort of institutions produce the evidence required for public policy. Think tanks and policy research centres certainly have an important role to play, but as Peru’s think tank scene has yet some way to go as a community, we have taken care to open the Alianza to members from other sectors as well.
From the think tank side, the Alianza has engaged with GRADE, the National Health Institute (INS), Soluciones Prácticas, and CIUP, each presenting how they produce evidence for their own sector (GRADE specializes on education, the INS on health, Soluciones Prácticas on rural development, and CIUP on economic and social research). They are all different kinds of think tanks, too: GRADE is a private research centre, INS is a public sector research centre, Soluciones Prácticas is part of an international NGO, and CIUP is a university based research centre.
CONCYTEC, a public institution that manages State’s funds for research and innovation, also joined us to explain their work. From the private sector, we’ve had Ipsos Peru, a company specialized in doing research for clients that range from extractive industries to local governments, as well as ministries and several other institutions from the public sector.
Finally, from the international community, we invited United Nations Office for Project Services (UNOPS), who presented their monitoring and evaluation system.
We can group these into three categories: academia and civil society, public sector, and private sector. We’ve found that this distinction helps us understand the type of evidence produced by these institutions.
What challenges does evidence produced by “academia” (research centres, think tanks, NGOs and other CSOs) face?
Several institutions in Peru conduct research on a regular basis and try to use it to influence public policy (although perhaps not as much as in other countries of the region). The quality of this evidence is guaranteed by the prestige that these institutes have. As they belong to an international academic community, their findings tend to be published in national and international journals or reports, which have (or strife to have) strict and rigorous controls on what is methodologically correct.
The use of evidence produced by academia faces several challenges.
Because of its strict methodological procedures, it is often unavailable at the right moment in the right time. As timing is usually an issue in the policy making process and academically minded researchers tend not to invest enough in communications, this can lead to academic evidence being under-used.
Does academic research in Peru converse with the policy making process? Yes and no. Mainly because there aren’t many spaces in which academia and public sector can share information (such as the Alianza).
Soluciones Prácticas, in its presentation, stressed the importance of a good communication strategy in order to guarantee that results and information produced is taken into account. That requires think tanks and academia to take into account who their audience is, whether it is the general public, policymakers, professionals from the international community, or any other.
It’s not unusual for think tankers in Peru to complain that their studies aren’t being taken into account. But it is also true that many researchers from important think tanks (such as GRADE, CIUP and IEP) tend to split their career between academia and public service. So, while there may not be formal spaces for engagement, the researchers themselves fulfil this role as boundary workers.
How is evidence produced by the public sector? What does it imply?
Another important (and increasingly so) source of evidence for public policy is the public sector itself.
First, there are public sector think tanks, such as the National Health Institute that has a new technical arm, UNAGESP. At its presentation to the Alliance, UNAGESP explained that it conducts research at the request of the Ministry of Health, but remains independent by receiving funds directly from the Ministry of Economy and Finance. This allows it to, if necessary, advise against Ministerial policy, based on the evidence it produces or gathers.
But are all public think tanks as independent as the INS? And, how public is the evidence generated. Unlike academic researchers whose career incentives point (increasingly) towards publication and dissemination, publicly funded and managed research in Peru tends to remain unpublished and unshared (even within the government).
Another kind of evidence produced by the public sector comes from the various evaluations that public institutions have to conduct, according to the new Budget Based on Results (BBR) system that is being implemented by Peru’s MEF.
Budget Based on Results is a system being implemented by MEF that implies restructuring the traditional way that public spending is managed in Peru. In its presentation to the Alliance, MEF explained how the BBR requires public policies to be evaluated and monitored during their four main stages: design, budgeting, implementation and accountability. This implies moving away from a causal and linear frame of mind (where evaluation comes at the end) to a more flexible scheme, where the programme’s objective can be subject to change if evidence proves this to be required.
In the BBR scheme, each part of the policy making cycle is being evaluated.
Evidence produced by the State or commissioned by the State may share the same methods as that produced by academia – and in fact, the same researchers may be involved – but it enjoys greater power in public policy-making processes such as that mandated by the BBR. A key argument used by the public sector is that these studies and evaluations benefit from information that academics and the private sector do not have access to.
How does the private sector produce evidence for policy making? Could it be made more public?
Private institutions conduct research for the public sector in the form of consultancies. Ipsos Peru, which presented its work to the Alliance, is an example of this. The information it produces, however, is private.
Currently, there is no formal mechanism that enforces public institutions to disclose results from research it conducts or requests. There is plenty of information being produced for public offices in the form of consultancies, but this information is rarely shared. Therefore, this information is being under-utilized, as it is only available to those who requested it. This is an important barrier in the use of evidence for public policy in Peru.
Then there is evidence generated by the private sector for its own purposes. For example, utility companies generate data on their clients and the use of their services; agro-industrial companies generate data on soil quality, new species of seeds and water quality; and tourism operators have data on travellers that is far more accurate than the data generated by the State. How is this evidence used? Is it?
Here is a good example of how utility companies can use evidence for the public’s benefit.
As we have seen, Peru is not a country that lacks production of evidence for public policy –certainly on a number of policy issues such as economic policy and trade. Whether it’s from CSOs, private institutions or public research centres, there is much demand for it, especially from the technical cadres within ministries. And this is extending to all levels of government, albeit slowly. However, several barriers still exist that prevent public servants from using it.
- Not all evidence is used – but this is OK
What barriers prevent or discourage the use of evidence in public policy in Peru?
One is timing. A sense of urgency in public planning is derived from the fact that terms of office are short (four years for local governments – if they are not disrupted for political reasons) and administrative processes are long. The use of evidence, either when producing it or when using existing studies, takes time and doesn’t always coincide with the demand for it.
So, evidence is being underused because it tends to be unavailable at the right moment. Openness and a culture of sharing information would help change this. Currently, there is no legal enforcement in Peru for public institutions to disclose the research they conduct or request from private institutions. This generates a situation in which public information becomes private, and limits the possibility of it being used. We think that if it’s paid by public money, it should be available for everyone. With more information available, the overall quality of evidence would also improve and, with it, so would the quality of public spending.
Another barrier is that traditionally there haven’t been any formal mechanisms that conditions public spending to the use of scientific evidence, so it’s common for public policy to be produced based on beliefs, ideologies and assumptions, rather than robust evidence. This, however, has been changing both as a consequence of the institutional changes that MEF is leading and the incorporation of new technical cadres within government.
The limited technical capacity of some public servants has been repeatedly mentioned as a key problem at several meetings. This capacity refers to their limited ability to understand research results and to be able to use them. Although this may be true, it can also be presented as a communications challenge to academia and think tanks, which can (and should) use more innovative methods to communicate effectively their results and investigations, taking into account the capacity of their audience (that is, if they want to remain relevant).
Finally, evidence is not the only factor that should inform public policy. Personal and public values (even political calculations) may play a role in the formulation of policy. We must recognize that policy makers are responsible to the public – which means that they ought to respond to their own interests, beliefs and values, as much as to the facts produced by research. In the end, policy is a matter of values: research can only tell us what is but not what to do.
- Not all public policy issues are supported by sufficient evidence
Briefly, and this will be a subject of future events, not all sectors receive the same attention from researchers and policy makers when it comes to funding and producing evidence. And different sectors’ hierarchies of evidence are different.
- What does the Alliance bring to the table?
In this context, the Peruvian Alliance for the Use of Evidence aims to create a space in which to discuss these issues with those directly involved in them. As it turns out, there is a growing interest on the subject, but few opportunities to learn more about it and take the debate forward.
The meetings organized have been received with wide enthusiasm between experts, and we seek to continue expanding the debate.
What comes next? The Alliance will continue its activities for the rest of the year. Public events have had an increasing interest, as well as the internal meetings. We have yet to invite members from media, other public offices and the international community.
This article first appeared on the On Think Tanks blog.