EIPM Readings #1: Scaling up evaluation as a source of learning and improving programming in Mexico

As part of the EIPM readings, Clara Richards summarizes the case of Mexico, a country that has developed a Monitoring and Evaluation system that has slowly changed the culture of using information from different types of evaluations to improve policy and programmes. The paper is called Mexico’s M&E system: Scaling up from sectoral to the national level. It is written by the World Bank as part of their Independent Evaluation Group (IEG) Working Papers series.

Mexico was able to move from limited use of evaluation to greater use in government decision-making. It demonstrates a gradual construction of performance-based management institutions, taking into account the international experience but also introducing a high degree of innovation and country-specific development. The paper gives an overview of the institutional reforms introduced, the policy decisions and the operational steps, taking into account the political context that allowed this development.

The emphasis on performance management in Mexico’s public administration was influenced by legislation that ensures long-term sustainability and impact evaluations of social programmes such as the popular Progresa/Oportunidades, an anti-poverty programme of the Mexican government. It was the first programme in Mexico to have independent evaluations of the programme. There was a special emphasis on collect accurate data to measure effectiveness and evaluate the impact. It played an important role in putting evaluation on the forefront of development work. This attracted national and international interest and the social sector’s M&E system developed further. A conclusion of this paper is that after the 2005 reforms, Mexico entered into an advanced stage of performance-based management that is grounded on strong institutional foundations.

As the authors point out, some of the factors that were key to this development are listed below:

• In the 1990s political events in Mexico led to the institutional strengthening of government accountability (prior to this there had been little incentive for transparency and accountability).
• A series of laws created the legal framework for M&E to get institutionalized. These created CONEVAL (National Council for the Evaluation of Social Policy) with the remit to measure national results in poverty reduction. CONEVAL was an opportunity to modernize government and decision-making processes
• High-level officials led the reforms with special interest to modernize the budget and to facilitate evaluation of public expenditure
• The role of Congress, linked to the end of the single-party regime, found a way to exert control over the executive to ensure accountability over social spending
• CONEVAL’s special characteristics: unique governance structure and the fact that satisfied the demand for transparency and accountability from groups such as the media and CSOs
• Institutions like SHCP, SFP and the President’s office were highly involved to ensure that evaluation methods can effectively match the demands

Lessons learnt:

It is useful to highlight the lessons derived from the Mexican case, and it can be useful for other countries to strengthen the use of information in policy making and programming:

• Mexico capitalized on the 2005 reform momentum and conformed to multiple partnerships. Of course, these created overlaps in agencies’ roles which in turn created tensions. However, interested government agencies worked to keep the process moving forward, without stopping to formally define their respective functions (this resonates with adaptive approaches described by Andrews and Prittchet, which I’ll include in my future blogs). The authors highlight the importance of keeping a balance between organic development and over formalizing roles and responsibilities.
• The introduction of M&E tools needs to respond to both political and technical requirements. The political transition to democracy was a window of opportunity, specifically because this created calls for accountability in government. But it was also a technically planned effort in response to Congress’ political decisions to demand performance evaluations of all government programmes. I’d like to highlight here the fact that that the Congress had a specific role to play and wonder if it’s useful for other legislative institutions to learn more about how to influence executive’s processes. Ian Goldman also highlights this here.
• There is a risk of overly engineering the M&E system; there is only so much information that can be absorbed. It is important to identify the sequence and types of evaluation that needs to be implemented. Excessive development of methods and production of information can weaken the demand for information!
• There is a specific section of the paper that discusses the actual use of the information produced. The case shows that proactive steps are needed to integrate performance information in decision making and accountability processes. I would like to highlight an important point the authors make regarding context; in countries with legal tradition, these processes are supported by laws but in countries with managerial tradition, the use of this information will depend more on largely accepted practices.
• I like the emphasis on collective action rather than on a single ‘champion’ to drive change: in this case, a combined effort of multiple agencies which was combined with the opening up of the political system helped to overcome a legacy of intense mistrust in government.
• Of course, there are still challenges to overcome:
  • Enforce the actual use of results
  • Risk of information overload and agency fatigue
  • Developing evaluation capacity, there is still a relatively short number of these experts in universities and consulting firms. Opportunity for universities to introduce more training-related knowledge products
  • Managing change processes, which often create confusion and frustration. Overcoming the resistance of the civil service, interested in maintaining the status quo is a major challenge
  • Institutional coordination, especially between political and technical levels of government agencies, between agencies, congress and control entities. Finding the balance between formalizing roles but nor over-regulating
  • Strengthen the role of planning for effective performance-based management, including harmonization with budgeting processes.
  • There’s a need to standardize protocols and process and develop capacities to manage information by management staff. If decision makers perceive that bad information is feeding into the system their credibility and long-term sustainability can be damaged.

The first part of the series was published last week in the Research to Action website. Please, do get in touch with me if you have come across something interesting that I should include in this reading list. Special thanks so far for those who have pointed to the useful reading material .