‘Context Matters’ framework for improving evidence use: what do policymakers and practitioners think about it?

In 2016, INASP and Politics & Ideas (P&I) produced the Context Matters Framework – a lens through which to examine the internal and external context in which an organization is working, identify the main factors affecting evidence use, and provide a way to systematically identify how they manifest in any given agency. It considers interrelated areas to help navigate complexity, so that one can better understand the organization, the blockages and opportunities for positive change towards evidence use.

In this post, Leandro Echt highlights some of the benefits and advantages of the Framework, as well as areas for improvement, from the perspective of a worldwide range of policymakers, researchers and other practitioners.

The ‘Context Matters’ Framework has been conceived as a living tool, an approach that is adapted and updated as we learn from its application in several contexts and from new perspectives. The Framework draws on the extensive academic literature as well as on interviews with more than 50 policymakers in Africa, Asia and Latin America, but in addition, we are constantly integrating new perspectives on the Framework from policy makers, researchers and practitioners spread around the world (through webinars, workshops or by actively requesting feedback). Two pilots with government agencies in Peru and Ghana have also provided us with valuable lessons about its applicability in different organizations.

Our intention is for the framework to build on – and contribute to – learning and wider community thinking about evidence-informed policy making. It is designed to dialogue and work alongside tools from other stakeholders, such as ODI’s Guidelines and good practices for evidence-informed policy-making in a government department. We are excited to be featuring our Framework in an upcoming book produced by international experts and the University of Johannesburg, and we have shared ongoing lessons with networks such as the LSE Impact Blog, the I2Insights and the GAIA communities. We have also published our learning in case studies, articles and interviews.

But how does the tool work from the perspective of those at the “coal-face” of evidence use? This is what policy makers, researchers and practitioners worldwide think about the Framework:

 How it adds most value

Raising organizational awareness on the value of evidence: Because it looks at evidence performance in several dimensions, creating a systemic picture of an organization’s work, it raise awareness on how evidence is a critical part of governmental agencies’ strategic and daily work.

Broadening the range of evidence: Some organizations and individuals work with a limited understanding of evidence, mainly linked to research and data. The Framework proposes to consider other types of knowledge that an agency is implicitly manages and that can also influence the work it does, such as citizen knowledge or evidence coming from reports and evaluation efforts.

Improving the research-policy interface: It helps researchers and research organizations understand the factors constraining and facilitating uptake of evidence by their public stakeholders, helping them anticipate the various aspects they need to consider when researching for policy as well as providing more informed policy recommendations.

Creating a sense of connection with the individual: The comprehensiveness of the Framework together with the way it is structured help users factor themselves in the complexity of the problem. As a human resource, as part of broader processes, as leaders, as knowledge providers, as a donor, etc, the framework interpolates every potential user as an important aspect in the broad and complex inter-connections of factors that influence evidence use in policy.

Providing a lens with which to surface “invisible” factors: The Framework looks not only at visible factors but also invisible. The explicit inclusion of the organizational culture, including incentives, motivations, values and beliefs related to the use of evidence have been highlighted as an important incorporation in the analysis of evidence-informed policy making.

Fostering participation and ownership: The capacity of the Framework to enable reflection, debate and lessons sharing through a consultative process (both with internal and external stakeholders) has been very valued by those who have been part of the process. A highly participatory nature also helps an agency to own the process, this increasing the potential for success in enhancing evidence-informed policy making.

Areas for development

Based on feedback from users, there are some aspects of the framework that we have prioritized for continued development:

Balancing comprehensiveness with practicality: While potential users acknowledge that the framework captures the complexity of the linkages between evidence and policy, it can also be a bit overwhelming because of its comprehensiveness and the many interconnected dimensions and sub-dimensions that it addresses. In concrete application of the framework with governmental agencies, we typically develop what we call “delivery frameworks” which pick a reduced number of dimensions to work with, based on the agencies’ needs and interests. Another way of facilitating the understanding of the framework could be merging some of the factors into composite ones.

Integrating power dynamics: Despite our strong belief that political economy is a critical issue in every aspect of the evidence to policy efforts, users struggle to identify how it deploys in the Framework. It might require additional effort from our side to make this more explicit, acknowledging the challenge of featuring such a transversal factor.

Developing a larger bank of case studies: As a matter of fact, potential users of the Framework demand more examples from its concrete application. Our experience in Peru and Ghana provide us with important lessons which we feature in several spaces. However, current and future use of the Framework to promote organizational change will certainly contribute to a better understanding of the Framework.

Supporting ongoing evaluation and improvement: Several potential users raise the point of how it would be to apply the Framework in an ongoing basis and keep track of its performance as well as compare themselves to other agencies. A checklist, an index or a score system as a tool of evaluation for governmental institutions could be further developed.

Offering solutions: Potential users of the Framework have expressed that while the Framework provides relevant support in the diagnostic stage, it is not very clear how it can help implement solutions to the main challenges identified. Thus, going from the diagnostic to a change plan seems feasible, but implementation of the changes is the main challenge. However, even though the Framework itself would not provide specific solutions to specific problems (this is left to the support of expert facilitators working jointly with agencies’ teams), we have developed potential uses of the Framework and identified more than 30 emerging practices in different developing countries that deal with challenges and opportunities implied in the main dimensions of the framework, which can inspire potential users in the search of solutions.

These perspectives provide us with a lot of food for thought to continue enhancing the Framework and its applicability in different organizational contexts. We are very thankful to all those who take some time to share their thoughts on the Framework with us. We are already using the Framework in joint work with other stakeholders, which will also give us more insights and experience to continue making this tool a relevant contribution to evidence-informed policies.

Do you want to share your thoughts on the Framework? Please do so by replying to this post at the bottom of this page, or feel free to write to contact@politicsandideas.org or info@inasp.info. Your feedback is most welcome!

Image: African policymakers and researchers discussing the Context Matters Framework at the Evidence-Informed Policy Making Seminar on Water-Energy-Food-Health co-organized by the European Commission, the African Academy of Sciences, UK Research and Innovation, the International Network for Government Science Advice and the South African Department of Science and Technology in Pretoria, South Africa (December, 2018).