Evaluation
NIRAS has many highly experienced evaluators on staff and in our expert networks, and a long-running track record of successfully delivering evaluations of all types and sizes – including impact, process and value for money evaluation, employing a wide variety of methods.
Our methodological expertise includes:
Theory-based approaches
Using realist evaluation, developmental evaluation and utilization-focused evaluation designs, and multiple methods, including:
- Contribution Analysis
- Process Tracing
- Contribution Tracing
- Rubrics and Scorecards
- Qualitative Comparative Analysis
- Outcome Harvesting
- Most Significant Change (MSC)
- Qualitative Impact Assessment Protocol (QuIP)
- Theory of Change (ToC) Analysis
- Social Network Analysis (SNA)
- Political Economy Analysis
Experimental, Quasi-experimental and Quantitative Methodologies
- Randomised Control Trial (RCT)
- Interrupted Time Series Analysis (ITS)
- Difference-in-Differences (DiD)
- PropensityScore Matching (PSM)
We also have experience of applying evaluation approaches at a range of different levels, from relatively small project/programme level evaluations to portfolio evaluations and meta evaluations. We have extensive experience in the management of regional and global evaluation framework contracts, including the UK FCDO Global Evaluation Monitoring Framework Agreement (GEMFA), the EU’s Framework Contract (currently SEA 2023) and for Sweden’s Sida since 2011. Our internal quality assurance processes ensure our services effectively meet client needs, and we also provide external evaluation quality assurance services, including for FCDO’s Evaluation, Quality Assurance and Learning Service (EQuALS).
In all our evaluations we seek to integrate Gender, Equity and Social Inclusion (GESI), including by applying an intersectionality lens in our data collection and analysis to help understand groups or individuals’ relative vulnerabilities or power
Dissemination of findings is a big part of our evaluation offer. We have extensive experience of presenting findings in engaging ways to different audiences using a variety of vehicles and tools, including presentations, webinars, blogs, infographics, videos and social media.
A participatory, iterative and adaptive developmental evaluation of the Ford Foundation’s BUILD model
The Building Institutions and Networks (BUILD) initiative was a five-year, USD1 billion investment in the long-term capacity and sustainability of 350 social justice and civil society organisations around the world, working on advancing civic engagement, freedom of expression, gender and racial justice, disability rights, natural resources and climate change, digital governance and equal access to technology. NIRAS conducted a developmental evaluation of the BUILD model, covering grantees in the US and Global South. With a participatory, iterative and adaptive design, the evaluation was able to flexibly respond to the changing contextual environment and the emerging findings. Employing a mixed-method approach, the evaluation placed a strong emphasis on strategic communication to build understanding, evidence, and influence.
The evaluation met both accountability and learning objectives, providing real-time feedback to inform programme adjustments, surfacing emerging issues as well as longitudinal impacts, and informed the next phase of BUILD. Inclusion and diversity are fundamental to the evaluation and NIRAS also reflected this in the selection of the evaluation team to ensure fair and balanced representation. Following the launch of BUILD 2.0 in 2021, NIRAS has been conducting a longitudinal evaluation of the new phase of the BUILD initiative.
A Randomised Control Trial (RCT)-based impact evaluation of a major climate resilience programme
The Building Institutions and Networks (BUILD) initiative was a five-year, USD1 billion investment in the long-term capacity and sustainability of 350 social justice and civil society organisations around the world, working on advancing civic engagement, freedom of expression, gender and racial justice, disability rights, natural resources and climate change, digital governance and equal access to technology. NIRAS conducted a developmental evaluation of the BUILD model, covering grantees in the US and Global South. With a participatory, iterative and adaptive design, the evaluation was able to flexibly respond to the changing contextual environment and the emerging findings. Employing a mixed-method approach, the evaluation placed a strong emphasis on strategic communication to build understanding, evidence, and influence.
The evaluation met both accountability and learning objectives, providing real-time feedback to inform programme adjustments, surfacing emerging issues as well as longitudinal impacts, and informed the next phase of BUILD. Inclusion and diversity are fundamental to the evaluation and NIRAS also reflected this in the selection of the evaluation team to ensure fair and balanced representation. Following the launch of BUILD 2.0 in 2021, NIRAS has been conducting a longitudinal evaluation of the new phase of the BUILD initiative.
Informing MasterCard Foundation’s work to address barriers for talented young Africans to succeed in higher education
NIRAS conducted both the mid-term and final evaluations of the initial two phases of the MasterCard Foundation Scholars Program (MCFSP) at the University of Edinburgh between 2020 and 2023. The MCFSP’s ultimate goal is to build a thriving network of transformative leaders contributing to social and economic transformation in Africa. These were mixed-method, theory-based evaluations with a focus on identifying key emerging lessons and informing ongoing implementation of the programme. The evaluations answered the questions of what was working well and what less well – by measuring changes resulting from the intervention through quantitative methods, based on analysis of management information and conducting online surveys – why and how – examining the processes leading to these changes and identifying key contextual factors through qualitative enquiry. During data collection a variety of channels for engagement were offered to capture stakeholder perspectives in the Global South as comprehensively as possible, and paid close attention to the extent to which there were gendered differences in the impact the programme had. Alongside the evaluation reports, NIRAS also created a suite of communications outputs to illustrate programme impact and put across key evaluation messages, including infographics, a high-level summary paper, and case studies profiling individual scholars.
Evaluating Norway’s inclusion of persons with disabilities in development aid programming
NIRAS wascommissioned by NORAD to carry out the Evaluation of Norway’s inclusion of persons with disabilities in development cooperation, covering the period 2010-19. This was a strategic, portfolio-level evaluation covering numerous projects and countries. The evaluation focused on:
(1) the organisation of and the capability of the Norwegian development administration to meet the normative commitments related to inclusion of persons with disabilities in development cooperation,
(2) the country level results in the education sector in Nepal, Malawi, South Sudan, and Uganda, and
(3) an in-depth study of the achievements made and remaining challenges regarding inclusion of persons with disabilities in selected projects in Nepal and Malawi.
A theory-based approach was used, involving reconstruction of theories of change to determine if the underlying assumptions on how to achieve disability inclusion were valid and effective,and tailored frameworks to assess disability inclusion in the aid administration (drawing on the UN Disability Inclusion Strategy monitoring framework) and to assess progress towards inclusiveness of education systems in partner countries (drawing on the International Disability Alliance Inclusive Education criteria framework).This approach was further explored in the in-country case studies, which explored extent to which theories of change aligned with and were for achieving outcomes in actual programming in Malawi and Nepal.
A developmental evaluation to test Gates Foundation’s SAGE model on gender integration
Since 2024, the developmental evaluation ofthe Sustaining Action for Gender Equality (SAGE) initiative aims to generate actionable insights and deepen Gates Foundation’s understanding of how gender mainstreaming can drive gender equality outcomes and amplify impact. Using a developmental evaluation approach, NIRAS is in the process of co-designing an evaluation that will help SAGE partners strengthen their strategies, inspire innovation, and enhance the effects of gender mainstreaming efforts during the grant period (2024-2028). In addition, our team provides capacity building through a MEL Clinic offering diagnostic assessments, tailored mentoring, coaching and tools to strengthen partners’ monitoring systems and enhance gender-responsive programming. Our innovative, iterative and real-time approach supports adaptive learning for partner organisations and program officers at the Gates Foundation.
