4.1. General points of attention

The OECD defines monitoring as ‘a continuing function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an on-going development intervention with indicators of the extent of progress and achievement of objectives and progress in the use of allocated funds’.[20] It is an ongoing process of data collection in which the data collected must be used not only for accountability purposes, but also to inform programming decisions and improve programme implementation in real time. Effective monitoring plays a crucial role in making programmes flexible and adaptable to changing contexts, which is particularly relevant in complex and volatile situations of conflict and violence.[21]

For this reason, monitoring programme implementation progress in situations of conflict goes beyond simply reporting on planned versus actual activities and results. There is much more to monitor than programme indicators, and when donors and implementers create a monitoring plan, they should include a variety of monitoring processes, such as: conflict monitoring; implementation and quality monitoring; monitoring theories of change and assumptions; risk monitoring; monitoring value for money; and monitoring for learning.[22] Yet, if often happens that donors and implementers base their monitoring process largely on programme log frames, indicators and M&E plans. If these are not updated regularly to address the changing dynamics of a conflict and to trigger activity adjustment, they are not appropriate for guiding monitoring efforts and cannot inform effective decision making.[23]

Data and information are the foundations of a good monitoring practice. However, too often people confuse data collection with monitoring. Programme managers need to exercise the discipline to routinely collect, analyse and reflect on information about their programme at both activity and outcome levels.[24] In other words, a monitoring strategy must include elements to allow for information uptake and utilisation. A robust and adequate monitoring plan that informs flexible and timely decision making on programming in situations of conflict should therefore contain analytical, organisational and procedural elements that both donors and implementing organisations need to have in place.

4.2. Monitoring in conflict settings in practice – options for donors

In conflict settings where security challenges are so severe that donors cannot be physically present on a (semi-)permanent basis, there are roughly speaking three options for monitoring the progress of programme implementation: (1) (self)-monitoring conducted by (local) implementing partners; (2) third-party monitoring; and (3) monitoring via modern technology. In practice, donors can choose either one of these options, or a combination of them, as they are not mutually exclusive.

4.2.1. Monitoring through (local) implementing partners

Notwithstanding the fact that there are many guidance documents and toolboxes that provide overviews of different tools and best practices to monitor programme implementation in fragile and conflict-affected settings, there is no standard framework. Also, the politicisation of aid and the high security risks of implementing aid activities in conflict zones mean that organisations are reluctant to go into detail about their monitoring practices or share examples of risks they did not adequately monitor or respond to.[25] The majority of the implementing organisations we spoke with rely on local implementing partners to conduct their monitoring of implementation. There are several challenges that need to be taken into account (also see table 3), two of which we highlight here:

The implementing partner can lack the capacity to properly monitor, for example it may have no experience with receipts or lack other similar documentation and training.

The implementing partner could be incentivised to inflate its monitoring reports to make itself appear in a more positive light.

In line with this, the two key factors for effective monitoring that were underlined time and again were trust and continuous capacity building. As one interviewee put it, ‘We have to have strong relationships of trust and respect between us and our partners.’[26] Situations will inevitably arise where implementing partners need to be pragmatic – for instance, when they must get around a roadblock and pay a fee – and it is key that partners are able to have an open conversation with the funder about how to engage in such situations. As these are oftentimes not black-and-white issues, emphasis is placed on creating an environment where partners can talk about the challenges and work through them together. However, that begs the question whether there is enough ‘distance’ between the partners to critically assess processes and results. Tools that are typically applied to balance this are internal anti-corruption policies, codes of conduct, whistle blowing policies, and vetting and compliance procedures for partners and contractors.

The interviews underlined that for this approach to be workable, continuous training of local staff is a must. An NGO active in Somalia, for instance, explained that they invest in their partners and train their senior staff in financial management and new financial systems. Every year they invite the finance officers of local partners to their headquarters in Nairobi and train them there. They underlined that they are involved in building the financial management capacity of their partners over time and did not simply provide one-off training sessions. They also developed new monitoring instruments in collaboration with their partners. For instance, they developed a standard receipt for the implementing partner to use in areas where there are no receipts and partners struggle to account for expenses.[27]

When asked about what happens when irregularities occur, interviewees indicated that a limited degree of aid diversion tends to be reluctantly endured as part of the cost of providing aid in conflict settings – by donors and implementing organisations alike. While formally zero-tolerance policies apply, in the day-to-day reality at activity level, aid diversion is tolerated in certain cases to achieve more important objectives, such as gaining the acceptance of host communities and key stakeholders, and access to beneficiaries. Conflicts always feature gatekeepers (e.g. militia, armed actors, former warlords becoming ‘deputy governor’, government actors, etc) with whom donors and implementing organisations must negotiate, and who will expect some sort of payment.[28] Implementing organisations have come up with innovative accounting solutions, where payments for passing checkpoints for instance are shown as ‘transport costs’. Just as commercial entities may account for payments to armed groups as additional ‘security consulting’ costs.[29] Interviewees emphasised that aid diversion is not only linked to activities built around the delivery of hardware and cash transfers, but also applies, for example, to capacity-building activities where the costs for people participating in training or a workshop are covered.[30]

There has been debate in the aid community on the trade-offs between aid diversion and aid impact in the knowledge that, in certain situations, some of the former may be necessary to achieve the latter. In the end, it is a matter of priorities. Funders, such as the Netherlands, who have a zero-tolerance corruption policy, will inevitable face corruption at the level of programme implementation – whether they like it or not. This cannot be sanctioned but it is nevertheless a persistent reality and therefore it makes sense to discuss upfront how to deal with it. ‘Principled pragmatism’ seems a sensible way to go about it but such a label needs to be operationalised to serve programme implementation purposes.[31]

For now, responses to irregularities that might come out of monitoring exercises related to aid diversion are mostly of an internal, forward-looking nature – for example, re-assessing corruption protocols, training staff once more in rules and regulations on procurement, etc. In essence, this is a management holding response to a persistent and structural phenomenon that will not disappear, based on the assumption that regular attention will reduce frequency of occurrence. There is little evidence that the assumption is warranted, however. Another option would be to turn to external auditors and third-party monitoring mechanisms to ensure independence and compliance. However, there are limitations to this as well, as explained in section 4.2.2.

Good practices for monitoring through local implementing partners include the following:

Focus on building up long-term partnerships and trust relations.

Invest in continuous capacity building, especially in terms of financial management.

Create an environment where donors and partners can talk openly about the challenges and nuances of operating in conflict settings.

Put protocols and procedures in place that reduce risks, such as anti-corruption policies and training, codes of conduct, whistle blowing policies, and vetting and compliance procedures for partners and contractors.

Develop specific and pragmatic guidelines for dealing with corruption and nepotism that take relevant protocols, procedures and policies as starting points, but adjust them to the local context.

4.2.2. Monitoring through third parties

One common approach to remote monitoring is third-party monitoring, in which donors contract external third parties to collect and/or verify monitoring data. Third-party monitoring has several advantages: it provides independent eyes and ears on the ground, allows validation of monitoring data from implementing partners where confidence in partner reporting is lacking, and can sometimes allow for more cost-efficient field monitoring. It is typically most useful for verifying quantitative and physical outputs of aid projects. However, there are downsides and challenges to take into consideration as well (also see table 3):

The time and resources required to make third-party monitoring work are often underestimated by commissioning agencies.

Third-party monitoring can lack the technical understanding to design effective monitoring tasks.

The quality of reporting is often seen as sub-par (and objectivity may be questioned).

There are reputational risks from third-party monitoring staff actions that need to be mitigated.

There is a significant risk transfer to monitoring staff (especially where monitoring providers lack adequate security systems).

Third-party monitoring can negatively affect context understanding and acceptance where aid agencies use it as a substitute for regular internal monitoring.

The use of third-party monitors (TPMs) has been perceived by some implementing partners as contributing to donor micro-management.[32]

Organisations that have used TPMs in Syria and Somalia and which were interviewed for this study reported having run into these challenges. Aid workers with experience in Syria mentioned that TPMs are not always neutral, as repeat visits to the same area or repeat contracts with donors can compromise impartiality. One aid worker asked, ‘How do I know my staff is not bribing the TPM to write down a positive report? I would rather rely on the implementing partner that I personally know than a TPM that I haven’t met. You don’t know who you are giving your money to or who can influence them. TPMs are often locals so how can you verify that they don’t have conflicts of interest or take bribes?’[33] In Syria TPMs have also at times damaged existing trust with the local community, as monitoring staff are often not trained in the do-no-harm principles. There have been cases of third-party monitoring staff taking photos and asking questions that have confused and troubled local beneficiaries to the point where they later refused assistance.[34] Similarly, organisations operating in Somalia have run into challenges with TPMs. One aid worker mentioned that in their experience most third-party monitoring providers were unwilling to travel outside of the larger cities like Mogadishu and Kismayo, while their monitoring capability is basic: a TPM can check whether a road has been built or a well dug, but struggles with monitoring less tangible results of, for example, peacebuilding processes.[35]

Although existing literature and the interviews conducted for this study are more critical of third-party monitoring than of local implementing parties monitoring, both share the same broad limitations: capacity, professionalism and trust can all at times be less than desired. Capacity might be inadequate or limited (e.g. technical and financial skills, staff training, geographic coverage) while the professionalism of the monitoring party to monitor objectively could be called into question. This (lack of) professionalism subsequently feeds into the relationship of trust with the donor and affects the way monitoring information is used. It is also important to note that some of the criticisms specific to TPMs come from local implementing partners, who are understandably reluctant to have an external party brought in to evaluate their work while they themselves might be experiencing similar limitations.

Overall then, TPMs can strengthen compliance in areas where access is limited, and can therefore meaningfully contribute to monitoring. However, third-party monitoring should ideally complement rather than substitute for monitoring conducted by an agency’s own staff.

Good practices for monitoring using TPMs include the following:

Ensure there is clarity on roles, tasks and expectations vis-à-vis other monitoring entities.

Focus on acting as sparring partner and active feedback rather than control and accountability only.

Anticipate the need for time and resources to set up and maintain effective third-party monitoring systems.

Ensure that the information collected can be used to inform decisions.

Use technological devices to increase control over field monitoring.

Strengthen security protocols and duty of care.

Coordinate use of TPMs and exchange on emerging lessons.

Regularly reassess third-party monitoring and its alternatives.[36]

4.2.3. Monitoring via modern technology

New developments in technology allow for remote monitoring of projects. Yet, technology in conflict settings faces several constraints and needs. For example, it must be able to function without constant electricity supply, without reliable mobile and Internet access, across large distances, and without advanced computing skills. Bearing those restrictions in mind, four technology applications are well suited for conflict settings: handheld devices for digital data collection, mobile phone-based feedback mechanisms, remote sensing with satellites or delivery tracking, and broadcasting with radios and other forms of media.

The benefits of these types of technologies include the following:

rapid and near real-time monitoring;

ability to collect different types of data to asses programming;

a chance to track indicators more systematically;

costs savings (no costs for transportation, printing, staff hours, etc);

opportunity to improve capacities and collaboration (it is easy to share a wide range of documents online – both monitoring information as well as training – though there are security constraints to consider here as well).

However, there are also several challenges associated with the use of technology-based remote monitoring:

systematic bias or errors;

selection bias, safety and security;

training, logistics, privacy and security;

complexities of data and interoperability;

interpretation of data;

verification of data.

Technology-based approaches are, for instance, being used in Syria to improve the accountability of international NGOs and civil society organisations to their beneficiary populations. In one example, beneficiaries were able to take photographs of supplies given to them by an implementing partner that they judged to be of low quality, and then uploaded the photos to social media and wrote to the funder of the programme. The funder subsequently followed up with the implementing partner to improve the quality of the goods.[37] Similarly, satellite imagery provided by companies like Planet Labs and Digital Globe is being used in both Syria and Somalia to track developments in the conflict (e.g. destroyed infrastructure) as well as humanitarian developments (e.g. construction of new shelter or agricultural project).[38]

Good practices for remote monitoring using technology-based approaches include the following:

Take the time to study the context before choosing tools.

Involve all users actively and establish informed consent practices.

Provide back-ups and alternatives.

Use security-conscious, free and open source software.

Minimise and limit data.

Invest in building acceptance and training.

Pool funds and risks.

Apply humanitarian principles to technology.[39]

4.3. General good practices for gathering and using monitoring information

The good practices listed above are more or less specific to each of the three options for monitoring programme implementation in conflict contexts. Yet, our review of existing resources and our interviews also point to the existence of several general good practices for gathering monitoring information that should be considered by donors and their implementing partners.

Moreover, it became clear from the same sources that monitoring is only helpful when there are specific mechanisms in place to act upon its results. As one interviewee for a study on remote monitoring in Syria put it: ‘What is the point… if nothing changes?’[40] This means that donors need to reflect on what information is useful to them, in order to ensure that they do not unnecessarily burden those responsible for monitoring and that they receive relevant and digestible monitoring reports. One interviewee working for a large donor in Syria reported: ‘We only really use 20% of the monitoring information we get. It’s a hell of a lot of work and often nothing gets done with it.’[41]

With these points in mind, two groups of general good practices can be distinguished. First, ensuring that basic monitoring housekeeping is in order at donor headquarter level. This enables donors to engage meaningfully in discussions about programme and monitoring design in conflict situations. Second, putting key heuristics and practices in place at field level. This strengthens the ability to monitor programmes operationally.

Ensuring basic monitoring housekeeping is in order at donor headquarter level:

A procedure must exist for ensuring that monitoring strategies and practices are incorporated into the planning phase of aid programmes and kept alive and useful during programme implementation. This includes clarity of purpose of the proposed monitoring (what will be done with it), options for the appropriate frequency of monitoring (including resource requirements) and the creation of a decision-making body that meets frequently to review monitoring results and decides on programme adjustment when needed.

A minimum set of clear, simple and predetermined indicators and data collection procedures is required for effective monitoring of aid activity in conflict contexts. Such indicators should include links to key donor policies and/or theories of change. A good monitoring system also requires several layers of checks and filters for continuous verification.

Monitoring must be made relevant to learning and continuous improvement in addition to programme accountability purposes. In other words, the inevitable trade-off that will occur between monitoring-for-improvement, monitoring-for-accountability and monitoring-for-learning needs to be mapped, discussed and mitigated as much as possible.

Long-term partnerships need to be built for both political-economy analysis in the service of programme design and implementation, and for third-party monitoring in the service of programme monitoring.

Ensuring that such basic housekeeping is in order will reduce risks enumerated in table 2 across the board and increase a donor’s ability to respond to the monitoring challenges listed in table 3. This is because such sound basic housekeeping increases the stock of donor capabilities to think through and engage in effective programme monitoring.

Putting key heuristics and practices in place at field level:

Develop on-the-ground networks to enable greater local insight, better monitoring and higher levels of accountability. Well-maintained networks and relationships with key stakeholders are essential for effective monitoring and triangulation. This requires a mix of analytical and diplomatic resources and will be a time-intensive endeavour.

Ensure that aid activity contracts contain sufficient incentives to enable and encourage implementing organisations to be transparent with regard to the progress of their work to both donors and beneficiaries.

Where modern technology is applied as monitoring tool, ensure that adequate research, and associated investment, takes place into the pitfalls and challenges of local data collection and analysis software.

Ensure that monitoring work is conducted with the full knowledge of implementing partners. Share reports to validate work and justify monitoring activities, and ensure there is a feedback loop between implementing partners and monitoring partners that is mediated by the funder.[42]

OECD-DAC (2010), Glossary of Key Terms in Evaluation and Results Based Management, Paris: OECD, pp.27-28.
OECD-DAC (2012), Evaluating Peacebuilding Activities in Settings of Conflict and Fragility – Improving learning for results, Paris: OECD, p.30.
Corlazzoli, V. and J. White (2013), Back to Basics – A Compilation of Best Practices in Design, Monitoring & Evaluation in Fragile and Conflict-affected Environments, London: DFID, pp.35-38.
Corlazzoli, V. and J. White (2013), Back to Basics – A Compilation of Best Practices in Design, Monitoring & Evaluation in Fragile and Conflict-affected Environments, London: DFID, p.34.
Church, C. and M.M. Rogers (2006), Designing for Results: Integrating Monitoring and Evaluation in Conflict-Transformation Programs, Washington: Search for Common Ground, p.83.
Building Markets (2018), What is the point… if nothing changes? Current Practices and Future Opportunities to Improve Remote Monitoring and Evaluation in Syria, New York: Building Markets, p.6. Also confirmed by Clingendael interviews with NGOs in Afghanistan, Syria, Somalia, May-June 2018.
Clingendael interview with NGO operating in Somalia, May 2018.
Ibid.
The UN’s humanitarian operations in regime-held areas in Syria offer an extreme example. See: Leenders, R. (2016), ‘UN's $4bn aid effort in Syria is morally bankrupt’, The Guardian, online: link, 29 August 2016 (accessed 9 August 2018).
See, for example, the experiences in relation to the Lafarge cement factory in Syria: Lund, A. (2018), The factory: A glimpse into Syria’s war economy, The Century Foundation: online, link, 21 February 2018 (accessed 9 August 2018).
Clingendael interviews with NGOs in Afghanistan, Syria, Somalia, May-June 2018.
Clingendael interview with independent Somali humanitarian consultant, June 2018. DFID describes the trade-off as follows: ‘Risk management entails trade-offs among competing risks, including missed opportunities to help people with urgent needs or to achieve longer-term stabilisation and development outcomes. An excessively risk-averse stance would not be consistent with the UK’s strategic objectives.’ See: Independent Commission for Aid Impact (2016), DFID’s approach to managing fiduciary risk in conflict-affected environments – A performance review, London: ICAI, p.16.
Price, R. (2018), Approaches to remote monitoring in fragile states (GSDRC Helpdesk Research Report 1420), Birmingham: GSDRC, University of Birmingham, pp.10-11.
Clingendael interview with NGO operating in Syria, July 2018.
Building Markets (2018), What is the point… if nothing changes? Current Practices and Future Opportunities to Improve Remote Monitoring and Evaluation in Syria, New York: Building Markets.
Clingendael interview with NGO operating in Somalia, May 2018.
Price, R. (2018), Approaches to remote monitoring in fragile states (GSDRC Helpdesk Research Report 1420), Birmingham: GSDRC, University of Birmingham, pp.10-11.
Building Markets (2018), What is the point… if nothing changes? Current Practices and Future Opportunities to Improve Remote Monitoring and Evaluation in Syria, New York: Building Markets, p.16.
Building Markets (2018), What is the point… if nothing changes? Current Practices and Future Opportunities to Improve Remote Monitoring and Evaluation in Syria, New York: Building Markets, p.21.
For more details see: Price, R. (2018), Approaches to remote monitoring in fragile states (GSDRC Helpdesk Research Report 1420), Birmingham: GSDRC, University of Birmingham, pp.3-9.
Building Markets (2018), What is the point… if nothing changes? Current Practices and Future Opportunities to Improve Remote Monitoring and Evaluation in Syria, New York: Building Markets, p.2.
Clingendael interview with donor organisation in Syria, May 2018.
Price, R. (2018), Approaches to remote monitoring in fragile states (GSDRC Helpdesk Research Report 1420), Birmingham: GSDRC, University of Birmingham, p3.