The public–private partnership (PPP) mechanism is often proposed as a means of delivering public services, particularly complex construction projects, in developing countries. This study, published in September 2022, developed a performance framework to evaluate the application of PPP projects based on 10 key performance indicators and 41 performance measures. The framework was reviewed by experts for coverage and relevance, then validated through two Pakistani road construction case studies.

By David Fellows and John Leonardo

Introduction

The Public Expenditure and Financial Accountability (PEFA) programme provides a framework for assessing and reporting the strengths and weaknesses of public financial management (PFM). The current 2016 Framework refines the previous 2011 Framework and is structured under a hierarchy of 6 Pillars, 31 Indicators (PIs) and 94 Dimensions. The PEFA Field Guide explains the components of the 2016 Framework and describes how an assessment team should score each dimension on a scale of A to D, a D score representing the lowest level of performance.

An initial assessment of the latest PEFA reports for countries published under the 2016 Framework suggested that many countries were not getting the PFM basics right. This led to a comparison of recent results with those from earlier PEFA reports prepared under the 2011 Framework to examine performance over time and the lessons for PFM improvement that such a comparison may offer (termed the ‘dual study’). It was decided to focus on dimension scores since the demands of PFM can change markedly depending on the aspects of the subject matter under consideration and the evident variations of score for the same country at dimension level within a range of PIs.

It was decided to confine this initial study to the analysis of D scores at the dimension level given the frequency of D scores, the very poor performance they represent and the importance of raising performance to a higher level. The Field Guide requires a D score when: ‘the feature being measured is present at less than the basic level of performance or is absent altogether, or that there is insufficient information to score the dimension’.

For the purpose of this study, D scores include dimensions marked D*, NR and some NA scores where evidence suggests a breakdown in PFM activity. It seemed evident that these attributions are often applied inconsistently and serve to obscure the extent of the poor performance of some countries by avoiding the use of justifiable D scores. A summary of all scores for the 2016 Framework and the dual study evaluations, as discussed in this report, can be accessed at Annex 1.

2016 Framework analysis

The 2016 Framework analysis consisted of the latest published evaluations for the 63 countries for which there were published reports at the time of this study. The D scores represent 32% of all dimension scores in this data set, 39% amongst low-income countries.

D scores were widely distributed throughout the framework with 45 of the 94 dimensions having an above average number of D scores.

The study also defined and assessed the key factors (termed descriptors) that contributed to PFM performance. The results, summarised at  Annex 2, suggested that most D scores can be explained by the absence of ‘Management Effectiveness’, ‘Integrity’ and in one case of ‘High Level Technical Knowledge’ although poor “System Design” was another potentially important contributing factor.

Annex 3 provides a full list of the 2016 Framework dimensions and D score data together with the descriptors contributing to each dimension.

Dual framework

Following the results of the 2016 Framework D score study it was decided to undertake a review of 45 countries that have undertaken at least one PEFA evaluation under both the 2011 and 2016 frameworks (the earliest and the latest studies we used for countries with more than two studies). This enabled a country’s performance to be compared over a five-year period.

The 2011 and 2016 PEFA frameworks differ in many respects. An equivalence table published by PEFA suggests that the two frameworks can be aligned to 37 “equivalent” dimensions on the basis that the respective dimensions were either “directly comparable” or “indirectly comparable”.

The PEFA equivalence table identifies 28 dimensions (or in some cases subsets) from the 2011 framework as “non-comparable (subject only)” to 2016 counterparts suggesting that the dimension descriptions and scoring routines differ markedly while the general area of relevance to the dimensions are similar. This leaves only 37 pairs of comparable dimensions.

On examination, the study team decided that 26 of the 28 pairs of dimensions judged “non-comparable (subject only)” were in fact very similar to the 2016 counterparts, the main difference being the way in which the later guidance is translated into clear-cut scoring criteria but that a good PEFA evaluator should have made reasonably similar judgements for both frameworks when reviewing all but two of these dimensions.

This exercise, therefore, recognises 63 equivalent dimensions while also providing results for PEFA’s 37 equivalent dimensions. It is suggested that the D score characteristics of both data sets are sufficiently similar to provide a reasonable validation for the larger 63 dimension equivalence thereby extending the usefulness of inter-framework comparisons. Details of the PEFA and PFMConnect equivalence tables are set out at Annex 4. The dual study of 2016 and 2011 Framework with D score data at dimension level is set out at  Annex 5.             

The dual study is highly concerning in terms of the lack of improvement amongst those dimensions receiving D scores. These data are further summarised and commented on below.

The dual framework study reveals a deteriorating performance with most dimensions exhibiting a greater number of D scores in the later evaluations. Only 13 (35%) of dimensions from the 37 dimensions study and 16 (25%) from the 63 dimensions study experienced reductions in D scores between evaluations.

When the dual evaluations for the same country were compared, see Annex 6, it was noted that most countries recorded a higher proportion of D scores for the same dimension in both evaluations demonstrating a reasonably consistent poor performance. A few countries displayed less consistent results.

Few countries in the 63 dimensions set recorded reductions in the number of D scores in 2016 framework results compared with the 2011 framework results. The top performers where significant PFM reform activities had been undertaken between the dual framework studies included: Philippines, Maldives, Mongolia and Tajikistan.

The results for the proportion of dimensions with above-average D scores that are common to both framework dimensions sets is concerning. Approximately one third of all dimensions had above-average D scores that were common to both frameworks for the same country for both datasets. In addition, over 70% of the above-average dimensions in both datasets were common to both frameworks showing limited improvement in the worst scoring areas over a five-year period.

Dimensions with regular poor performance are widely distributed (titles in red at Annex 6). This suggests pockets of poor management that remain in place without effective challenge and this is consistent with the descriptor analysis.

Conclusions

This study offers a range of findings that pose questions about the approach, effectiveness and sustainability of PFM reforms instituted by national and subnational governments often in collaboration with development agencies. The concerns about management effectiveness and integrity highlighted in this study must be seen to question the most basic aspects of any organisation.

The study focusses on D score analysis, but it could be useful to extend the analysis to C-level scores where the performance of countries still remains below good international standards. This could reveal new characteristics of national PFM performance and extend the range of analytical techniques applied to performance data.

The data analysis evidences the credibility of PFMConnect’s extended 63 dimension equivalence model that offers significant potential for more detailed studies of specific countries or regions.

Further work on descriptors to reveal contributory factors to variations in performance seems worthy of further development.

The failure of some governments to publish PEFA studies in full reinforces concerns about the need for greater attention to integrity. Another improvement that could be readily and widely implemented is legislative scrutiny of audit reports (PI 31).

Recommendations

We recommend that country-specific studies should be undertaken based on PEFA assessment reports (both 2016 Framework studies for the full 94 dimensions and dual studies where the data are available) examining D scores at dimension level to establish potential causes of poor performance and identify ways in which performance may be improved. Issues to consider with respect to areas of poor performance, include:

• The commitment to personnel development and support, including: in-service training, management development, oversight, feedback on performance, and system design.
• The adequacy of transparency and accountability and evidence of corrupt activity.
• The quality of relevant communication and support levels among different departments and units of the finance ministry.
• The reasons for persistently poor or erratic performance and the fit with other findings.
• The observations of managers and staff on reasons for poor performance and barriers to improvement.

We recommend that country studies should be designed as the initial phase of PFM development programmes. In this context, a report by the Swedish International Development Cooperation Agency (SIDA) offers some observations about the conditions for effective PFM reform. These include the importance of change agendas being aligned with Government priorities and the need to treat PFM reform as a learning process with strong emphasis on coordination and systematic evaluation of the activities performed by teams responsible for delivery.

Groups of countries or subnational bodies may wish to collaborate in reform programmes enabling challenges and learning to be shared and systems of mutual support developed. We have previously advocated the use of digital communication as a cost-effective and time-saving way of sharing knowledge and ideas between nations (incl. expert advisors).

Any country, region or development institution wishing to participate in further work in this field is invited to discuss their interest with the authors.

An article based on this study has been published by the IMF’s PFM Blog.

PFMConnect is a public financial management consultancy with a particular interest in the use of digital communication to support learning and sharing expertise amongst the international development community.

David Fellows began his career in UK local government where he became President of the Society of Municipal Treasurers and a pioneer of digital government. He has held appointments in the UK Cabinet Office and the National Treasury of South Africa (david.fellows@pfmconnect.com).

John Leonardo is a PFM expert with extensive worldwide experience. He has undertaken PFM assignments in Africa, Asia, the Caribbean and the Pacific where he undertook PEFA assessments. Both authors are directors of PFMConnect, a public financial management consultancy (john.leonardo@pfmconnect.com).

 Posted by David Fellows[1]                

This is an extraordinarily important time for coherent policy, planning, and performance – the “3 Ps” – for delivering the SDGs and other core public policy objectives.

The SDGs present an extensive range of essential service improvements that are applicable across the world. The threat posed by climate change has become a major international issue with immensely ambitious remedial targets and huge spending requirements. Governments are also under pressure to introduce gender responsive budgeting and digitalize their public finances, reforms that offer huge benefits but also challenges and costs in the short to medium-term. At the same time, the Covid-19 pandemic has devastated many economies and produced huge fiscal burdens, increasing the challenge of delivering the SDGs and better environmental outcomes.

A coherent delivery framework

It is important that governments take decisions within a strategic framework that represents an appropriate timeframe and deals clearly with policy goals, service responses, resources deployed, and outcomes achieved. The various elements of this framework include:

1. A vision having a 10-year perspective expressed in terms of outcomes.
2. Objectives set with a 3-5 year delivery time frame, consistent with achieving the vision.
3. Delivery targets for each of the next 3-5 years in terms of service outputs relevant to the performance outcomes.
4. 3-5 year budgets for agencies or programs that reflect the delivery outcomes and performance targets that each budget represents.
5. Annual accounts that set out executive responsibilities, annual performance outcome and delivery targets and the actual performance achieved.
6. Training and recruitment plans that enable public agencies to operate the systems and deliver the services that have been approved.

Delivering change

Successful reform is an elusive concept. Any initiative worth doing must have a benefits realisation plan specifying the steps necessary to ensure that progress is being made and that the end results are achieved.

Services and changes to service provision should be protected by risk management strategies that seek to mitigate internal or external events and shocks that may otherwise hamper delivery or destroy valuable assets.

Review and accountability

The various elements of the framework must be consistent with each other. When major new commitments are proposed, or it becomes obvious that major targets are no longer achievable then a review of the framework should be undertaken. In addition, there should be an annual review of the framework as part of the annual budget preparation process, perhaps as part of a wider spending review. Policies, plans, performance, and the results of review processes should be made public. There is no aspect of the planning and delivery process that cannot benefit from public scrutiny and comment. It is the responsibility of all public institutions in a democratic country to make themselves open and responsive to such a dialogue.

The PFM challenge for developing countries

The relatively poor condition of PFM in developing countries shown in the chart suggests the difficulties that developing countries face in planning, managing, and maintaining their existing budget systems. The SDGs and other global pressures to increase spending represent additional challenges for PFM systems to face. Multilateral decisions on the SDGs and climate change must therefore take account of the consequences for developing nations given the likely dependence of successful outcomes on their cooperation.

Conclusion

The immense pressures on governments worldwide to fulfil the global obligations and pressures described above often require concerted action. If governments are to succeed without making over-extended commitments, wasting time and money on impractical solutions, they must make decisions within the rigours of a fully operational policy, planning, and performance framework. Multilateral agreements, economic, social and technological considerations will all feed into framework construction but the integrity of the framework is key.

Framework development will inevitably present hard choices but that is a strength of the process. It should also provide a coherent basis for democratic accountability if, as a result, drastic life changes are required, freedoms are curtailed, and personal costs are increased.

This article was first published by the International Monetary Fund’s Public Financial Management Blog on 20 September 2021.

---

[1] David Fellows began his career in UK local government where he became President of the Society of Municipal Treasurers and a pioneer of digital government. He followed this with appointments in the UK Cabinet Office and the National Treasury of South Africa. He is a Director of PFMConnect.

By David Fellows and John Leonardo

Media reports on Public Private Partnerships (PPPs) for the first half of the current year reveal both opportunities and confusion in the public sector about the use of this method of public service provision. Here we seek to explore the benefits and challenges of these arrangements and identify key questions that should be considered by decision-takers.

What is PPP?

PPP is a form of service delivery where private contractors are recruited by public sector clients to provide services while incurring the necessary capital costs ostensibly at their own risk. The specification of extensive service detail by clients and attempts by contractors to mitigate risk in respect of major capital investments on which returns are generated over long contractual periods result in extensive documentation covering a welter of issues. A health centre or even a school would be too small an item on its own to warrant the development of such a complex format, therefore, such services are usually wrapped into programmes of multiple provisions. Railways, roads, airports and power stations offer prime opportunities for this approach.

The Public Service Benefits of PPP

The PPP model can offer new insights and innovative approaches to public service delivery through contractors’ specialist knowledge and experience. For the public sector the PPP model also removes the challenge of obtaining skilled personnel to undertake design, construction and managerial tasks, eliminates the risk of budget overruns and avoids the state’s need to raise loans to defray capital costs. The latter, in turn, restricts the state’s overall debt burden and this can be of considerable benefit to countries that face serious difficulties in raising loans by conventional means or have agreed debt ceilings that they would otherwise breach. In some cases PPP may offer the only way that services can be provided.

The Transfer of Risk

PPP is not the only way, however, that private sector capacity can be accessed by the public sector. In many countries private contractors are engaged in the design, construction and operation of state-owned facilities through more conventional works and service contracts. The major innovation offered by the PPP model is the bundling together of capital funding, design, construction and operational services. The consequent transfer of risk from public to private sector is therefore regarded as its defining characteristic.

The scale of the services involved, the performance measures stipulated, the performance-linked payment regimes, the penalties stipulated for poor performance, the responsiblities for budget overruns, the size of the initial investment, and the lengthy contractual periods required to meet the necessary return, do indeed provide a formidable nexus of risk for contractors.

As a result, contractors seek to mitigate such risk both by charging a premium for risk assumption and by introducing a wide range of obligations on the public service client that help to secure the contract’s cost base and revenue stream. In simple terms, this is often achieved by specifying service demand levels and operational circumstances within narrow margins. The client is then required to offer restitution for variations outside the stated parameters. This risk reversal clearly serves to diminish the public sector benefits of the PPP vehicle as the client both pays for the risk transfer and pays if the mitigation criteria are breached.

The Consequences of Risk Reversal

The PPP model may assist in the containment of public debt levels but this tends to be overridden by risk reversal and some are beginning to argue that excessive risk diminution should require the basic capital investment to be scored as borrowing against the client state’s capital account. In addition, the market’s view of the client’s credit worthiness is beginning to take into account the PPP deals to which it is committed.

PPP Decision Criteria

The issues raised in recent media reports (see extracts in the Appendix taken from PFMConnect’s Public Private Partnership (PPP) Board on Pinterest ) centre around the opportunities and challenges offered by the PPP model as discussed above although they are not always explained in these terms. The questions arising are mostly for the public sector:
(i) Does the service need to be provided by the public sector or can it be provided by the private sector on its own initiative given a more conducive regulatory environment?
(ii) If the public sector needs to subsidise services for certain sections of the population can those services be provided by the private sector on a commercial basis to the public at large with public sector institutions buying in service provision for people in designated categories or by providing income support for such people leaving them to buy necessary services without any direct linkage between state and service provider?
(iii) As an alternative, should the public sector fund capital investment and buy in design, construction and operational services as necessary depending on its skill base and the nature of service provision and is this feasible in given circumstances?
(iv) Is it reasonable in the given circumstances to assume that long-term service provision can be contracted for without the expectation of substantial changes in service needs and delivery requirements (ie a basic requirement for the PPP model)?
(v) Can the costs and other conditions of a particular PPP contract be justified in terms of need/benefit, cost and priority? Note: in making this assessment it may be worth comparing the PPP costs and risks for the public sector with the costs and risks of the more conventional approach in (iii) above. It may be seen that in many cases that the PPP option may be more expensive than a more conventional approach although the feasibility of the alternative should also be considered.
(vi) If a PPP contract is seen as an essential requirement does the contract contain appropriate termination and variation clauses; has the client reviewed experience elsewhere for similar services under similar contractual arrangements and drawn the necessary lessons from the review; has an appropriate cost-benefit analysis (including risk assessment) of the scheme been completed and fully considered at an appropriate political level; has a benefits realisation strategy been prepared; and has the available advice from recognised authorities been considered on the subjects of procurement arrangements and contractual provisions ?

Conclusion

Most states make extensive use of private sector services in their array of delivery mechanisms. The issue here is not whether private sector contractors should be employed to provide public services but the applicability of the PPP model for this purpose.

It is persuasively argued that the PPP model can bring innovation to public service delivery and facilitate developments that could not be funded in others ways. Importantly, risk may be transferred to private contractors at a price that yields benefit to both parties, although risk transfer may be undermined in negotiation by contractors daunted by its scale and potential ramifications.

The problem of risk transfer, the complexity inherent in PPP negotiations, the difficulty of assuaging public anxieties once expressed and the record of PPP contracts that have been let unwisely must alert public authorities to the need for caution with this model. It is suggested that at the outset of every new proposal clients should ask themselves whether this really is an acceptable way forward and if so whether there is good reason to believe a reasonable deal can be struck to the benefit and satisfaction of both parties. Sometimes the answer should be no.

APPENDIX
Some Recent Media Coverage of PPP (see PFMConnect’s PPP Board on Pinterest)

Some of the Pins that illustrate these issues are, as follows:

USA Transportation Secretaries says private funding isn’t the sole answer – January17

In January the Skift Daily News Letter reported on the commitment of the new USA Administration to PPP infrastructure developments. It referred to the nominee for transportation secretary, Elaine Chao, having said at her Senate confirmation hearing that she wanted to “unleash the potential for private investment” although it reminded readers that she also said that infrastructure plans would include direct federal spending as well.

Skift implied that this latter point chimed with the sentiments of outgoing Transportation Secretary Anthony Foxx who had cautioned prior to his departure that public-private partnerships were useful but could only address about 10 to 20 percent of America’s transportation deficit and that the USA was “still going to need a fair amount of public funding”.

The Skift commented that roadway PPPs typically rely on revenue from tolls or sales taxes dedicated to that purpose to provide investors with a profit and that such projects have had a mixed record in the U.S. Several private toll roads have gone bankrupt, but express toll lanes on major highways constructed in part with private capital have had more success.

The public sector can’t deliver the new urban agenda alone – Feb17

When New York City decided to offer public Wi-Fi kiosks on city streets, it turned to Sidewalk Labs, an Alphabet company. Sidewalk Labs will absorb the cost with revenue from digital advertising and data collection from users.

This kind of partnership will be key to implementing the United Nations’ suite of new agreements to improve life in cities around the world, according to a new report from the World Economic Forum entitled ‘Harness Public-Private Cooperation to Deliver the New Urban Agenda’. The report calls for governments to set up business-friendly systems with clear guidelines for how the private sector can engage with the public sector in a transparent manner that fosters trust and mutual cooperation.

The USA needs a federal centre of expertise to equip the public sector to make PPPs – March17

The American Prospect referred to a report by the new Economic Policy Institute dealing with public-private partnerships. The report commented that design and construction have long been placed with the private sector under traditional contractual arrangements. In contrast to this, under a PPP for major roadworks the private company gets a percentage of a revenue stream, such as a toll or payments based on performance incentives, such as keeping a road well-maintained.

North Carolina agreed to a “non-compete” clause in the PPP contract for a toll road. This required the State to pay a penalty if officials moved to improve transit or nearby roads so some people could avoid paying the tolls. The same company undertook a PPP contract to provide a toll road in Indiana that ran into difficulties and filed for bankruptcy. The report commented that “The politicians behind PPPs are often close allies or financial beneficiaries of a project’s private promoters”.

The report advocated that the federal government should take the lead in amassing the necessary expertise for structuring public-private partnership contracts to the benefits of federal, state and local governments. It notes that such a proposal has been mooted by the federal Department of Transportation but was skeptical about the prospects for its imminent implementation.

Firehouse Broadcasting, Indiana – July 17

Firehouse Broadcasting reported on the dissolution of a PPP valued at $325m for the provision of a highway over a period of 35 years. Senate Appropriations Committee Chairman Kenley criticized the state for focusing too much on low bids and not enough on the background and credibility of the bidders. He suggested that signs that the company could not financially complete the project existed before the state agreed to work with them.

The state now plans to take over the project entirely. Governor Holcomb’s fiscal team says that because the state’s credit is better than the developer’s, costs for the project will actually decrease when the state takes out bonds for the project, possibly saving taxpayers around 30 million dollars.

PPP in the Philippines suffers delays amid lack of continuity policy – Feb 17

The Manila Times quotes the Fitch Groups’ BMI Think Tank as commenting that ‘The Philippines has one of the most robust PPP frameworks in Asia but projects continue to suffer delays … of the 56 PPP projects launched since 2010, only four are complete as of January 2017, while many others have been repeatedly delayed due to financing, land acquisition and contract negotiations’.

BMI reported Secretary of Finance, Carlos Dominguez, as promising that he would ‘dramatically review the PPP process’.

Scaling up Infrastructure Investment in the Philippines: the Role of PPP – June 17

The Asian Development Bank reported that lack of infrastructure development had held back the economy of the Philippines but reforms in the country’s approach to PPP were now helping to redress the situation. As a result, 11 projects worth P200 billion had been awarded and there was now a pipeline of 40 bankable projects ranging from expressways, airports, seaports, water, urban rail, information technology, and social sector projects (classrooms, hospitals, prisons).

The report recommends that: the stock of development project should now be vetted in line with government priorities; a medium-term financial framework should be adopted as the basis for investment; appraisal processes should be strengthened with PPP and directly financed projects being similarly assessed and selected; risk assessment to be fully embraced and included in VFM analyses; early termination decision process to be reviewed; and the legal framework to be revised to improve the attractiveness of the PPP environment to the private sector.

Philippines Government auctions to be limited to single rebidding – July 17

Business World reported that during a round table the Socioeconomic Planning Secretary of the Philippines, Ernesto M. Pernia, had stated that the Government will do away with multiple rebiddings in an effort to halve the current 30-month procurement process before the project breaks ground.

It is clear from the report that PPP projects had regularly fallen victim to a variety of disputes between competing contractors, between contractors and government and between landowners and government. Courts orders were being sought to enforce delays while disputes were settled, some taking many years.

Challenges in Jaipur High Court to PPP for health centres – May 17

The Times of India reported on high court challenges and ambivalence at state government level concerning the inclusion of substantial numbers of health centres in PPP deals. Challengers have suggested that such arrangements should be made in ‘pilot mode (offering) a chance to learn from the experience’. It was reported that last year the government in Karnataka had stopped PPP deals concerning 52 hospitals finding that the quality of services had not improved substantially although costs borne by government had risen.

In the same month the World Bank Group’s International Finance Corporation reported on the benefits gained by India’s Jharkhand State from PPP investment in hospital pathology and dialysis technology that it had enabled patients to receive medical scans at local facilities with the state bearing the costs for poorer patients.

Big push for private players as India’s Government unveils new metro policy – August 17

BloombergQuint reports that India’s Union Cabinet has determined that private participation either for complete provision of metro rail or for some unbundled components (like automatic fare collection, operation & maintenance of services) will form an essential requirement for all metro rail projects seeking central financial assistance. Project evaluation will be based on the economic rate of return for large projects taking into account the wider social value generated.

The new policy also mandates evaluation of various modes of mass transit like the bus rapid transport system, light rail transit and tramways.

Guidance on PPP contractual provisions, 2017 edition – June17

The World Bank has published an updated version of its Guidance on PPP Contractual Provisions in response to an encouraging reception to its initial guidance issued two years ago. Announcing the new edition the World Bank acknowledged that the complexity of public-private partnership (PPP) transactions often involves considerable time and expense in preparing and finalising a PPP contracts. It admitted that the development of complete PPP contracts on an international basis was unrealistic but there was merit in focusing on common contractual provisions.