Efficiency assessment methods of public procurement
The main purpose of public procurement is to ensure the fulfillment of the basic functions of the state in full, through the acquisition of goods, works and services, in compliance with the key principles of public procurement. As a result, the efficiency of the public procurement system affects the efficiency of public administration as a whole. In turn, the efficiency of individual parts of the public procurement system, including the legal and regulatory framework, the internal processes of the major procuring entities, and the technical capacity and competencies of the staff responsible for procurement, also determines the efficiency of the procurement system.
Improving the efficiency of public procurement is impossible without taking into account social, economic, political, environmental, technological and other development factors and trends. Regular assessment of public procurement efficiency is required to take these factors into account in a timely manner. Such evaluations include both an analysis of the efficiency of the public procurement system as a whole and of its individual parts, and are aimed at improving the systems, their individual parts or processes, in order to achieve better results for citizens and other consumers of public services. At the same time, the efficiency assessment can be carried out either independently by the countries or with the involvement of independent experts.
Methodology for Assessing Procurement Systems (MAPS)
The Methodology for Assessing Procurement Systems (MAPS) is an internationally recognized tool for a comprehensive and complete assessment of public procurement systems. The Methodology was developed by the Organization for Economic Cooperation and Development (OECD).
It is aimed to help the country, with or without the support of external partners, to conduct an assessment of the national procurement system to identify its strengths and weaknesses.
The MAPS is open for use by everyone and can be applied by all countries, both independently and with the involvement of external partners (including international organizations, in particular the World Bank). To obtain a quality certificate, the assessment must be carried out in accordance with a set of rules and according to a quality standard validated by the MAPS Secretariat.
- 2015 OECD Recommendation on Public Procurement;
- 2011 UNCITRAL Model Law on Public Procurement;
- 2014 EU Directive on Public Procurement;
- general procurement principles used by multilateral development banks, countries and providers.
MAPS evaluates the following elements of the procurement system:
- Value for Money;
- Transparency;
- Fairness;
- Good Governance.
MAPS includes:
- a user's guide with instructions on how to conduct a MAPS assessment;
- instructions on how to analyze the country context;
- the indicator framework in four thematic pillars (each pillar includes indicators, sub-indicators – qualitative and quantitative – and assessment criteria);
- a glossary;
- a suite of supplementary modules that allow a focused analysis of specific areas: sustainability, professionalism, e-procurement, public-private partnerships, etc.;
- support tools, such as templates, that can provide support in conducting a MAPS assessment.
European Union
Within the European Union, the assessment and comparison of national public procurement systems is carried out by the European Commission.
To this end, 12 efficiency indicators were developed. Indicators depend on many factors that are specific to each country: for example, the structure of the economy or the different ways of bidding. In addition, some factors affecting the quality of public procurement can be assessed only partially, or using indirect methods. These factors include: level of corruption, administrative burden or competence of the procurement staff. As a result, the table of indicators contains information that gives only partial insight into the efficiency of public procurement systems in EU countries.
Overall performance is the sum of the scores on all 12 individual indicators.
By default, satisfactory performance on an individual indicator raises a country's overall score by one point, and unsatisfactory performance lowers it by one point.
The three most important indicators have triple weight:
- single bidder;
- no calls for bids;
- publication rate.
The European Commission explains that these indicators are linked to competition, transparency and access to markets, which are the fundamental principles of efficient public procurement. Indicators No. 7–12 account for 1/3 of the weight because they measure the same factors from different perspectives: small business participation (indicators No. 7–9) and data quality (indicators No. 10–12).
The European Commission assesses the above 12 indicators based on:
- qualitative policy judgment on what constitutes good practice;
- recent (as of the date of analysis) data for individual EU countries.
For each indicator, an EU member country can receive the following scores:
- “satisfactory”;
- “unsatisfactory”.
№ | Название показателя | Удовлетворительно | Неудовлетворительно |
|---|---|---|---|
1 | Единственный участник торгов (Single bidder) | ≤ 10 % | > 20 % |
2 | Закупки без торгов (No calls for bids) | ≤ 5 % | ≥ 10 % |
3 | Стоимостная доля опубликованных извещений о закупках (Publication rate) | > 5 % | < 2.5 % |
4 | Совместные закупки (Cooperative procurement) | ≥ 10 % | < 10 % |
5 | Критерии выбора победителя (Award criteria) | ≤ 80 % | > 80 % |
6 | Скорость принятия решения (Decision speed) | ≤ 120 дней | > 120 дней |
7 | Контракты, заключенные с малыми и средними предприятиями (SME contractors) | > 60 % | < 45 % |
8 | Заявки со стороны МСП (SME bids) | > 80 % | < 60 % |
9 | Разделение контрактов на лоты (Procedures divided into lots) | > 40 % | < 25 % |
10 | Предоставление неполной (нечеткой) информации о контрактах (Missing calls for bids) | ≤ 3 % | > 3 % |
11 | Неуказание регистрационного номера поставщика в закупочной документации (Missing seller registration numbers) | ≤ 3 % | > 3 % |
12 | Неуказание регистрационного номера заказчика в закупочной рекомендации (Missing buyer registration numbers) | ≤ 3 % | > 3 % |
OECD
The OECD public procurement performance assessment is based on the OECD Recommendation of the Council on Public Procurement. In order to develop a unified approach to assessing the performance of procurement systems across the OECD and to measure the level of implementation of the OECD Recommendation of the Council, in 2023 the OECD presented Public procurement performance: A framework for measuring efficiency, compliance and strategic goals.
The OECD Framework for Measuring includes 259 indicators. Of these, 83 are core indicators and 176 are aspirational indicators. In terms of the OECD procurement efficiency measurement system, it is worth noting the relationship of individual indicators to specific stages of the public procurement cycle, as well as to the different actors involved in the procurement process (national governments, procuring entities, major procuring units).
Compliance indicators (Compliance KPI). This category of indicators is aimed to assess whether procurement processes and outcomes comply with national or any other applicable laws, including integrity and competition laws. Compliance indicators can be divided into various subcategories throughout the procurement cycle, such as:
- publication/transparency requirements,
- preliminary control/audit results,
- sanctions against unscrupulous suppliers,
- integrity and accountability,
- appeals/court proceedings;
- delayed payments.
Efficiency indicators (Efficiency KPI). Aimed at assessing whether procurement processes are achieving the best procurement outcomes and efficiencies, as well as best value for money. From an OECD perspective, efficiency is usually defined as the ratio of outputs to inputs, while effectiveness is the ratio of specific outputs to specific inputs and depends on the quality of the provision of goods, works and services. Specifically, efficiency indicators can assess:
- cost saving (in terms of money and time),
- degree of integration of market procedures into the public procurement process,
- duration of procurement processes (including the bid evaluation stage), etc.
Also, the category of efficiency indicators includes various subcategories, such as procurement planning, implementation of various efficiency tools such as framework agreements or Dynamic Purchasing Systems (DPS), contract modifications, development of professional competences and systems enabling payment for goods, works and services.
Strategic indicators (Strategic KPI) assess how public procurement processes and results contribute to the achievement of strategic national objectives set by governments, such as:
- climate change mitigation,
- promoting innovative development,
- job creation,
- social aspects of sustainability (human rights, including labor rights, gender mainstreaming or inclusion of vulnerable groups),
- SME development, etc.
As a result, this group of indicators may include a share of sustainable goods, works and services, a share of procurement provided by SMEs (in terms of quantity and volume), or a share of procurement of innovative goods, works and services. In addition, some evaluation frameworks may include indicators that assess the impact of a strategic approach to procurement, such as reductions in greenhouse gas emissions or energy consumption. Different subcategories of strategic indicators may reflect different policy objectives. Finally, the OECD emphasizes that some strategic national objectives can have an impact that transcends national borders. Thus, the public procurement evaluation system should take into account the impact not only within the country, but also outside it. In particular, when it comes to the reduction of greenhouse gas emissions, the application of these indicators for the assessment of goods, works and services should be carried out along the entire value chain.
INTOSAI
One of the indicators of the public procurement system efficiency is also the results of auditing the performance of individual contracts. The key role in this process is played by Supreme Audit Institutions (SAIs), as well as regional and municipal audit and accounts bodies. At the same time, different SAIs have different mandates and authorities, as well as different audit approaches. Many SAIs do not conduct special public procurement audits, addressing these issues in the context of broader control and expert-analytical engagements.
Despite differences in audit practices, most SAIs adhere to the principles and approaches of the International Organization of Supreme Audit Institutions (INTOSAI) as reflected in the 2016 INTOSAI Framework of Professional Pronouncements (IFPP).
The Framework reflects the hierarchy and synergies of the organization's main documents and includes:
- INTOSAI Principles (INTOSAI-P);
- INTOSAI Standards (ISSAI);
- INTOSAI Guidance (GUIDs).
Within the INTOSAI Framework of Professional Pronouncements, the main document to assist auditors in auditing public procurement is the INTOSAI Guidance for Audits of Public Procurement (GUID 5280) developed by the INTOSAI Working Group on Public Procurement Audit (WGPPA)[1] and adopted during the XXIV INTOSAI Congress in Rio de Janeiro.
Objectives of the INTOSAI Guidance for Audits of Public Procurement
The main purpose of all INTOSAI Guidance is to assist auditors (inspectors) in applying the relevant INTOSAI standards when auditing relevant industries. INTOSAI Guidance for Audits of Public Procurement covers two types of audits — performance audits and compliance audits. Thus, the purpose of the Guidance is to assist auditors in applying ISSAI300 (performance audit) and ISSAI400 (compliance audit) standards when auditing public procurement at all stages of the procurement process.
Highlights of INTOSAI Guidance for Audits of Public Procurement
Like most INTOSAI professional pronouncements, INTOSAI Guidance for Audits of Public Procurement has a number of distinctive features useful to auditors:
- applicability by all SAIs in any jurisdiction and legal environment (the experience of different SAIs in terms of their organizational structure, mandate and operating environment, as well as the experience and approaches of international organizations were used in the development of the Guidance);
- coverage of the entire audit cycle (the Guidance can be applied at different stages of the audit, from planning and execution to preparation of the audit report and follow-up);
- the possibility of applying the guidance to combined audits (the Guidance covers both performance and compliance audits);
- special attention is paid to the preparation of the audit report and the development of recommendations based on the audit results (with a focus on improving the efficiency of the procurement process);
- integrated approach to public procurement (public procurement is seen both as a tool for achieving strategic development goals and from the point of view of the process-based approach);
- taking into account the impact of new technologies, as well as the possibility of auditing public procurement in emergency situations;
- a large amount of additional materials (includes examples of audit questions at different stages of procurement, lists of risks, etc.).
[1] The Accounts Chamber of the Russian Federation is the Chair of the INTOSAI Working Group on Public Procurement Audit.