Редакция журнала «Позитивные изменения»
Позитивные изменения. Том 3, № 4 (2023). Positive changes. Volume 3, Issue 4(2023)
На наш взгляд, предлагаемая модель может быть адаптирована для задач и других организаций, а также территорий. Инвариантным может остаться общий подход – сочетание заданных посредством УОП критериев и индикаторной оценки качественных изменений. При этом критерии УОП могут отражать те показатели, которые важны для данной конкретной территории или организации. Критерии «гигиены» (с 8 по 11 в данной модели) являются универсальными и могут быть использованы для любого проекта как базовый «чек-лист» с последующими рекомендациями. Второй компонент также может быть реализован посредством протокола качественной оценки изменений, либо другого подхода, позволяющего выявить индикаторы изменений.
Таким образом, можно говорить о том, что представленная модель является моделью «с открытым кодом» – она может быть дополнена и переработана в соответствии со стоящими перед конкретной организацией или территорией задачами.
Assessing Territorial Development Projects: Our Norilsk Endowment Fund Case Study
Elena Avramenko, Vladimir Vainer, Natalia Gladkikh, Ivan Smekalin
DOI 10.55140/2782–5817–2023–3–4–78–91
This article introduces an assessment model designed for projects financed by Our Norilsk Endowment Fund[131], focusing on how these projects contribute to territorial development and align with the current regional agenda. The model is multifaceted, comprising two primary elements. The first is an assessment against the Unique Assessment Profile (UAP) criteria, which encapsulate essential regional indicators, sustainability of project outcomes, and the integrity of the assessment process. The second element is the actual social impact assessment, which utilizes qualitative change indicators developed specifically for each supported area. This element also includes criteria for expanding the indicator set when a project’s impact transcends the predefined boundaries.
Elena Avramenko[132]
Expert of the project “Development of a Social and Economic Impact Assessment Model for NGOs” by the GLADWAY Foundation, Lean 6 Sigma Green Belt master
Vladimir Vainer
Positive Changes Factory, Director
Natalia Gladkikh
PhD in Psychology, Leading Expert The Institute of Social and Economic Design, Higher School of Economics
Ivan Smekalin
Analyst, Positive Changes Factory
Our Norilsk Endowment Fund is a non-profit organization dedicated to promoting the social advancement of Norilsk city and the Taimyr Dolgano-Nenets Municipal District (hereinafter the Territory, or Taimyr). Its primary goal is to facilitate long-term regional investments and provide financial support to non-profit initiatives focusing on vital sectors such as education, healthcare, culture, and sports, amongst other socially significant areas. The Fund’s operations are centered around involving committed citizens and stakeholders in the social development of the region. This makes Our Norilsk Endowment Fund a model organization, prioritizing institutional support for projects driving territorial development. This quality was one of the factors in formulating the assessment model for projects financed by the Fund.
This paper details the impact assessment model used by Our Norilsk Fund, developed through a collaborative effort with the team and experts from the Positive Changes Factory. In our opinion, this model is adaptable for other regions, and its various components – such as the evaluation criteria, indicators, and approaches for developing diagnostic tools – could be effectively utilized in formulating assessment models for federal projects and programs.
OVERVIEW OF THE ASSESSMENT MODEL
The evaluation model comprises two primary components:
1. Developing a Unique Assessment Profile[133], which delineates the project’s expected and actual contribution to changes based on seven universal socio-economic impact criteria. These are dependent on the region’s unique characteristics as well as global best practices in assessing territorial development projects. The assessment profile further includes four supplementary criteria to reflect the quality of the evidence base and the social technology utilized in the project. It also assesses the existence and effectiveness of a project data collection and monitoring system.
2. Project assessment via social impact indicators. For this, a comprehensive list of potential indicators was created, covering all financial support domains provided by Our Norilsk Fund. Moreover, a detailed algorithm has been developed and described for augmenting this list, should the project target different changes / activities or if the Fund’s list of support areas is expanded. The assessment relies on two key factors: the number of beneficiaries likely to experience the changes described by the indicators, and the quality of these changes, which is evaluated through a qualitative change assessment protocol. Additionally, an algorithm for conducting an overall assessment of all projects has been formulated. This comprehensive evaluation consists of three modules: aggregate analysis of project assessment profiles (representing the first component of the model), determining the change significance index based on the indicator analysis (the second component), and finally conducting an overall qualitative change assessment at the regional level. The first two modules merely summarize the data furnished by the projects. The third module, however, requires a separate study in the form of a regional survey based on the qualitative change assessment protocol methodology.
PROCEDURE FOR FORMULATING THE PROJECT ASSESSMENT MODEL
1. Any applicants seeking financial support from the Fund must assess their project against 11 criteria outlined in the Unique Assessment Profile (UAP). Each criterion is rated on a 1–5 scale, with 1 representing the lowest expression and 5 the highest. The application includes comprehensive descriptions for each criterion. For instance, the criterion “Development of the Region’s Human Resource Potential” is described as follows:
• Very low manifestation (1): The project does not incorporate measures to enhance professionalism and retain human resources in the region.
• Low manifestation (2): The project includes limited measures, which are not central to personnel retention.
• Medium manifestation (3): The project incorporates some measures for professional development and retention, but there is room for improvement.
• High manifestation (4): The project incorporates significant measures for retaining human resources in the region.
• Very high manifestation (5): The project presents a comprehensive strategy aimed at increasing personnel professionalism and retention, effectively addressing the issue of migration.
2. Once the application is approved, an evaluation model is created based on the ratings given to the individual profile criteria during the application. This model includes an assessment against the universal criteria (1 to 7) for all the projects that have declared the presence of the respective area of impact, with scores 3 to 5 for the respective criteria. For the additional UAP criteria (8 to 11), recommendations are provided only for low scores (1 and 2) to develop necessary modules like evidence base, monitoring system, etc., to be incorporated in the project outcomes.
Our Norilsk Fund is a model organization, prioritizing institutional support for projects driving territorial development. This quality was one of the factors in formulating the assessment model.
3. Throughout the project, the team collects data based on the declared UAP criteria (model component 1) and the developed social impact indicators (model component 2).
4. The Fund aggregates project data from all projects for subsequent use in the overall assessment of their impact and effectiveness.
ASSESSMENT CRITERIA FOR THE SOCIO-ECONOMIC IMPACT OF PROJECTS (UNIQUE ASSESSMENT PROFILE, UAP)
Based on the analysis of projects supported by Our Norilsk Fund and other sources, such as publications and regional development project assessment experience, the following universal criteria for project assessment have been identified:
1. Development and Retention of Human Resources in the Territory
This criterion measures the project’s effectiveness in enhancing professional opportunities for both the project team and its target audience, thereby aiming to reduce staff turnover rates. Considering the significant migration issues in Norilsk and Taimyr and the high propensity of people to leave the region, this metric is deemed universal for all projects. Projects with a specific focus on retaining human resources in the region are likely to score higher on this scale. Evaluating this aspect in projects requires an analysis of staff turnover dynamics within participating organizations, taking into account dismissals related to ‘professional burnout.’
The evaluation model comprises two primary components: developing a Unique Assessment Profile and project assessment via social impact indicators. Additionally, an algorithm for conducting an overall assessment of all projects has been formulated.
Typical indicators for assessment:
• Employee Turnover Rate: Calculated as the number of employees who left the company during a period divided by the average number of employees in that period, multiplied by 100 %.
• Attrition: The number of employees resigning during the period due to issues the project aims to address.
• Absolute Reduction in Staff Attrition: The difference in attrition rates before and during the project.
• Monetization of Personnel Retention: The decrease in attrition (from before the project to during the project) multiplied by the hiring cost per employee in the region.
Periods subject to evaluation: The assessment covers three phases: the project implementation period, a comparable period before the project start, and a similar duration following the project’s completion. Indicators should demonstrate a decline in staff turnover during and after the project period.
2. Developing a New Standard for Consuming Social and Cultural Services
This criterion measures how much the project contributes to either introducing new or enhancing the quality of existing social and cultural services. Essentially, it determines whether the project leads to the creation of new products or services or the transformation of existing ones.
To evaluate the establishment of a new consumption standard for social and cultural services, it is essential to analyze the changes in revenue and income before, during, and after the project. This assessment would typically involve examining comparable financial data, such as ticket sales for events or information on the variety and pricing of provided social services. This analysis helps gauge the effectiveness of the new standard and its impact on the organization’s financial metrics.
Key indicators for evaluation:
• Average Revenue (RUB/month): calculated for the project implementation period, a comparable period before the project start, and a similar duration following the project’s completion.
• Average Net Income (RUB/month): Similar to revenue, the average net income is calculated for the project implementation period and comparable periods before and after the project implementation.
The average indicators can be followed up after the project is completed, to monitor residual changes over extended periods of time:
• Average Income Growth: (Average Income During Project / Average Income Before Project – 1) × 100 %
• Average Net Income Growth: (Average Net Income During Project / Average Net Income Before Project – 1) × 100 %
3. Replication to Secondary Audiences
This criterion reflects the project’s capacity to disseminate its outcomes to secondary audiences, which are not directly involved in the project’s core activities. Secondary audiences are those who benefit from the project’s results without significant additional investment or with comparatively lower investment than the primary audience.
This type of changes is assessed over two indicators. The first one is expressed as an estimate of the cost of training the secondary audience, compared to that of the primary audience. It is characteristic of projects aimed at disseminating knowledge and skills to specific professional groups. The second indicator is a cost-benefit analysis between the original project and its subsequent extension. This parameter applies to projects that create and replicate new social models.
A set of indicators for assessment:
• Numerical coverage of the secondary audience (number of people who received new knowledge / skills from the project participants).
• Cost of training per participant in the primary (secondary) audience: equals the total cost of training activities divided by the number of people trained. The formula is based on a comparable volume of training for the primary and secondary audience. Both the company’s own funds and grant funds are factored in.
The cost of training per participant must be lower in the secondary audience than the cost of training for the primary audience.
• Absolute difference in the cost of training per participant: cost per primary participant less cost per secondary participant.
Relative difference in training = (Cost per primary audience member / Cost per secondary audience member – 1) × 100 %.
• Economic benefit of replication = absolute difference in cost of training per participant × number of secondary audience participants.
This is an “open source” model, allowing for customization and refinement to meet the specific challenges and objectives.
• Number of confirmed replications of the project models in other organizations.
• Subject to data availability, the assessment should also include third-party replications of the model created under the grant project. This includes monetization of personnel retention, changes in earnings, profits, and economic benefit from translating the training to tertiary audience, based on the figures published or obtained during consultations. The total third-party effects are considered part of the economic benefit of the original project.
4. Financial Sustainability of the Project (Sustainability of Project Results)
This criterion evaluates a project’s ability to continue its operations and maintain the results even after the financial support from the Fund is cut off.
In order to assess financial sustainability of a project, the dynamics of service provision at the end of project financing need to be analyzed. The following indicators need to be considered for this purpose:
1. Number of services provided.
2. Revenue from services provided.
Periods subject to evaluation:
• The project implementation period.
• A comparable period before the project commencement.
• A similar period following the project’s completion.
• To assess financial strength, we can calculate the sustainability ratio using the following formula:
• Sustainability ratio = amount of revenue from services provided after the end of the project / amount of revenue from services provided during the project.
Any ratio above 0.8 is considered sustainable. It implies that the project maintains a level of service revenue post-completion close to that achieved during its active phase.
5. Increasing the Share of Own Contributions in Subsequent Projects
This aspect reflects the extent to which an organization’s own funding increases in future projects, thereby demonstrating the project’s contribution to creating a socio-economic foundation that reduces future dependence on external financial support.
Primary function of this model is to provide an opportunity to consider the changes being made from various perspectives.
Dynamics of the organization’s own funding in each subsequent project are assessed using the following formula:
• Own contribution share growth = contribution share of new project / contribution share of previous project – 1) × 100 %
In case of a singular project, establishing an organization’s own development fund is advisable. All forms of contribution need to be factored in (premises, equipment, other in-kind resources) in cash equivalent.
6. Increase/Decrease in Statistics on the Problem Being Addressed at the Territory Level
This criterion reflects the extent to which the project aims to bring about changes that are reflected in the existing municipal statistics of Norilsk and Dudinka.
To evaluate the increase or decrease in statistics concerning the problem at the regional level, an analysis of the dynamics of municipal statistics is necessary, focusing on the selected indicators the project attempts to impact. The changes can be assessed by comparing the baseline and final values of the chosen indicators. If the indicators are increasing, the growth can be monetized by using the economic benefit assessment method, which takes into account additional revenue or decreased costs resulting from improved statistics.
Indicators assessed at the territory level and relevant to the scope of project implementation to date include those for which several years’ worth of data is[134]. These figures can be used to compare and assess the changes in various indicators, the degree to which those indicators were impacted by the projects, etc., namely:
• the number of students attending additional education institutions;
• the count of children in orphanages;
• the seating capacity of cultural and leisure centers;
• the number of attendees at cultural and leisure events;
• the count of movie screenings/viewers;
• the number of library visitors;
• the frequency of visits to museums and other similar institutions;
• the enrollment numbers in municipal sports establishments;
• the student count in children’s sports schools;
• the number of visitors to special interest clubs.
If a project claims to rank high on this criterion’s scale, the following data will be needed for assessment:
• The specific indicator targeted by the project for which statistics are published;
• Indicators before, during, and after the project implementation.
These dynamics must reflect the project’s focus – reducing negative indicators and enhancing positive ones.
7. Territorial Coherence
This criterion reflects how well the project is tailored to address the specific issues of high relevance to the territory.
Evaluating territorial coherence requires analyzing the reach of the audience that gained access to the social technology aimed at solving relevant problems of the area.
The assessment can utilize quantitative analysis of projects that address issues whose significance for the territory is affirmed by external sources, such as those mentioned in the region’s Quality of Life Index and other similar indices. For instance, “N% of successfully completed grant projects were directed at resolving issues confirmed to be of high relevance for the region.”
The following set of criteria, numbered 8 to 11, falls into the ‘hygienic’ category, i.e., reflecting general recommendations applicable to any social project.
8. Presence of an Evidence Base for the Social Technology Used
This criterion reflects whether the project is founded on social technology or practice substantiated by scientific theories, research findings in the relevant field, and the experiences and opinions of the beneficiaries.
9. Presence of a Monitoring System This criterion reflects whether the project involves regular data collection throughout its implementation. This includes maintaining statistics (such as inquiries, number of services rendered, visitor counts, calls), as well as gathering feedback through questionnaires, interviews, etc
This model is adaptable for other regions, and its various components – the evaluation criteria, indicators, and approaches for developing diagnostic tools – could be effectively utilized in formulating assessment models for federal projects and programs.
10. Presence of Indicators for Assessing Project Effectiveness
This criterion reflects whether the project has developed and is currently employing a system of performance indicators for each implementation phase. These indicators should be clearly described, with an elaborated assessment methodology, including tools and data collection procedures.
11. Presence of a Data Collection and Media Presence Monitoring System
This criterion reflects whether the non-profit organization, both overall and at the project level, has established and implemented a system for evaluating media effectiveness. This involves collecting and analyzing data regarding the project’s coverage in traditional and social media. The data should be regularly gathered, analyzed, and the results promptly utilized in future media interactions.
For the ‘assessment hygiene’ criteria (8–11), recommendations are provided only for those criteria receiving low scores (1 or 2). Here are sample recommendations for criterion 8 (Presence of an Evidence Base for the Social Technology Used):
Table 1. Example of a set of indicators for a priority area
If the project is being implemented for the first time without prior testing or implementation experience (e.g., in a different territory or for a different target audience), it is advised to:
• Conduct desk research based on scientific and other data sources. This research should aim to establish an evidence-based vision of the project’s outcomes using the embedded social technology;
• Conduct a stakeholder survey, including potential beneficiaries, is recommended to identify potential challenges and threats, and to preliminarily evaluate the nature and quality of the changes expected from the project, and the conditions required for the changes to take effect. Such surveys are mandatory if the beneficiaries include audiences with feedback limitations (such as children, individuals with mental or cognitive impairments).
If the project has been implemented before but lacks a sufficient evidence base, conducting surveys of at least three stakeholder groups involved in the project is recommended. These could include project team members, beneficiaries, funding organizations, partner entities, governmental bodies, etc. The survey aims to identify potential areas for project growth, outcomes, and conditions necessary for achieving these results. A desk study, as in the case of initial project implementation, is also recommended.
The outcome should solidify the evidence base for the social technology or practice used, backing it by scientific theories and concepts, research findings, and beneficiary feedback.
PROJECT ASSESSMENT ACCORDING TO SOCIAL IMPACT INDICATORS
The second component of the model involves project assessment based on social impact indicators.
These indicators, developed from analyzing previous bids and existing thematic support areas, reflect the qualitative impact of the project. In the Table 1 an example of a set of indicators for a priority area.
When the project team believes that existing indicators fail to adequately capture the project’s impact, or if there are changes in the thematic areas supported by the Fund, the list can be supplemented. Such modification is conducted according to a defined algorithm. For new indicators, a universal formula is suggested: “What changes have occurred in the area of…? How do you assess the project’s contribution to these changes? How do these changes apply to you?” The quantitative assessment scales for these changes are akin to those used for the primary (universal) list of indicators.
It is recommended to engage project stakeholders in formulating the indicators, which may include:
• State entities: municipalities, individual politicians (party representatives, lawmakers), and executive branch representatives;
• civil society: volunteer centers, NGOs, and universities;
• market representatives: business and state corporation representatives;
• direct beneficiaries.
It is crucial to ensure triangulation in the study, validating findings through interviews with representatives from at least three stakeholder groups (e.g., beneficiaries, their families, state administration representatives, businesses involved in the project, etc.).
An impact assessment questionnaire is recommended for gathering feedback from the beneficiaries. This can be used for online and offline interviews or surveys.
OVERALL ASSESSMENT MODEL
The Fund was also tasked with developing an Overall Assessment Model, encompassing the assessment of all projects that receive grant support. The final report collates all project coverage indicators and types of impact. The overall assessment of the cumulative project impact on regional development is designed to record three categories of data: UAP-based universal social impact criteria, overall change significance index based on indicator analysis, and an optional integrated qualitative impact assessment protocol. These are operated as follows:
• Projects are assessed against universal social impact criteria based on UAP;
• Data is aggregated into the change significance index;
• A separate study is conducted to learn the cumulative project impact on regional development, using a qualitative impact assessment protocol.
Econometric modeling analysis can be employed to further refine the overall assessment of social impact indicators based on the UAP. This approach can be viable once a comprehensive database of indicators is collected, enabling the establishment of causal relationships.
CONCLUSION
Summing up, we would like to mention a few characteristics of the model developed, which are noteworthy in our opinion.
The model, while not designed as a decision-making tool for supporting specific projects, has the potential to serve this purpose. Its primary function is to act as a “system of mirrors,” allowing for a multifaceted examination of the project-initiated changes from various perspectives. This approach helps to better appreciate the “beauty” of the changes being made in the lives of the beneficiaries and the territory as a whole.
It is essential not to interpret the UAP (Unique Assessment Profile) criteria as definitive markers of “good” or “bad” projects. For instance, a project contributing to workforce retention in the region should not be automatically deemed “good,” and vice versa. These criteria simply suggest that if a project aims to tackle workforce retention, in addition to other significant changes, specific relevant data should be requested and considered, especially if it addresses a key regional issue effectively. A project not addressing this does not inherently become “worse”; it simply focuses on different changes. Similarly, not all projects are based on mechanisms to achieve economic sustainability etc., but if they do, that aspect should be analyzed separately to potentially recommend such solutions for replication in other regions. Most importantly, this approach allows the project itself to gain deeper insights into its impact and to plan future activities based on these insights.
In our opinion, the model proposed can be adapted for the tasks of other organizations and for different territories. The combination of UAP criteria and qualitative change indicators remains a constant feature. However, the specific UAP criteria can vary to reflect the important indicators for a particular territory or organization. The “hygiene” criteria (8 to 11) are universal and can serve as a basic checklist for any project, offering foundational guidance. The second component, focusing on qualitative change assessment, can be implemented through various methodologies suited to identifying change indicators.
It is therefore safe to describe the model as “open source,” allowing for customization and refinement to meet the specific challenges and objectives of different organizations and territories.
