Редакция журнала «Позитивные изменения»
Позитивные изменения, Том 3 №1, 2023. Positive changes. Volume 3, Issue 1 (2023)
No Standards, Just Calling. Is Project and Program Evaluation a Profession?
Yulia Vyatkina
DOI 10.55140/2782–5817–2023–3–1–16–27
The topic of professionalizing project and program evaluation (in the social sphere, culture, science, environment, etc.) has been on the agenda of international organizations for more than a dozen years, or about ten years in the Russian practice. There are numerous publications and discussions between evaluation specialists talking about the status of evaluation as a profession at the regional and international level. In our country, this topic is addressed at the annual conferences of the Association of Specialists in Program and Policy Evaluation (ASPPE or ASOPP). So is project and program evaluation a profession, and if so, how can we find out and tell?
Yulia Vyatkina
Editor, Positive Changes Journal
THE STAGES OF TURNING AN OCCUPATION INTO A PROFESSION
Before we start talking about the professionalization of evaluation (referring specifically to evaluation of projects and programs), let us consider how an occupation becomes a profession.
Harold Wilensky, an American sociologist who studied the functioning of various organizational structures in the modern society, proposed in the 1960s a model describing the formation of a profession in several stages[15].
The process starts with a certain occupation acquiring full-time status. That is, with enough people engaged in some kind of activity from morning till night and making a living off of it. Full-time employment is the mandatory starting point, otherwise the occupation cannot turn into a profession. In addition, it is possible when there is a demand for services.
The second stage is the emergence of a school of professional training for the people engaged in this type of activity. Schools like this are often associated with universities. The next step is a professional association – an association of people engaged in this activity on a permanent basis. The association takes actions forcing the state to adopt certain regulations establishing the procedure for licensing and certifying activities, thus establishing a kind of professional monopoly. Also, the professional association affects the professional training of specialists, promoting the development of a theory, methods, strategies of activity and competences of the specialists.
Next, a code of ethics is developed, which helps prevent abuse of authority and power. Self-regulated professional associations demonstrate to the public that they control the code of ethics.
The final step is the formation of a positive image in the society. When you have established an image, you can say that you have a new profession.
WHAT IS A PROFESSION?
There are two approaches to defining a profession. In the first case, it is defined as an activity, an occupation, a job. In the second, as a group of people.
A profession as an activity is a paid job, especially one that involves lengthy training and formal qualifications. Accordingly, a professional is a person who has undergone such training, received confirmation of his/her qualifications and is engaged in that job.
Caroline Heider, former Director General and Senior Vice President, Evaluation, World Bank Group, highlights the key features of established professions[16]:
• There is an established stream through the tertiary education system that each member of the profession has to complete, and some professions require regular training to update knowledge and skills;
• The profession requires that new graduates practice in close collaboration and under the supervision of more experienced practitioners;
• There are strong, recognized professional associations that enforce and reinforce standards;
• Legislation is in place that sets out requirements and provides the basis for legal recourse in the case of malpractice.
IS EVALUATION A PROFESSION?
We decided to do a little research of our own and turned to experts who have been working in evaluation for quite a long time. Our speakers responded that it is too early to call evaluation a mature profession yet. By the way, its degree of maturity varies from country to country.
“I think it is a profession in its nascent stages. There are many signs indicating this. If you look at the percentage of those who graduated from higher education institutions and majored in «Evaluation», that number is really small anywhere in the world. Russia does not have such specialization at all. People come to evaluation from different fields and have different backgrounds,” says Alexey Kuzmin, Owner, Process Consulting Company.
It’s always worth remembering that when we talk about a profession, we have to understand that it has to have a number of attributes. Some of these are present in Russia, some are not, says Irina Efremova-Garth, First Deputy Director of the Donors Forum.
One of the key signs of professionalization of an activity is the self-organization of the related community. For example, in Russia we have the Association of Specialists in Program and Policy Evaluation (ASPPE or ASOPP), established in 2014 with the mission to develop and promote evaluation as a profession. The Siberian Civic Initiatives Support Center and the Garant Center of Social Technologies hold summer schools on evaluation; the PROOCENKU Club has been working actively, with the colleagues actively sharing their experiences and lessons learned during its meetings. An equally important attribute of professionalization is the existence of principles and standards formulated by the specialist community. In Russia, as in other countries where professional associations operate, these principles have been developed and adopted by the ASPPE[17]. Their distinctive feature is that each of the six principles contains recommendations not only for evaluators, but also for the clients and the participants of the evaluation process. Another sign of professionalization is the certification system, wherein a specialist in the field of evaluation would receive official confirmation of his/her experience and knowledge. There is no certification of evaluators in our country; at the same time, it does not exist in many countries of the world either, with the exception of Canada and Japan.
The list of signs can also include growing interest of universities in the professional training of evaluation specialists: for example, Moscow State Psychological and Pedagogical University has launched a Master’s degree program “Evidence-Based Design and Evaluation of Programs in the Field of Social Risk Management in Childhood”. One of the factors contributing to the development of evaluation in Russia was the strategic audit organized and conducted by the Accounts Chamber of the Russian Federation. It is carried out to assess the feasibility, risks and results of achieving strategic goals, as well as the impact of external and internal factors on the progress towards these strategic goals.
“All these facts serve as an evidence that evaluation is developing as a profession,” Irina Efremova-Garth summarizes.
Elena Malitskaya, president of the Siberian Civic Initiatives Support Center (SCISC) notes that there is a growing demand from the state for specialists in evaluation in the country. “However, this demand is quite narrow – for a specific type of evaluation only, related to a review of applications, projects and programs,” Elena Malitskaya says.
Marina Mikhailova, director of the Garant Center, agrees that the demand for specialists in evaluation and the respective training is growing.
“So far, evaluation remains an expertise, not a profession. But 15–20 years ago, it was merely a discussion between specialists; today the situation has changed greatly. On the one hand, there is a lot more talking about evaluation. On the other hand, every applicant in the presidential and other grant competitions needs to evaluate the results of their projects. In addition, there are more educational products that teach self-evaluation,” Marina Mikhailova says.
There are contests rewarding people and organizations that pay attention to evaluation, she continues. The expert community started looking at the annual reports to see whether they contain information about the evaluation of the organization’s performance. They also started realizing that evaluation is not the tool of a few super-specialist, but something everyone has to do, one way or another. Sometimes employees of organizations are just unaware of this.
Evaluation has come a long way towards professionalization, but it still has a long way to go, says Natalia Kosheleva, president of the Association of Specialists in Program and Policy Evaluation, a consultant on monitoring and evaluation of socially oriented programs and projects.
“Any profession is constantly evolving in some direction. Professions are becoming more and more niche, evaluation is no exception. It’s a small profession, but it’s growing pretty fast,” she says.
WHERE DO EVALUATION SPECIALISTS COME FROM?
There are not as many people around the world with a degree in evaluation as there are lawyers or economists. Our colleagues in the West have Master’s and PhD degree programs in evaluation, but given its interdisciplinary nature, the number of people in this profession is not determined by the number of those who received a University degree, experts say.
So where do specialists in evaluation come from, and how do they build a professional career? Alexey Kuzmin says that the Association of American, Canadian, and European specialists in evaluation, which has several thousand members each, consists of people with very different backgrounds.
“Let me list: just in terms of the underlying education, there are psychologists, sociologists, educators, medics, engineers, computer technology specialists, economists, managers, statisticians. Of course, if you graduated from a University program where you received training in sociological research methods, it’s a good help,” he says.
The expert himself came to the evaluation in the late 1990s from organizational development consulting. In 1999 he became a member of the American Evaluation Association, attended conferences and seminars abroad, read literature in English, and in 2001 he decided to attend a PhD program in the United States, specializing in organizational behavior and program evaluation.
“A lot of people overseas that I know of have professional careers as internal evaluation specialists. This is, of course, common in large companies. The common denominator for all stories is that at some point a person faces the need for evaluation, and something resonates inside, they find it exciting. And that’s where you start a turn in this direction – searching for information, training, searching for specialists who do this,” says Alexey Kuzmin.
Elena Malitskaya also cites 1998–1999 as the years in which her work became closely tied to evaluation. “At that time, we at the Siberian Civic Initiatives Support Center were conducting our own grant competitions. So we had a need for monitoring, evaluation and training. We just started integrating evaluation into our work routine and into the work of our network of resource centers. In 2000, we held the first school of evaluation and international conference in Novosibirsk, which eventually grew into regular activities,” Elena Malitskaya says.
Alena Bogomolova, head of the Resource and Methodological Center of the Road Home Charitable Foundation and a board member of the Association of Specialists in Program and Policy Evaluation, is one of those who received training from SCISC and The Garant Center at the International School of Evaluation. Alena Bogomolova is a sociologist by education. She says she had done many practical projects during her University years that were related to evaluation in one way or another, except that it was called differently at the time. “It was called sociological support in the early 2000s, at least in Cherepovets. But the set of activities was the same as what we now call evaluation, or evaluation design. I have studied before, I am studying now, and I will continue to study, because evaluation is a constantly evolving profession,” she says.
Irina Efremova-Garth and Natalia Kosheleva received their Master’s degrees from American universities. Irina earned her MPA (Master of Public Administration) from the School of Public Administration at the University of Delaware.
“In my two years of training, my schedule included classes on various research methods, on evaluation in general, on evaluation in nonprofit organizations. It was so delicious, so interesting, and so exciting that all my plans to go into the government went straight to the bottom of my priority list. I returned to Moscow realizing that I would be involved in evaluation and nonprofit organizations,” the expert continues.
She says she has been in different roles: the customer of evaluation, internal evaluator, and external evaluator. Her work at IBM had given her some interesting insights: data-savvy and digitally savvy nonprofits are much more likely to achieve their mission, to be able to prove their contribution to addressing the social problem at hand.
Natalia Kosheleva learned about evaluation during her Master’s degree studies at Indiana University’s School of Public and Environmental Affairs.
“It was March 1996. I came across a vacancy announcement for a specialist to evaluate two environmental projects funded by IREX (International Research and Exchanges Board). That was the first time I saw evaluation might be a thing. The beauty of an American university is that it has an open library. I just went in, found some books on the topic, read them, got an idea about the general principle, and applied. And they hired me,” Natalia Kosheleva explains.
Irina Sinelina, a specialist in program and project evaluation at International Labor Organization (ILO), received her Master’s degree in Nonprofit Management in the United States and interned at a nonprofit organization in New York. Then she returned to Moscow and got her first job at a nonprofit organization, where she was to analyze the effectiveness of trainings, before coming to work at the International Labor Organization, as an evaluation officer.
“Like all UN organizations, ILO has an internal evaluation system. This is done for the purpose of donor accountability, increased experience, and work improvement. Like the rest of the UN system, we follow the standard criteria developed by the Organization for Economic Co-operation and Development (OECD). There are five basic principles: relevance, effectiveness, efficiency, sustainability, and impact. Two years ago, another one was added – coherence,” Irina Sinelina says.
Marina Mikhailova learned the principles and approaches to evaluation (including organizational development evaluation and personnel evaluation) from Process Consulting and the Siberian Civic Initiatives Support Center, and international schools.
“All the small pieces got together, one after another, to build the big theoretical experience. In practice, it is, first of all, extensive experience in evaluating projects submitted for grant competitions. We, at the Garant Center, have been organizing these contests since 2001 in Arkhangelsk Region. Later I took part in reviewing applications for other contests and evaluating the results of the projects we had supported. I also had some experience in evaluation from the times I was the director of a recruitment agency. as such, I had to evaluate personnel and the management system in the organization,” Marina Mikhailova continues.
KNOWLEDGE AND COMPETENCIES
The topic of knowledge and competencies in evaluation is closely related to the topic of learning. If we know what the competencies are needed, we know what to teach them, and how to write job descriptions.
Since the late 1980s, various professional associations of evaluators started publishing lists of competencies. For example, the UN Evaluation Group believes that an evaluator should have basic professional competencies, technical skills, management skills, interpersonal skills, and promote a culture of evaluation.
Canada, a country that has established a certification system for evaluators, suggests the following areas of competence for a practicing evaluation specialist[18]:
1. Reflexive – competencies related to fundamental norms and values in the field of evaluation, as well as awareness of one’s own expertise and needs for professional growth.
2. Technical – competencies related to specific aspects of evaluation: planning, data collection, data analysis, reporting.
3. Situational – competencies related to evaluative thinking applied to analyze and consider the interests, problems, and circumstances specific to each evaluation.
4. Managerial – competencies related to the project/evaluation management process (budgeting, resourcing, management).
5. Interpersonal – competencies related to communication, negotiation, conflict resolution, cooperation.
The Association of Specialists in Program and Policy Evaluation has not yet produced a similar list of competencies. Natalia Kosheleva, ASPPE President, emphasizes an understanding of how programs and projects work as one of the core competencies of evaluation specialists in Russia. Equally important is the ability to collect, analyze and present data. Communication and interpersonal competencies are necessary for those involved in external evaluations. Finally, visual communication skills are increasingly demanded lately.
Alena Bogomolova says that an evaluator needs to be able to exchange opinions and knowledge freely.
“You need groups of people or platforms where you can talk about what you’re getting/not getting and discuss everything before embarking on the journey of evaluation: the best ways to proceed, questions to formulate, design to use. You need to be able to talk to other people about it. Because it is very difficult to develop in a vacuum. What else is important? In my opinion, it is vital to have your own value-based professional position, from which you will make an evaluation,” she says.
Marina Mikhailova agrees: “To be a professional, you have to be part of the professional community, sharing experiences, learning from others, communicating. And it is important that would-be evaluators follow ethical standards. Still, evaluation is not an audit, not an inspection. And here it is important to remember that evaluation is not about giving an A or an F; it is about answering the right questions and making the right decisions. An evaluator is a person who has come to help sort things out,” Marina Mikhailova explains.
Evaluation is an incredibly interesting field, Irina Efremova-Garth notes, because an evaluator must have skills from many different areas.
“You have to be extremely proficient at doing different kinds of research, extremely good at data crunching. You must have very good management skills. After all, evaluation is rarely done by just one person. It usually requires a team, so you have to be able to build processes properly, allocate roles in the team and manage results. You have to have reflection, critical thinking skills,” the expert adds.
Irina Sinelina believes that it is important to have a good basic education in liberal arts, which builds analytical skills, and a good knowledge of English, because much of the material about evaluation is written in English. The rest you can learn by practice.
PROFESSIONAL STANDARD
As mentioned earlier, one of the stages in the development of a profession is a professional standard. Why do we need a professional standard in any activity? As noted in Decree No.23 of the Russian Government dated January 22, 2013 professional standards are applied[19]: in developing the personnel policy, preparing job descriptions, in developing vocational education programs and standards.
According to Alexey Kuzmin, an attempt was made in 2015 to develop a professional standard for a specialist in program and policy evaluation in the social sphere. It was initiated by an independent business expert, Alexander Ovchinnikov. The application for development was registered at the Ministry of Labor of Russia website. A working group was established, including representatives of Process Consulting Company and ASPPE, basic research and analysis of the professional activities were conducted, the first draft of the professional standard was developed, and a public discussion took place; however, the work on the document was halted at some point, and the professional standard was not approved by the Ministry.
“I think it is a little premature to develop such a professional standard today. There is no profession yet, it has not yet been established. The big picture, as seen by the Ministry of Labor, still lacks both supply and demand for this profession,” Alexey Kuzmin explains.
Natalia Kosheleva suggests that the ministry’s job was to develop professional standards for basic professions, something that a program and policy evaluation specialist clearly isn’t so far.
“The Ministry of Labor had different expectations at the time, and it had a mission to cover all mass occupations. Ours is definitely not one of those. I think that was the reason. But the standard may indeed emerge at some point, as the profession develops further. When we gain enough visibility for the state to decide we need regulation. It’s like the story of licensing all kinds of activities. To be seen, there must be a lot of you,” says Natalia Kosheleva.
The National Council for Professional Qualifications under the President of the Russian Federation also reviews drafts of professional standards in Russia. Based on the National Council’s conclusions, a decision is made to approve professional standards. As explained by Artem Shadrin, General Director of the National Agency for the Development of Qualifications, when developing a professional standard, it is important not to duplicate existing ones, approved under a similar name. It would be good to get the support of some reputable organizations working in this area, as well as the relevant federal ministry, which would confirm the need for developing such a document.
Once a professional standard has been approved by the Ministry of Labor of Russia, it is then sent to the National Council for Professional Qualifications under the President of the Russian Federation. If approved by the National Council, which represents leading employers’ organizations, such as the Russian Union of Industrialists and Entrepreneurs, the Chamber of Commerce and Industry, trade unions and federal ministries, the standard is then enacted by order of the Ministry of Labor and registered with the Ministry of Justice.
Why does the professional standard story remain unfinished? What could be the reasons?
“First, the Ministry of Labor may see no need for such a professional standard. This is a classic unsolved problem. For example, as for a director per se, there is no such professional standard, and the ministry believes that it is not necessary. Likewise, there is no professional standard for an entrepreneur. On the one hand, this doesn’t seem like a job function; but, on the other hand, it can be considered a type of professional activity. The Ministry of Labor has not yet agreed to develop a professional standard for this activity,” Artem Shadrin says.
He explains it may also happen that an initiative group tasked with developing a professional standard fails to have it approved by the relevant ministry, or the project might not win support from professional associations.
“You cannot get a standard approved without a consensus in the professional community. That’s the challenge: developing a professional standard is a search for touching points. Its development is a long story, but nevertheless, nothing is impossible. As long as there is a desire and support from the relevant ministry, which is always important,” Artem Shadrin adds.
WHAT DOES THE EVALUATION SPECIALIST AS A PROFESSION STRIVE TO GIVE TO THE SOCIETY?
The American philosopher Thomas Schwandt wrote an article in 2017, in which he outlined the need to discuss the ethos of evaluation[20]. That is, the lifestyle of a social group that associates itself with evaluation. According to him, right now, most of the professional community is just talking about competencies and training. Specifically, he writes:
“…Most glaring is the lack of active discussion about what professionalism in evaluation means and what the profession itself seeks to bring to the society.”… “The critical question for the global evaluation community is how that community understands its role in relation to its clients and the society at large.”
What do members of the Russian program and policy evaluation community think about this, and how would they answer the question posed by Thomas Schwandt?
“To me, the profession of an evaluator means objectivity and a competent approach to management decision-making. Because evaluation is based on the principle of triangulation. Before you make any decisions, you have to consider your issue from different perspectives. There is triangulation by attribute, by method, by data source. You cannot make a decision based on any single fact,” Alena Bogomolova reasons.
Irina Efremova-Garth believes that evaluation gives the public a sense of confidence in the quality of management and the appropriate level of effectiveness of the programs implemented by the state, a company, or a nonprofit organization.
“If we talk about the nonprofit sector, the focus is often on how many projects have been supported, how many beneficiaries are covered, and how many grants have been awarded. These are important indicators, but they don’t give us any information about what changes are taking place because of that nonprofit’s work. At best, these numbers only describe the grant manager’s performance. But they do not explain how our lives or the lives of certain groups change for the better,” the expert says.
When you have the evidence base and can show what actions have been taken to address a social problem and what their effect was, this is a highly valuable step, because it helps build confidence in the society.
Irina Sinelina of the International Labor Organization believes that the public benefit provided by evaluation is the accountability in the use of money.
“The ILO, as an international organization, uses taxpayers’ money that comes from different countries. Evaluation is an opportunity to understand how well the funds were spent, and how they can be spent better in the future, how the projects can be improved, and how change can be effected. We collect data, we learn, we study the degree of usefulness of our activities. This knowledge guides our future decisions”, Irina Sinelina explains.
Natalia Kosheleva agrees that evaluation helps use resources more effectively to improve people’s lives. She recalls that evaluation of social projects and programs originated in the United States and was first used in projects aimed at fighting poverty and overcoming social inequality.
“Globally, evaluation is about improving the lives of the most vulnerable people. When I began to specialize in evaluation, I realized it was really an analysis that allowed me to improve the quality of the program. It is a tool for informing your actions, learning from experience, and improving from empirical data. After all, this is not just any evaluation: this is about evaluating social projects, ensuring a more efficient use of resources and a higher effectiveness of social projects,” she says.
WILL THE EVALUATION BE DEMANDED IN THE FUTURE
The profession of evaluator will certainly be in demand in the future all over the world, the experts agree. “It’s already the gold standard in the public and private sector,” Irina Sinelina says.
“Evaluation has everything it takes to become a profession, and it will certainly become one. But this will be the result of an evolutionary process, through accumulation of knowledge, understanding of the reasons why it is needed, the growing demand for evaluation professionals, establishment of a community that can raise those professionals, and the emergence of educational products,” Marina Mikhailova forecasts.
The future is often associated with artificial intelligence. “Who is a project and program evaluator?”, we asked ChatGPT. This is what the AI told us: “A project and program evaluator is a professional expert engaged in assessing the value of projects and programs. He or she can work in various sectors of the economy, as well as in the legal market. To be successful, the professional needs to be educated and experienced in financial analytics, be able to work with various forms of assets, to develop valuation models, analyze information, and produce recommendations on improving property values.”
As we can see from this definition, the neural network does not yet understand the difference between a property evaluator and a project and program evaluator. This means that the Russian professional community still has a lot of work to do to make itself known. After all, the profession is still in its infancy. And it still has a long journey ahead, building the demand, ensuring full employment of specialists, setting up professional training schools, developing and approving professional standards, the code of ethics, and establishing an image for itself. As they say, time will tell how the professionalization of evaluation will develop and whether artificial intelligence can compete with it.
This article was written using presentations by Alexey Kuzmin and a video from the ASPPE conference “Valuation in a Changing World: Partnerships and Infrastructure”, session “Professionalization of Evaluation in Russia and the World”.
