Research Productivity Assessment at HSE
The HSE Academic Council approved a new regulation on Research Productivity Assessment (RPA) at its November session. Dmitry Dagaev, Deputy Vice Rector and Associate Professor of the HSE Department of Higher Mathematics, who is responsible for coordinating evaluations of research activities, talked with The HSE Look about the role of RPA at our university and the changes that will be introduced in 2018.
What is RPA and why has this procedure been launched at HSE?
The idea to create a tool for monitoring the publication activities of HSE researchers was conceived in 2013 at a seminar of the HSE programme for Young Academic Professionals. The participants, both researchers and faculty, noted that, while faculty members who hold teaching positions must regularly account for their publication activities when participating in competitions for positions, no similar requirements were in place for those holding purely research positions. This generated a certain imbalance in the university’s policies, which are aimed at attaining excellence in research and establishing the tools used towards these ends.
However, before changing our tools and policies, it was necessary to understand the situation. So, the first RPA was launched in 2013 only as a monitoring. It was first tested on a small number of research units, and, in late 2013, it was implemented for all research staff at HSE. Effective as of 2014, evaluations of research productivity have been carried out regularly, and information about individual performance is delivered confidentially to persons undergoing evaluation. These results are also presented to their direct supervisors.
What do evaluations entail?
RPA takes into account a researcher’s articles published in the two full years preceding the evaluation year. Everything published in the current year up until the date of the evaluation is also counted towards the final assessment result. It is not possible to guarantee equal research productivity every year in terms of total publications, due to a variety of factors, including long peer-review processes at prestigious journals and the time needed to address complex problems in given research projects. Therefore, we definitely wanted to accommodate this. However, two years and several months are quite enough to have some of one’s work published.
Generalized results are made public, and each year HSE researchers are performing better: at the start, only 33% met the established criteria, while last year we were at 86%. Since the tools for enhancing publication activity are proving to work well, the university keeps raising the bar for passing the RPA process, so as to encourage HSE’s staff to develop publications in high quality journals.
What happens if you fail an evaluation?
It is important to understand that failure to meet RPA criteria does not necessarily mean automatic dismissal from HSE. Nevertheless, this may serve as a signal for an employee and his or her supervisor that things need to change in order to continue working as a researcher. Each situation when colleagues do not meet the criteria shall be treated on an individual basis. Any formal decisions can only be taken by the head of the involved unit and the Vice-Rector responsible for coordinating said unit. With this in mind, an individual plan is usually drawn up by the researcher, so that their supervisor can assess the results after a certain time, and see if any positive change is feasible. One year is quite enough time to make progress in this respect and demonstrate it through one’s publication performance.
HSE has many tools to support research activities that are run on a competitive basis. Which of them rely on the RPA process?
We are pleased that the results of the RPA procedure are now being widely used at HSE by many units and programmes. Firstly, the HSE Academic Fund uses RPA results when considering applications. For example, for several years, positive evaluation results have been a prerequisite for applications. Secondly, the professional development programme for Young Academic Faculty also uses RPA for the purpose of admission to some of its tracks, as well as measuring the effectiveness of individual participation in the programme.
Thirdly, international laboratories at HSE rely on RPA procedures in quite a strict way. Since they are meant to be the drivers of groundbreaking research, we have set a higher bar than for other research units and require that all of their academic staff meet the evaluation criteria. Furthermore, the HSE Centre for Fundamental Studies has their own criteria for measuring the performance of research groups and projects receiving its funding. Nonetheless, they integrate RPA as one of the criteria in applications. A certain percentage of research project members should have positive RPA results if a project is to be considered for funding.
What changes will be introduced in 2018?
There are several aspects in which the RPA process shall be changed so that it can be more convenient and transparent for academic staff at HSE.
Firstly, we are diversifying the ways whereby academic staff can pass the evaluation process. Previously, we had 3-4 different tracks with their own rigid requirements. We often received suggestions regarding how to make them more flexible, so as to match publication conventions and the timeframes in different research. When looking at the alternative solutions, we realised that the HSE Academic Fund already uses a flexible approach in its calls for applications. And this can work for our purposes as well. Depending on your position, you need to acquire a specific number of points (for example, 14 points for associate and assistant professors, as well as research fellows). Points are assigned for different types of publications (a book, a book chapter, an article, a textbook, etc.) and the quality of the publisher or the journal (whether they are on the priority list and indexed by Web of Science and Scopus or not). All categories are outlined in the RPA regulations. This approach thereby allows each faculty member to employ different strategies in order to pass the evaluation process.
Secondly, we are moving from manual review of all publication lists submitted by faculty members to a more automated process. Currently, a number of committees are involved in reviewing applications for evaluation, and a heavy burden is placed on those colleagues who should submit their documents by a certain date, and those who should check them and report the results within quite a short timeframe. Instead, we will use software to collect all information about publications for the two previous years and up to the current date on personal pages. It then analyzes and organizes the publications according to the different categories outlined by the RPA regulations. This allows academic staff to check at any moment how they are performing in line with the evaluation criteria rather than anxiously wait for their results in December. Naturally, such a system requires that we do not neglect updating our personal pages and that we duly add information about new publications as soon as they come out. After the information on the personal pages is verified, it is analyzed by the software, and if a publication is difficult to categorize, we can review it manually.
Thirdly, we are continuing to raise the quality bar, and this has been a major trend of 2017. For example, effective this year, publications in Russian journals only count towards your RPA score if the journal has been included on HSE’s ‘priority list’. We are aiming to foster an environment, which encourages colleagues to publish work in high-quality journals. In this respect, we are eager to continue fine-tuning the evaluation criteria, so as to help shape the expectations of faculty, while also letting them adapt their individual strategies.
What are the exemptions from the evaluation process? Have there been any recent changes in this regard?
HSE employees may apply for opting out of the yearly evaluation on several grounds. For instance, if they are research assistants who are currently studying at a Bachelor’s or Master’s programme, or if they are on a maternity or parental leave, or have returned from it less within the last two years. Previously internationally hired faculty and researchers have also been exempt from the evaluation process. However, starting in 2018, we are also including them in the RPA process. This will be carried out only asa monitoring tool, with no changes in regards to the terms of their contracts.
The reasoning for this is simple: HSE is eager to have a more well-rounded understanding of its research performance so that it can develop its policies and fine-tune those tools that help us be more globally competitive as a research university. In this respect, we must be able to observe the whole picture in order to do that.
What is most important to remember for anyone taking part in the RPA process?
I think it is important to keep in mind that HSE wants us to choose the best venues for publishing our work so that it can have a strong impact in regards to our respective research areas. The crux of the issue is not in matching formal criteria, but in making our research known internationally. However, the academic market is currently set up in such a way that, the higher a given journal’s impact factor and its quartile score, the more experts in the particular field read it. Thus, our formal requirements correspond to the goals of producing high-quality research. And this is an objective which both HSE and the faculty members share.
In summary, it is essential to find a journal that suits best one’s article and is well-respected by researchers in your field.
I would also like to note that if there are any questions regarding the RPA process, they can be addressed to me via e-mail at ddagaev@hse.ru.
Read more articles in The HSE Look December 2017 issue