What is changing in the assessment of scientific quality?

One of the objectives from the position paper Room for Everyone’s Talent (2019) is: focus on quality. In the paper, we argue that the implicit and overly one-sided emphasis on traditional, measurable output indicators (such as number of publications, h-index and journal impact factor) is partly responsible for the high workload. In addition, we argue that their use can upset the balance between academic disciplines and is inconsistent with the principles of the San Francisco Declaration on Research Assessment (DORA). This had been reaffirmed with the publication of the Agreement on Reforming Research Assessment (2022). The signatories of this statement have committed to refraining from the inappropriate use of benchmarks based on journals and publications, in particular the Journal Impact Factor and h-index.

This is because while bibliometric indicators tell a story, they do not tell the whole story. Commonly used indicators often give a one-sided picture of academic quality. A lot of research has been done in recent years on the creation, use and negative effects of various metrics. The conclusion has been that the disadvantages often outweigh the advantages. In addition, these indicators are not comparable across academic disciplines. Consequently, they do not do justice to the diversity in academic domains and academic practice. Relying too strongly on such indicators can disrupt the diversity and societal impact of research. At the same time, it impedes the implementation of open science. It is important, therefore, to recalibrate and broaden the system of rewarding research.

As knowledge institutions, we are looking for a more evidence-based approach. In the Strategy Evaluation Protocol 2021-2027 for example, qualitative assessment of research groups is already the starting point. Also, an increasing number of institutions – including the NWO and ZonMw – are working with an evidence-based CV in which academics describe their profile and achievements in a coherent way, supported by key results (evidence) that fit the candidate’s profile, discipline and context.

The evidence-based CV certainly provides space for quantitative substantiation, so it is not the case that the use of quantitative indicators is disaproved, as is sometimes claimed. Quantitative indicators can still be used, as long as it is done responsibly. When meaningful metrics are a valuable addition and an academic can explain why a quantitative indicator is relevant, this provides far more context and can definitely be included in the assessment.

In the assessment of all academic work, we therefore want to place a greater emphasis on quality, content, academic integrity, creativity, contribution to science and/or society and the specific profile in which an academic is active. How we define quality and how we should assess quality is still undeveloped in the domains of teaching, impact and leadership. The joint programme monitors developments at the institutions and tries to bring together developments at the national level where possible.

You can read more about the assessment of quality in our Knowledge Base (mainly in Dutch).