2.10 Research Quality Assessment in an era of change: is it possible to develop and operationalize a framework that embraces the current debate and needs of the community?

Comments

• • Johan van de Worp
  • 1682335611

  Here you can read a summary from this workshop (with thanks to the reporter for making it):

  Tona presents the research quality assessment framework, and the approach to develop the framework.

  During this workshop participants stepped into the shoes of an assessor, and from this position (jointly) discussed a framework [under construction] to assess research quality. The framework itself responds to the need to operationalize the reform of research assessment. It is a result of earlier co-creation processes in which a diverse group of academics have been involved.

  Within the framework four dimensions which are relevant for assessing research quality are presented, each a ‘leg’ within an X-frame. The two ‘legs’ at the base of the X represent the dimensions “Rigor” and “Culture” – pillars of any research. The two upper ‘legs’ representing “Relevance” and “Originality” will receive more or less importance during an assessment, depending on the context of the assessment.

  Before going into break-out groups all participants were asked to indicate – via mentimeter – what they thought constituted ‘rigor’, ‘culture’, ‘relevance’ and ‘originality’. The answers were relatively uniform and showed that there is a joint understanding what these terms mean within the context of doing research. This included the relatively new dimension ‘culture’ (i.e., collaboration/teamwork; social safety; learning environment). Only the word 'societal impact’ - which was mentioned within the relevance dimension - raised discussion: not all research would immediately have societal impact. Would all research need to be measured on this?

  Within the break-out groups the participants discussed the four dimensions, including sub-elements. Were any missing? What would further be needed to conduct an assessment using this framework? And what would additionally be needed to conduct a transparent assessment?

  Some of the results are:

  • No additional dimensions were identified. It was suggested that the dimension ‘climate’ should be at the basis of everything.
  • The term impact is confusing – it gives the impression that every researcher immediately has to have impact. This supported the use of ‘relevance’ rather than ‘impact’ as one of the dimensions.
  • Some sub-dimensions raised discussion - should ethics be assessed as part of ‘rigor’ or as part of ‘culture’, or as part of both dimensions?
  • An important requirement for a good and transparent assessment: time.

  
• • Claartje Chajes (Communitymanager RRview)
  • 1683551631

  @mariaverschoor, @klodianadaphnetona, @irenevanstaveren, @leandervanderspek, @thomasschlatholter, @karinpfeffer, @cashenckens, @balinthardy, @liafluit, @marileendogterom would anyone of you like to share with us how you experienced the framework? Does it make you optimistic about the possibilities it offers?