Inspiring Canadian institutions in transforming assessment

  • Dec 2023
  • Giovanna Lima
  • 1
  • 5
  • 81
Giovanna Lima
R&R festival 2023
  • Julma Braat
  • Jeroen Jansen
  • Bianca Langhout
  • Eveline Braber
  • Claartje Chajes
  • Marta Teperek

How can the Dutch experience inspire other national systems? What are the differences that I encountered in the Netherlands when compared to Ireland, where I was previously based? These were some of the questions I had in mind when preparing for the 2023 Canadian Science Policy Conference. Last month, on Tuesday November 14th, I was delighted to be a panellist on “Transforming the assessment of research excellence”, organised by Concordia University and the Natural Sciences and Engineering Research Council of Canada (NSERC). Sharing my perspectives derived from working to advance societal impact in partnership with the Recognition & Rewards (R&R) team at Erasmus University Rotterdam (EUR), and as a Research Impact Officer at Trinity College Dublin, where I got to author the Researcher Impact Framework (see https://doi.org/10.25546/98474), I was hoping to inspire and support change.

Together with experts from Canadian funding and higher education institutions and Professor Stephen Curry, chair of the steering group of the San Francisco Declaration on Research Assessment (DORA), I highlighted the importance of collaboration and coordination, and how R&R can serve as inspiration for Canada to reform research assessment. Particularly, I noted the role of the national coordination, the importance of a community of practice, and the many experimentation avenues that are in place in multiple Universities and Schools. Drawing from our experience at the EUR, I also highlighted the importance of establishing internal supports, especially in the form of hiring colleagues to implement these programs to capacity building with academic staff.

Acknowledging the complexity of the Canadian federalised system, I shared other relevant European examples, like the national chapters for CoARA and Campus Engage in Ireland as successful coordination initiatives to be considered. Regarded as one of the best panels in the conference, the questions discussed by the panellists included:

  1. How would you define research excellence and its relationship to research assessment reform?
  2. What are some common challenges or gaps existing in Canada and abroad in the implementation of research assessment reform?
  3. What successful strategies have institutions adopted to address the challenges and gaps, and transform their culture to align with emergent understandings of research excellence? What progress has Canada already made that can be built upon?
  4. How can Canadian institutions cooperate in concrete ways to move forward together towards redefining or broadening the concept of research excellence? What is the role of funding agencies in this process? What could be the role of international organizations?

What else would you have highlighted from R&R to inspire other national systems implementing reforming of academic assessment? I’d love to hear from you! Please reach out to me at giovanna.lima@eur.nl and our R&R team at recognitionandrewards@eur.nl with your ideas of what makes R&R an inspirational initiative.

Attachments

Download
title added by options
20231114_133840.jpg Dec 2023 Giovanna Lima

Comments

5 comments, latest: 20 December 2023

  • Thanks so much for sharing Giovanna and congratulations for organising such a well-received discussion! Much appreciated and very interesting. I like the questions you asked during the panel and I was actually wondering if you might still remember (or could share) some of the answers shared in response to questions 3 & 4?

    Marta Teperek

  • Hi Marta, thanks for the question! I do have my own prepared notes for those questions, which I paste below. The reply that I remember most for question 4 was from Stephen Curry who said "Follow the Dutch", as I had indicated many of the R&R principles/initiatives as a reply to question 3, as you can see below. I ended up not replying to question 4 as he had mentioned most of the points I had listed (with the exception of the critical voices point). Let me know if I can help in any other way! Warm regards, Giovanna

    --

    Q3) What successful strategies have institutions adopted to address the challenges and gaps, and transform their culture to align with emergent understandings of research excellence? What progress has Canada already made that can be built upon?

    • Thinking we’re implementing quite a broad systemic change, we need to adopt the same values and principles we’re using to redefine excellence to our own strategies: we need to co-create the change with the community, be open and transparent about, be ethical, prevent unintended consequences so we’re not perpetuating or creating new negative externalities.
    • For me, this means we need to:
      • Involve the community in decision-making and in developing fit for purpose approaches
      • Try as many multiple steps as possible
      • Take an incremental approach – an evolution, not a revolution
      • Share what you’re doing with colleagues – internally and externally
    • In practical terms, this translates into many steps that have been taken:
      • Establishing coordinating networks and communities of practice – be a national programme like Recognition & Rewards, be a CoARA national chapter (https://coara.eu/coalition/national-chapters/)
      • Establishing internal supports – from hiring colleagues to implement these programs, to capacity building with academic staff
      • Developing tools and resources – the Researcher Impact Framework is such a tool, developed for the research community to adapt to narrative-based approaches to assessment

    Q4) How can Canadian institutions cooperate in concrete ways to move forward together towards redefining or broadening the concept of research excellence? What is the role of funding agencies in this process? What could be the role of international organizations?

    • The first easy step towards cooperation is establishing a community of practice in which you can share knowledge, experiments, drafts; it is crucial to have some sort of cooperation and coordination between the approaches. You can also create working groups by themes, like those working in promotions and others in grant proposals.
    • You can also search and establish international cooperation – and holding a panel like this one is part of it. We all know science has both a local and an international aspect to it, and we know how researchers are mobile. At the same time, different communities and universities have different profiles, so expect some variation. Being inspired by (inter)national movements and adapting is key.
    • When holding shared initiatives, invest in the coalition of the willing with prizes and awards held at national or regional level, but keep the critical voices in the room fostering dialogue and awareness.
    • The role of the funding agencies is crucial. Funders should understand the major trickle-down effects they have across the system – while it’s important funders experiment and lead trial initiatives, it may also not be fair with researchers if changes are implemented too fast and at scale without the other players in the system not being able to catch-up.
    • Funders should also be aware that changing the criteria without changing how assessors make decisions will not necessarily lead to successful results.
    • Funders also can and should take a coordinated approach – see for example the UKRI funders group (https://www.ukri.org/what-we-do/supporting-healthy-research-and-innovation-culture/research-and-innovation-culture/joint-funders-group/).
    • Funders can also establish supports and training for professional and academic staff is definitive for successful adoption of these changes. See for example NWO course on impact (https://impact.nwo.nl/en/working-with-an-impact-plan).

    Giovanna Lima

  • Thanks so much Giovanna for sharing your notes - very valuable and I can clearly see it must have been a very informative session indeed :)

    Marta Teperek

Do you want to provide a comment? Please sign up first