On 13 March 2024, the FNR hosted the first lecture in the forum “Trends Under Review”. Featuring Noémie Aubert Bonn speaking on the topic of why we urgently need to reform research assessment, a recording of the first lecture is now available, along with key take aways from the panel discussion that followed the lecture. The next lecture takes place on 22 May 2024.
Learn about the upcoming lecture on 22 MayThe “Trends Under Review” discussion forum aims to foster academic discussions in an open forum around current topics in research policies. The event allows external experts to engage with people in the Luxembourgish research community around important topics that affect everyone working in research.
This session focused on the topic of research assessment, with Noémie Aubert Bonn (Hasselt University, University of Manchester) giving a keynote address. The panel was composed of Jennifer Dusdal (University of Luxembourg), Anja Leist (University of Luxembourg), Laurent Vallar (Luxembourg Institute of Health), and Jinyuan Wang (University of Luxembourg). The following is a summary of the panel discussion, highlighting the main points and some quotes from the panellists.
Panel discussion summary
Need for different roles in research and dividing tasks
- Broadening what is valued is good, but we shouldn’t expect everyone to do everything.
- Distribution of tasks and sharing tasks is important. Everyone should be able to do what they do best.
- The main issue is how to acknowledge these roles. Only counting publications and patents isn’t good enough.
- “If we could divide tasks, we would make a huge impact.”
- “Leadership is an important aspect of academic careers and grant funding, but not everyone needs to be a leader.”
- Scientific goals are important, but different jobs and levels require different skills.
- Team assessment, team awards, and recognition of teams will be crucial moving forward.
There is an increasing pressure on researchers, leading to misconduct
- “If we recognize researchers for what they do, there will be less need and drive to lie.”
- There is a huge demand for assessment because of the increasing amount of scientific manuscripts, proposals, and precarious short-term contracts.
- The concept of “Slow Review” can help create better reviews but the funders need to take important steps to valorise the time needed for this.
- Researchers are expected to do more and better research, get funding, supervise teams, publish, and do peer review. However, there is not enough time to do all of this and not all of it is rewarded.
- We should find ways to reduce the imposed pressure of evaluation – e.g. focusing on recommendations, reflections, and suggestions over rankings and absolute measures.
Evaluators are key in assessment, but require training and recognition
- “If journals have open reviews, you receive more useful comments and feedback for your research.”
- Peer review training exists in Luxembourg, but it is less valued as an important skill because it is not rewarded in the current system.
- There is wide variation in how research groups train their teams in peer review. Guidelines from institutions/funders are important, but not sufficient.
- “I have the feeling that most people assess honestly because they feel karma is going to bite them.”
Research assessment is different between disciplines
- There are differences between disciplines in terms of receptiveness to reforms
- One set of indicators doesn’t fit all domains, and leads to differences in funding and the perception of “success.”
- We need to find a broader set of qualitative and quantitative indicators to embrace the broad range of how research is done/valorized in different domains.
- It is important to encourage the collaborations between and among disciplines. Collaborations between and among disciplines can help reduce the negative impact from assessment differences. And collaboration itself can serve as one indicator for assessment.
What’s going on in Luxembourg?
- The University of Luxembourg is a CoARA member since 2022, and has published their CoARA action plan in January 2024.
- The Ministry is heading the evaluations of research institutions and defines the research assessment on an institutional level. It is unclear what the impact on the landscape will be: “It is important to inform the researchers in the system on what should be achieved with the exercise.”
- “Are we collecting data just to collect data?” – Links between the mission of the research and research activities, such as data collection, should be there and made explicit
- Current Key Performance Indicators don’t reflect the actual activity and goals of the research institutions. “An institution is evaluated on the number of samples stored. We can store millions of samples, but what is the impact if they aren’t being used?”
Where do we go next?
- We should continue to be critical about research assessment and evaluation – do we really measure what we want to with the indicators we have?
- Leadership should listen to all levels of researchers to make policies – “Are they satisfied with the current evaluation system? What do they need to succeed?”
- The Luxembourg university and research institutions are mature environments for pushing the frontiers of knowledge, but we can do better in societal impact – “We need to incentivize and systematically aim for generating impact instead of accidentally creating impact.”
- There needs to be a push from the top and a push from the bottom, but we need more awareness raising in the middle, where people have been in the old system for a long time.
- “Researchers need to learn as early as possible how the system works and where the problems are. They should then be rewarded for getting involved and contributing to improving it.”