PRESTO: Peer assessment in relay format

The learning relay is based on peer assessment. This goes one step further than peer feedback: students rate each other's work with 1 to 5 stars. And that assessment counts. Therein lies the complexity of the system. Because how do you make sure that students don't simply give each other 5 stars? Then everyone is happy, but no one has learned anything.

Earn back the stars

Peer review has many advantages, but also disadvantages," says Pieter Bots. "Feedback is a service to another person. Students generally don't want to spend too much time on it, and besides, most also want to be liked. This encourages uncritical high grading. In this system, a quick good rating works against you: the stars you give out you lose yourself. You then have to 'earn them back' by rating your work."

Relay idea

But this is precisely how it pays to give a bad rating, isn't it? Bots: "Behold the moral dilemma. Do you choose for yourself or for someone else? The trick is the relay idea. You build on the work of an anonymous predecessor, which you first improve to the best of your ability. Then you perform your own follow-up task and only then do you give point-by-point feedback and an assessment. After all, you only really see the shortcomings of someone else's work when you have to work on it yourself."

Waterproof

Bots developed the relay method more than seven years ago as a compulsory practical in the System Modelling course in the first year of the bachelor's degree in Technical Management. Initially to save himself a lot of revision work, but gradually he used this method to tackle some major bottlenecks of large-scale (digital) education. Including little or poor feedback and low student motivation to practise certain skills.

Scalable form of education

Under the name PRESTO - Project-Estafettes for Scalable Applicable Open and Online Education - the project received support from the Open and Online Education Incentive Scheme 2017. Within that framework, the learning relay was generalised in 2017 and 2018 into a scalable teaching format, which is basically suitable for all courses with step-by-step assignments. It was reprogrammed, made responsive, standardised and made available as open source software via GitHub.

Rules of the game

"The incentive scheme has given us the leeway to optimise this form of learning and make it accessible to a much broader target group. Above all, it allowed us to improve and document the software. All in all, the development of the system has cost me many times the hours I would otherwise have spent on revision. In principle, also to make it watertight on all sides. That turned out to require a considerable set of ground rules after all."

Objection and appeal

One such set of ground rules is that students who feel disadvantaged by their successor can object to their assessment. The validity of such an objection must then be checked by an independent party - a senior student. Students who disagree with the decision of such a referee can appeal it, after which the matter is settled by the lecturer. To prevent objections and appeals from raining down, a threshold has also been built in for this purpose: those who do not come up with valid reasons are given penalty points.

Challenge

Bots: "I have tried in every possible way to make the system as fair as possible, in many ways it even follows the structure of our judiciary. But what some students still find unfair is that their own result depends partly on the work of others. If that other person performs badly, they have to work harder. But other students actually see that as the challenge: the game element encourages them to work more thoroughly than they would otherwise have done, which ultimately also helps them master the material."

Academic thinking

"It is a somewhat controversial method, but I have great confidence in this form of peer feedback for large-scale teaching. Also because students are introduced early on to an academic way of thinking and working. In science, too, you rarely start a research project from scratch, but continue with the results of others. In principle, the learning relay can be used worldwide. It is a young but mature form of learning, ready to stand on its own feet."