Quality assurance of open educational resources

Are you working on open educational resources? Make sure that the learning materials are good quality and that they meet the right criteria. Creating a quality model can help in assessing learning materials. This roadmap will guide you through the different steps to create, implement and maintain a quality model for open educational resources.

Hard werkende studenten in bibliotheek

Step 3. Develop an initial version of the quality model

Now you can get to work with the team on developing a quality model. This step includes a number of sub-steps.

Learn in detail about other quality models

Firstly, learn all about existing, generic quality models and the models used in other professional communities. There are several national and international examples of quality models and research on which to base your own model. Read more about the different frameworks and models listed as references below.

Bronnen

1. Kwaliteit van open leermaterialen. — 2013 (PDF)

Generiek framework voor een kwaliteitsmodel

Dit onderzoek is uitgevoerd door Robert Schuwer in opdracht van SURF en vormt de basis van dit stappenplan. Wat is belangrijk als je kwaliteit van open leermateriaal zichtbaar wilt maken? Zorg voor een adequate beschrijving van het leermateriaal en de context waarvoor het leermateriaal geschikt is. Maak de waardering van het materiaal zichtbaar door community’s van gebruikers hun oordeel over open leermateriaal te laten geven. Een andere, indirecte manier om de kwaliteit weer te geven, is de reputatie van de instelling die open leermaterialen publiceert.

Open source

2. A framework for assessing fitness for purpose in open educational resources. — 2016

Een framework om docenten te helpen bij het beoordelen van open leermaterialen

In ‘A framework for assessing fitness for purpose in open educational resources’ presenteren Insung Jung, Teruyoshi Sasaki en Colin Latchem een framework dat is gebaseerd op de kwaliteitsaanpak van MERLOT en een survey onder ruim 200 experts over criteria. Het framework is vooral bedoeld om docenten te helpen bij het beoordelen van open leermaterialen. De onderzoekers besteden veel aandacht aan de didactische aspecten van leermaterialen.

Open source

3. The TIPS Framework Version 2.0 — 2014

Een generiek framework met 4 categorieën en 38 criteria

The TIPS Framework Version 2.0 presenteert een framework gebaseerd op literatuurstudie en feedback van experts. Het onderscheidt vier categorieën met totaal 38 criteria, die de naam voor dit framework, TIPS, verklaren:

(T) Teaching and Learning Processes
(I) Information and Material Content
(P) Presentation, Product and Format
(S) System Technical and Technology

Open source

4. Annotationskonzept für Bildungsressourcen. — 2018

Beoordelingsinstrument op basis van framework

Ook de Duitstalige publicatie Annotationskonzept für Bildungsressourcen biedt een framework gebaseerd op een literatuurstudie. Zij komen tot vier categorieën van criteria:

  • Pädagogik
  • Inhalt
  • Technik
  • Lizenz

Dit framework vormt de basis voor een beoordelingsinstrument om open leermaterialen per criterium te kunnen beoordelen. Er geldt een schaal van 0 (Das Kriterium ist für die Ressource nicht zutreffend oder kann nicht bewertet werden) tot 5 (Die bewertete Ressource erfüllt das Kriterium vollumfänglich). Tijdens het schrijven van dit stappenplan, in oktober 2018, was het instrument nog in ontwikkeling.

Open source

Quality models from professional communities

There are two professional communities in the Netherlands who have jointly developed a quality model they now use to assess learning materials. The community of nursing higher education (HBO-V) have been working together since 2017 on providing a high-quality collection of open educational resources. The second community is of ICT education at tertiary level, HBO-I.

You can find more information about these communities and their quality models under "References" below.

Bronnen

1. Liever kort en praktisch dan een te uitgebreid kwaliteitsmodel — Interview Marloes van den Broek, Fontys Hogescholen 2018

Marloes van den Broek over het kwaliteitsmodel HBO-V

Binnen de opleidingen hbo-verpleegkunde loopt een initiatief om leermaterialen voor de opleiding open beschikbaar te maken. Het ontwikkelen van een kwaliteitsmodel voor het leermateriaal maakt daarvan onderdeel uit. Het initiatief is gestart met vijf instellingen binnen een boegbeeldregeling van het Ministerie van OCW. Vanuit de Stimuleringsregeling Open en Online onderwijs krijgen ze de mogelijkheid uit te breiden naar alle hbo verpleegkunde-opleidingen in Nederland. In dit interview verteld Marloes van den Broek, docent en projectlid van het project 'samen hbo-v' hoe ze gekomen zijn tot het huidige kwaliteitsmodel.

Open source

2. Kwaliteitsmodel HBO-V — 2017

Het kwaliteitsmodel van HBO-V bestaat uit 9 must have criteria en 15 nice to have criteria. Na een periode dat het kwaliteitsmodel door de community is gebruikt is de community begin 2019 bezig met een herziene versie.

Open source

3. Bouwstenen voor ICT-onderzoeksonderwijs delen — Interview Danny Plass, Saxion

Een aantal hogescholen werkt samen aan het ontwikkelen van nieuw leermateriaal voor onderzoeksonderwijs in het ICT. Een kerngroep van vier HBO's ontwikkelde een kwaliteitsmodel en een template voor het maken van zogenaamde 'bouwstenen'. In dit interview verteld Danny Plass, docent en projectleider van het initiatief hoe ze gekomen zijn tot het huidige kwaliteitsmodel.

Open source

4. Kwaliteitsmodel HBO-I — 2018

Het kwaliteitsmodel van HBO-I is gebaseerd op het model van HBO-V. Het beschrijft:

  1. Vakinhoud
  2. Didactiek
  3. Presentatie en taal

Tevens maken ze onderscheid in must have en nice to have.

Open source

Embedding the quality model into the quality system

A quality model is just one element of a whole system of quality assurance. How can you best tackle the quality assurance for open educational resources systematically? Three examples based on a quality model:

MERLOT

MERLOT, an initiative of California State University, is not just a platform for sharing open educational resources, but also 'a community of colleagues, volunteers and MERLOT members who collaborate in various ways in order to provide the user of open educational resources with a series of services and functionality to improve their educational experience.'

MERLOT's quality assurance consists of a number of different elements:

  • Any institution that wants to add learning materials has to be a MERLOT member.
  • Learning materials are peer reviewed against a set of criteria. The results of the peer review are shared openly as reports, linked to the learning materials. The peer review is carried out retrospectively, once the learning materials are already accessible via the site.

Wikiwijs

Wikiwijs is a Dutch platform for creating, sharing and searching for open educational resources. Wikiwijs quality assurance includes the following aspects:

  • You must have an account before you can upload learning materials.
  • A limited quality model (framework) has been developed, which is applied to different disciplines. In addition, there are also quality models for specific learning materials such as learning materials for pupils with profound, multiple disabilities. These models are managed by an organisation specialising in the relevant field.
  • Wikiwijs has learning materials specialists for each discipline and for each specialisation. They check the available learning materials retrospectively, based on the quality model for the discipline or specialisation.
  • If learning materials comply with the relevant quality model, this is shown by a quality mark attached to the learning material.

Klascement

Klascement is a learning materials network in Flanders. Klascement's quality assurance includes the following aspects:

  • You must have an account before you can upload learning materials.
  • Before learning materials go online, an editorial committee checks the learning material by applying a generic quality model.
  • If learning materials do not meet the quality criteria, then the submitter of the learning materials is informed of this. If the learning materials do comply with the quality model, then they are published to the platform.

2. Create an inventory of criteria for the learning material

The project team draws up a list of quality criteria. This can be done in a variety of ways. Some suggestions:

  • Check whether there are national or international agreements in place about the contents of a curriculum or about learning materials within this discipline (an educational profile). For example, you can ask professional associations about this. These types of agreements are a good source of criteria for the discipline.
  • In many quality models the criteria are subdivided into different categories, such as subject matter, teaching methods and form/presentation. The latter includes for instance accessibility criteria. For some categories of criteria there are standards or guidelines. Accessibility guidelines for content can be found in the Web Content Accessibility guidelines from the W3C Consortium and on the website digitoegankelijk.nl published by the Dutch government. It is a good idea to contact some specialists in the area of these types of guidelines and ask them to be a member of your project team.
  • A very common criterion for form/presentation is that the learning materials must include metadata. Standards also exist for metadata, which were created by Edustandaard.
  • Are you looking for a good basis for teaching method criteria? Study what features in the learning materials would match the didactic vision of the group for whom the model is being created.
  • Formulate a criterion that is as SMART (Specific, Measurable, Attainable, Realistic and Timely) as possible. For example: do not say "The text must not be too small", but instead: "The text must have a minimum 10pt font size."
  • Make each criterion simple. An example of a complex criterion might be: "The learning material must be viewable on a regular PC, Mac or tablet and the text must have a minimum 10pt font size". Turn this into two simple criteria, namely: "The learning material must be viewable on a regular PC, Mac or tablet" and "The text must have a minimum 10pt font size".
     
You can set a lot of great-sounding criteria, but such a huge list will most likely put lecturers off. Consequently, we had to take another critical look at our model and ask ourselves: what really needs to be there?
Marloes van den Broek

3. Define the weighting of each criterion

The importance of each of the criteria depends on the context. Normally, the learning material will not have to meet absolutely all the criteria to be of adequate quality. You need to discuss this with all members of the project group. Split the criteria into different categories, depending on their relative importance. State whether a criterion is a requirement ("must have") or an option ("nice to have").

  • A criterion is a requirement if the learning material has to comply with it to be labelled "good quality".
  • A criterion is an option if the learning material is not required to comply with this criterion, but it would definitely be improved if that were the case.

4. Create a written procedure for each criterion

Formulating the criteria themselves is only part of the job; they also need to be measurable. Try for each criterion to spell out how the criterion can be checked. Initially, it is enough to write out a procedure for those criteria that are regarded as requirements.

For example: a criterion might be "The text does not include any spelling mistakes". You can choose to have the full text proofread for this. In practice, that would probably take too long. A possible procedure would then be: "Take a random sample of one page. If that page does not include any spelling mistakes, then the criterion is met."

We set fairly basic requirements for resources. For instance, they should not contain any spelling errors. In addition, there are several subject-specific and didactic criteria against which we assess resources. For example, the resource must stimulate the student’s prior knowledge, and the learning goal for which it is suitable must be indicated.
Danny Plass