October last year I shortly introduced an instrument we developed which enables different stakeholders such as teachers, students, course designers and researchers to analyse the educational scalability of their course design when it comes to formative feedback. (You can read here more about it)
I am glad to inform you that both the instrument and the corresponding paper are accessible online now!
You can find the paper “A Framework towards Educational Scalability of Open Online Courses” here
You can find the design analysis instrument here
If you have any questions regarding the paper and/or the instrument feel free to contact me here: julia.kasch(at)ou.nl
Don’t know what I am talking about? No problem, let me explain:
What did we do?
We developed an instrument which is a form you can download, print and then use to analyse the instructional design of an online course. In our paper we did several pilot studies with the instrument to check whether the instrument was clear for various stakeholders and also tested it ourselves. The instrument helps us to get insight into the instructional design choices that are made in open online learning. Since there are no guidelines on how to design an online course it is interesting to see how various online courses are designed. We were especially interested in the design regarding formative feedback since this often is the most difficult aspect to offer students in courses with large student numbers. What are common and best practices when it comes to formative feedback design in open online courses? Are there certain patterns? De online courses provide formative feedback and if so how and to what extent? For example: How did they provide their students with feedback? Who provided the students with feedback? What kind of feedback did they receive and how elaborated was the feedback?
Why did we do this?
We were not satisfied with existing instruments in the literature. Those instruments we found were developed as a kind of checklist with which you could see whether a course contains certain design elements or not. However, when using these checklists you still don’t really know much about the design. We were not only interested in the ‘which?’ about also the ‘how?’. Which design elements were used and how were those implemented?
How did we analyse the courses?
We semi-randomly selected open online courses from five different domains which where: Teaching & Education, Programming, Health & Medicine, Business & Management and Humanities. To be eligible for our study a course had to be given in English, contain some form of formative assessment and feedback and be of course available during the runtime of our study. Two raters independently analysed the same courses and the interrater reliability (Cohen’s Kappa) was calculated.
Preliminary results
Overall the educational design in the courses was very similar regarding the learning activities, interaction possibilities and feedback methods. All courses provided videos, reading materials and quizzes. In two out of the five courses, we found (potential) scalable support and feedback methods that were able to provide large numbers of students with support and feedback, without increasing or depending on a high teacher time involvement. For more detailed information have a look in our paper.
A follow-up study on a bigger scale (including more online courses) will follow this year!
Julia Kasch
This work is licensed under a Creative Commons Attribution 4.0 International License.