PhD Defense Julia Kasch – Scaling the Unscalable? Interaction and Support in Open Online Education
On Friday, October 9th I defended my PhD dissertation – corona style. At that time this meant that my promotor, supervisor and 20 guests were allowed with me in the room. The committee was present online which worked pretty well.
This post provides a quick overview of the dissertation. The complete dissertation will be shared online as soon as possible.
In this dissertation we focused on three main themes:
(1) scalable formative feedback and interaction
(2) scalable peer-feedback and peer-feedback training
(3) peer-feedback dispositions.
With ‘educational scalability’ as the guiding theme, we investigated through various studies how in courses with high student numbers, meaningful interaction and formative feedback can be provided in a sustainable way. The need and relevance of educational scalability becomes evident in higher education where the disproportional student-teacher ratio and increasing student numbers challenge the provision of elaborate formative feedback and interactive teaching methods. In the current Covid19 pandemic crisis, this problem became even visible as a general challenge due to restrictions for higher education institutions and schools. The research context of this thesis, namely (open) online education, where teachers are challenged by high student numbers and lack of controlled and synchronous communication and interaction, resembles a similar set of challenges. By investigating scalable feedback and interaction methods, we aim to support teachers and designers in their teaching practices, build upon the existing feedback literature and provide new perspectives and instruments on how to analyse scalable designs.
(1) Scalable formative feedback and interaction
The underlying theory and focus of this dissertation regarding ‘educational scalability’ is laid out in chapter 2 where we define educational scalability as the capacity of an educational format to maintain high quality despite increasing or large numbers of learners at a stable level of costs. This theory is based on common theoretical concepts from distance education and learning design and builds on the Iron Triangle. The Iron Triangle states that there is always a trade-off between the three dimensions (scale, costs and quality). Due to the possibilities of online education such as providing education at large scale, some state that online education is able to stretch the Iron Triangle by enabling us to provide high quality education on a large scale with low costs.
To investigate this claim we developed a theoretical framework on ‘educational scalability’ where we define scale, costs and quality in the context of open online education. We developed a course design analysis instrument, which translates theory to practice and enables teachers and course designers to analyse the scalability of their (open) online course regarding formative feedback and interaction.
In chapter 3 we used the course design analysis instrument to analyse 50 MOOCs. The main elements we focused on were complexity and clarity of the learning goals, type and complexity of learning activities, interaction types that were provided during learning activities (student-teacher, student-student, student-content) and lastly how and when formative feedback was provided to the students. We were able to find various best practices of scalable formative feedback and interaction. Peer-feedback activities were provided where students received clear prior instructions on the peer-feedback process and tools (e.g. rubrics). Instructions were provided on how to upload feedback by using the feedback tool. Additionally, students were informed about the expectations they have to meet such as time investment, being constructive and why providing peer-feedback has educational value to them.
Personalized feedback was provided at large scale through (live) hangout sessions where the teacher discussed the most asked questions, mistakes and/or misconceptions. In some MOOCs, students could contribute through a chat to live interviews with guest speakers and ask questions.
Elaborate automated feedback was provided during quizzes and simulations. Students’ learning process was supported by automated comments that informed them why their answers were correct/incorrect and how to improve. Simulations were provided in which students could apply and practice their knowledge and skills in a controlled environment of diverse complexity levels. Automated process and content hints supported students.
Due to the lack of scalable course design guidelines, a list and examples of best practices can serve teachers and course designers. An elaboration and continuation of a list with scalable best practices would be desirable. Especially, when created by and shared between different high education institutions. The results in chapter 3 strengthen the idea that the Iron Triangle can be stretched in MOOCs. To achieve educational scalability (high quality with low costs on high scale) it is not enough to offer scalable formative feedback and interaction. The educational quality of the feedback and interaction are determining the educational scalability of a course.
(2) Scalable peer-feedback and peer-feedback training
In MOOCs, peer-feedback activities are a common way of students learning from and with each other. The MOOC analysis in chapter 3 showed a consistent lack of clear instructions and prior training for peer-feedback. Only by informing and preparing students properly, students can be expected to engage in the activity and provide high quality feedback. Chapter 4 presents a study in which we developed an online peer-feedback training for a MOOC on Marine Litter. The goal of this study was to prepare students for a peer-feedback activity via an instruction video, exercises and examples. Students were informed about the relevance of peer-feedback itself and were made aware of the benefits they would gain from it.
Effects of the peer-feedback training were studied by focusing on student perceptions. Via online questionnaires we were able to investigate changes in student perceptions regarding their ‘Willingness’ (intention); ‘Usefulness’ (subjective norm), ‘Preparedness’ (perceived behavioral control) and general ‘Attitude’. The results of this study showed that initially, students had a positive attitude towards peer-feedback. After having participated in the training, an increase in all four perception variables (willingness, usefulness, preparedness and general attitude) was found. These positive results lead to a follow-up study which is described in chapter 5. In an experimental study we provided MOOC students a peer-feedback training prior to a peer-feedback activity. The same but slightly improved training as in chapter 4 was used. Again, we investigated the effects of peer-feedback training on students’ willingness, perceived usefulness, perceived preparedness and general attitude. An improved version of the previous questionnaires was used. Additionally, we studied students’ prior peer-feedback experiences and how these relate to their current peer-feedback perceptions.
Taking into account students’ prior peer-feedback experiences is relevant since it is expected to have an impact on how open students are to participate in a training and peer-feedback activity. The results of the questionnaires support this. Students’ prior peer-feedback experience results in significant differences in students’ perception (willingness, usefulness, preparedness, general attitude). Students without prior peer-feedback experience scored higher on willingness, usefulness, preparedness and general attitude compared to students with some prior experience. Students who reported to have much experience, showed the highest perception scores. Due to the low completion rates of the post questionnaire we were not able to study the effect of training on student perception.
(3) Peer-feedback dispositions
In the last two chapters of this dissertation (chapter 6 & 7) we studied peer-feedback in a broader context, i.e. independent whether it takes place online or face-to-face. We zoomed in on students’ openness to provide and receive peer-feedback (i.e. their peer-feedback orientation). The quality and success of a peer-feedback activity highly depends on students’ engagement. Students are asked to provide high quality feedback, receive feedback from fellow peers and ideally reflect on it and use it. This requires content knowledge and feedback skills but moreover openness for the process as a whole. In the literature, limited attention has been given to the personal needs and beliefs of students with regard to peer-feedback.
In chapter 6 we introduce a study in which we studied the different dimensions that influence students’ feedback orientation towards peer-feedback. The Feedback Orientation Scale (FOS) by Linderbaum and Levy (2010) was used as the basis for this study. Semi-structured interviews were conducted with students, teachers and researchers and thematic analysis was used to analyse the interview data.
Participants indicated a broad range of student elements that can influence students’ peer-feedback orientation. The most prominent elements were about the usefulness of receiving and providing peer-feedback, the social bond between students, fairness and skills. These findings were used in the follow-up study, described in chapter 7, to develop a peer-feedback orientation scale (PFOS). Data on the identified elements was collected via an online survey among Dutch higher education students. The data was analysed through an exploratory factor analysis (EFA) which revealed a five-factor solution including the dimensions: accountability, communicativeness, utility, self-efficacy and receptivity. These five factors are seen as the underlying and contributing factors of students’ openness to provide and receive peer-feedback. Through the Peer-Feedback Orientation Scale (PFOS) we provided an initial quantitative instrument that can be used by teachers and researchers in order to understand students’ dispositions toward receiving and providing peer-feedback.
This work is licensed under a Creative Commons Attribution 4.0 International License.