User:Jtneill/Teaching/Online Quizzes and Exams with Moodle

This document overviews how I've been using Moodle quizzes for academic assessment and feedback, with consideration of quiz design, settings, advantages, student feedback and ideas for improvement.

 Explore the unit's quiz info:
 * Moodle: Quizzes.
 * Description: Quizzes

Overview
During Semester 1, 2010, I sought to improve student engagement and satisfaction with Survey research and design in psychology, a third-year undergraduate research methods and applied statistics unit, by providing more regular assessment and feedback in form of weekly online quizzes (instead of a paper-based mid-semester exam) and an online final exam (instead of a paper-based final exam).

Online quizzes offer considerably more flexibility than more traditional paper-based mid-semester and final exams. For example:
 * 1) Modularisation: Any number and arrangement of quizzes can be created. Use of multiple short quizzes in place of, say, a mid-semester or final exam facilitates chunking of content, learning activities and assessment.
 * 2) Temporality: Students can participate in quizzes 24/7. It's easier to arrange early or late online quizzes than is the case with paper-based quizzes or exams.

Main quiz settings
The UC implementation of Moodle, LearnOnline was used. The main design features and settings for the quizzes were:
 * 1) Nine 10-item quizzes, covering each of the main topics in the unit, were available for a week at a time (students could sit in their own time and use unit materials)
 * 2) Quiz time limit was (usually) 10 minutes
 * 3) Quizzes were worth 2% each
 * 4) Students could do the quizzes in their own time
 * 5) Quiz items were drawn randomly from categories within custom-developed test-banks which were adapted from past mid-semester and final exams.
 * 6) The order of quiz item answers was randomised.
 * 7) The main question types used were those which could be automatically marked:
 * 8) Multiple-choice (one correct answer) and True/False
 * 9) Multiple-choice (more than one answer allowed) - make sure to include penalties for incorrect answers, otherwise all answers can be ticked to get full marks
 * 10) Matching
 * 11) Numerical (make sure multiple variations of the correct response are specified as acceptable)
 * 12) Cloze questions (fill in the blank using pull-down menus)
 * 13) Students were permitted to see responses and answers immediately after submission, but only general feedback and overall mark was viewable after this window is closed (see Figure below).
 * 14) A final exam was drawn from quiz test-banks, with item selection based significantly on the item discrimination indices derived from the quizzes (available within Moodle).
 * 15) Quiz extensions were only offered to students under exceptional circumstances.
 * 16) Students were offered bonus marks if they could correctly report any quiz errors to the unit convener.



Quiz security is a common concern for teaching staff. Of particular consideration were the following:
 * Security
 * 1) There is a need to balance the pedagogical importance of providing students with detailed quiz feedback so that they can review/improve whilst minimising the potential for students to cheat by sharing answers with other students who haven't yet sat the assessment.
 * 2) Quizzes were worth relatively little (2%) in order to provide relatively little incentive for cheating and relatively greater incentive for learning (because similar items would be used in the final invigilated exam, therefore it was worth learning by doing the quizzes by one's self).
 * 3) The final exam, covering the whole semester, was worth twice as much as all the quizzes combined, and was conducted through the formal university exam system and invigilated in computer labs. The online exam was open for a limited period of time and was password protected. Thus, the final exam was worth more marks and was more tightly restricted and invigilated than the quizzes (which could be done any time during each week).



Student performance

 * 1) Students' overall mean marks for the quizzes and exam were ~65%
 * 2) Distribution of marks (overall discriminability was good, i.e., allowing for appropriate proportions of Pass, Credit, Distinction and High Distinction marks without moderation) - see histograms

Staff tasks
Considerable time is required to initially set up good quality, large, customised testbanks for each of the main topics in a unit. In particular, time is needed to: However, once this up-front investment has been made, quizzes offer a relatively efficient, scalable method of providing ongoing assessment and feedback.
 * 1) Create a practice quiz
 * 2) Develop quiz instructions
 * 3) Develop or converting the quiz items e.g., into GIFT format and importing or creating the items directly in Moodle. However, this was a good opportunity to review and revise questions.
 * 4) Create/convert/upload/insert images
 * 5) Develop quiz feedback
 * 6) Test quizzes
 * 7) Manage each quiz (e.g., student questions and technical problems such as resetting a quiz for students when it doesn't record responses/mark properly)
 * 8) Deal with student questions and feedback
 * 9) Apply extensions (using groups and groupings and adjusting dates or a password)

Staff benefits
Advantages to teaching staff of using online quizzes included:
 * 1) Marking time is significantly reduced, although this is offset initially by the set-up time and the ongoing maintanence.
 * 2) By providing smaller, regular assessment components, students offered the unit convenor more feedback and dialog about individual quiz items, leading to richer student-driven improvement in many items and aspects of the quizzes than was previously the case with paper-based exam formats.
 * 3) It is easier to track individual student progress by using multiple, small-weight quizzes.
 * 4) Automatically-marked online quiz marking is scalable - an increase in students doesn't necessarily add significantly to marking time

Student evaluation and feedback
Unit Satisfaction Survey (USS) results were higher than for the previous year (approx. 85% compared to 75%). The quizzes, I think played an important part in this improvement, particularly in relation to large improvements in feedback-related scores. However, there was mixed open-ended feedback from students about the quizzes, suggesting value but with room for further improvement and refinement.

Positive student feedback responses about online quizzes and exams included:
 * 1) Quizzes encouraged students to keep pace with the unit
 * 2) Students appreciated receiving regular feedback.

Negative responses about online quizzes and exams included:
 * 1) Difficulty and ambiguity of the quizzes
 * 2) Being assessed on all nine quizzes (without being able to drop any marks)
 * 3) Limited opportunity to review quiz items
 * 4) Being assessed on all semester content in the final exam.

Reflection on this student feedback has led to these proposed changes in the use of online quizzes for this unit in 2011:
 * 1) Open quizzes for two weeks at a time (rather than one week)
 * 2) Review and revise quiz questions, considering:
 * 3) Particularly items with the lowest % correct - e.g,. is it too difficult or lacking in clarity?
 * 4) Check how well lecture and tutorial content matches quiz assessment items?
 * 5) Improve automatic feedback provided in response to incorrect answers.
 * 6) Expand the number of items (e.g., create variations) in the quiz test banks to minimise potential for cheating (sharing answers)
 * 7) Provide students with more opportunity to learn and improve - some options are:
 * 8) Provide practice quizzes
 * 9) Allow two attempts at each quiz
 * 10) Allow adaptive mode (where students can improve answers to receive partial marks)
 * 11) During the exam preparation period, allow students to review their quiz responses and correct answers

Conclusion
The initial jump into using online quizzes requires considerable preparation and management time, but yields pedagogical and efficiency benefits.