Cheat-Proof Auto-Graded Exams Are Real: Here is How to Design Yours
With the pandemic still raging across the world, organizing exams is challenging. Some universities have resorted to outdoor exams in school yards [1] or even stadiums [2,3]. That’s when weather or air quality conditions allow [4]. Interestingly, taking exams outdoor have also been considered in the past as they make cheating harder [5].
Nearly one year into this pandemic, many countries are still implementing partial or complete closures of universities. Given this context, online exams seem to be on the right track towards preventing large gatherings of students. However, if videoconferencing platforms have been the go-to for online lectures, online exams are far from being widely accepted given the alleged risk of cheating they pose [6,7,8].
Most learning management systems (LMS) come with hidden, yet powerful features which allow for the design of cheat-proof synchronous online exams. Combined to auto-graded questions and online proctoring, you will treat your students with the most definitive exam you have ever designed.
In this article, I detail all the steps it takes to design a time-limited, cheat-proof, auto-graded, synchronous online exam on Moodle. Similar steps can be taken on most other Learning Management Systems such as Sakai.
Randomness is The Key
Here is a concrete example of what I mean by cheat-proof. Assume your exam consists for 3 questions. The trick is to create multiple instances of each question by changing the values they use. Changing the values will guarantee the same level of difficulty for all students.
If you create 6 instances for each question, your LMS will generate 6³ random instances of your exam. One instance will consist of 3 questions, each randomly chosen out of the 6 instances. That’s 216 different random instances of your exam!
That means that if two student team up to solve the exam, the likelihood that the LMS will present them with the same instance of the exam goes as follow:
- 3 identical questions out of 3: .0046
- 2 identical questions out of 3: .0278
- 1 identical question out of 3: .1667
If you are brave enough and can come up with 7 instances for all 3 questions that will give you 2,401 randomly-generated exams. Instead of creating more instances for each question, you can also decide to add a new question and create 6 instances for this question. You will get 1,296 instances of your exam.
Here is a simple formula to help you decide whether you should add a new question or create more instances of each question: v^q > N, where v is the number of question instances, q the number of questions, and N the number of students. Below I have plotted the number of random exam as a function of the number of question instances for varying number of questions.
Given the small likelihood of students being presented with similar exams, randomly-generated exams are right up the alley of instructors who consider organizing exams online. For most instructors, it will appear worth spending the time needed to design multiple instances for each question given the advantages against cheating.
Coupled to the right type of questions, instructors will also benefit of LMSes’ auto-grading capabilities. Such questions include Fill the blank questions (also called Embedded-answers or Cloze questions in Moodle).
For the purpose of illustration, I will consider Moodle. Similar steps can be taken on other LMSes such as Sakai. I will show how to design an examen consisting of embedded-answers questions as I believe they provide greater flexibility.
Cheat-proof Quiz
Instead of creating first a quiz and then adding questions to the quiz, you will first add a new Category to the Question bank, one for each question of your exam. Once the question categories added, you will then create the instances of each question by adding a new question to the Question bank under the corresponding category.
In summary, here are the main steps you will take:
- Create one question category for each question of your exam.
- Create multiple instances of each question within the corresponding category.
- Create the exam (i.e., a quiz).
- Add each question category created in the first step.
Et voilà! Let’s now detail each of these steps.
Question Categories
For Moodle to insert a randomly-chosen instance of a question, you need to provide Moodle with a specific Question category, one per question. A Question category will contain all instances of each question.
To create a question category:
- Click on the “Administration” icon to show the Administration navigation block of your course. Click “More…” at the end of the Administration block. You will be presented with the “Course administration” page of your course. (Picture 1) (The Administration icon is represented by a grey blue gear located in the top right corner of the main page of your course.)
- In the 3rd section entitled “Question bank”, click “Categories”. (Picture 2)
- From there, you will create a new category for each question of your exam. Use a self-explanatory name such as “Final Exam – Q1”. Click “Add category”. (Picture 3)
Repeat the last step for all questions of your exam.
Once all question categories added, we will now add the question instances to those categories.
Question Instances
To create instances of a question :
- In the Question bank, you can create a new question under the categories you have created in previous step. Click “Create a new question”. (Picture 4)
- You will be prompted with a popup window where you can select the type of question you would like to create. (Picture 5)
- Add a self-explanatory name such as “Q1 – V1” for instance 1 of question 1. (Picture 6)
- For the purpose of illustration, I will present the “Embedded-answers (Cloze)” questions in the next section.
Repeat these steps to create as many instances as needed per question.
Exam Quiz
To create the exam:
- In the Administration block, select “Turn editing on” for your course page. (Picture 1)
- “Add an activity or resource” in the Topic/Section where the link to the exam should appear.
- You will be presented with a list of activities or resources. Select “Quiz”. (Picture 7)
- Fill the Name of the quiz (i.e., Final Exam). Set the options of the quiz (Timing, Grade, Review options, …). (Picture 8)
- Click “Edit quiz”. (Picture 9)
- Click “Add” in the left bottom of the page and select “a random question”. (Picture 10)
- Select the category created in Step 1. (Picture 11)
Repeat last two steps for all the question categories of your exam.
From there, you can set the maximum grade of the exam, the points of each question, and ask Moodle to shuttle the questions if needed. (Picture 12)
Auto-graded Quiz
Moodle offers a wide variety of auto-graded question types including true/false, multi-choice, and short answer questions. Below, I present the Embedded-answers question type as it provides a higher level of flexibility.
Embedded-Answers (Cloze) Questions
There are four main types of embedded-answers question: short answer, numerical, multi-choice, and multi-response.
When students encounter a Short Answer embedded-answers question, they are presented with an empty box where they can fill the text of their answer. Their answer will be matched against the answers you provided when you created the question. You can also update the answers you provided once the quiz closed and ask Moodle to regrade all student attempts.
To create an embedded-answer question, Moodle will present you with a text editor where you will add the text of your question (Picture 6). The top of the editor is a toolbar with a list of icons to create headings, bold or italic text, color and highlight text, create numbered and bulleted lists, or adding media or links to a page.
Below I show on the left, the text of an embedded-answers question and on the right, the resulting output the students will see. To see the preview, click “Save changes and continue editing”: A clickable magnifying glass icon will appear followed by “Preview”. This question consists of two subquestions each followed by the expected embedded-answer. You can add as many embedded-answers as subquestions in the main question you are creating.
Contrary to the question itself which can be specified in a WYSIWYG way, the embedded-answer follows a WYSIWYM (what you see is what you mean) approach. You need to follow a strict syntax for Moodle to understand how to grade the question.
The syntax of an embedded-answer is the following:
{1:SA:=Correct Answer}
An embedded-answer is delimited by an opening and a closing curly bracket ‘{’ … ‘}’. Inside the brackets there are three parts:
- The first part is a strictly positive integer (1, 2, 3, etc.) that indicates the weight of the answer. This weight is used in the calculation of the question points. In case of a question with multiple subquestions, the total points for the question will be computed as the weighted average of all subquestion points using the weight assigned to each answer.
- The second part is delimited by semicolons (‘:’) and indicates the type of the embedded-answer (SA, MC, NM, and MR).
- The third and last part gives the answer(s) against which students’ answers will be matched and the feedback students can receive depending on their answers.
The table below lists the different types of answers and feedback with the corresponding question text.
Below I explain the syntax of the Moodle code for each type of answers and feedback listed in the previous table.
One correct answer:
{1:SA:=correct answer}
where ‘=’ marks the correct answer.
Multiple correct answers:
{1:SA:%100%correct answer 1~%100%correct answer 2}
where ‘~’ is the separator between each answer and %100% the percentage of the subquestion points for the corresponding answer. You can use negative points in case of wrong answers.
Correct and partially correct answers:
{1:SA:%100%answer1#Correct~%50%answer2#Partially correct}
where ‘#’ marks the beginning of a specific feedback for the preceding answer. Specific feedback are shown to students who take quizzes setup with multiple tries. A penalty can be setup before the start of each next try.
Partially correct and wrong answers with feedback:
{1:SA:%100%answer1#Correct~%50%answer2#Partially correct~%-200%*#Wrong answer}
where ‘*’ indicates the feedback students will receive in case of any answer other than the preceding listed answers.
Type of embedded-answers questions
The table below lists all types of embedded answers:
Below, I give the Moodle code for each type of embedded answers and the preview of the question:
Short Answer (SA) embedded answers
The length of the text box is computed based on the length of the longest answer including the wrong answers. In case of multiple SA subquestions, the trick to fix the length of all the text boxes is to add the same dummy wrong answer to all embedded answers. In the example below I also show the use of tables for a better control over question and answer boxes alignment.
Final Words
Student evaluation through synchronous, time-limited, proctored exams is the core of education systems. And in the times of pandemic, that is a problem. Media have been going to town on stories of mass cheating based on students testimony priding themselves on the creative and not-so-creative tactics they have used to break the rules when taking online exams.
Organizing online exams require extra preparation and for any new proctoring technology, some students will work hard to find ways to get around the rules. Whether successful or not, these attempts should not discourage the adoption of online technologies in education as they provide pathways for flexible and affordable higher education for learners who need it the most.
[1] http://hanoitimes.vn/students-take-exam-in-playground-in-hcm-city-13552.html
[4] https://www.nytimes.com/2016/12/21/world/asia/china-smog-exam-pupils.html
[5] https://www.dailymail.co.uk/news/article-2838213/Thousands-Chinese-students-sit-exam-outside.html
[8] https://www.nytimes.com/2020/12/23/nyregion/west-point-cheating-scandal.html
