Tips & Tricks for Writing Effective Fill-In-The-Blank Questions

A comprehensive guide to Moodle Embedded-Answers (Cloze) questions

Moodle as many other learning managing systems allows instructors to design online assessments. In Moodle, online assessments are called quizzes. Quizzes can be used to administer tests, assignments, quizzes, or exams.

A quiz consists of questions you can either create from scratch or select from the question bank. You can also let Moodle randomly select previously created questions from the question bank.

Moodle offers a wide variety of question types. Multiple choice, true-false, and matching questions are the most common type of question as they are thought to be the only questions that can be graded automatically. However, other question types such as Fill in the blank questions can also be graded automatically while offering greater flexibility. In Moodle, Fill in the blank questions are called Embedded-Answers or Cloze questions.

In this article, I present how to write effective Fill in the blank (aka Embedded-Answers) Questions with Moodle. You can also download the cheat sheet for Embedded-Answers Questions at bit.ly/moodle-cloze.

• I first introduce the four (4) main types of Embedded-answers questions.
• I then present the Moodle text editor and the markup language for writing Embedded-answers.
• I show the various feedback that students will receive after answering an embedded-answers question.
• I give a handful number of tips to consider when developing embedded-answers questions.
• Finally, I conclude this article with an example of quiz I give to the undergraduates who take my introductory course to Computer Networking at NYU Shanghai and Sorbonne Université.

The 4 Types of Embedded-Answers Questions

Moodle Embedded-Answers questions are similar to Fill in the Blank questions on Sakai and other LMSes. There are four (4) main types of embedded-answers questions:

• Short Answer;
• Numerical;
• Multi-Choice;
• Multi-Response.

Below I present a preview of each type of embedded-answers questions.

Moodle offers 4 main types of embedded-answer questions.

For instance, when students encounter a Short Answer (Question 1 in the above figure) or a Numerical question (Question 7 in the above figure), they are presented with a text box in which they can enter their answer. Their answers will be compared to a list of predetermined answers provided during the authoring of the question. All predetermined answers come with the number of points they are worth. You can update the list of predefined answers and/or their points after the quiz due date and ask Moodle to regrade all student attempts.

Moodle Text Editor

To create an embedded-answers question, Moodle will present you with a page comprised of two text areas. The first text area is a text editor where you will add the text of your question and the markup text for the answers. The second text area is where you can optionally add the general feedback students will see after submitting their answers.

Moodle’s question text editor.

The top of the question text editor is a toolbar with a list of icons to create headings, bold or italic text, color and highlight text, create numbered and bulleted lists, or adding media or links to a page.

By default, the toolbar is collapsed with only one row of the most commonly used tools showing. The default toolbar can be expanded to show all of its features by clicking the “Show/hide advanced buttons” button (e.g., the first on the left of the collapsed toolbar).

Moodle’s text editor extended toolbar.

Contrary to the question itself which can be specified in a WYSIWYG way, the embedded-answers follow a WYSIWYM (What you see is what you mean) approach. You need to follow a strict syntax for Moodle to understand how to grade the question. I will present the syntax of Moodle markup language in the next section.

Once you have entered your question into the first text area, you can see a preview of your question: Click “Save changes and continue editing”; A clickable magnifying glass icon will appear followed by “Preview”. A pop-up window will show a preview of the question you are authoring. The bottom of the window will give the following choices:

The bottom of the Preview question window.

“Fill in correct answers” will let you preview the correct answers you have programmed. You can also fill arbitrary answers and check how Moodle will grade the question after clicking “Submit and finish”.

Below, I show on the left, the markup text of two Short Answer subquestions and on the right, the resulting output the students will see. This question consists of two subquestions, each followed by the expected embedded-answer. You can add as many embedded-answers as subquestions in the main question you are creating.

Short Answer Question text and preview.

The Embedded-Answers Markup Language

The minimal syntax of an embedded-answer is the following:

{:SA:=Correct Answer}

This corresponds to an embedded-answer accepting a single correct answer with no specific feedback. The maximum score will be awarded only if students give the answer “Correct Answer”. Any extra space or spelling mistakes will be counted as wrong.

The full syntax of an embedded-answer is the following:

Embedded Answer question markup breakdown.

An embedded-answer is delimited by an opening and a closing curly bracket ‘{’ … ‘}’. Inside the brackets there are four parts, the first and last being optional:

• The first part is a strictly positive integer (1, 2, 3, etc.) that indicates the weight of the answer. This weight is used in the calculation of the overall question grade. In case of a question with multiple subquestions, the total points for the question will be computed as the weighted sum of all subquestion points using the weight assigned to each subquestion. If all subquestions have the same weight, this first part may be skipped.
• The second part is delimited by semicolons (‘:’) and indicates the type of the embedded-answer (SA, NM, MC, MCH, MCV, MR, and MRH).
• The third part gives the predefined answer(s) against which students’ answers will be matched. Each answer is preceded by its score given as a percentage of the total score awarded for the correct answer. Consecutive predefined answers are separated by the tilde symbol (‘~’). A correct answer is preceded by the equal sign (‘=’) or by ‘%100%’. This will indicate that the corresponding answer will be awarded the maximum score. A partially correct answer is typically preceded by a value greater than 0 but lower than 100 (e.g. ‘%50%’). The wildcard character (‘*’) can be used as a catch-all for all wrong answers. If wrong answers are awarded zero points, ‘ ~%0%*’ is optional. You can also award a negative score in case of any wrong answer (e.g. ‘%-50%*’).
• The fourth and last part is optional. It indicates the specific feedback students will receive if enabled in the Quiz settings. The hash symbol (‘#’) that follows a predefined answer indicates the specific feedback the students who gave that answer will receive.

Short Answer Questions

The table below presents various combinations of correct, partially correct, and incorrect answers and specific feedback with the corresponding Moodle markup text.

Types of answers and feedback for Short Answer questions.

Below, I explain the syntax of Moodle’s markup text for each type of answers and feedback listed in the previous table.

One correct answer:

{1:SA:=correct answer}

where ‘=’ marks the correct answer.

Multiple correct answers:

{1:SA:%100%correct answer 1~%100%correct answer 2}

where ‘~’ is the separator between each answer and %100% the percentage of the subquestion total points for the corresponding answer. Any other answer than ‘correct answer 1’ and ‘correct answer 2’ will be considered wrong and awarded 0 points.

Correct and partially correct answers with specific feedback:

{1:SA:%100%answer1#Correct~%50%answer2#Partially correct}

where ‘#’ marks the beginning of a specific feedback for the preceding answer. Specific feedback are shown to students if opt-in in the Review options of the quiz. Any other answer than ‘answer1’ and ‘answer2’ will be considered wrong and awarded 0 points.

Correct, Partially correct, and incorrect answers with specific feedback:

{1:SA:%100%answer1#Correct~%50%answer2#Partially correct~%-50%*#Wrong answer}

where ‘*#’ indicates the feedback students will receive in case of any answer other than the preceding listed answers. %-50% indicates that half of the points allotted to the subquestion will be deducted from the overall question score if the student selects any answer other that the correct and the partially correct answers.

Embedded-Answers Questions

The table below lists all types of embedded answers:

Types of embedded-answers questions.

Below, I give the Moodle markup text for each type of embedded-answers and the preview of the question:

Moodle markup and preview for all types of embedded-answers questions. Answer A is the correct answer, Answer B a partially correct answer, and Answer C the wrong answer. Answer A is awarded the maximum score, Answer B half of the maximum score, and Answer C no points.

Embedded-Answers Tips and Considerations

Adjusting the Layout of Embedded-Answer Questions

The length of the text box in SA and NM embedded-answers is computed based on the length of the longest answer including the wrong answers.

In case of multiple SA or NM subquestions, the trick to fix the length of all the text boxes is to add the same dummy wrong answer to all embedded-answers.

In the example below, I’ve added the dummy wrong answer ‘~%0%XXXX’. I also show the use of tables for a better control over the alignment of the subquestions and the corresponding answer boxes.

Adjusting the length and layout of multiple embedded-answers questions with dummy answers (‘XXXX’) and tables.

You can also control the size and the borders of the table, by tweaking the HTML code of the question. Click the last icon in the text editor extended toolbar (</>) to show the HTML code for the question you are authoring:

• To extend the width of the table, add tag width followed by the size in pixels (i.e., “500”).
• To enable borders, add tag border followed by the thickness of the border lines in pixels (i.e., “1”).

The HTML code should look like this:

<table width="500" border="1">

Adjusting the layout of a table.

Use Keyword ‘Empty’ for Questions with Non-Applicable Answers

Student answers are matched against the list of expected answers embedded in the Cloze question. To grade a question, Moodle expects to find an answer. A question submitted with no answer is considered as not answered and won’t be awarded any points.

To indicate to Moodle that the blank answer is a correct answer, you can define the use of a specific keyword such as ‘empty’ that Moodle will interpret as a blank answer. The points for this question will be awarded if ‘empty’ has been answered.

Avoid Copy/Paste from MS Word

If you copy the answers from a Microsoft Word document, this may add some unwanted code to the HTML code that Moodle will consider as part of the answer. As students fill their answer in plain text, Moodle may flag their answer as wrong because of the code wrongly added to your embedded-answer. To avoid having to clean the html code of your question, use a plain text editor.

Keep Questions Short and Add Break Pages

Once a question answered, there are two main ways for Moodle to store the answers. Students can either click “Submit all and Finish” or navigate within the quiz provided that questions are distributed on multiple pages. In the latter, Moodle will log the answers until submission. Depending on the Time setting of the quiz, answers can be submitted either automatically upon expiration of the quiz time limit or only if students click “Submit all and Finish” before the closing date of the quiz.

In the layout quiz settings, you can ask Moodle to put one question per page. You can also add page breaks where needed in the ‘Edit quiz’ page. Adding page breaks will let Moodle log the answers as students navigate through the pages of the quiz.

Regrade a Quiz after its Closing Date

You can make changes to the quiz and have Moodle regrade all attempts. Such changes include changing the maximum points of a specific question or the maximum grade of the quiz. You can also add answer options to Short Answers and Numerical questions.

From the Quiz summary page, you can access the attempts summary page where a table will list all attempts and for each attempt, the detailed points awarded per question. You can review how the students performed for each question of the quiz.

If after reviewing student answers, you realize that a handful number of students responded with a partially correct answer you didn’t anticipate when creating the quiz, you can add that answer in the answer options of the embedded question. You can then have all attempts regraded to account for that additional answer.

Review Options

In the table below, I present what the student sees after answering an embedded-answers question depending on the review options of the quiz. In this example, the question has the two following subquestions:

Subquestion 1: {1:SA:=Correct answer 1#Correct Answer!~%0%*#Wrong Answer!}

Subquestion 2: {1:SA:=Correct answer 2#Correct Answer!~%0%*#Wrong Answer!}

Students who enter the correct answer will receive feedback ‘Correct Answer!’ and feedback ‘Wrong Answer!’, otherwise.

Question text and preview.

For each example, I suppose that a student has answered the first subquestion with the correct answer (e.g., “Correct answer 1”) and the second with a wrong answer (e.g., “Wrong answer 2”).

In the left column, I show the Review options you can set in the Quiz settings and in the right column, the resulting output as seen by the students. These outputs are similar whether the review options are set to ‘During the attempt’, ‘Immediately after the attempt’, ‘Later, while the quiz is still open’, or ‘After the quiz is closed’. The only difference is when the various feedback will be available for review to the students.

The points awarded for the question (e.g., mark), whether the answer is correct or not, the specific feedback for the provided answer, and the right answer are shown in a yellow box revealed when passing the mouse over each answer. The general feedback is shown below the question in an orange text box.

List of the review options for embedded-answers question and the resulting output as seen by students.

A Comprehensive Example

To show the range of possibilities embedded-answers questions can offer, I present a quiz I give in my introductory course to Computer Networking. The topics covered by this assessment include IP subnetting, ARP, ICMP, and Proxy ARP.

You can download this quiz in Moodle XML format here (right-click and select ‘Save link as’) and import the quiz in your course question bank.

Part 1

Consider the network depicted in Figure 1. This network consists of two (2) Ethernet segments interconnected by a bridge noted X.

A, B, C, and D are end hosts. The IP and MAC addresses are shown in the figure. The figure also shows two observation points which capture the traffic on both directions on each segment.

Figure 1: X is a bridge.

Host A (172.16.10.100) wants to ping host D (172.16.20.200). The following command is run on Host A:

ping –c 1 172.16.20.200

Question 1.1

Host A starts by sending an ARP request. Complete Table 1.1 with the MAC addresses found in the Ethernet frame header encapsulating the ARP request and the ARP reply messages as captured at each Observation Point.

Table 1.1.

Question 1.2

Complete Table 1.2 with the addresses found inside the ARP messages captured by Observation Points 1 and 2.

Table 1.2.

Question 1.3

Host A can now send its Echo Request to host D. Complete Table 1.3 with the addresses included in the Ethernet and IP headers of the Echo Request and the Echo Reply captured at Observation Point 1.

Table 1.3.

Question 1.4

Suppose host A has sent pings to all other machines in this network. Complete Table 1.4 with the entries that can be found in host A’s ARP table. Use keyword empty for cells with no applicable value.

Table 1.4.

Part 2

Consider now that the two segments are connected by a router noted R. The IP and MAC addresses of R are shown in Figure 2.

Figure 2: X is a router.

Host A (172.16.10.100) wants to ping host D (172.16.20.200). The following command is run on Host A:

ping –c 1 172.16.20.200

Question 2.1

Host A starts by sending out an ARP request. Complete Table 2.1 with the MAC addresses found in the Ethernet frame header encapsulating the ARP request and the ARP reply messages as captured at each Observation Point.

Table 2.1.

Question 2.2

Complete Table 2.2 with the addresses found inside the ARP messages captured by Observation Points 1 and 2.

Table 2.2.

Question 2.3

Host A can now send its Echo Request to host D. Complete Table 2.3 with the addresses included in the Ethernet and IP headers of the Echo Request and the Echo Reply captured at Observation Point 1.

Table 2.3.

Question 2.4

Suppose host A has sent pings to all other machines in this network. Complete Table 2.4 with the entries that can be found in A’s ARP table. Use keyword empty for cells with non-applicable value.

Table 2.4.

Question 2.5

Assume that host A is misconfigured with subnet mask 255.255.0.0. Which machines can host A ping?

Question 2.6

We now assume R is configured as an ARP proxy. As in previous question, we assume host A is configured with subnet mask 255.255.0.0.

Host A (172.16.10.100) wants to ping host D (172.16.20.200). The following command is run on Host A:

ping –c 1 172.16.20.200

A starts by sending an ARP request.

Complete Table 2.6 with the addresses found inside the ARP messages captured by Observation Points 1 and 2.

Table 2.6.

Host A can now send its Echo Request to host D. Complete Table 2.7 with the addresses included in the Ethernet and IP headers of the Echo Request and the Echo Reply captured at Observation Points 1 and 2.

Table 2.7.

Suppose host A has sent pings to all other machines in this network. Complete Table 2.8 with the entries that can be found in host A’s ARP table. Use keyword empty for cells with non-applicable value.

Table 2.8.

Final Words

Moodle offers a wide variety of questions that Moodle can grade (and regrade) automatically. Once created, questions are added to a question bank from where they can be reused or exported. These questions include Multiple choice, true-false, and matching questions but also Embedded-answers questions which offer a great level of flexibility.

Embedded-answers aka Cloze questions allow instructors to assess student understanding of more advanced topics. Once submitted, instructors can update the predefined answers. They can for instance add more partially correct answers to account for unanticipated student answers. Moodle can then regrade the student attempts accordingly. Feedback include the overall score and the question points but also a general and specific feedback shown to students depending on their answers.