Last week I held a workshop at the Tekom/TCWorld 2011 conference in Wiesbaden, Germany. The title was Creating effective online tests. It was about how to write better multiple choice tests. It’s a topic we are currently dealing with at my company, as part of our partner certification programme. The following contribution is the text that appeared in the conference proceedings (handed to every attendee). You can also download the presentation (PDF).
Tests play an important role in the educational aims of organizations, and are one of the main means of certification. As a technical writer or trainer developing online tests, you need to be aware of the requirements for creating effective ones. A good test can a) better reflect the stated knowledge requirement and b) improve the accuracy of the assessment.
Luckily, we do not have to start with a blank sheet when composing tests. There is a wealth of experience and established practices which we can and must draw upon.
The aim of your test
Before you start writing test questions, you should think about what level of knowledge is being tested. Don’t just compose questions blindly. Usually, a test is taken after a period of study or training. They want to obtain knowledge of a subject. The concluding test should reflect the level of knowledge the person should have. This means that a person who passes the test has demonstrated that he/she possesses the requisite knowledge of the subject. This increases the value of the test and turns the pass certificate into more than just a piece of paper.
Avoid questions that test irrelevant or unimportant topics, like “What is the keyboard shortcut to change the image size?” This does not test the student’s knowledge of how to change the image size, only how to perform this operation faster. It is not significant.
The structure of the test “item”
An item is the term used to describe the grouping of the question and the possible answers. Unless you want to be stuck with the time-consuming task of manually marking tests, your questions must be one of the following types, which allow them to be marked automatically by various software or Learning Management Systems (LMS):
- Multiple choice
In a way, all of these item types are multiple choice. We will look at the “classic” multiple choice item here.
An item is composed of:
- The stem (the question).
- The alternatives (the correct answer plus the distractors, or incorrect answers).
What value does x have in (6 * 1.5) ÷ (36 ÷ x) = 3 ?
Here, answers a., b. and c. are distractors. The correct answer is d.
Types of multiple-choice items
Some people assume that multiple-choice items limit what you can test, and allow for the possibility of guessing the correct answers. Although these concerns are sometimes justified, there are ways in which you can create items that test a person’s:
- Comprehension of a subject. The stem states a process and the alternatives list different outcomes. Example: Q. A purchase order contains 3 items. Items 1 and 2 have GR-based IV active. Item 3 does not. Only Item 1 has been received. After entering the purchase order in MIRO, SAP proposes which items in the invoice? A. Item 1 only.
- Ability to apply knowledge to solve a problem. The stem states a process and the outcome, and the alternatives list the possible means by which the outcome is achieved. Example: Q. Only items that have been delivered should be added to the invoice in MIRO. This is achieved by: A. Activating GR-based IV in the purchase order.
- Analytical skills. The stem states a fact and the alternatives list possibilities for what the fact is. Example: Q. What is the GR-based IV setting used for? A. To force the item to be delivered before it can be added to the invoice, and therefore paid.
The analytical type of multiple choice item is perhaps the easiest to compose, but do not rely entirely on it. In the above three examples, the first two arguably force the person taking the test to think more. When deciding on which item type to use, try to match it to the way a person would think about it in practice. For example, for topics where problem solving is particularly important, formulate these problems as comprehension item types. In this way, you test what people have to know to perform their work.
When writing items, keep these criteria in mind:
- Clarity. Your items must be clear to the reader. This minimizes the time required to understand the question and the correct answer, and reduces the risk the reader will answer incorrectly due to not understanding the item correctly. Everyone has encountered questions where you are not sure what is being asked, and have to guess the intention of the person who wrote them.
- Images. Use images where possible to help the reader understand questions. Images are particularly helpful for complex questions and can help avoid the need for long, descriptive questions.
- Brevity. Keep your items as short as possible to convey the meaning. Do not include anything irrelevant.
- Positive questions. Form questions in a positive form. For example, use “What is…” and not “What is not…” If you use the negative form, highlight the negative word (bold, italics, underline), to make it clear to the reader what is being asked.
- Plausible distractors. Distractors should be plausible enough to promote doubt among the reader who is not sure what the correct answer is. Using distractors that are not plausible do not test the reader. On the contrary, they assist those who do not know the correct answer to determine it by eliminating obvious implausible distractors.
- Do not combine distractors. That is, avoid distractors that are “a. and c.” If the reader knows one is incorrect, he/she knows both are incorrect.
- Avoid “All/None of the above.” Again, these can assist the reader to determine the correct answer (when they do not know it) by a process of elimination.
- Mutually exclusive. If there is to be only one correct answer, ensure the distractors are incorrect. That is, that there is no circumstance in which a distractor may be considered correct, “under such-and-such condition.”
- Simple vocabulary. This is particularly important in today’s international business world, where tests are taken by people whose mother tongue is different that the language test. Again, it is a matter of ensuring the reader understands the item completely in order to test him/her fairly.
Writing tests is a challenge and should not be underestimated. You must:
- Make the difficulty of the items reflect the aim of the test and the level of knowledge required.
- Have a nice balance of multiple choice item types (comprehension, application, analysis), where the type of each item ideally reflects the type of thinking that is required in practice, on the job.
- Write items accurately and observe the best practices. Some of these are listed above. More can be found in the reference below. Your items must be exact, leaving no room for misunderstanding. Otherwise, people may answer questions incorrectly, reducing the accuracy of the test.
There are lots of references you can find online for writing good tests. The following is an excellent starting point. It is concise, well written, and includes many examples.
How to Prepare Better Multiple-Choice Test Items: Guidelines for University Faculty. (1991) Burton, Sudweeks, Merrill & Wood. Available online at http://testing.byu.edu/info/handbooks/betteritems.pdf