TechComm for Life

Last week I held a workshop at the Tekom/TCWorld 2011 conference in Wiesbaden, Germany. The title was Creating effective online tests. It was about how to write better multiple choice tests. It’s a topic we are currently dealing with at my company, as part of our partner certification programme. The following contribution is the text that appeared in the conference proceedings (handed to every attendee). You can also download the presentation (PDF).

Introduction

Tests play an important role in the educational aims of organizations, and are one of the main means of certification. As a technical writer or trainer developing online tests, you need to be aware of the requirements for creating effective ones. A good test can a) better reflect the stated knowledge requirement and b) improve the accuracy of the assessment.

Luckily, we do not have to start with a blank sheet when composing tests. There is a wealth of experience and established practices which we can and must draw upon.

The aim of your test

Before you start writing test questions, you should think about what level of knowledge is being tested. Don’t just compose questions blindly. Usually, a test is taken after a period of study or training. They want to obtain knowledge of a subject. The concluding test should reflect the level of knowledge the person should have. This means that a person who passes the test has demonstrated that he/she possesses the requisite knowledge of the subject. This increases the value of the test and turns the pass certificate into more than just a piece of paper.

Avoid questions that test irrelevant or unimportant topics, like “What is the keyboard shortcut to change the image size?” This does not test the student’s knowledge of how to change the image size, only how to perform this operation faster. It is not significant.

The structure of the test “item”

An item is the term used to describe the grouping of the question and the possible answers. Unless you want to be stuck with the time-consuming task of manually marking tests, your questions must be one of the following types, which allow them to be marked automatically by various software or Learning Management Systems (LMS):

• Multiple choice
• True/False
• Yes/No

In a way, all of these item types are multiple choice. We will look at the “classic”  multiple choice item here.

An item is composed of:

• The stem (the question).
• The alternatives (the correct answer plus the distractors, or incorrect answers).

For example:

What value does x have in (6 * 1.5) ÷ (36 ÷ x) = 3 ?

a. 3
b. 6
c. 9
d. 12

Here, answers a., b. and c. are distractors. The correct answer is d.

Types of multiple-choice items

Some people assume that multiple-choice items limit what you can test, and allow for the possibility of guessing the correct answers. Although these concerns are sometimes justified, there are ways in which you can create items that test a person’s:

• Comprehension of a subject. The stem states a process and the alternatives list different outcomes. Example: Q. A purchase order contains 3 items. Items 1 and 2 have GR-based IV active. Item 3 does not. Only Item 1 has been received. After entering the purchase order in MIRO, SAP proposes which items in the invoice? A. Item 1 only.
• Ability to apply knowledge to solve a problem. The stem states a process and the outcome, and the alternatives list the possible means by which the outcome is achieved. Example: Q. Only items that have been delivered should be added to the invoice in MIRO. This is achieved by: A. Activating GR-based IV in the purchase order.
• Analytical skills. The stem states a fact and the alternatives list possibilities for what the fact is. Example: Q. What is the GR-based IV setting used for? A. To force the item to be delivered before it can be added to the invoice, and therefore paid.

The analytical type of multiple choice item is perhaps the easiest to compose, but do not rely entirely on it. In the above three examples, the first two arguably force the person taking the test to think more. When deciding on which item type to use, try to match it to the way a person would think about it in practice. For example, for topics where problem solving is particularly important, formulate these problems as comprehension item types. In this way, you test what people have to know to perform their work.

Best practices

When writing items, keep these criteria in mind:

• Clarity. Your items must be clear to the reader. This minimizes the time required to understand the question and the correct answer, and reduces the risk the reader will answer incorrectly due to not understanding the item correctly. Everyone has encountered questions where you are not sure what is being asked, and have to guess the intention of the person who wrote them.
• Images. Use images where possible to help the reader understand questions. Images are particularly helpful for complex questions and can help avoid the need for long, descriptive questions.
• Brevity. Keep your items as short as possible to convey the meaning. Do not include anything irrelevant.
• Positive questions. Form questions in a positive form. For example, use “What is…” and not “What is not…” If you use the negative form, highlight the negative word (bold, italics, underline), to make it clear to the reader what is being asked.
• Plausible distractors. Distractors should be plausible enough to promote doubt among the reader who is not sure what the correct answer is. Using distractors that are not plausible do not test the reader. On the contrary, they assist those who do not know the correct answer to determine it by eliminating obvious implausible distractors.
• Do not combine distractors. That is, avoid distractors that are “a. and c.” If the reader knows one is incorrect, he/she knows both are incorrect.
• Avoid “All/None of the above.” Again, these can assist the reader to determine the correct answer (when they do not know it) by a process of elimination.
• Mutually exclusive. If there is to be only one correct answer, ensure the distractors are incorrect. That is, that there is no circumstance in which a distractor may be considered correct, “under such-and-such condition.”
• Simple vocabulary. This is particularly important in today’s international business world, where tests are taken by people whose mother tongue is different that the language test. Again, it is a matter of ensuring the reader understands the item completely in order to test him/her fairly.

Conclusion

Writing tests is a challenge and should not be underestimated. You must:

• Make the difficulty of the items reflect the aim of the test and the level of knowledge required.
• Have a nice balance of multiple choice item types (comprehension, application, analysis), where the type of each item ideally reflects the type of thinking that is required in practice, on the job.
• Write items accurately and observe the best practices. Some of these are listed above. More can be found in the reference below. Your items must be exact, leaving no room for misunderstanding. Otherwise, people may answer questions incorrectly, reducing the accuracy of the test.

References

There are lots of references you can find online for writing good tests. The following is an excellent starting point. It is concise, well written, and includes many examples.

How to Prepare Better Multiple-Choice Test Items: Guidelines for University Faculty. (1991) Burton, Sudweeks, Merrill & Wood. Available online at http://testing.byu.edu/info/handbooks/betteritems.pdf

Introduction

Last year, my development department held a series of meetings to discuss ways in which we could help significantly increase the sales of the products we make. This year, we’ve put a lot of what we discussed into practice, and the Technical Communication team has played a key role.

Our department believes we make excellent products, and we have the customers to back this claim up. As a development lab though, we do not sell our products. This is the job of sales people in our company’s subsidiaries around the world, and among our partners. We know our products can sell, but can we make sales sell more?

The Holy Grail

I like to think of the Holy Grail as giving subsidiaries and partners the “magic power” to sell, implement, and support our products better, and more independently from the development lab. However, it’s not really magic, it’s about giving them the skills and knowledge found within the development lab itself.

How can the Holy Grail be achieved?

This is what we have achieved this year:

• Fully-featured demo systems
  Pre-configured, with sample data, and complete with full instructions. Great for sales and pre-sales meetings, as well as instructor-led training courses.
• Detailed sales presentations and other material
• Comprehensive best practices
  Initially created as a package to sell the most effective solutions and implement them in the shortest amount of time, best practices also help educate people about the product in general.
• Copious amounts of helpful documentation
  Documentation that not only explains the technical nitty-gritty, but that contains quick start and general topics for those new. The documentation doesn’t answer every question, but it covers the basics well.
• Training material
  Lots of presentations that can be used to give instructor-led trainings.
• A video library
  Yes! Is there anyone who doesn’t now believe in the power of videos? When you want to refresh your knowledge on something, would you rather browse help topics, try your luck by experimenting in the software/hardware, or sit back and watch a 5min video explain everything? I have a big demand now for more videos, from people who realise their value.
• We have attended courses to improve our business process knowledge and the environment in which our products are used.
• An easy-to-use, but powerful intranet presence, which encapsulates all of the above. Sales and technical staff can dig into our resources, find what they want fast, and learn as much about our products as they want, in all kinds of different ways.

Conclusion

We can’t attend every sales meeting, workshop, or implementation around the world. We have a division of labour and we think this labour power can now be utilised to the maximum, thanks in no small part to the efforts of our technical communicators (but also to our developers, solution managers, and support staff). The next time you think about what a technical writer does, or should or can do, consider the bigger picture and the Holy Grail.

I recently had to manage a translation of user guides from English to French. This was the first time I had done this for RoboHelp projects that we use to publish AIRHelp. Our localization provider did not have RoboHelp, but this proved to be no problem. After all, what requires translating is a series of HTML and XML files, which comprise the RoboHelp project. Instead of sending the provider the entire RoboHelp project and letting them sort through a mass of files, I only sent the ones requiring translation. This is how I organised the translation:

Files to send

Copying your translated files and checking your project

1. Make a copy of your English-language RoboHelp project.
   You will use this for the new language.
2. Delete the contents of the !SSL! subfolders.
   That is, remove the English outputs copied from the previous step.
3. Copy the translated files to the new project, overwriting the existing files.
4. Open the project.
5. Rename the project to <project name>-<language>.xpj.
   This prevents projects in different languages getting mixed up!
6. Go to File > Project Settings.
   Items to change or check:
   a. The Project Title for the new language.
   b. Language.
7. Open your AIRHelp SSL properties.
   Items to change or check:
   a. General > Help Title
   b. Generate > Version (set it to 1.0, or whatever should be the first version number).
   c. General > Help ID (it should be unique, so include a language reference, for example, fra for French).
   d. Template > Description (if it has not already been translated)
   e. View > Specify the URL for Online Help
   f. Server > URL.

Generating the output and checking the content

Once you have ensured your new translated project is ready, generate the AIRHelp!

Go from this...

..to this.

Things to check in the output:

• If possible, ask someone in your organisation who is a native speaker of the language to check for errors. This is more of a technical check that the specific terminology used for your product is correct.
• Check the index and TOC. The index may contain double or similar entries, depending on how words have been translated.
• Instances of your snippets and variables.
• AIRHelp window title.
• The AIRHelp skin strings (like Contents, Index, About, and so on) should appear in the correct language. This is the result of changing the project’s Language setting, which copies the RoboHHRE.lng file from the C:\Program Files\Adobe\Adobe Technical Communication Suite 2\Adobe RoboHelp 8\RoboHTML\Language\<Language> folder to your project. The LNG file contains the AIRHelp skin strings.

I was curious about file sizes between different output formats using different screencasting applications. So I created a sample 1minute recording (video only, no audio) using ScreenFlow, Camtasia and Captivate, and compared the results. I tried to use the default settings for each of the outputs, avoiding the need to manually specify data encoding rates.

Results

Although the results below vary quite a bit, this is due to the default presets. Each program gives you the ability to modify the quality of the output. So for example, I could modify the dimensions and data rate of the output video in ScreenFlow to make it the same as the SWF file in Captivate, and the resulting file size was comparable, although the quality was not quite as good as the SWF, but still ok.

Conclusion

I think the key is to set the data rate as low as possible. If you’re watching movies (I mean, the Hollywood variety), then having high data rates is the key to watching a great quality video. But screencasts only have a miniscule amount of animation in comparison, meaning there is no reason to have high data rates. Maybe I’m wrong though. If so, please let me know. But in my opinion, screencast viewers don’t expect super high-quality recordings—they don’t care that window frame movements or menu selections are not HD—just as long as they can easily follow what’s happening on the screen.

If you have reached your own conclusions please let me know. I’d like to know if you have preferences for output sizes and quality.

In my last post on Screencasting software for the Mac, I criticised Adobe Captivate 5′s lack of publishing options, which were essentially limited to Flash. Well, would you believe that two days later Adobe released Captivate 5.5, which allows you to publish to the standard MP4 video format, as well as directly to YouTube! These are long-awaited features. Both ScreenFlow 2 and Camtasia for Mac 1.2 have options to publish to MOV and directly to YouTube.

Preset MP4 publishing options in Captivate 5.5.

MP4/MOV are ideal formats for publishing standard screen recordings, that is, when you want to demonstrate something. If you need Adobe Captivate’s many interactive features, then you must still publish to Flash. A lot of my recordings are just normal screencasts, and previously I’ve always had to publish to SWF (I have Captivate 4), or to PDF (which acts as a wrapper for the SWF file). SWF files, while small in size (allowing for easy distribution), can be problematic to play. I often get asked how to play these files. Although they usually play in a browser, sometimes users will get browser security warnings when attempting to play the SWFs. So you can imagine my relief when Captivate 5.5 was released. MP4′s can also be played on iPhones and iPads.

Below is a short video demonstrating the MP4 and YouTube publishing options:

There have been many screencasting software comparisons. Well, I’m going to add one more. Why?

1. My employer will soon be implementing its very first Learning Management System (LMS). The TechComm group currently uses Adobe Captivate 4 for screencasts. But although our team will be one of the main contributors of content to the LMS, many other people also want to contribute. For example, to develop training packages for customers that include videos based on their custom installations. So our team is going to provide some advice on what tools are available.
2. I wanted to decide what I should purchase myself, for my Mac at home, to produce simple screencasts for this blog!

Tested applications

I’ve used Captivate for about two years. I recently downloaded trial versions of the following screencasting software for Mac OSX to test them out:

• Camtasia 1.2
• ScreenFlow 2
• Captivate 5
  [Postscript: On 27 May 2011 Adobe released Captivate 5.5, which includes the ability to publish to YouTube and MP4, one of my main criticisms of previous versions. I have created a new post: Adobe Captivate 5.5 – New publishing outputs.]

Methodology

My main testing criteria were:

• Ease of use
• Interactivity features
• Publishing formats

I did not really look at the basic features which most screencasting applications have (and should have), like slide transitions, callouts, adding own slides with text and images, etc., and how pretty these are in the different apps.

Results

For anyone familiar with screencasting software, the results are not surprising. The main differentiators between these three applications are: Interactivity and publishing formats.

Results based on my test criteria:

• Ease of use
  All three applications are easy to use. I really like Captivate’s storyboard organisation of the screens (it basically takes a series of screenshots, rather than full-motion video)—just like in PowerPoint or Keynote, but with the addition of audio, video, and interactive elements. When I record screencasts, I record the video first, and add audio later. This takes the pressure off and you can concentrate more on both tasks, rather than having to create the perfect video and audio simultaneously. When it does come to recording the audio, it is much easier to synchronise it with the video using Captivate. ScreenFlow and Camtasia record videos that appear in a timeline. After clicking the record audio button, you have to manually play the portion of the video you want to add the narration to. Apart from this drawback though, the layouts of ScreenFlow and Camtasia are similar and are both easy to use.
• Interactivity features
  With Captivate your videos don’t have to be passive. You can make them interactive, allowing users to click on different parts of the screen in order to proceed. In addition, you can create quizzes. You can create quizzes as well with Camtasia, but only in the Windows version. Apart from this, Camtasia and ScreenFlow create plain videos, albeit good looking ones.
• Publishing formats
  This is where ScreenFlow and Camtasia come out on top. They both allow a wide range of publishing options, including Flash, MOV, M4V, WMV (ScreenFlow only), AVI (Camtasia only) and MP4 (Camtasia only). Significantly, you can also use both to publish directly to YouTube. Captivate is limited to Flash (SWF and F4V) only [See Postscript, above]. You can publish your projects as applications (APP and EXE for Mac and Windows, respectively), and the Windows version allows you to publish to AVI, but the resultant file size is incredibly high (for example, 60MB for one 3min video I recently created, compared to just 3MB as a SWF). Therefore, if you really want the maximum sharing possibilities, Captivate is ruled out. However, Adobe now has the Media Encoder application available, which allows you to convert F4V files (not SWF) to MP4. When I tried it though, the simulated sounds of the mouse clicks and keyboard typing that Captivate adds were out of sync in the MP4.
  Of course, any interactivity that relies on Flash won’t work in normal video formats, but for those basic screencasts that don’t contain interactive objects, Captivate sorely needs some standard video output options.

Fazit

For work I’ll continue to use Captivate 4. It’s a power tool and easy to use. And I hope that the future Captivate 6 has more output options. For home/blog use I’m going to purchase Camtasia for Mac. At $99USD it’s affordable (and three times cheaper than the Windows version). It’s the same price as ScreenFlow, which would also be a reasonable investment but it doesn’t have quite as many features as Camtasia. Captivate retails for $799USD, a price only for enterprise and professional users.

Here are some quick presentations of each application:

ScreenFlow 2 for Mac

Camtasia 1.2 for Mac

Captivate 5 for Mac