The traditional multiple choice questions we use in assessment are often abstract and measure only whether people recall facts they heard in the last 5 minutes. Converting these questions to scenario-based questions can increase the level of difficulty, measure higher level thought, and provide relevant context.
Let’s say you’re creating training for managers on how to provide reasonable accommodations for employees. You drafted a set of traditional multiple choice questions as a quiz for the end of the course, but they’re all very low level. You want to improve the quality of your assessment with some scenarios.
This is a question from your current quiz that measures recall of a fact from the training. The rest of the assessment is similar.
Example 1 (Original)
What reasonable accommodation is recommended for a temporary disability or medical issue affecting work?
- None; reasonable accommodations are only used for permanent or long-term disabilities.
- Unpaid time off can be offered as an accommodation for temporary issues.
- Paid time off should be offered, even if it exceeds the amount of paid time off other employees receive.
- Schedule adjustments are the only accommodation offered for temporary issues.
Align to Objectives
What are your objectives? Does your assessment align to them? If not, rewrite it.
In this example, the objective is “The learner will follow the procedure for providing reasonable accommodations.” The objective is application level; you need to apply this procedure. (You could argue for analysis or evaluation here too, but let’s assume it’s application.)
The question assesses recall; the objective requires application. Therefore, this question should be rewritten at a higher level.
When would people use this?
The first step to shifting from traditional to scenario-based assessment is asking when people would use the information. When would managers need to know about handling temporary disabilities? A common situation would be due to an illness or surgery. Maybe an employee needs a reduced schedule due to fatigue from chemo. Maybe an employee needs time off to recover from back surgery.
For each multiple choice question, ask yourself how learners would use that information on the job. When would they need to differentiate between those options?
If you can’t come up with any situation in which people would need this information on the job, why are you asking that question? If you have a question with just irrelevant information, skip down to the section on complete rewrites below.
Scenario as Introduction
One method to revise the question is to add a scenario to introduce the choices. This provides context. It shifts the question from just recalling information to using that information to make a decision.
Let’s see how this works with the previous example. The scenario introduces the question. The choices are essentially the same as before, but now it’s a decision about how to work with an employee you manage. Instead of measuring recall, this question measures if learners can apply the reasonable accommodations policy.
Example 1 (Revised)
Simon, a graphic designer on the team you manage, is having surgery. He requested 2 weeks time off to recover after his surgery. How should you respond?
- Let Simon know he can use his accrued vacation time. Reasonable accommodations are only used for permanent or long-term disabilities.
- Provide two weeks unpaid time off.
- Provide two weeks paid time off.
- Allow Simon to adjust his schedule as long as he still works his full normal hours.
Notice that this scenario isn’t long; it’s only 2 more sentences than the original question.
Sometimes adding a scenario at the beginning won’t work, and you need a complete rewrite of the question. If the question is something unrelated to your objectives or that people will never use on the job, you have to start over and replace the question.
Look at this example. Would a manager ever need to know this history on the job? Will they be more effective at offering accommodations if they can memorize this date?
Example 2 (Original)
In what year was the Americans with Disabilities Act or ADA passed by Congress?
We have all seen questions like this on quizzes before. They’re easy to write, but they don’t assess anything meaningful. Replacing it with a scenario-based question would give you a more accurate assessment.
Example 2 (Replacement)
One of your employees, Miranda, brought documentation from her ophthalmologist about her vision and how it affects her driving. Her night vision is deteriorating. Miranda has requested a change in her work schedule. She wants to start and end her work day later to avoid driving in the early morning when it’s still dark. How do you respond?
- Agree to adjust Miranda’s schedule.
- Tell Miranda to contact HR to start the official accommodation process.
- Tell Miranda that the schedule change is not possible since it creates too much burden on the rest of the team.
- Ask Miranda if any other accommodations would work instead of a schedule change.
What Do You Want to Learn?
What else would you like to learn about writing these kinds of assessment questions? Do you have questions I could answer in a future post? Let me know in the comments.