Job Training: "Assessment Development First Steps"
The following post gives an overview of an instructional design course. For access to the full course, you can download it and view it as a Microsoft PowerPoint slideshow by clicking the link below.
I strongly recommend not opening it through Google Slides or going through the slides outside of the presentation mode.
NOTE: The content and setting of this course was based on experience working in English language standardized assessment at Edutech Lab Inc. However, it was not made for or used by Edutech Lab. The course's content includes information that is standard practice within language assessment rather than being specific to this company.
Background
This portfolio item consists of a lesson for new employees and part-time employees at a standardized testing company. These individuals will utilize this training as part of their first test development assignments.
Their introductory assignment will begin with learning the basic terminology and concepts of English as a Second Language (ESL) assessment writing and then reviewing the manual and resources provided for the specific part of the test they have been assigned. They will then participate in an editing meeting with the item lead (the assessment specialist in charge of this section of the test) and another member of the item team. Finally, they will begin writing their own items and be given feedback on their work from the item team.
Training Needs Analysis
Project Overview: Assessment Development First Steps
Project Description:
Provide training to new and junior assessment specialists to gain enough tools to understand item type manuals and actively participate in an item editing meeting.
Objectives:
Writers will be able to communicate about individual parts of an item and identify common issues with questions and response alternatives across item types.
Audience:
New full-time assessment specialists come from a background in English as a Second Language Education. They likely have an understanding of examinee proficiency levels, but they may not have written tests at the quality level required for standardized assessment.
Junior assessment specialists spend a majority of their time on rating examinee responses rather than writing new tests. They may have some understanding of examinee levels for productive skills (writing and speaking), but do not have a clear understanding of receptive skills (reading and listening). They have experience in academic writing, but have not written tests before.
Constraints:
- The training should consist of a single lesson that writers can guide themselves through.
- Training should contain examples that can be generalized to various item types.
- At a minimum, writers should attain a level of understanding that will allow them to follow discussions in an editing meeting.
- Given the small size of the company, we cannot invest resources into additional software to develop or deliver the training materials.
Research Findings:
Current training at this standardized company involves giving item writing assignments to writers and expecting them to complete these assignments solely with the use of item manuals and resources. After they have completed their assignment, they are given feedback on some items and asked to rewrite items that require extensive changes.
This leads to slow development for all writers, but in particular, for the junior employees who may not work on item writing again for over a month between assignments. Full-time assessment specialists on the other hand, have contact with item development in the writing, editing, and proofreading stages of development before being assigned more writing.
Additionally, junior writers frequently only communicate with editing teams when they are being given feedback or asked for rewrites. This has led to an increase in junior writers who voice anxiety about writing assignments or who request to focus solely on their exam rating duties.
Final Product Overview
The training is a self-guided lesson using Microsoft PowerPoint. It contains 3 sections: introduction and terminology, writing questions, and writing response alternatives (answers).
Since more interactive technology was not available, navigation through the course was designed to be more interactive rather than simply reading slides passively. This allows learners to skip past information they already know and engage more actively with the content. For example, in the introduction to terminology, employees can click each term to get a "pop-up" with more information.
Similarly, rather than reading a list of definitions for question types, employees can select the types they are interested in to see pop-ups with information and examples.
Other sections engage learners by asking them to consider a problem before going on to the answers.
