Technical Notes: A.3 Test development
The development of the assessment instruments is an interactive multi-step process that involves representatives of the participating jurisdictions, various expert committees, and an international consortium of contractors. Generally, the first step is to develop or revise a framework to guide the construction of the assessment. Items that fit within the framework are submitted by participating jurisdictions as well as developed by each study's international consortium of test developers. Potential items are reviewed by representatives of each jurisdiction for possible bias and relevance to the study's framework. All items are meant to reflect the national, cultural, and linguistic variety among participating jurisdictions. Approved items are field-tested, and items that perform well are identified for inclusion in an "item pool" that is used to create the actual or "main" assessment instruments.
Design of instruments
The main assessment instruments consist of (1) test booklets made up of instructions and blocks or clusters of items (small sets of items from the final pool of items) and (2) questionnaires for students, schools, teachers (in PIRLS and TIMSS), and parents (in PIRLS and PISA, although the United States has not administered the parent questionnaire). The test booklets for PIRLS, PISA, and TIMSS are constructed such that no student responds to all of the items. This construction is consistent with other large-scale assessments, such as the U.S. National Assessment of Educational Progress (NAEP). To keep the testing burden to a minimum, and to ensure broad subject-matter coverage, the assessments use a "rotated block design" that includes items from different content domains and, in the case of PISA and TIMSS, different subject areas. The number of booklets for each assessment varies as does the amount of time students have to complete each booklet they receive.
In each assessment, a portion of the items are identical to items used in the prior administration of the assessment. These items allow a study to "maintain trend" (i.e., be able to make reliable comparisons among the results of different administrations of an assessment over time) and to provide for corrections through equating, if necessary.
Source versions of all instruments are prepared in English and, in the case of PISA, in French as well. These are translated into the primary language or languages of instruction in each participating jurisdiction. In addition, it is sometimes necessary to adapt the instrument for cultural purposes, even in nations such as the United States that use English as the primary language of instruction. (For example, British terms such as "lift" might be replaced with "elevator" in the U.S. version.) The national translation and adaptation of all instruments are reviewed and approved by the sponsoring organization of each international study to ensure that the translations and adaptations did not change the substance or intent of the question or answer choices.