reading

Cover
Title Page
Notes on Contributors
Foreword
Acknowledgements
1 Introduction to Handbook
1. Motivation for Handbook
2. Handbook Structure
3. Closing Words
Part I: Frameworks
1. 2 The Role of Theories of Learning and Cognition in Assessment Design and Development
  1. A Brief History of Evaluation Criteria for Theories of Learning and Cognition
  2. Principled Assessment Design as an Evaluative Lens for Theories of Learning and Cognition
  3. Principled Assessment Design Evaluation Criteria for Theories of Learning and Cognition
  4. Interface of Evaluation Criteria with Validity
  5. Conclusion
  6. References
2. 3 Principled Approaches to Assessment Design, Development, and Implementation
  1. Motivations for Principled Approaches
  2. Overviews of Principled Approaches
  3. Elements of Principled Approaches
  4. Use of the Elements in Principled Approaches
  5. Discussion
  6. Acknowledgments
  7. References
3. 4 Developing and Validating Cognitive Models in Assessment
  1. Definition of Cognitive Models
  2. Model Development and Validation Methods
  3. Offline Evidence: Methods and Approaches
  4. Conclusion
  5. References
4. 5 An Integrative Framework for Construct Validity
  1. Unified Framework for Construct Validity
  2. Example: Test Form Development with Generated Items
  3. Conclusion
  4. References
5. 6 The Role of Cognitive Models in Automatic Item Generation
  1. Introduction
  2. The Relentless Demand for Test Items
  3. Automatic Item Generation: A Three‐Step Method
  4. Identifying Content for Item Generation Using Cognitive Modeling
  5. Evaluating the Cognitive Models Used in Automatic Item Generation
  6. Summary and Directions for Future Research
  7. Acknowledgments
  8. References
6. 7 Social Models of Learning and Assessment
  1. Introduction
  2. Defining “Social” Models of Learning
  3. Examples of Assessment Informed by Social Models of Learning
  4. Example 1: Carbon: Transformations in Matter and Energy (Carbon TIME)
  5. Example 2: Contingent Pedagogies
  6. Goals for Learning and their Justification
  7. Example 3: Data Modeling
  8. Comparing the Examples: Key Sources of Variation
  9. Looking to the Future: Additional Considerations for Assessment from a Sociocultural Perspective
  10. References
7. 8 Socio‐emotional and Self‐management Variables in Learning and Assessment
  1. Introduction
  2. Importance of Social, Emotional, and Self‐Management Variables in School and in Life: Correlational Studies
  3. A Social, Emotional, and Self‐Management Framework
  4. Role of Socio‐Emotional, Attitudinal, and Self‐Management Variables on Summative and Formative Test Scores
  5. Assessing Socio‐Emotional, Attitudinal, and Self‐Management Variables
  6. Conclusions and Future Directions
  7. References
8. 9 Understanding and Improving Accessibility for Special Populations
  1. Accessibility of Educational Assessments
  2. Cognitive Models of Learning and Performance
  3. Putting It All Together: Assessing Accessibility for Individual Students
  4. Conclusion
  5. References
9. 10 Automated Scoring with Validity in Mind
  1. An Overview of Automated Scoring Applications
  2. Automated Scoring with Validity in Mind
  3. Design Decisions in Automated Scoring
  4. Discussion
  5. Acknowledgments
  6. References
Part II: Methodologies
1. 11 Explanatory Item Response Models
  1. Explanatory Item Response Models: An Approach to Cognitive Assessment
  2. EIRM Framework and Modeling Principles
  3. Data Set and Research Questions
  4. Software and Model Specification
  5. Example 1: Test Construction and Item Generation
  6. Example 2: Diagnostic Assessment
  7. Example 3: Correction for Confounding
  8. General Discussion
  9. Acknowledgments
  10. References
2. 12 Longitudinal Models for Repeated Measures Data
  1. Reading Comprehension Data
  2. Latent Growth Curve Models
  3. Growth Mixture Models
  4. Conclusions
  5. References
3. 13 Diagnostic Classification Models
  1. Technical Foundations for DCMs
  2. Evaluating Cognitive Theories through DCM Criticism and Refinement
  3. Design Considerations for Diagnostic Assessments
  4. Concluding Remarks
  5. References
4. 14 Bayesian Networks
  1. Evidentiary Motivations and Logical Bases
  2. Statistical Definition and Estimation
  3. Building and Using a BN
  4. Understanding Models and Results across Communities
  5. Conclusion
  6. References
5. 15 The Rule Space and Attribute Hierarchy Methods
  1. Test Design with the RSM and the AHM
  2. Statistical Pattern Classifications with the RSM and the AHM
  3. Demonstration of RSM and AHM Using a Real Data Example
  4. Summary and Future Directions
  5. References
6. 16 Educational Data Mining and Learning Analytics
  1. Emerging Complex Assessment and Learning Environments
  2. Case Studies in Assessment of Cognition via EDM
  3. Conclusions
  4. References
Part III: Applications
1. 17 Large‐Scale Standards‐Based Assessments of Educational Achievement
  1. Challenges of Using Cognitive Modelsfor Large‐Scale Assessments
  2. Examples of Best Practices from Current Testing Programs
  3. Conclusions and Recommendations
  4. References
2. 18 Educational Survey Assessments
  1. Introduction
  2. Building Cognitive Models
  3. The Role of Cognitively‐based Assessment in ESAs
  4. Definition and Guiding Examples
  5. Three Methodological Perspectives
  6. Summary, Trends and Upcoming Development
  7. References
3. 19 Professional Certification and Licensure Examinations
  1. Overview of Certification and Licensure
  2. Assessment Frameworks for Certification and Licensure
  3. Meeting Practical Certification and Licensure Assessment Challenges
  4. Conclusions
  5. References
4. 20 The In‐Task Assessment Framework for Behavioral Data
  1. Introduction
  2. Evidence‐Centered Design
  3. In‐task Assessment Framework
  4. Example I‐TAF Instantiations
  5. Leveraging the I‐TAF Framework to Instantiate ECD Models
  6. Discussion
  7. Acknowledgments
  8. References
5. 21 Digital Assessment Environments for Scientific Inquiry Practices
  1. Foundations for the Design of the Inq‐ITS System
  2. Case Studies with Think‐Aloud Components
  3. The Inq‐ITS System
  4. Summary
  5. References
6. 22 Assessing and Supporting Hard‐to‐Measure Constructs in Video Games
  1. Hard‐to‐measure Constructs and Why They are Hard to Measure
  2. Well‐designed Games as Vehicles for Assessing and Supporting These Constructs
  3. Evidence‐centered Design and Stealth Assessmentin Well‐designed Games
  4. Examples of Stealth Assessments of Hard‐to‐Measure Constructs in Two Games
  5. Validation of In‐Game Measures
  6. Discussion
  7. Acknowledgments
  8. References
7. 23 Conversation‐Based Assessment
  1. Performance‐Based and Conversation‐Based Assessments
  2. An Augmented Taxonomy for Assessment Tasks
  3. An Illustration of Conversation‐Based Assessments
  4. Discussion
  5. References
8. 24 Conclusion to Handbook
  1. Starting Points for Reflection
  2. Final Words
  3. References
Glossary
Index
End User License Agreement

List of Tables

Chapter 02
1. Table 2.1 Content features of tasks that elicit evidence on students’ understanding of shape composition/decomposition, length, and area measurement.
2. Table 2.2 Typical learner performance in each of six stages from the “area” strand of the LP.
Chapter 03
1. Table 3.1 Foundation and organizing elements of principled approaches to assessment design, development, and implementation and their relationship to the assessment triangle.
2. Table 3.2 Foundation elements of five principled assessment design, development, and implementation approaches.
Chapter 04
1. Table 4.1 Research phases in a large‐scale operational assessment program.
Chapter 05
1. Table 5.1 Prediction of item difficulty in generated bank of 150 ART items.
2. Table 5.2 Number of items by memory load and object display type.
3. Table 5.3 Descriptive statistics for item features in generated ART‐E1 items.
Chapter 06
1. Table 6.1 Five generated items produced using the “Structures” cognitive model in Figure 6.2.
2. Table 6.2 Six generated items produced using the “Jaundice” cognitive model in Figure 6.4
Chapter 07
1. Table 7.1 Key features for three distinctive examples of sociocognitive models of learning and assessment.
Chapter 08
1. Table 8.1 Noncognitive skills framework.
Chapter 09
1. Table 9.1 Dimensions of accessibility on the mathematics problem solving assessment for Susana.
Chapter 11
1. Table 11.1 Example Q‐matrix for basic addition items.
2. Table 11.2 Illustration of long data form.
3. Table 11.3 Goodness of fit statistics of models used across all chapter examples.
4. Table 11.4 Estimated covariate effects and standard errors (SE) for Models 2 and 3.
5. Table 11.5 Estimated success probabilities for three sample students.
Chapter 12
1. Table 12.1 Cross‐classification of longitudinal statistical models.
2. Table 12.2 Fit indices for a series of unconditional GMMs.
Chapter 13
1. Table 13.1 Example Q‐matrix.
Chapter 14
1. Table 14.1 Conditional probability table for p(cpc₁|CPC).
2. Table 14.2 Conditional probability table for p(cpc₂|CPC).
3. Table 14.3 Conditional probability table for p(CPC).
Chapter 15
1. Table 15.1 Expected response patterns for the hierarchy shown in Figure 15.1.
2. Table 15.2 The Q‐matrix for the fraction subtraction example.
3. Table 15.3 The attribute pattern (AP) and expected item response pattern (EP) matrix for the fraction subtraction example.
4. Table 15.4 Inputs and outputs used to train the MLP.
Chapter 17
1. Table 17.1 Materials reviewed for chapter.
2. Table 17.2 Excerpt from NAEP‐TEL specifications (Table 3.1).
3. Table 17.3 Example of AP biology learning standards (p. 11).
4. Table 17.4 Excerpt from the NYSTP‐CC Standards Interpretation Guide (ELA Writing, Grade 6).
5. Table 17.5 Excerpt of PARCC evidence statements for ELA Writing for Grade 6.
6. Table 17.6 Excerpt from NYSTP‐CC Regents Algebra Exam Standards Interpretation Guide.
Chapter 19
1. Table 19.1 Task model grammar statement components.
Chapter 20
1. Table 20.1 Conditional probability of action set label occurrence given establishing shared understanding.
2. Table 20.2 Correctness of establishing shared understanding by student and task level.
Chapter 21
1. Table 21.1 Next generation science standards.
Chapter 22
1. Table 22.1 Conditional probability table for indicator no. 8 and specific skill in Level 9.
Chapter 23
1. Table 23.1 Example responses from Comparing Notes conversation.

List of Illustrations

Chapter 02
1. Figure 2.1 Using principled assessment design to foster coherence in the assessment triangle and support of a validity argument.
2. Figure 2.2 Learning progression for “geometric measurement of area.”
3. Figure 2.3 The validity argument for the interpretation of assessment results based on the “geometric measurement of area” learning progression.
Chapter 03
1. Figure 3.1 Logical and procedural relationship between two foundation elements in principled design: specifying assessment targets and identifying intended test score interpretations and uses simultaneously.
2. Figure 3.2 Process model for principled assessment design, development, and implementation to support intended interpretations and uses of test scores.
3. Figure 3.3 Conventional processes (white boxes) and processes based on principled approaches (foundation elements are numbered in the three boxes with grey background) for assessment design, development, and implementation showing overlap and differences.
Chapter 04
1. Figure 4.1 A development and validation process for cognitive models in assessment.
2. Figure 4.2 Scanpath of one reader in the multiple choice‐only condition of Feng et al. (2012) applying a piecemeal reading strategy to answer the first multiple choice question. Circles indicate fixation locations, with numbers indicating their temporal sequence. Information necessary to answer the question is marked in gray.
3. Figure 4.3a Eye gaze heatmap for students who correctly identified the main idea in the summary (Feng et al., 2013).
4. Figure 4.3b Eye gaze heatmap for students who failed to identify the main idea in the summary (Feng et al., 2013).
Chapter 05
1. Figure 5.1 Relationship between aspects of construct validity and other variables.
2. Figure 5.2 Item from ART.
3. Figure 5.3 Cumulative percentages of scores on the ART‐E1 and CFIQ by race‐ethnicity.
Chapter 06
1. Figure 6.1 A general cognitive model for automatic item generation.
2. Figure 6.2 A cognitive model for the “Structures” science concept.
3. Figure 6.3 Item model for the “Structures” cognitive model in Figure 6.2.
4. Figure 6.4 A cognitive model for managing jaundice.
5. Figure 6.5 Item model for the “Jaundice” cognitive model in Figure 6.4.
Chapter 08
1. Figure 8.1 Heckman‐Kautz model of cognitive test performance.
Chapter 09
1. Figure 9.1 Interaction between students’ personal characteristics and features of educational assessments influence accessibility.
2. Figure 9.2 Guiding questions to consider when identifying students’ personal characteristics.
3. Figure 9.3 Understanding the consequences of inaccessibility on the accuracy of measurement.
4. Figure 9.4 Framework for integrating cognitive models of learning and performance in test development to improve accessibility.
5. Figure 9.5 Template for creating test specifications that align with purpose and targeted construct.
Chapter 10
1. Figure 10.1 Three main models of the CAF in ECD.
Chapter 11
1. Figure 11.1 Boxplots comparing item difficulties estimated in four ways.
Chapter 12
1. Figure 12.1 Flow chart depicting a principled data‐analytic process for LGM analyses.
2. Figure 12.2 Prototypical path diagram for a linear LGM.
3. Figure 12.3 Schematic of a generic Michaelis‐Menten function with three growth factors.
4. Figure 12.4 Schematic of the interaction effect between syntactical skills, language status, and growth factor.
5. Figure 12.5 Flow chart depicting a principled data‐analytic process for GMM analyses.
6. Figure 12.6 Prototypical path diagram for a GMM with Michaelis‐Menten function.
7. Figure 12.7 Elbow plot for reading comprehension data.
8. Figure 12.8 Mean growth trajectories in three‐class GMM for reading comprehension data.
9. Figure 12.9 Individual and mean growth trajectories in three‐class GMM for reading comprehension data (fall of second grade to spring of fifth grade, with assigned middle school reading remediation status).
Chapter 13
1. Figure 13.1 Item discrimination () and difficulty (p). This figure shows the probability of correct response conditional on mastery status – mastery or non‐mastery – for four simple structure items. Each bar shows the predicted probability of correct response for the corresponding class, and the dashed line indicates the p‐value, or overall proportion of examinees who answered the item correctly. Item discrimination is the difference in predicted probability of correct response for the two classes of examinees.
2. Figure 13.2 Example of complex item from the Examination for the Certificate of Proficiency in English. This figure shows the attribute pattern‐specific log‐odds (left) and probability (right) of correct response for an example item that measures Attribute 1 (morphosyntactic rules) and Attribute 3 (lexical rules). The attribute patterns are indicated by [α₁α₂α₃α₄] where if attribute a is mastered in the pattern; if attribute a is not mastered in the pattern; and . if the item does not measure attribute a, meaning the response probability is the same for all values of α_a, making the mastery status of a irrelevant for this item.
3. Figure 13.3 Example items from the Diagnosing Teachers’ Multiplicative Reasoning Fractions test. This figure shows the attribute pattern‐specific log‐odds (left) and probability (right) of correct response for two example items. The attribute patterns are indicated by [α₁α₂α₃α₄] where if attribute a is mastered in the pattern; if attribute a is not mastered in the pattern; and . if the item does not measure attribute a, meaning the response probability for the item is the same for all values of α_a, making the mastery status of a irrelevant for this item.
4. Figure 13.4 Comparison of results for two attributes structures of English grammar attributes. This figure shows the attribute structure and results when the Examination for the Certification of English Proficiency’s (ECPE) three attributes were modeled with the LCDM imposing no hierarchy compared to the HDCM imposing a linear hierarchy.
5. Figure 13.5 Structure for mathematics attributes. On the left, this figure shows the original attribute structure for a set of elementary mathematics attributes, which included no attribute dependencies. The attributes are addition (Add), subtraction (Sub), multiplication (Mul), division (Div), and modeling (Mod). The pattern of results shown in the middle illustrates that the data reflect a possible attribute hierarchy, which is depicted on the right.
6. Figure 13.6 Sample examinee‐level diagnostic information.
7. Figure 13.7 Sample examinee mastery base rate information for multiplicative reasoning attributes. The percentages of examinees exhibiting mastery for each attribute are labeled on the pie charts.
Chapter 14
1. Figure 14.1 Example tasks providing evidence for the CPC claim.
2. Figure 14.2 Simple BN for relating CPC to performance on two tasks, cpc₁ and cpc₂.
3. Figure 14.3 Two BNs for two students, Alice and Bob, relating CPC to performance on two tasks, cpc₁ and cpc₂.
4. Figure 14.4 BN generalization of Figure 14.3 to allow an arbitrary number of students.
5. Figure 14.5 BN generalization of Figure 14.3 to allow for an arbitrary number of students and tasks.
6. Figure 14.6 BN generalization of Figure 14.5 to allow different model structure extensions.
7. Figure 14.7 Two‐dimensional BN for an arbitrary number of students.
8. Figure 14.8 BN extension of Figure 14.7 with structural parameters.
9. Figure 14.9 Complex BN for multiple mathematical skills, tasks, and students.
10. Figure 14.10 BN representations for knowledge tracing and performance factor analysis models.
11. Figure 14.11 BN representations of models for text analyses.
Chapter 15
1. Figure 15.1 A six‐attribute hierarchy.
2. Figure 15.2 Hypothesized cognitive model for fraction subtraction example.
3. Figure 15.3 Rule space plot of expected response patterns.
Chapter 18
1. Figure 18.1 Screenshot of a scene in the “Wells” task developed as part of the NAEP TEL assessment.
2. Figure 18.2 Screenshot of a scene in the “Mystery Plants” task developed as part of the NAEP Science ICTs.
Chapter 19
1. Figure 19.1 Three IRT models under an examinee‐centered perspective.
2. Figure 19.2 A hierarchical IRT model for item families differing in cognitive demands.
3. Figure 19.3 A sample task model map for a certification or licensure test.
Chapter 20
1. Figure 20.1 Conceptual assessment framework within evidence‐centered design.
2. Figure 20.2 Generic ontology fragment.
3. Figure 20.3 Generic behavioral ontology fragment.
4. Figure 20.4 Generic cognitively enhanced ontology fragment.
5. Figure 20.5 Generic in‐task feature.
6. Figure 20.6 Generic chain‐of‐evidence.
7. Figure 20.7 Screenshot of the “Ohm’s Law” task.
8. Figure 20.8 Electronic circuits ontology fragment.
9. Figure 20.9 Example feature for adding volts.
10. Figure 20.10 Example chain‐of‐evidence for “Ohm’s Law”.
11. Figure 20.11 Collaborative problem solving ontology fragment.
12. Figure 20.12 Example feature for sharing information.
13. Figure 20.13 Example chain‐of‐evidence for establishing shared understanding.
14. Figure 20.14 Screenshot of the “Tetralogue” task.
15. Figure 20.15 Modified collaborative problem‐solving ontology fragment.
16. Figure 20.16 Modified feature for sharing information.
17. Figure 20.17 Modified chain‐of‐evidence for establishing shared understanding.
Chapter 21
1. Figure 21.1 Assessment report for teachers reported out by science inquiry practice and sub‐skills.
2. Figure 21.2 Screenshot of the Inq‐ITS system with all phases of inquiry shown. Students generate a question, run trials to test it, interpret data, and select trials to warrant claims with evidence.
Chapter 22
1. Figure 22.1 Viral Internet image, circa 2014.
2. Figure 22.2 Stealth assessment cycle.
3. Figure 22.3 Screen capture of Plants vs. Zombies 2.
4. Figure 22.4 Competency model for problem solving in Plants vs. Zombies 2; [R] means reverse coded.
5. Figure 22.5 Initial Bayes net model with prior probabilities of problem solving for level 9 in Plants vs. Zombies 2.
6. Figure 22.6 Competency model of creativity and sample indicators; [R] indicates reverse coded.
7. Figure 22.7 “Swamp People” level in Physics Playground (a) with typical ramp solution (b), less common springboard solution (c), and rare lever solution (d).
8. Figure 22.8 Bayes net fragment for creativity for “Swamp People” level.
9. Figure 22.9 Ramp solution (typical) to the “Swamp People” level.
10. Figure 22.10 Springboard solution (less common) to the “Swamp People” level.
11. Figure 22.11 Lever solution (rare but effective) to the “Swamp People” level.
12. Figure 22.12 Pendulum solution (unusual and ineffective) to the “Swamp People” level.
13. Figure 22.13 Physics Playground dashboard showing score board per agent of force and motion.
Chapter 23
1. Figure 23.1 Sample items on openness and interactivity dimensions. MC = multiple‐choice, CR = constructed response, CBA = conversation‐based assessment.
2. Figure 23.2 Task flow for Volcano scenario with grey squares representing CBAs.
3. Figure 23.3 Screenshot during Comparing Notes conversation.
4. Figure 23.4 Screenshot during Evaluating a Prediction conversation.
5. Figure 23.5 Fragment of a conversation space diagram for the Volcano scenario.

The Handbook of Cognition and Assessment

Frameworks, Methodologies, and Applications