BMC Session 4: Quizzes that tell you something: Guttman analysis in the age of GenAI

|

The Future-Ready Assessment Ecosystem: Institutions

 

Assessment organisations are navigating rapid technological change, evolving learner expectations and increasing demands for efficiency, quality and governance. Building a future-ready assessment ecosystem requires institutions to rethink how assessment is designed, delivered and managed.

 

This week focuses on the organisational challenges and opportunities shaping modern assessment. Sessions will explore AI in assessment operations, strategic planning, data-driven decision making and AI governance, providing practical insights into how institutions can build resilient, responsible and scalable assessment systems.Generative AI (GenAI) can produce the products of thinking with   minimal cognitive effort from the student, and emerging evidence shows that   students who use it as a substitute for thinking perform worse once access is   removed (Bastani et al., 2025; Stromberg et al., 2026). Educators therefore   need evidence of where students are in their learning, not just what they can   produce (Masters, 2013).

About This Session

Quizzes are widely used but vulnerable to GenAI and need not be abandoned.   Repositioned as diagnostic instruments that reveal readiness to learn rather   than reward performance, they retain their value, provided the inferences   drawn from them are valid (Dawson et al., 2024).

Guttman analysis (Guttman, 1950), adapted for classroom use by Griffin et   al. (2017), orders response data by item difficulty and student performance.   The resulting chart shows where learning is clustered and what students are   ready to learn next, their zone of proximal development (Vygotsky, 1978).   Uptake in higher education appears limited, constrained by workload (Datnow   & Park, 2018) and by the interpretive demands of reasoning diagnostically   from data (Mandinach & Gummer, 2016).

This session addresses both barriers by demonstrating two pedagogy-informed   prompts. One generates diagnostic quiz items from a rubric or developmental   assessment framework (Griffin et al., 2017) and item-writing guidelines   (Haladyna et al., 2002; Scully, 2017). The other generates a Guttman chart   from response data. Both are structured through context, epistemic grounding,   procedure and reflexivity, on the premise that output quality depends on the   frameworks that instruct it. GenAI undertakes the drafting and data   reordering; the educator retains judgement over validity and teaching   decisions. A worked example models interpretation of the chart, and the   prompts and a sample dataset are provided. The approach is equally applicable   in K–12 settings.