Computerized standardsetting using 3DC
The Data Driven Direct Consensus procedure (3DC) is a new and very flexible method for standard setting developed at Cito. As the name implies, it borrows considerably from the Direct Consensus procedure but it adds psychometrical rigour and guidance for the panellists by providing extra information about the relative difficulty of the clusters in relation to each other and to the test score.
Efficient and innovative standardsetting
The 3DC method can be used for both dichotomously and polytomously scored item types. Cito offers a free user-friendly application to apply the 3DC method digitally.
How does it work?
In a typical workflow, a psychometrican constructs an ability scale based on assessment data. Items are then grouped in clusters of similar items. The 3DC application then constructs a digital form, on which participants of a standard setting session can indicate where they think the standard should be set.
A number of content experts are then invited to a standard setting session. For this session, a small network of laptops is set up, often using the panellists own laptops. A group leader guards and guides this process. The group leader is supported by helpful overviews and graphics in the application.
The application also allows participants to browse through and view the items in each cluster. It supports items in QTI, html or image format.
See also:
- Keuning, J. Straat, J.H. & R.C.W. Feskens (in press). The Data-Driven Direct Consensus (3DC) Procedure: A New Approach to Standard Setting. In “Standard setting: International state of research and practices in the Nordic countries”, Springer “Educational Measurement” series (eds. Veldkamp & von Davier)
Chapter - Standard Setting in Education