The 35th Language Testing Research Colloquium
 
Untitled Document
 
 
Title: Rating Scales and Raters in Speaking Assessment
Date: Monday, July 1, 2013
Instructors: Dr. Gad S. Lim & Dr. Evelina D. Galaczi, Cambridge English Language Assessment
 
Performance assessment of speaking ability requires assessment criteria for scoring performances and raters to apply those criteria to performances. This workshop focuses on these two aspects of speaking assessment.
 
First, the workshop will use a range of activities for participants to explore the principles that underpin and inform the development of rating criteria and scales (e.g. comparison of various assessment scales, writing and critiquing descriptors, case studies of scale and criteria development, interpreting statistical output). Topics covered include determination of relevant criteria vis-á-vis test construct, different types of scales and their respective advantages, qualities of descriptors, guidelines for dealing with special cases, considerations of usability and impact, and methods (intuitive, qualitative, quantitative) for developing and validating rating scales.
 
Second, the workshop will consider the rating of speaking performances by human assessors and the issues that surround this activity. The discussion will cover various models of the marking process and of related issues including rater bias, severity and consistency, halo effect and central tendency, rating when the rater is also the interlocutor, rater expertise, differences in rating using different types of assessments scales. The workshop will also address rater training procedures and processes for assuring rating quality.
 
Throughout the workshop, emphasis will be given to context as a primary determinant of which approaches and options are most suitable for the purpose, and participants will consider how different contexts (e.g. standardized testing, classroom assessment, formative and diagnostic assessment) will result in the ideas and principles being applied and expressed differently.
 
Agenda:
9:00 a.m.- 10:15 a.m.Fundamental concepts and considerations in the development of assessment criteria and scales
10:30 a.m.- 12:30 p.m.Methods (intuitive, qualitative, quantitative) for developing and validating assessment scales
1:30 p.m. - 2:45 p.m.Fundamental concepts and considerations in rating speaking ability
3:00 p.m. - 5:00 p.m.Procedures for training markers and assuring rating quality
9:00 a.m.- 10:15 a.m.Fundamental concepts and considerations in the development of assessment criteria and scales
10:30 a.m.- 12:30 p.m.Methods (intuitive, qualitative, quantitative) for developing and validating assessment scales
1:30 p.m. - 2:45 p.m.Fundamental concepts and considerations in rating speaking ability
3:00 p.m. - 5:00 p.m.Procedures for training markers and assuring rating quality

About the Instructors:
Dr. Gad S. Lim is Senior Research and Validation Manager at Cambridge English Language Assessment, where he leads research on performance assessment of the productive language skills. In this capacity, he led the development of new writing assessment scales for the Cambridge suite of exams, as well as the quality assurance and training process for examiners. Previously, he worked at the testing division of the University of Michigan English Language Institute, where he was part of the team that developed assessment criteria for ECCE Writing, and managed the production of speaking tests. He has taught and trained teachers in Asia, North America, and Europe, has published in Language Testing, International Journal of Testing, and in various other outlets, and is writing a book on raters and rating scales in writing and speaking assessment.
 
Dr. Evelina D. Galaczi is Principal Research and Validation Manager at Cambridge English Language Assessment. Her current role involves leading on speaking research and draws on her extensive experience as a researcher in speaking assessment and as an ESOL teacher, teacher-trainer and program administrator. She also holds the role of Managing Editor of the Studies in Language Testing series. Evelina has authored and co-authored works in the Language Assessment Quarterly and Assessment in Education journals and in Examining Speaking. Her research interests include scale development, rater training, and qualitative approaches to assessment research. She is a current member of the Language Assessment Quarterly Editorial Advisory Board and is a Coordinating Committee Member of the BAAL Testing, Evaluation and Assessment SIG.
 
 
Untitled Document
 
Untitled Document