評分者趨中效應指標表現效果探討

Abstract

本研究目的在於探討五種不同的指標在判斷評分者趨中效應程度時的表現效果並提出使用建議。本研究分成兩個部分，包含模擬研究及實證研究，模擬研究部分的三個自變項分別為五種不同的指標、評分者評分樣本數和評分者嚴苛度的程度；依變項為不同指標的準確性、敏感性以及不同指標基於的IRT模式在估計受試者能力值時的精準度。實證研究部分，分析不同指標在偵測評分者趨中效應程度時結果是否一致，並針對不一致的結果進行分析討論。 本研究結果顯示無論評分樣本數多少，五種指標的準確性排序結果都相同。五種指標的表現效果會隨著評分樣本數的增加而愈好，且當評分者嚴苛度適中時，五種指標的表現均有所提升（相較於評分者是寬鬆或嚴苛時），在對受試者能力估計時，兩種模式存在一定差異。因此建議在判斷評分者趨中效應程度需使用何種指標依照評分樣本數和評分者嚴苛度作為判斷依據：當評分者樣本數較少時（例如本研究採50人），無論評分者嚴苛度為何，建議使用基於MFRM模式的r_(measure,res)和r_(exp,res)；當評分者樣本數較多時（例如本研究採100人），若評分者嚴苛度適中，則r_(measure,res)、r_(exp,res)、r_(score,measure)、SD和基於MF-RC模式的ω_k'均表現良好；但若評分者較寬鬆或嚴苛時，建議使用r_(measure,res)、r_(exp,res)和ω_k'。建議後續研究可以探討不同指標的切截分數。

This research illustrates effects of four indicators in the framework of IRT and one indicator in the framework of Non-IRT to detect the rater’s centrality by using simulation methods. This research includes two parts: The simulation study and empirical study. In simulation study, three independent variables are used including five different indicators、the number of ratees and the severity of raters. Dependent variables are Spearman's rank correlation coefficient which is used to judge the effects of indicators 、RMSE for two IRT models which can be used to judge the accuracy of ability estimations. In empirical study, five indicators are used to judge which raters are of high rank of centrality, some discussions are provided if the conclusions are inconsistent while using different indicators. The results show that when the numbers of rating samples are growing the effects of five indicators tend to be better, when rater’s severity is moderate consequences are better as well. Besides there are differences between using MFRM model and MF-RC model to estimate the abilities of students. There are some advice for using indicators to detect the rater’s centrality: If the rating samples are too small(e.g., 50), r_(measure,res) and r_(exp,res) are recommended. If the rating samples are large(e.g., 100), when rater’s severity is moderate, r_(measure,res)、r_(exp,res)、r_(score,measure)、SD and ω_k' which is based on MF-RC model are all performing acceptable, however when rater’s severity is too lenient or harsh, then r_(measure,res)、r_(exp,res) and ω_k' are recommended. Some researches about the cut score of indicators to detect the rater’s centrality can be done in the future.