Machine learning for the peer assessment credibility

The peer assessment approach is considered to be one of the best solutions for scaling both assessment and peer learning to global classrooms, such as MOOCs. However, some academic staff hesitate to use a peer assessment approach for their classes due to concerns about its credibility and reliability. The focus of our research is to detect the credibility level of each assessment performed by students during peer assessment. We found three major scopes in assessing the credibility level of evaluations, 1) Informativity, 2) Accuracy, and 3) Consistency. We collect assessments, including comments and grades provided by students during the peer assessment process and then each feedback-and-grade pair is labeled with its credibility level by Mechanical Turk evaluators. We extract relevant features from each labeled assessment and use them to build a classifier that attempts to automatically assess its level of credibility in C5.0 Decision Tree classifier. The evaluation results show that the model can be used to automatically classify peer assessments as credible or non-credible, with accuracy in the range of 88%.