Honour thesis assessment: the role of guidelines in achieving inter-rater agreement

Engineering honours projects are often self selected or negotiated by students to be in areas of their own interest. While this encourages motivation and engagement in a self directed research project, it also considerably increases the diversity of honours project types. Such project diversity raises questions about the most suitable form of guidelines to provide good inter-rater agreement. Project diversity can also result in academics assessing projects outside their primary area of research specialisation. This is particularly true of transdisciplinary projects that cross over conventional discipline boundaries. To investigate these issues, a team of academics from two Schools at the University of Tasmania assessed a collection of engineering theses using a variety of different guidelines. All guidelines were found to produce poor inter-rater agreement, however inter-rater agreement was improved when both assessors were of the same discipline. An account of the academics’ comments on use of the guidelines reveals conflicting opinions of good and bad features. Guidelines that were viewed as easy to use and less subjective were found not to substantially improve inter-rater agreement. The implications of these findings are discussed and suggestions made in relation to improving assessment guidelines for honours theses.