Inter-rater Agreement is the degree to which two raters using the same scale give the same rating in identical situations.
Inter-rater Reliability is the relative similarity between two or more sets of ratings.
- Reliable raters agree with the "official" rating of an evaluation example.
- Reliable raters agree with each other about the exact ratings to be awarded.
- Reliable raters agree about which evaluation example is better and which is worse.
Inter-rater Agreement is essential to:
- Bridge the credibility gap.
- Train raters.
- Monitor system performance.
- Link professional development to evaluation results.
Within the evaluation process, Inter-rater Agreement is used during:
- Teacher and Building Leader observations.
- Teacher and Building Leader artifact reviews.
- Teacher and Building Leader summary ratings #1 and #2.
- Teacher and Building Leader final summative evaluation ratings.
Inter-rater Agreement trainings at the district level need to include:
- Team discussions and agreements about Inter-rater Agreement.
- Team review of data collection and analysis.
- Training design at preservice and inservice levels.
- Regular training check-ups.
- Coaching and observation.
- Monitoring ratings.