Interrater Agreement Resources

Inter-rater Agreement Overview Video by Scott Myers, Director of TLA, KSDE 

KSDE is working to add the following resources to this webpage:

1.       An Interrater Agreement Guide book – Including teaching tools for various audiences

2.       Inter-Rater Agreement In Kansas Abilene 7-15-2014.pptx

3.       Evidence/Examples that clarify (both for video and written documents)

            Inter-Rater Agreement Training Video - KSDE Video of Elmont Elementary, USD 345
            

Interrater Agreement

Inter-rater Agreement is the degree to which two raters using the same scale give the same rating in identical situations.

Inter-rater Reliability is the relative similarity between two or more sets of ratings.

  • Reliable raters agree with the "official" rating of an evaluation example.
  • Reliable raters agree with each other about the exact ratings to be awarded.
  • Reliable raters agree about which evaluation example is better and which is worse.

 

Inter-rater Agreement is essential to:

  • Bridge the credibility gap.
  • Train raters.
  • Monitor system performance.
  • Link professional development to evaluation results.

 

Within the evaluation process, Inter-rater Agreement is used during:

  • Teacher and Building Leader observations.
  • Teacher and Building Leader artifact reviews.
  • Teacher and Building Leader summary ratings #1 and #2.
  • Teacher and Building Leader final summative evaluation ratings.

 

Inter-rater Agreement trainings at the district level need to include:

  • Team discussions and agreements about Inter-rater Agreement.
  • Team review of data collection and analysis.
  • Training design at preservice and inservice levels.
  • Regular training check-ups.
  • Coaching and observation.
  • Monitoring ratings.
Kansas Professional Learning Team (KPLT)
The Kansas Professional Learning Team (KPLT) is a new group committed to dispersing correct information and training throughout Kansas K-12 regions and districts. The 28 member group is from all areas of the state and most currently already disperse KSDE information. Now the group will interact collectively to create dissemination processes that are coordinated. Evaluations information is one area this group will work together to convey. Please contact a local KPLT member for up-to-date information and trainings. KSDE staff will continue to be available.

District/Organization

Name

Blue Valley USD 229

Tonya Merrigan, Executive Director, Curriculum & Instruction

Garden City USD 457

Dr. Steve Karlin, Deputy Superintendent

Hays USD 489

Shanna Dinkel, Curriculum Director

Kansas City USD 500

Dr. Sherry Samples, HR Director

Lawrence USD 497

Sarah M. Oatsvall, Assistant Director, Teaching & Learning

Olathe USD 233

Dr. Eric Punswick

Topeka USD 501

Diane Cox

Wichita USD 259

Alicia Thompson, Assistant Superintendent (Elementary)

ESSDACK

Tamara Konrade

ESSDACK Region

Ben Proctor, Hesston

Keystone

Mary Getto

Keystone Region

Josh Woodward

NWKESC

Kurt Brown, Assistant Special Education Director

NWKESC Region

Diana Wieland, Colby

Smoky Hill

Darla Smith, Smoky Hill; Larry Patrick, alternate

Smoky Hill Region

Carol Arjona, Geary County

SCKESC

DeAnne Heersche, SCKESC

SCKESC Region

Teresa Tosh, Haysville

SEKESC

Michael Koonce, Greenbush

SEKESC Region

Amber Wheeler, Columbus

SWPRSC

Kelly Gillespie, SWPRSC

SWPRSC

Jason Johnson

SWPRSC Region

Ardith Dunn

Dodge City USD 443

Mischel Miller

KASB

Doug Moeckel

KNEA

Idalia Shuman

MTSS Core Team

Todd Wiedemann

KLFA

Dayna Richardson

Learning Forward Kansas

Jill Lachenmeyer

The Kansas State Department of Education does not discriminate on the basis of race, color, national origin, sex, disability, or age in its programs and activities. The following person has been designated to handle inquiries regarding the non-discrimination policies: KSDE General Counsel, 900 SW Jackson, Topeka, KS 66612; 785-296-3201