Inter-Rater Reliability

ATI conducted a study of inter-rater reliability on Galileo scales in partnership with three Head Start programs in the state of Ohio. Observations were conducted on three randomly selected children from each of 318 classrooms in these programs.

The following table presents the results of the study indicating adequate correlations between the observations for the two observers.

The means and standard deviations for observations for the two observers are nearly identical. This is consistent with the expectation that observer-measurement error for aggregated data will be low. These results provide evidence of inter-rater reliability for the Galileo scales.

It is important to note though, that inter-rater reliability may vary based on the specific staff members conducting observational assessments as well as well-established phenomena such as observer drift. For this reason, ATI offers inter-rater reliability training designed to assist programs in training staff members to conduct reliable assessments.

Learn more about inter-rater reliability professional development.