Research using longitudinal ratings collected by multiple raters: One methodological problem and approaches to its solution

This study examines the adverse consequences of use of hierarchical linear modeling (HLM) to analyze ratings collected by multiple raters in longitudinal research. The most severe consequence of using HLM that ignores rater effects is the biased estimation of both level 1 and level 2 fixed effects and the potential for incorrect significance tests. Three statistical approaches (that is, the cross-classified random effects model [CCREM], the multitrait- multimethod model [MTMM], and the local linear regression [lowess]) are proposed as alternatives to HLM. Monte Carlo studies confirm that CCREM performs better than HLM to control for rater effects. Future studies are needed to develop robust models of MTMM and lowess to correct for rater effects in various settings of data creation. Strengths and limitations of using the corrective approaches and implications of correcting for rater effects in longitudinal research are discussed.