Whether Called Standards, Quality Indicators, or Criteria: Is the Education Profession Addressing the Same Elements in Defining Quality Research

Schirmer, Barbara R., Alison S. Lockman, and Todd N. Schirmer

The purpose of this investigation was to compare standards for educational research developed by five national organizations (AERA, Council for Exceptional Children’s Division for Research, Division 16 of the APA/Society for the Study of School Psychology, AACTE, and What Works Clearinghouse of the Institute for Education Sciences) in order to determine the extent to which these five sets of standards overlap and, therefore, define quality research similarly. The investigation was designed to answer the following questions: (1) Do the standards for educational research as developed and articulated by five national organizations address all key research elements? (2) What are the differences and similarities between the five sets of standards? (3) Do any of the organizations provide evidence that use of the standards improves the quality of published and/or funded research? The mode of inquiry was an analysis of each set of standards to determine which indicators of quality each addressed and then to compare and contrast these quality indicators to determine overlaps and gaps. We developed an a priori scheme of major characteristics that educational research routinely addresses. For each set of standards, we first parsed them by research design with categories for all designs. We then parsed the standard by taking the verbatim language and assigning it to the appropriate characteristic. This analysis resulted in a rubric in which we could compare each set of standards by design and characteristic. Results offer information on overlaps and gaps that can be used in moving toward a set of standards that are broad and deep enough to cover the critical dimensions of quality research yet provide a manageable tool for researchers and practitioners.