External evaluation
Since its inception in 1999, external evaluations of the eMINTS program have been conducted to determine the effects of eMINTS professional development (PD) on teacher and student outcomes. Qualitative research and formative evaluations also contributed to a better understanding of the facilitating factors and challenges associated with school/classroom implementations of eMINTS.
Purpose and philosophy
eMINTS National Center evaluation processes use a variety of quantitative and qualitative data collection and analysis methods to provide both formative and summative results. Analysis of quantitative data and information derived by those methods is best understood when complemented by the rich insights available through qualitative analysis.
eMINTS National Center program evaluation also assists schools, districts and other educational organizations that are implementing eMINTS programs in developing, capturing and communicating useful and usable information for key stakeholders and other audiences.
All eMINTS National Center evaluation processes fully comply with the applicable policies and standards of the University of Missouri Institutional Review Board.
More than a decade of research
eMINTS program external evaluations from 1999-2009 document the effects of eMINTS professional development on teacher and student outcomes. A summary of findings was compiled by research specialists at Learning Point Associates in summer 2010.
A Summary of External Program Evaluation Findings for the eMINTS Program from 1999-2009 (Summer 2010)
[+ show][- hide]
Since its inception in 1999, annual external evaluations of the eMINTS program have been conducted to determine the effects of eMINTS professional development (PD) on teacher and student outcomes. Qualitative research and formative evaluations also contributed to a better understanding of the facilitating factors and challenges associated with school/classroom implementations of eMINTS. This document summarizes 10 years of eMINTS PD research and evaluation and assesses the quality of evidence reported.
Reports
View or download reports from Education Development Center (2006-2009) and Missouri’s Office of Social and Economic Data Analysis (1999-2006) below. Click the [show+] link to read a brief summary.
eMINTS program evaluation report 2009: an analysis of the persistence of program impact on student achievement (September 2009)
[+ show][- hide]
Over two years of evaluation, EDC’s analyses have demonstrated consistent patterns of program impact. Mastery of program concepts; fidelity of professional development; and amount of time planning lessons during the visits from instructional specialists are all associated with higher levels of student achievement, and those relationships persist a year after these teachers have completed their two-year eMINTS professional development programs. In addition, this new set of analyses (2009) also suggests that students who have spent two years in the classrooms of eMINTS Comprehensive teachers outperform their peers in the same schools who have less experience with these teachers. All of these analyses provide strong evidence that, when the program is delivered as intended, it can have a persistent positive impact on the students and teachers it reaches.
eMINTS program evaluation report: an investigation of program fidelity and its impact on teacher mastery and student achievement (September 2008)
[+ show][- hide]
This evaluation provides evidence that the eMINTS program is being implemented with a high level of fidelity and integrity by both the eMINTS staff and participants and graduates of the PD4ETS program; that teachers are mastering most, but not all of the core eMINTS concepts; and that maintaining a high level of Program Fidelity/Integrity is important for ensuring that teachers are mastering the core program concepts, which may then result in higher levels of student achievement.
Analysis of 2006 MAP results for eMINTS and non-eMINTS students (2007)
[+ show][- hide]
The analyses of the 2006 MAP Communication Arts and Mathematics data presented in this report indicate that the eMINTS program is having a small but definite impact on student outcomes in the four grades that were examined. While there were little or no differences evident between students in eMINTS and non-eMINTS classrooms when the basic analyses were conducted, once regressions were done that controlled for the many demographic variables that can affect student achievement, the analyses suggested that students in eMINTS classrooms in grades three through six perform slightly better on the MAP assessments than students in non-eMINTS classrooms, and in two cases (third and fifth grade Mathematics) perform significantly better on the MAP assessment.
No significant differences were found in achievement between eMINTS and non-eMINTS students based on gender, race/ethnicity or IEP. However, positive trends in student performance were found for eMINTS students who qualified for Title I services and FRLP. This finding suggests that the program is having a greater impact on the highest-need students. It is important to note that eMINTS is now funded in Missouri primarily through the Title II.D grant program, which only provides funds for schools that meet federal poverty criteria. Because of this limitation, a large percentage of the students in eMINTS classrooms come from economically disadvantaged backgrounds. The results from these analyses suggest that the program may be particularly appropriate for the specific student population that it is serving.
Analysis of 2005 MAP results for eMINTS students (January 2007)
[+ show][- hide]
The analysis of student MAP scores in the FY04 cohort of eMINTS schools shows significant differences by eMINTS enrollment status on the MAP Communication Arts and Mathematics tests. Analyses of MAP scores for special education students, students receiving Title I services and students receiving free and reduced lunch suggest that eMINTS enrollment significantly increases their scoring on the MAP tests. These results support previous analyses of eMINTS cohorts.
There is no doubt that for the FY04 eMINTS cohort, enrollment in an eMINTS classroom is beneficial for increasing MAP scores for both the Communication Arts and Mathematics tests.
Principals exhibiting student-centered leadership correlates with positive school climate indicators in Title I eMINTS schools and higher MAP math scores in Title I eMINTS classrooms (July 2006)
[+ show][- hide]
Results of 40 school observations in Title I eMINTS FY03 Schools indicated that positive school climate could be distinguished by the nature of interaction between the building principal and students. Building principals who were classified as student-centered positively affected the climate of the school. In this context, student-centered leadership was defined as maintaining high visibility with the students: visiting classrooms, relating to students in school settings, monitoring student achievement and progress. Four correlations were flagged for statistical significance between student-centered principals and school climate indicators. Student-centered principals were more likely to lead schools where student work was displayed in the classrooms, where visitors were welcomed, where office staff related positively to students and where teachers attended to and instructed student behavior in public areas. Analysis of student MAP scores also shows significant differences by principal leadership status on the mathematics tests. These results support previous analyses of the importance of principal leadership in eMINTS schools.
Professional development of eMINTS teachers: a comparison of delivery methodology between eMINTS staff and PD4ETS participants (November 2005)
[+ show][- hide]
This report presents results from an examination of the professional development of eMINTS teachers in the FY03 cohort. The purpose of the study was to assess differences and similarities between the efficacy of professional development delivered by eMINTS Staff (Cluster Information Specialists – CIS), and by participants in the eMINTS Professional Development for Educational Technology Specialists (PD4ETS) program, a train-the-trainer program. Previous analysis of eMINTS teachers’ perceptions of their professional development found that CIS played a crucial role in the development of teachers. Evidence from the current study suggests that both types of professional development delivery produce similar results and assist in preparing eMINTS teachers to support student learning.
The effects of eMINTS enrollment on Title I students (April 2005)
[+ show][- hide]
Previous analyses of Missouri Assessment Program (MAP) data have consistently indicated a positive interaction between the eMINTS and Title I programs. This report examines the MAP performance of students who receive Title I-funded educational assistance and are enrolled in eMINTS classrooms, using data from the FY03 eMINTS cohort of schools. While the results were mixed, there was evidence of a smaller achievement gap between Title I and non-Title 1 eMINTS students. These quantitative results are supported by observations from Title I teachers regarding the ways in which eMINTS classroom enrollment supports the performance of the students with whom they work. These results suggest that a closer coordination between the Title I and eMINTS programs will have a beneficial impact on the performance of Title I students.
Analysis of 2004 MAP Results for eMINTS Students (January 2005)
[+ show][- hide]
The analysis of student MAP scores in the FY03 cohort of eMINTS schools shows significant differences by eMINTS enrollment status on the MAP Communication Arts test and smaller, but positive, differences on the MAP Mathematics test. Analyses of instructional practices suggest that participation in the eMINTS program helps all teachers more effectively support higher levels of student achievement. Finally, analyses of MAP scores for Black students, special education students and students receiving Title I services suggest that eMINTS enrollment helps reduce the achievement gaps between these groups and other students. These results support previous analyses of eMINTS cohorts.
An investigation into selection bias in FY02 eMINTS schools (November 2004)
[+ show][- hide]
This report analyses the impact of the nonrandom selection of eMINTS schools, teachers and students in the FY02 eMINTS cohort. The report summarizes the evaluation-design choices made by the program in light of the simplistic study design endorsed by advocates of scientifically based research. It also proposes a modeling strategy to estimate the potential impact on MAP scores of the nonrandom selection of teachers and students. The analyses suggest that no identifiable selection biases exist among the schools, teachers and students participating in the FY02 eMINTS cohort.
An exploratory study of the black-white achievement gap in eMINTS-student MAP results (November 2004)
[+ show][- hide]
Text
Analysis of MAP results for expansion classrooms (September 2004)
[+ show][- hide]
This policy brief examines the quantitative impacts of the eMINTS expansion project. The report analyzes 2001 and 2002 Missouri Assessment Program (MAP) results for eMINTS students in classrooms with teachers who have completed the eMINTS professional development training and are serving as Mentors to students in classrooms with teachers undertaking the eMINTS training (Mentored teachers), students in Mentored teachers’ classrooms, as well as non-eMINTS students in classrooms in that grade in the same school building. Additional analysis explores the relationships of teacher status, student average MAP scores, and classroom instructional practices categorized according to the eMINTS lesson typology. Results suggest that the structure employed in the expansion project may have helped newly mentored teachers adopt inquiry-based instructional practices more quickly, but the pace of these adoptions did not necessarily help student performance.
Classroom climate, instructional practice and mentorship experience in the eMINTS expansion classrooms: a two-year study (September 2004)
[+ show][- hide]
This report presents data from observations of classroom climate, instructional practice and mentorship experience in the eMINTS Expansion program classrooms. Data from the entire two-year program is analyzed. The analysis describes clear relationships between a positive classroom climate and the use of effective behavior management strategies. The analysis indicates that these classroom characteristics are also related to the observed instructional practices of teachers. Differences between the classrooms of mentor teachers and those of mentored teachers over the two years of the expansion program are considered. The two-year data suggests an overall improvement in the observed classroom climate of eMINTS classrooms.
Analysis of 2003 MAP results for eMINTS students (May 2004)
[+ show][- hide]
This report presents Missouri Assessment Program (MAP) test results from the FY02 eMINTS cohort. The analysis highlights statistically significant differences on the Communication Arts, Mathematics and Social Studies MAP tests, but not on the third grade Science test. These differences can be seen for all students, for low income students, and for special education students. There are also significant differences for students enrolled in Schoolwide Title I schools.
Assessing instructional practices in eMINTS classrooms (July 2003)
[+ show][- hide]
The eMINTS evaluation has highlighted the role of inquiry-based instructional practices in supporting high levels of student achievement. As the eMINTS program has developed, “instructional practice” has been assessed in three ways: through direct observation using the eMINTS Lesson Typology, through coding each lesson along Bloom’s Taxonomy of Educational Objectives and as part of a general assessment of the classroom environment using the “Hallmarks of an Effective eMINTS Classroom” rubric. This paper assesses the validity of each of these measures by analyzing their interrelationships. Data for this analysis is taken from a set of classroom observations conducted with teachers participating in the eMINTS expansion project. Results show significant correspondence between each measure at both the lesson and classroom levels.
Classroom climate, instructional practices and effective behavior management in eMINTS expansion classrooms (July 2003)
[+ show][- hide]
Many ingredients go into being an effective eMINTS teacher. Results of 99 classroom observations in eMINTS Expansion Schools indicated that effective and less effective behavior management could be distinguished on the basis of teachers’ sensitivity to five areas in creating their classroom environment: exhibiting work that was student-produced; arranging space for flexible use; providing elements that afford a soft, sensory, tactile option; maintaining physical comfort; and interacting in a respectful and encouraging manner. Teachers observed conducting facilitated lessons were also more likely to have classrooms with these positive aspects. The combination of a positive classroom and the use of facilitated, inquiry-based instructional practices helped to construct a well-ordered and effective learning environment.
“Proficient” and “Emerging” educational communities (July 2003)
[+ show][- hide]
The eMINTS expansion project was designed to identify successful strategies for extending the impact of eMINTS instructional resources beyond the third and fourth grades. This report summarizes the early experiences of the expansion schools as they developed the capability to expand and sustain the eMINTS innovation in their buildings. Analysis of these schools’ experience revealed three common factors that are key to creating an educational environment that supports the eMINTS innovations: an instructional plan, efforts to involve students in the school community and programs to bring community volunteers into the school. Expansion schools that used these three factors as resources to create learning environments where students can achieve, showed strong positive results on the 2002 MAP test and are identified as successful, proficient eMINTS schools.
Analysis of 2002 MAP results for eMINTS students (January 2003)
[+ show][- hide]
This policy brief examines the quantitative impacts of the eMINTS Program on the second cohort of participating classes. The analysis compares 2002 Missouri Assessment Program (MAP) results for eMINTS students and non-eMINTS students in the same school building and for eMINTS students and all students statewide. eMINTS students in the fourth grade scored significantly higher than non-eMINTS students. Similar results were not seen among third grade students. eMINTS students in classes characterized by teachers who consistently implement the inquiry-based teaching practices emphasized by the eMINTS professional development program scored higher than students in other types of classes. While there were fewer significant differences in some aspects of the program, these results in general substantiate the positive trends associated with eMINTS enrollments reported in the policy brief “Analysis of 2001 MAP Results for eMINTS Students,” published in January 2002.
Leadership orientations of FY00 eMINTS principals (October 2002)
[+ show][- hide]
This report classifies FY00 eMINTS principals according to their leadership orientation. School visits were conducted during the 1999-2000 and the 2000-2001 school years. On-site visits including interviews with the principals and the eMINTS classroom teachers, illustrate four general leadership orientations among the FY00 principals. Principals with a collaboration orientation acted as catalysts to the full implementation of eMINTS. Principals with a regulation orientation limited the full implementation of eMINTS. Principals with a disconnection orientation had little impact on the implementation of eMINTS in their buildings. In addition to these three main orientations, a fourth, “hybrid” orientation was identified. These leadership orientations are clearly related to the ways eMINTS teachers conduct their lessons, and have clear impacts on student performance. Students in schools with collaborative principals score higher than students in schools with other types of principals. This report describes FY00 principal leadership orientations and to what extent leadership orientation affected the implementation of the eMINTS program.
Analysis of 2001 MAP results for eMINTS students (January 2002)
[+ show][- hide]
This policy brief examines the quantitative impacts of the eMINTS Program on the first cohort of participating classes. The analysis compares 2001 Missouri Assessment Program (MAP) results for eMINTS students and non-eMINTS students in the same school building. eMINTS students scored consistently higher on the MAP than non-eMINTS students. eMINTS students classified as having special needs scored higher than special needs students in non-eMINTS classes. Finally, higher MAP results among eMINTS students were found to be associated with the instructional practices.
Typology of eMINTS lessons based on classroom observations conducted during 2000 & 2001 (November 2001)
[+ show][- hide]
This report summarizes classroom observations conducted in FY00 classrooms. It describes a general typology of instructional lessons based on 85 classroom observations conducted in 2000 and 2001. The observations, which serve as the empirical basis for the typology, were conducted during site visits made by the eMINTS evaluation team in two waves, between March 1 and May 20, 2000 and between November 11, 2000 and April 18, 2001. During each lesson, evaluation team members observed teachers’ interaction with students, their use of computers and their general conduct of the lesson. The discussion outlines the rationale for the lesson typology and the key characteristics of the categories that constitute the typology. Comparisons between observations conducted in 2000 and 2001 shows a general increase in the occurrence of facilitated lessons.
Final results from the eMINTS teacher survey (July 2001)
[+ show][- hide]
This report presents results from the eMINTS teacher surveys administered between 1999 and 2001. During this period the eMINTS Program refined its professional development instruction and improved its service delivery procedures. Over the survey period the results show general improvement in teacher computer usage in areas related to instruction. By the spring of 2001 eMINTS teachers report that they are doing more with computers, that they feel more skilled in using computers, and that they use computers more frequently in their work. Teachers also feel that they are more effective teachers, as indicated by the overall increase in the average score on the MSIP Instructional Efficacy scale. The pattern of responses suggest that this improvement in teacher opinion is related to programmatic improvements in its second year.
Results of the MSIP student questionnaire, spring 2001 (September 2001)
[+ show][- hide]
Results from a spring administration of the MSIP Elementary Student Questionnaire show that fourth grade students enrolled in eMINTS classrooms rate their school climate and expectations for learning higher than students not in eMINTS classrooms. However, there were no statistically significant differences among third grade students. These results will be used in estimating the impact of the eMINTS experience on student performance on the MAP test.
Analysis of 2000 MAP results from Jennings (MO) School District (June 2001)
[+ show][- hide]
This policy brief examines the qualitative and quantitative impacts associated with the installation of computer classrooms in the Jennings School District.
Teacher survey responses from the teacher profile unit teaching plan (January 2001)
[+ show][- hide]
FY01 teachers and their approach to lesson plans.
Teacher survey responses at eMINTS’ kick-offs (December 2000)
[+ show][- hide]
The results of a comparison between FY01 and FY00 teacher surveys collected at
the beginning of their participation in the program suggest that FY01 teachers do
not differ substantially from FY00 teachers in their initial computer skills and usage
patterns. However, there are statistically significant differences between the cohorts
with respect to their initial perceptions of instructional efficacy and the effectiveness
of selected teaching activities.
FY00 teachers’ summary impressions of their experiences with eMINTS and factors they identified for its successful implementation (October 2001)
Parents’ perceptions of the eMINTS program and its effect on their children (September 2001)
Teacher perceptions of student changes in FY00 eMINTS classrooms (February 2001)
Teacher experiences with technical support (December 2000)
Teacher comments about working in an eMINTS classroom (December 2000)
Teacher perceptions of Cluster Information Specialists (August 2000)
Teacher perceptions of eThemes services (August 2000)
