eMINTS evaluation

Evaluation: a tool for learning how to do our work better. Education Development Center (EDC) Purpose and philosophy Three components of eMINTS evaluation   Reports from EDC, 2006-   Universal Design for Learning   Reports from OSEDA, 1999-2005   For more information about eMINTS or the evaluation processes and results, contact the eMINTS National Center at emints-info@emints.org or (573) 884-7202.

Three components of eMINTS evaluation

eMINTS program evaluation consists of three distinct components:

1. Evaluation of the input provided to teachers – the eMINTS professional development program (including materials and the delivery mechanism) to insure that the program contents and processes are delivered in a manner that is consistent with the program and assures fidelity to the program’s goals and intent.
2. Evaluation of the teacher and student product output – an electronic portfolio of specific artifacts (both teacher and related student products) to document the effect of the input (professional development) on the teacher’s practice, the teacher’s mastery of key eMINTS objectives and impact on the subsequent student learning as verified by student products related to the teacher artifacts.
3. Evaluation of student academic performance - analysis of student performance as measured by standardized academic assessment instruments.

Evaluation of input

Evaluation of the input (professional development) is achieved through several processes including an external examination of the professional development materials and a rigorous observation of professional development sessions.

The external examination of the professional development materials is attained by the ongoing submission of eMINTS participant and facilitator guides to the International Society for Technology in Education (ISTE). The ISTE NETS (National Educational Technology Standards) Product Review Program was developed by the NETS leadership team in conjunction with key ISTE partners Intel Innovation, Microsoft and PBS TeacherLine. The program provides recognition of products, services, or resources through specific levels of alignment with ISTE mission and standards. eMINTS professional development materials are submitted for examination of their alignment with the NETS-T (National Educational Technology Standards for Teachers). The eMINTS Comprehensive Professional Development program materials are the first program in the world to accomplish full alignment with all ISTE NETS-T.

Another portion of the evaluation for fidelity is completed by a rigorous observation protocol that is applied to all eMINTS professional development providers, both eMINTS staff members and specialists trained through the eMINTS Professional Development for Educational Technology Specialists (PD4ETS) program. The observation protocol was developed, field tested and validated through an external consultant, the Center for Children and Technology affiliated with the Education Development Center (EDC). The observation instrument includes an overall evaluation of the delivery of the professional development and a series of “snapshots” conducted periodically throughout the observation. The snapshots quantify observable activities and essential conditions that have been authenticated as critical to a robust delivery of eMINTS professional development content.

A record of the classroom visits provided by either eMINTS staff member or specialists trained through the eMINTS PD4ETS program is also included in the evaluation of program input (professional development). The eMINTS design and development team places a high value on the importance of quality classroom coaching and mentoring as a part of eMINTS professional development programs. The record of classroom visits is achieved via an online summary of all visits provided to teachers along with a quantitative summary of the types of activities participants engage in during classroom visits.

Formative evaluation is completed through online surveys of participants that are completed semi-annually. The surveys are tabulated and results are shared with the appropriate individuals for ongoing program improvement and re-direction.

Evaluation of teacher and student output

An evaluation of teacher and student output in terms of artifacts and products that can be rated and analyzed is an important feature of insuring that high quality professional development input results in high quality teacher practice and student achievement.

Teacher mastery of eMINTS competencies is a key feature of the evaluation of the overall effectiveness of eMINTS professional development. The formation of a highly qualified teacher requires not only high quality pre-service teacher education but also ongoing clearly articulated plans for professional development, such as those based on the eMINTS instructional model. Demonstration of teacher mastery of eMINTS competencies is achieved through the evaluation of three key teacher products:

1. A complete and original WebQuest aligned to state standards and district curriculum requirements.
2. A complete lesson or unit plan based on the eMINTS Constructivist Lesson Plan format.
3. A classroom website containing all the required elements for demonstrating student and parental involvement in instruction and collegial collaboration.

In addition to the teacher products, student artifacts related to the implementation of the WebQuest and lesson or unit plan are included in an electronic portfolio. A minimum of two student artifacts each for both the WebQuest and lesson or unit plan is required.

Analysis of student academic performance

Statistical analyses of student academic performance completes the suite of evaluation processes included in the eMINTS evaluation program. All participating schools and districts submit student performance data from the standardized test measures used by their state or district. The results are stripped of any personally identifiable information and subjected to the appropriate statistical tests for comparison with students in similar demographics who are not participating in eMINTS programs. Statistical analyses can also include pre-post assessment (if available).

The statistical analyses of student academic performance are correlated with both the input (professional development) and the teacher/student products and artifacts (output) to provide a complete portrayal of the entire eMINTS implementation. Evaluation summaries are reported only at the highest possible aggregate level to insure anonymity of teachers, students and districts. However, individual summaries by school or district can be provided with appropriate contractual arrangements.

Reviewed August 14, 2006.