Reports - 2002 Independent Assessment

AAM 2002 Evaluation—November, 2002
(Adobe PDF file 497KB - requires Adobe Acrobat Reader)

Executive Summary

The purpose of the third year evaluation of the Adventure of the American Mind (AAM)
program was to ascertain the effectiveness of the training at four program sites and the
consequent impact in the program graduates’ classrooms. Believing that the basic strengths and challenges of the AAM program had been discovered in surveys of program graduates in previous years, the AAM project director asked that qualitative
evaluation methods be employed to “dig deeper” into the feelings and opinions of program participants. Consequently the evaluation included focus groups, visits to school sites where AAM graduates were teaching in primary or secondary schools, and the scoring and ranking of 40 randomly selected lesson plans produced by program graduates. Four AAM training sites were targeted for this evaluation: Furman University, Mars Hill College, Montreat College, and Western Carolina University. After the low attendance at Montreat’s focus group and the cancellation of Western Carolina’s focus group, the project director and evaluation team used site visits to enhance and guarantee the number of participants interviewed. An additional evaluation was conducted in April 2002 for the Arizona cluster, and is reported separately in Appendices F and G.

Four Sources of Data Collection

1. Twenty-seven focus group participants
2. Twelve teachers during site visits in their classrooms
3. Observations of teaching in four classrooms
4. Assessment of 40 randomly selected lesson plans

Evaluation Findings
Overall, this program has enabled 139 teachers in 77 schools to develop and sharpen their understanding of technology integrated instruction and access to electronic resources, especially those provided by the Library of Congress.

1. Laptop computers were the central motivation for becoming involved in the AAM
   program.
2. The ability to connect to the Internet varied among the schools. The schools with T1 lines appeared to have the least amount of difficulty. Filters and incompatible systems and software between school districts and AAM were problematic.
3. The combination of classroom instruction and online coursework is effective, but
   classroom instruction must continue to be the primary method of instruction.
4. Online coursework needs to be refined based on feedback from graduates.
5. Participants felt supported and respected by the AAM faculty and staff.
6. Participants often feel they have inadequate technical support in their schools. Personnel are viewed as ineffective, and financial resources are limited for LCD projectors and televisions.
7. Participants’ computer skills vary widely from the novice to comfortable experts.
   Instruction has not adequately provided for such variance.
8. Participants entering the program with strong computer skills are able to learn more about instructional design techniques and consequently create better lesson plans.
9. Program participants have difficulty learning the technology and instructional design at the same time.

Recommendations

1. Continue to provide participants with laptops or another comparable technological tool as incentive, if the program is to continue in its present form.
2. Continue working on connectivity for the schools in western North Carolina, especially
   problematic areas such as restrictive filters and compatible systems and software.
3. Continue to screen program participants’ computer skills and divide the classes
   accordingly. If classes cannot be divided because of budget constraints and availability of instructors, then the number of technical support per cluster needs to be increased.
4. Continue to support AAM participants by giving them instructional models that encourage student led inquiry.
5. Continue to provide participants with the time needed to construct sound instructional design.
6. Continue to develop and require appropriate formal and informal assessment tools with lesson plan designs.
7. Require program graduates to participate in the evaluation process. This requirement needs to be inserted in their contracts to ensure more participation in future focus groups.

Dr. Chris Dennen, (former) Project Director of Education Research Consortium of Western North Carolina (ERCWNC), selected the Center for Assessment and Research Alliances (CARA) at Mars Hill College to conduct this evaluation. Under the CARA Director Dr. Thomas Plaut, Dr. Deborah Morris conducted the focus groups and site visits. James Brown, M.Ed., Professor of Education at Mars Hill College, designed and conducted the lesson plan assessments.

An Adventure of the American Mind is made available through a Library of Congress grant funded by the U.S. Congress. AAM is administered through the Education and Research Consortium of the Western Carolinas Inc.
For site-related problems, e-mail aam@ercwc.org..