Evaluation Reports

The National AAM Director’s Response
to the March 31, 2004 AAM Evaluation Report from the
Education Development Center for Children and Technology
October 27, 2004

By Christopher C. Dennen Ph.D.

Introduction

In 2003 the Education and Research Consortium (ERC) contracted with the Education Development Center for Children and Technology (CCT) to perform a summative evaluation of the first four years of the Adventure of the American Mind (AAM) program. A 60-page report was delivered on March 31, 2004. The full report is available online in the Reports section of the AAM Web site (www.aamprogram.org).

In addition to sections on Methods and Findings, the CCT report also included a Recommendations section. This report is the official AAM response to the CCT’s suggestions and observations.

The CCT’s Recommendations section is reproduced in full below. This is followed by the National AAM Director’s point-by-point responses in the following section.

From the CCT Report

Recommendations

Our review of the Partner programs did not yield any easy answers about "efficiencies" or the "exportability" of particular models. As stated above, while the Partner programs varied, they were more similar than different. All had intensive requirements for and made intensive investments in teachers. The benefits we observed (increases in teacher knowledge, changes in teacher practice, and student learning) were closely related to these intensive program requirements and investments. Large-scale changes in the model (reducing the time teachers spend learning together; making the training more modular so that the technology, primary source and pedagogy elements are experienced separately; or training groups of teachers from one school) may cause the program to have a different impact on teachers and their teaching.

The following recommendations stem directly from the Phase I findings. They are offered here as suggestions for thinking through the design of future iterations of AAM, not as prescriptive remedies. Some may not be applicable given structural changes in the program as it evolves. And, it should be recognized that the AAM program has already incorporated several of these recommendations into Phases II and III.

Participant Recruitment & Selection

• Consider targeting teachers of the Humanities broadly conceived (history, social studies, English language arts, geography, visual art, design, architecture, communication) as participants. Our data consistently suggest that these are the areas of study most likely to be enhanced by LOC resources in significant ways, for the benefit of students and student learning.
  
• Consider targeting upper elementary through high school teachers (grades 5-12) as program participants, since teachers below 5th grade found it hard to make meaningful enhancements to their curricula beyond exposing their students to documents as illustrations or examples.
  
• Consider targeting library media specialists as participants, in partnership with teachers in the same school. Such school-based partnerships take advantage of librarians’ information skills and time flexibility, and teachers’ knowledge of curriculum and their access to students in classrooms.

Incentives and Technology Supports

Consider providing incentives – especially technology incentives – in future program models. In Phase I the laptops were not just a nice perk but put the technology directly into the hands of the classroom teacher, with important consequences for the work in many cases. A serious commitment to educators and schools also sends a strong symbolic message that enables the program to demand, and receive, a lot from teachers.

Consider ways to improve existing technology infrastructure in schools. For example, if a certain number of teachers complete the AAM professional development program, donate a projector – which in Phase I became the second most sought-after tool after the laptop – or a wireless lab to the school for teachers to check out and use in their classrooms.

Program or Course Enhancements

• Provide all teachers with more explicit and detailed instruction in strategies for teaching and learning with primary documents.
  
• Make the lesson development process a more robust part of the program. Lessons were stronger when teachers were given careful coaching during the design process, and had opportunities to draft, get feedback, and revise them. Opportunities to test or "try out" key portions of a lesson, before revising and submitting it, are also crucial in order for the resulting lesson to be useful in real classrooms.
  
• Fold into the training simple self-diagnostics – short evaluation exercises – so that learners and instructors can see their progress toward clear benchmarks of knowledge and skill.
  
• Give participants some formal training in "how to mentor" before they have to embark on a mentoring requirement; consider providing incentives to mentees as well.
  
• Use web-based tools and exercises to deliver more dynamic experiences with primary sources to teachers and students. The web resources prepared by the home school partner, for example, can be leveraged for broader K-12 educational use and interactive exercises like those available via the websites History Matters or Picturing Modern America can enhance learning for teachers and students.
  
• Add to the home school program some kind of parent and student training in the use of digitized primary source material.
  
• Continue to conduct formative research on the home school partner program, making sure to include questions about access, usability, and practicality. Participants responded well to the web-based materials but, for example, it was unclear whether home access in terms of equipment and Internet connectivity was robust enough.

Program Outputs

• Create a formal AAM "alumni" program through which teachers can continue to be involved in the work. By creating ongoing ties (via web forums, electronic "posters" and local and regional "meet-ups") participants may be more likely to continue implementing their knowledge in their classrooms. In addition, alumni could be utilized in future programs by mentoring participants.
  
• "Brand" AAM more clearly so that teachers form a professional identity that embraces the Library of Congress, using online primary sources, and enhanced teaching. Phase I graduates did not identify strongly with AAM, but rather with their local campus program and staff. Consider calling teachers "Fellows" or "Partners" of AAM or the LC. Use technologies to foster ongoing sharing and communication around the work.
  
• Examine the utility and usefulness of allocating resources to a lesson plan database. Many such lesson databases exist but there are few broad data suggesting that teachers use them frequently or effectively. Moreover, most Phase I teachers did not use existing AAM lessons or Learning Page lessons in their work, except when asked specifically to look at them as models during the training. They were more likely to reference existing databases such as NC Learn. Therefore, it is questionable whether they would perceive an AAM lesson and unit plan database to be valuable.

We recommend conducted iterative formative research on any lesson database tool. In addition, since most teachers come to American Memory looking for "stuff" – content resources to use with students – and since they often have difficulty finding appropriate materials using the search tools, more useful than lesson plans might be an index of individual American Memory documents that have been deemed to be rich and compelling for students in different grades and subject areas to examine – a "treasures" of American Memory feature. Individual documentary "treasures" might be accompanied by 10 – 50 minute exercises that get students actively reading, debating and responding to the documents in writing, role-playing, etc.

Director Training and Preparation

• Create a formal director training program. Continue to strive to build greater coherence across all programs, while preserving adaptability to local circumstances. In Phase I, while ERC provided objectives for the directors, each director decided his or her personal vision is for the program.
  
• As one director said, "Everyone started it a little differently and does it a little differently. It’s almost like you had to find out where it is you feel comfortable in what you do, so there are pros and cons to that."
  
• Short of standardizing course or workshop structure, strive to distinguish the program features that are core and required (and the rationale behind each of these), from those that are more optional.
  
• Have faculty instructors/directors visit the Library of Congress in DC prior to teaching the course.
  
• Implement mechanisms for formal and informal sharing of knowledge and practice by directors.

No research report can conclude without a final note about "further research needed." Well designed formative and summative evaluation research should have a powerful role in the AAM program as it evolves. The project’s ambitions – to have teachers acquire and then quickly turn over to their students the ability to learn using new technologies and the online resources of the Library of Congress – requires that teachers learn in three domains at once. Coupled with this is an ambitious programmatic goal: To deliver training in many different sites around the country simultaneously via an expanding network of partners and affiliates. At both instructional and programmatic levels, AAM would benefit greatly from having in place, at the outset of significant new work, a set of evaluation diagnostics that would help teachers, instructors, and administrators monitor, assess, and improve their progress toward clear benchmarks of knowledge and skill.

Working with researchers to craft an evaluation "toolkit" of flexible techniques that programs can use to assess their work would pay off in many ways. First, it would address what is currently an "assessment gap" in the AAM program. Second and more important, it would give teachers – and ultimately students themselves – insight into the many ways students learn, and learn differently, using these new materials. Rigorous, flexible assessments would go a long way toward making the AAM program as a whole what Bransford, Brown, & Cocking (2000) cite as most effective for teachers and students: a knowledge-centered, assessment-centered, and learner-centered professional development program.

The National AAM Director’s Response and Commentary

From the Recommendations Section

"Large-scale changes in the model (reducing the time teachers spend learning together; making the training more modular so that the technology, primary source and pedagogy elements are experienced separately; or training groups of teachers from one school) may cause the program to have a different impact on teachers and their teaching."

[Responses and comments are shown indented and in an alternate font.]

This comment refers to the change from Phase I of the program where the focus was a high quality, but low volume approach to teacher training, to Phase II where the emphasis is on training much larger numbers of teachers at the same overall cost as Phase I.

One of the two main goals of this pilot project, as stated in the grant agreement is to:

"Demonstrate and evaluate a training program exportable to other communities at minimal cost."

While Phase I was an effective educational strategy it was not cost-effective. Phase II is designed to achieve an eight-fold (800 percent) decrease in the cost of training each teacher, thus allowing the AAM program to meet a key goal.

We recognize the CCT’s concerns that changes in the program may result in changes in the outcomes. Their concerns are our concerns, and we have planned accordingly. The CCT report expresses their apprehensions in detail below, and these are answered on a point-by-point basis.

From the Participant Recruitment & Selection Section

"Consider targeting teachers of the Humanities broadly conceived (history, social studies, English language arts, geography, visual art, design, architecture, communication) as participants. Our data consistently suggest that these are the areas of study most likely to be enhanced by LOC resources in significant ways, for the benefit of students and student learning."

Our experience supports this conclusion to the degree that teachers in the humanities found an easier connection to the LOC resources; however, with the implementation of Phase II, the teacher population is more self-selecting. Because the LOC is rich in resources for the humanities, we are seeing more teachers of those subjects enrolling in workshops.

Another important observation we made, in the first four years, is that it is not the amount of resources that the LOC offers that determines a teacher’s success in our program; rather, it is their enthusiasm for using the materials. We therefore don’t want to exclude teachers based on the subjects they teach, because we have seen great success from teachers of subjects – like math – where the LOC is not especially strong.

"Consider targeting upper elementary through high school teachers (grades 5-12) as program participants, since teachers below 5th grade found it hard to make meaningful enhancements to their curricula beyond exposing their students to documents as illustrations or examples."

This finding is not supported by our experience. I suspect that this was true with the first cohorts of teachers in Phase I because our directors were still learning about the uses of primary documents in the classroom. We are now finding that there are many uses for primary source materials appropriate to K-5 students’ cognitive levels.

Elementary school students can effectively engage in the analysis of photos, music, and written documents, and they find these activities interesting and stimulating. It is important, just as it is with all other areas of the curriculum, that these activities are age-appropriate.

"Consider targeting library media specialists as participants, in partnership with teachers in the same school. Such school-based partnerships take advantage of librarians’ information skills and time flexibility, and teachers’ knowledge of curriculum and their access to students in classrooms."

I agree with this recommendation. Three partners, at this time, are specifically developing the knowledge and programs to address this group. Metropolitan State College of Denver is charged with finding ways of including libraries in its program. At Illinois State University, the AAM program is being administered through the University’s library. Mars Hill College is targeting media specialists and librarians for its program. Most other partners have trained many media specialists and librarians, and we will continue to increase our focus on this group.

From the Incentives and Technology Supports Section

"Consider providing incentives – especially technology incentives – in future program models. In Phase I the laptops were not just a nice perk but put the technology directly into the hands of the classroom teacher, with important consequences for the work in many cases. A serious commitment to educators and schools also sends a strong symbolic message that enables the program to demand, and receive, a lot from teachers."

Phase II continues the technology incentive program, but with a more cost-effective approach. As noted above, the laptop program was not only a nice perk, but also necessary in order to provide the teacher with the tools needed to implement AAM in the classroom.

Although the laptop program was necessary, it was prohibitively expensive when scaling the program to a larger size. Fortunately, technological progress in the school systems and cost competitiveness in the retail sector have combined in a timely way to open other approaches for AAM. In 2000 many teachers in Western North Carolina had limited access to computers; now most have relatively easy access. In 2000 a desktop computer system cost $1,500 to $2,000; now the cost is less than $500. This means most teachers currently have the computer access they need to utilize the knowledge gained through AAM.

It is the job of the AAM program to monitor all aspects of our changing educational culture and modify the program appropriately. Changing the incentive program is one example of how this has been done.

"Consider ways to improve existing technology infrastructure in schools. For example, if a certain number of teachers complete the AAM professional development program, donate a projector – which in Phase I became the second most sought-after tool after the laptop – or a wireless lab to the school for teachers to check out and use in their classrooms."

This is exactly the approach used in Phase II.

From the Program or Course Enhancements Section

"Provide all teachers with more explicit and detailed instruction in strategies for teaching and learning with primary documents."

Many resources designed to do this are now on the AAM Web site. Some examples are:

• The Lesson Plan Database, currently at 60 lessons, with 20 more being added now
• Papers like "Primary Sources in Perspective" by Dr. Mark Newman
• Books like Uncovering Our History: Teaching with Primary Sources. American, by Susan Veccia
• The new online Lessons section with self-guided, self-paced lessons for 4th to12th grade students

"Make the lesson development process a more robust part of the program. Lessons were stronger when teachers were given careful coaching during the design process, and had opportunities to draft, get feedback, and revise them. Opportunities to test or ‘try out’ key portions of a lesson, before revising and submitting it, are also crucial in order for the resulting lesson to be useful in real classrooms."

Creating lesson plans is no longer a requirement in Phase II, so teachers are no longer under pressure to create a lesson plan in order to graduate from the AAM program. As a result they are now free to develop lessons and activities that fit their needs (rather than as a course requirement).

Early reports from the field indicate that teachers may be making more frequent use of LOC materials as a result of the Phase II approach. This may be because in every workshop they are exposed to easy and effective methods for incorporating LOC sources into the classroom.

The Phase I focus on creating lesson plans over the course of a semester may have inadvertently given the impression that using the LOC in the classroom was a substantial undertaking, thus discouraging use of primary sources. In contrast, Phase II makes incorporating primary sources easier.

"Fold into the training simple self-diagnostics – short evaluation exercises – so that learners and instructors can see their progress toward clear benchmarks of knowledge and skill."

Workshop evaluations (summative and formative) are standard practice with Phase II. One major emphasis with Phase II evaluations is continuous improvement. Therefore, constant feedback from teachers is critical so directors can act and react during training.

"Give participants some formal training in ‘how to mentor’ before they have to embark on a mentoring requirement; consider providing incentives to mentees as well."

As part of Phase I, many of the partners did incorporate formal mentoring training, some from the beginning, and some by the fourth year the program. If Phase I were to continue we would have made such training a standard part of AAM. Phase II does not have mentoring as a course requirement.

"Use web-based tools and exercises to deliver more dynamic experiences with primary sources to teachers and students. The web resources prepared by the home school partner, for example, can be leveraged for broader K-12 educational use and interactive exercises like those available via the websites History Matters or Picturing Modern America can enhance learning for teachers and students."

This is an excellent suggestion and one we will be implementing. The Home School Program is currently preparing a version of its online curriculum for the classroom teacher. Once the current curriculum (version 1.0) is field-tested, our next round of improvements will include this suggestion. It will be implemented in both the home school and grade 4-12 classroom curricula. Development is scheduled to begin in the winter of 2005. Product release will be in the summer of 2005.

"Add to the home school program some kind of parent and student training in the use of digitized primary source material."

This type of training is now part of the home school program and provides step-by-step examples in using LOC resources. Both parent and student are walked through the process of accessing and using LOC materials. Initial feedback from parents and students strongly indicated that this approach was needed.

As we continue to develop the Home School Program we will be adding more open-ended options for home school students who have been through the initial training.

"Continue to conduct formative research on the home school partner program, making sure to include questions about access, usability, and practicality. Participants responded well to the web-based materials but, for example, it was unclear whether home access in terms of equipment and Internet connectivity was robust enough."

We are continuing to assess the home school program in all its aspects. The assessment process is dynamic and heuristic. We continue to refine and elaborate the process based on our experience so that our insights and the products we create continue to improve.

With regard to participants’ Internet accessibility, cultural and economic trends continue to favor improving conditions. From a strategic perspective the most responsible course of action is to create a program that will meet the needs of most home school families over the next few years. The technical landscape is rapidly improving in the home as it is in the public schools. We are making every effort to address the needs of the future as it arrives, and not create programs that meet the needs of the past.

From the Program Outputs Section

"Create a formal AAM ‘alumni’ program through which teachers can continue to be involved in the work. By creating ongoing ties (via web forums, electronic ‘posters’ and local and regional ‘meet-ups’) participants may be more likely to continue implementing their knowledge in their classrooms. In addition, alumni could be utilized in future programs by mentoring participants."

Every partner has programs that support its alumni; for example Montreat College has incorporated an AAM alumni program since 2000. These programs demonstrate the positive value of this suggestion.

Our alumni programs could be developed into a more formal structure; therefore I will bring this recommendation to the AAM Executive Committee and ask them to create an action plan. We will implement this in 2005.

"‘Brand’ AAM more clearly so that teachers form a professional identity that embraces the Library of Congress, using online primary sources, and enhanced teaching. Phase I graduates did not identify strongly with AAM, but rather with their local campus program and staff. Consider calling teachers ‘Fellows’ or ‘Partners’ of AAM or the LC. Use technologies to foster ongoing sharing and communication around the work."

We have done many things to create AAM brand recognition. Among these are the standardization of the AAM logo, brochure, reports, color schemes, use of the term "AAM program" by the partners, and ongoing support activities for our graduates.

I would like to further the branding process; however, because this may require moving beyond the boundaries of our current program, I see this as a decision that needs to be made by the ERC Board of Directors and approved by the Library of Congress. It is not clear at this time what the intent or desire of the ERC Board or LOC is with regard to additional AAM branding.

"Examine the utility and usefulness of allocating resources to a lesson plan database. Many such lesson databases exist but there are few broad data suggesting that teachers use them frequently or effectively. Moreover, most Phase I teachers did not use existing AAM lessons or Learning Page lessons in their work, except when asked specifically to look at them as models during the training. They were more likely to reference existing databases such as NC Learn. Therefore, it is questionable whether they would perceive an AAM lesson and unit plan database to be valuable."

The lesson plan database did not exist until the end of Phase I, so naturally Phase I teachers could not use it. Furthermore, at least one partner (Western Carolina University) reports that their teachers did regularly use the Learning Page Lessons for a reference. This CCT finding may be an anomaly of the sampling process.

The creation of the lesson plan database is a requirement of the current grant. So for that reason alone we will continue to develop this resource. In asking groups of teachers in Phase II training, I have found that more than half have accessed the lesson plan database for reference and inspiration. It seems logical that lessons specifically designed to integrate LOC primary source materials would be of value to teachers learning how to do this themselves.

I agree that an objective evaluation of this resource should be undertaken. Therefore, I will recommend that the CCT include this as a component of the 2005 evaluation.

"We recommend conducted iterative formative research on any lesson database tool. In addition, since most teachers come to American Memory looking for ‘stuff’ – content resources to use with students – and since they often have difficulty finding appropriate materials using the search tools, more useful than lesson plans might be an index of individual American Memory documents that have been deemed to be rich and compelling for students in different grades and subject areas to examine – a ‘treasures’ of American Memory feature. Individual documentary ‘treasures’ might be accompanied by 10 – 50 minute exercises that get students actively reading, debating and responding to the documents in writing, role-playing, etc."

This is similar to a suggestion by the Federation of Independent Illinois Colleges and Universities through one of their member schools. They call the index of individual American Memory documents a "pathfinder" which is a quick reference list to resources linked to specific academic needs. For example, a pathfinder might be a list of primary source materials highly suited for use with a states’ standard course of study at a specific grade level and subject. The LOC contains a section called Path Finders that references collections by topic. The AAM pathfinders will need to be more specific and linked to particular academic requirements. We will be developing these in 2005.

With regard to the short exercises or activities, we are designing a new database and a new set of evaluation criteria for Phase II. These shorter learning exercises are referred to as "learning experiences" in Phase II, and the new database is called the Learning Experiences Database. This new collection of shorter learning activities combined with the pathfinders seems to fulfill the suggestion given above.

From the Director Training and Preparation Section

"Create a formal director training program. Continue to strive to build greater coherence across all programs, while preserving adaptability to local circumstances. In Phase I, while ERC provided objectives for the directors, each director decided his or her personal vision is for the program."

As one director said, ‘Everyone started it a little differently and does it a little differently. It’s almost like you had to find out where it is you feel comfortable in what you do, so there are pros and cons to that.’"

This is a valuable point; however, it is not clear if the AAM program will continue to grow or if the growth from 2002 to 2004 was a one-time event. The history of the program is one of unanticipated growth. Looking forward to the 2005 year, we are planning for just one new partner.

With the addition of each new partner we have refined and standardized the director training program. We currently have:

• An 80-page new Partner Handbook
• A standard orientation in North Carolina
• A standard orientation at the LOC

"Short of standardizing course or workshop structure, strive to distinguish the program features that are core and required (and the rationale behind each of these), from those that are more optional."

This is exactly the process we are engaged in now. Based on our experience with Phase II, the directors are attempting to identify the core pedagogy to be used and outcomes that every AAM program must achieve.

"Have faculty instructors/directors visit the Library of Congress in DC prior to teaching the course."

We have been formally doing this since 2003, and informally since 2000.

"Implement mechanisms for formal and informal sharing of knowledge and practice by directors."

This is one of the primary reasons for the semiannual directors’ meetings. As the structure of these meetings has evolved, we have created more and more opportunities for the directors to network in formal and informal settings. In addition we now have standing and ad hoc committees to advise the national director on a wide range of issues. These provide additional formal settings for collaboration among the directors.

In addition we have always had a culture that values and rewards collaboration among the directors. This tradition continues and is demonstrated throughout the year by joint presentations at conferences and through workshop development, informal networking, and mentoring.

"No research report can conclude without a final note about ‘further research needed.’ Well designed formative and summative evaluation research should have a powerful role in the AAM program as it evolves. The project’s ambitions – to have teachers acquire and then quickly turn over to their students the ability to learn using new technologies and the online resources of the Library of Congress – requires that teachers learn in three domains at once. Coupled with this is an ambitious programmatic goal: To deliver training in many different sites around the country simultaneously via an expanding network of partners and affiliates. At both instructional and programmatic levels, AAM would benefit greatly from having in place, at the outset of significant new work, a set of evaluation diagnostics that would help teachers, instructors, and administrators monitor, assess, and improve their progress toward clear benchmarks of knowledge and skill.

"Working with researchers to craft an evaluation ‘toolkit’ of flexible techniques that programs can use to assess their work would pay off in many ways. First, it would address what is currently an ‘assessment gap’ in the AAM program. Second and more important, it would give teachers – and ultimately students themselves – insight into the many ways students learn, and learn differently, using these new materials. Rigorous, flexible assessments would go a long way toward making the AAM program as a whole what Bransford, Brown, & Cocking (2000) cite as most effective for teachers and students: a knowledge-centered, assessment-centered, and learner-centered professional development program."

We are currently working with the CCT to achieve this goal.

An Adventure of the American Mind is made available through a Library of Congress grant funded by the U.S. Congress. AAM is administered through the Education and Research Consortium of the Western Carolinas Inc.
For site-related problems, e-mail aam@ercwc.org..