October 2006
Independent Project Evaluators
Dr. Mary Dingle
Dr. Jennifer Mahdavi
The Effective Reading Intervention Academy (ERIA) is a project developed and implemented by California Services for Technical Assistance and Training (CalSTAT). CalSTAT is a special project of the California Department of Education, Special Education Division and is part of the California Institute on Human Services located at Sonoma State University. It is funded through the Special Education Division and the California State Improvement Grant (SIG). The SIG, a federal grant, supports and develops partnerships with schools and families by providing training, technical assistance and resources to both special education and general education.
One focus of the SIG is to communicate common messages to the field about selected topics. These common, or core messages, articulate critical research findings and essential components of effective application. All core messages have been identified by experts in the field and have been approved by the California Department of Education, Special Education Division. Reading is one of the identified common messages and is further defined by five focus areas. These areas include, (1) a focus on effective reading instruction, (2) early identification and prevention, (3) assessment that drives instruction, (4) assess to core curriculum and (5) practices that are linked to research.
To provide training and technical assistance in this area, CalSTAT contracted with three area consortiums to deliver specialized intervention and educational strategies for reading instruction in the upper elementary and middle school grades using the ERIA model. ERIA is based on the work of Dr. Anita Archer. It incorporates seven programs, focusing on decoding skills, fluency, comprehension, study skills, and paragraph and essay writing. Each academy features research-based curricula and techniques that work in supplemental programs, general and special education, summer school, after school programs, Title I, and academic year and interim sessions. Teachers learn skills grounded in research-based practices, which they can use to increase the reading skills of struggling readers. In addition teachers learn how to develop an intensive reading intervention program for elementary and middle school students.
To support the implementation of ERIA in the three consortiums, CalSTAT provided funding for initial trainings from nationally recognized experts and provided an outside, expert coach to work with the individual schools in each of these consortiums to assist in program implementation. One outside coach worked with the West Orange County Consortium and Antelope Valley Special Education Local Plan Area (SELPA). A literacy expert from the Diagnostic Center, Central Valley served as the outside coach for the Fresno Consortium. ERIA was implemented in West Orange County and Antelope Valley in January 2005. Implementation of ERIA in Fresno began in September 2006.
In January 2006, the independent project evaluators began work with CalSTAT to evaluate the Effective Reading Intervention Academy (ERIA) that had been implemented in the West Orange County Consortium and the Fresno Consortium to describe the coaching models that had been developed by the two outside, expert coaches.
This evaluation has two main goals. The first goal is to describe the ERIA coaching models that were developed by the two outside coaches, including an evaluation of the strengths and challenges of each model. The second goal is to describe an ideal ERIA model that would inform efforts to scale up this project in the coming years.
This initial evaluation design was based on the premise that the main focus of ERIA was coaching individual teachers in their own classrooms. This design was quickly revised to include a much broader scope of activities that were defined as coaching, including follow-up training on assessment procedures and programs, meetings with administrators, classroom coaching and debrief sessions which was more closely aligned to the coaching models at work in ERIA.
The primary investigator (PI) began this evaluation by talking with the outside coach from the West Orange County Consortium and the Antelope Valley SELPA about the scope of her work in ERIA. The PI also read the previous year's evaluation report, familiarized herself with the ERIA manual, and did a short review of the literature focused on professional development models that included coaching. She then traveled to a middle school in the West Orange County Consortium and observed two coaching sessions, attended a debrief session with the teachers, their administrator, and the coach, and met with the school principal and a district representative. She also spoke via telephone with the outside coach from Fresno about her work with the participants of ERIA project.
From this point, several focus groups were organized. Participants in the Fresno consortium were invited to attend a focus group held at the Diagnostic Center, Central California in March 2006, organized by Jane Ketchem. Two additional focus groups were held for the participants of the West Orange County Consortium in May 2006, organized by Michel Hsiang. The PI interviewed the coach from the West Orange County Consortium after facilitating these focus groups. Initial attempts to contact participants from the Antelope Valley SELPA were not successful, and they were not included in this evaluation.
Interviews of the outside coaches, the observations of the middle school classrooms, and participation in the debrief session are also utilized as background information and to clarify specific points.
The focus group questions were organized around five identified areas. These included the history and vision of ERIA, the current status of the project, the successes and challenges, coaching, and sustainability issues. Each identified area included one or two specific questions and additional suggested probes.
We held five concurrent focus groups for the Fresno Consortium in April 2006 with a total of 28 participants from nine schools. There was a range of 4 to 7 participants in each group. The participants included 5 general education teachers, 9 special education teachers, 8 administrators, 4 reading specialists and one representative from the Tulare County Office of Education. Years of experience in education ranged from 4 years to 31 years. Fifty-four percent of the participants (n=14) held a Master of Arts Degree (MA).
The focus groups were facilitated by members of the California Diagnostic Center, Central California staff and two faculty members from the Department of Educational Leadership and Special Education at Sonoma State University. The facilitators met together and reviewed the focus group protocol and decided on a common format for the focus groups. Each question was allocated a specific amount of time, and participants were asked to respond in a round robin format, with the initial respondent changing each time. Each focus group was videotaped, however due to technical difficulties, only four of the videotapes were able to be transcribed.
In the West Orange County Consortium, two focus groups were held in May 2006. Sixteen administrators, one teacher and a reading coach attended the first focus group. Participant years of experience ranged from 2 years to 39 years. Ninety-four percent of this group (n=17) held MA degrees. Participants were again asked to respond to the focus group questions in a round robin format, with the initial respondent changing each time. The focus group responses were audio-taped. The original focus group protocol was shortened to accommodate the number of participants. The modified focus questions for this group included the current status of ERIA in their schools, the strengths and challenges of the ERIA model and questions about sustainability of the model in future years.
The second group of West Orange County participants was comprised of 12 teachers. Eleven teachers were general education teachers, and one teacher was a special education teacher. Years of experience ranged from two years to thirty-one years. Fifty-eight percent of the teachers (n=7) held MA degrees. Most of the teachers taught at middle school (67%), however there were also three upper elementary school teachers (grades 4-5) and one high school teacher who were members of the second focus group. Focus group questions for this group focused on implementation of ERIA in their classrooms and question about coaching. One reading coach attended both of the focus groups in West Orange County.
The independent investigators also developed an online survey that was sent out via email to the project participants. Questions clustered around five main themes: (1) the value and utility of the components of ERIA, (2) the assessment process, (2) implementation issues, (4) coaching and (5) support. There were three versions of the survey, one that was distributed to the administrators, one to the teachers and one to the reading coaches. There were approximately 50 questions that participants answered using a Likert scale with a range of responses from one to four. There were also opportunities to provide open-ended responses. Twenty-one ERIA participants responded to the survey, including 6 administrators, 13 teachers, and 2 reading coaches, a response rate of 11%. The three respondent groups were collapsed to provide a more robust number of responses to consider in this analysis.
The focus groups' audio and videotapes were transcribed and analyzed for thematic content. As the data was analyzed, a general explanation of the ERIA model emerged. Initial codes were assigned to the data, and by continuing to examine the responses to the focus group questions, researchers were able to make connections between and among the categories to form a more complete picture of the ERIA project.
The survey responses were also analyzed and frequency scores and means were calculated to provide more detail to the initial evaluation questions. Although mean scores were used as a level of analysis, these were only to suggest trends. There was not enough statistical power to determine whether the mean scores held any significant difference from each other.
This report will describe the result of the analysis of the focus group and the results of the survey. The main body of the report comes from the analysis of the focus group participants. Quotes from the focus group participants are used throughout the report, in italics. These quotes are found at the beginning of many sections in yellow boxes, in the text, and in parentheses. The survey information is identified each time it is used in the report, to avoid confusion about the data sources.
The ERIA coaching models that developed in the two consortiums were very similar. The participants in the West Orange County focus groups and the participants in the Fresno focus groups reported similar ERIA activities at their schools that were developed and implemented with the support of CalSTAT. These included attending the two-day academy (workshop) given by Kevin Feldman and implementation of the ERIA model. Implementation included assessment of students; data based decision-making comprised of grouping students and implementing intervention curriculum and strategies; and outside, expert coaching that included working with teachers in their classrooms and ongoing trainings and meetings at the school sites. Some schools also had an onsite coach who was instrumental in the development of ERIA model that developed at their schools.
Figure 1 provides a visual representation of the survey respondents' participation in the various components of the ERIA model. All of the survey responders attended the academy, and many attended trainings and debrief planning sessions with the outside coach. Only 3 of the 12 survey respondent teachers had individual coaching in their classrooms with the outside coach, and only one of the teachers had worked with an onsite coach at their school to implement ERIA. The respondents valued all components of the ERIA model, however, the academy and ongoing trainings show the highest mean score (Table 1). The debrief sessions with the outside coach were also highly valued. The survey respondents reported that each of these components prepared them to implement the ERIA model in their schools (Table 2).
Figure 1: The frequency with which survey respondents participated in ERIA activities (n= 20).
| Academy | 20 |
| Trainings | 12 |
| Classroom Coaching with Outside Reading Coach | 5 |
| Debrief Sessions with Outside Coach | 9 |
| Classroom Coaching with Onsite Reading Coach | 2 |
| Debrief/Planning Sessions with Outside Reading Coach (Administrators Only) | 3 |
Table 1: Mean scores for the perceived value of each component by survey respondents.
| n | Min | Max | Mean | |
|---|---|---|---|---|
| Academy | 20 | 2 | 3 | 2.9 |
| Trainings | 14 | 2 | 3 | 2.9 |
| Classroom Coaching by Outisde Coach | 10 | 2 | 3 | 2.8 |
| Debrief Sessions with Outside Coach | 10 | 2 | 3 | 2.3 |
| Classroom Coaching with Onsite Coach | 6 | 2 | 3 | 2.3 |
| Debrief/ Planning Sessions with Outside Coach (Administrators Only) | 4 | 2 | 3 | 2.8 |
Table 2: Mean scores for the extent this component prepared the school or the respondent to implement the ERIA model.
| n | Min | Max | Mean | |
|---|---|---|---|---|
| Academy | 20 | 2 | 4 | 3.5 |
| Trainings | 14 | 2 | 4 | 3.3 |
| Classroom Coaching by Outisde Coach | 6 | 2 | 4 | 3.3 |
| Debrief Sessions with Outside Coach | 10 | 2 | 4 | 3.2 |
| Classroom Coaching with Onsite Coach | 2 | 4 | 4 | 4 |
| Debrief/ Planning Sessions with Outside Coach (Administrators Only) | 2 | 3 | 4 | 3.5 |
The support structure that CalSTAT organized to implement ERIA began with the two-day ERIA "academy" given by Kevin Feldman in West Orange County in January 2005 and Fresno in November 2005. During the two-day academy, Dr. Feldman explained the overall picture of reading intervention and the role of assessment. He also provided specific training on how to implement the ERIA assessments, curricula and instructional strategies to achieve the overarching goal of supporting struggling readers.
Though most participants reported on the positive impact of the speaker, some participants reported a lack of clarity about the project and the expectation for their involvement in ERIA. Some mentioned that they first became involved in ERIA because they went to hear Kevin speak. One individual explained, "When we first got invited to hear Kevin Feldman, we didn't know what we were getting ourselves into, and all of the sudden it became ERIA." Another focus group member in West Orange County pulled me aside after the focus group and told me that we should make sure that we explained to future participants what they were getting into. She had also gone to hear Kevin Feldman speak ("and who wouldn't take advantage of an opportunity to hear Kevin speak?") and found out at his presentation that she was a participant in the ERIA program.
The academy also became the foundation for a wonderful collaborative community on many different levels. Ten school sites from each area attended the academy, each school with a team of ten teachers and their administrator. The administrators in the West Orange County Consortium commented on the unique opportunity to work at a regional level, across districts. Many participants discussed the collaborative work within districts, schools and grade level teams. The academy helped build a common focus and a common language for teachers and administrators to begin working together that was identified as a major strength of the program, and is discussed in more depth later in this report.
Another critical component of the ERIA identified multiple times were the actual intervention tools that comprise ERIA. Assessment and data-based decision making that included grouping students, implementing the ERIA programs and strategies, were all at various stages of development within the schools. In some schools, only targeted grades or targeted groups of students were considered for the ERIA intervention, while at other schools the entire student body was assessed and grouped according to reading ability. The schools in West Orange County Consortium were further along in their implementation, due to the fact that this was the second year they had been working with ERIA. Most of these schools had their assessment procedures functioning, and had been grouping students for intervention in a variety of ways, and were implementing one or more of the ERIA curriculum and/or instructional strategies. Many of the schools in Fresno had also moved to this stage, having been to the academy six months prior, but some of the schools were still exploring the different curricula, and learning to manage the data and how to group the students.
The participants understood the importance of an initial assessment to screen students to determine their reading abilities as an essential tool to make intervention decisions. In the Fresno Consortium, the outside coach had also trained many schools to use the Dynamic Indicators of Early Literacy Skills (DIBELS) assessment prior to ERIA project, so the assessment process was already in place with these schools. In the West Orange Consortium, many schools were also involved in an assessment process, although not always those identified in the ERIA model. Some of these schools continued to use their current assessments if the tool provided the needed data to make intervention decisions. For most of the participants in the schools described above, schools, the initial, or screening assessment in ERIA was a familiar process. According to the survey respondents and the focus group participants, DIBELS, the San Diego Quick and REWARDS were the most commonly used assessments (Figure 2).
Figure 2: Which assessments were used to gather student data (n=21)?
| DIBELS | 16 |
| Reading Fluency Monitor | 9 |
| San Diego Quick | 16 |
| Phonics for Reading | 4 |
| REWARDS | 16 |
| Read Naturally | 9 |
| The Six-Minute Solution | 9 |
| Other | 3 |
For other schools, not currently using a screening assessment, there were many challenges involved in this initial assessment process. The first set of challenges identified was clustered around logistics. Who collected data? How was the data collected? Who compiled the data to be analyzed? In schools that had a reading specialist or reading coach that was "on board" with ERIA, this individual was often responsible for assessing the target students. In some schools ERIA was housed in special education, and these teachers took the main responsibility to assess students and collect data. In many other schools, the teachers assessed the students, as is suggested from the survey data in Table 3.
Table 3: Mean scores for the extent that individuals were involved in the ERIA assessments.
| n | Min | Max | Mean | |
|---|---|---|---|---|
| Principal | 21 | 1 | 4 | 2.4 |
| Outside Coach | 19 | 1 | 4 | 2.1 |
| Onsite Coach | 6 | 1 | 4 | 2.5 |
| Peer and Other Teachers | 10 | 1 | 4 | 3.5 |
| Self (teachers only) | 2 | 1 | 4 | 3.7 |
Once the assessments were given, some schools had a central location where a designated individual colleted the data in an electronic format, but not all the schools had participants who had the skills, the time, and/or the foresight to make this an integral part of the data collection. The outside coach from West Orange Consortium described one school where the participants had these technological skills:
The next set of challenges identified was processing the results of the data. This included becoming familiar enough with the assessment to understand the information provided. One school reported the shock of seeing the actual screening results for the first time because they hadn't realized how many students were reading below grade level, and although not all schools shared this experience, many participants agreed that this was an eye-opening experience.
According to the survey respondents, teachers were the individuals most involved in the analysis of student data, followed by onsite coaches and principals (Table 4). Most respondents valued the input of teachers and onsite coaches over that of outside coaches and principals, however there was little differentiation in the mean scores (Table 5).
Table 4: Mean scores for the extent that individuals were involved in the analysis of student data.
| n | Min | Max | Mean | |
|---|---|---|---|---|
| Principal | 18 | 1 | 3 | 2 |
| Outside Coach | 18 | 1 | 3 | 1.8 |
| Onsite Coach | 18 | 1 | 3 | 2.4 |
| Peer and Other Teachers | 21 | 2 | 3 | 2.4 |
| Self (teachers only) | 13 | 2 | 3 | 2.8 |
Table 5: Mean scores for the extent that individuals valued the input of others in the analysis of student data.
| n | Min | Max | Mean | |
|---|---|---|---|---|
| Principal | 12 | 1 | 3 | 2.3 |
| Outside Coach | 16 | 1 | 3 | 2.3 |
| Onsite Coach | 12 | 1 | 3 | 2.6 |
| Peer and Other Teachers | 19 | 2 | 3 | 2.7 |
| Self (teachers only) | 13 | 2 | 3 | 2.8 |
The final set of challenges in the initial assessment process clustered around how to make placement decisions based on the assessments, how to group students and what curriculum and instructional strategies to use. These challenges will be discussed in the following sections that focus on curriculum and instructional strategies.
Many schools were in the process of collecting the year-end data during the time of the focus groups. The challenges at this point included scheduling the time to assess students around other mandated assessments, convincing teachers to allow time for students to take assessments, and analyzing the data based on sub-groups and type of intervention implemented to determine the success of the intervention.
Most of the participants described the assessment process in general, without differentiating between screening, placement, progress monitoring or evaluation. Only a few focus group participants described progress monitoring of their intervention students during the intervention, although this is an integral part of most reading intervention programs and many of the curricula being used included embedded assessments. When progress monitoring was mentioned, it was more of an evaluation tool and there wasn't any discussion of how these assessments were being used to make instructional decisions during intervention, except for one school that had previously used the Success for All Reading Program, and reassessed and regrouped students at regular intervals during the school year. For many of the schools, assessment seemed to be collected systematically only as requested by CalSTAT. This finding may only be an artifact of the focus group questions, but is also an area to probe in future trainings and evaluations.
Most participants of the focus groups had positive impressions of the assessment process. As well, 57% of the survey respondents reported that the assessments were easy to administer (n=12), while forty-three percent said that they were moderately challenging to administer (n=9). Forty two percent of the survey respondents (n=11) reported that the assessments were easy to interpret, while an additional fifty-two percent reported that they were moderately challenging to interpret (Figure 4).
Figure 4: Ease of administering and interpreting assessments (n=21).
| Number responding "Difficult" | Number responding "Challenging" | Number responding "Moderate" | Number responding "Easy" | |
|---|---|---|---|---|
| To what extent did you find the reading assessments easy to administer in your school? | 0 | 0 | 9 | 12 |
| To what extent did you find the reading assessments easy to interpret? | 0 | 0 | 11 | 9 |
The focus group participants reported that they appreciated the focused assessments to evaluate student progress from the beginning to the end of the year. They felt that the assessments allowed them to "hone in what the needs were of each student." They learned "what it told me and what good it does." The assessment process also assisted the schools to group students and choose intervention programs.
The assessment process led to making data-based instructional decisions. The flexibility of the ERIA model allowed each school to design a specific model that met the unique needs of the students and the school structure. This is an important design attribute because the participant schools were very diverse in term of student needs, school grade range, and capacity for change. Schools also had different structures for making these instructional decisions. Instructional decisions clustered around how to group students for intervention and curricula and strategies that were implemented.
In some schools the decisions on how to implement ERIA began with discussions between the administrator and the outside coach as the administrators began to think about how to implement the ERIA model into their specific school context. Other schools used leadership teams comprised of reading specialists, lead teachers and administrators as decision makers. Grade level teams and literacy teams were responsible for instructional decisions in other schools. One grade level team left the academy with a "blueprint" outlining how to begin the ERIA program in their classrooms. In another subset of schools, individual teachers made decisions in their own classrooms.
Some schools had more difficulty with the decision-making aspect of implementing ERIA. One newly assigned principal spoke about the challenges of a new position at a new school, and how difficult it was to focus and support ERIA at such an early stage of his administration. Other schools had similar challenges. One school had changed districts, and other schools were designated by the state as "Program Improvement" schools, based on state test results and had state mandated interventions already in place. Other schools had just implemented new instructional programs and were adjusting and becoming familiar with the new curriculum. Capacity for change and for implementation of the ERIA model is an important consideration. However, it is important to also note that the schools that had not been able to implement ERIA fully had a plan in place of what the next steps would be, and each school was still very interested in continuing towards full implementation and working with the ERIA project.
The most common models for grouping students for intervention includes (1) providing intervention to the entire class for a short time (e.g., Six Minute Solution for the first ten minutes of class), (2) pulling kids from class in a typical resource type model and (3) platooning (as one district calls it in the Fresno consortium) where students are grouped for instruction across classroom and/or grade level and move to different classrooms for a set intervention time. At the middle and high school levels (and one elementary school), students were scheduled to attend an entire class period to work on intervention strategies (called a reading class). One middle school had every teacher in the school focus on reading intervention in their respective advisement class, from physical education to pre-algebra. Another middle school included reading intervention on the elective wheel, so that each student had the opportunity to participate in a reading intervention class, but that they also had the opportunity to participate in the other elective subjects typically offered at the middle school level. Other schools had after school programs and summer programs that utilized the ERIA model.
One model that was not discussed was grouping students within the teacher's own classrooms for small group intervention. This may be an artifact of the focus group questions (it may not include the challenge that other models do, and thus not an issue) or an artifact of the upper elementary grade levels in ERIA, where small group work may not be as common as in primary classrooms.
REWARDS was the curriculum most often reported being currently used by the focus group participants, closely followed by the Six Minute Solution and Reading Naturally. The focus group participants also reported using Phonics for Reading and Skills for Success, but to a much lesser extend than first two curricula. Survey respondents reported using the entire ERIA curriculum, with REWARDS also being reported by survey participants as the most frequently used curriculum (Figure 3).
One school in the West Orange County Consortium focused on active participation strategies as a major ERIA focus for the teachers, and reported excellent results from the concentrated coaching in this area. The survey respondents also reported that this was the most frequently used component of ERIA (Figure 3). Most schools were pleased with the results of using these curricula and strategies, however, both outside coaches did spent time retraining and/or supporting teachers in learning how to implement and sustain these programs.
Figure 3: Strategies and curriculum used by ERIA participants (n=19).
| Active Participation | 16 |
| Rewards | 13 |
| Phonics for Reading | 5 |
| Read Naturally | 4 |
| Six Minute Solution | 8 |
| Novel Partners | 5 |
| Skills for Success | 3 |
| Paragraph and Essay Writing | 4 |
What became clear through the focus group interviews was how these programs aligned with the teachers' previous knowledge, beliefs, and practice. Many teachers and administrators had previously worked with Kevin Feldman and Anita Archer and were familiar with their philosophy and curriculum ideas, in particular ideas about direct instruction and intervention. Other teachers, particularly those with a special education background were already using REWARDS and Read Naturally.
Some schools reported that the curricula also "filled a gap" in their literacy programs. Administrators described the needs of a "strategic" group of readers, not those at the lowest level of reading ability, but students who were still not reading at grade level. REWARDS and the Six Minute Solution, and in some cases Phonics for Reading, seemed to fit easily into many of the intervention structures developed by the schools and focus instruction to specific areas of need for groups of students. The results of the instruction, as assessed through the embedded assessments, showed progress in the targeted skill areas, which made the students, teachers and administrators feel very successful.
ERIA supported each consortium with an outside coach. Coaching activities, as reported by the recipients, included consultation and mentoring; observations and in class teacher coaching and debrief sessions; and on-going training in assessment and program implementation.
Consulting usually included discussion with the administration or the site point person (sometimes the onsite coach) to identify priorities and needs that focused the coaching activities. In the role of consultant, outside coaches kept up an ongoing dialogue, provided on going support, mentored schools and individual teachers by "pointing in the right direction" and meeting with leadership teams and staff as appropriate. They assisted with solving scheduling and organizational challenges, including how to "keep the data in order." The coaches also observed and coached in the classroom, often modeling how to give assessments or use curriculum and strategies and would meet the teachers afterward the coaching session for a debrief session. The focus group participants reported that the two-day academy, while wonderful and inspiring, was also so filled with information that by the time it came to implement the different aspects of ERIA, they felt the need for another training session, so the coaches also gave workshops and/or additional training as needed to retrain teachers in specific ERIA components.
Figure 4: How many times did you meet with the outside coach (n=20).
| Number responding "0 times" | Number responding "1-3 times" | Number responding "4-6 times" | Number responding "7 or more times" | |
|---|---|---|---|---|
| Workshops | 6 | 6 | 1 | 1 |
| Individual Coaching | 10 | 5 | 1 | 0 |
| Debrief Sessions | 6 | 10 | 3 | 2 |
The survey respondents also reported a range of experiences and amount of contact with the outside coach (see Figure 4) and most respondents expressed a high degree of support and level of satisfaction with the level of support provided by the outside coach (Figure 5). This however, was not true for some. Two teachers and one onsite coach reported no support from the outside coach. Two teachers, one onsite coach and an administer were not satisfied with the level of support received from the outside coach. Most respondents, however, felt that the coach was helpful in implementation of the ERIA model, interpretation of the assessment reports and training participants to use the curricula and strategies (Figure 6).
Figure 5: Level of support and satisfaction with the level of support from the outside coach (n=19).
| Number responding "Not at all" | Number responding "A little" | Number responding "Some" | Number Responding "A lot" | |
|---|---|---|---|---|
| Level of Support | 3 | 3 | 4 | 9 |
| Level of Satisfaction | 4 | 2 | 2 | 10 |
Figure 6: Helpfulness of reading coach in specific activities (n=17).
| Number responding "Not at all" | Number responding "Somewhat" | Number responding "A lot" | |
|---|---|---|---|
| Implementation of Model | 1 | 6 | 11 |
| Interpretation of Assessments | 4 | 3 | 10 |
| Forming Reading Groups | 4 | 7 | 6 |
| Curriculum and Strategies | 3 | 6 | 9 |
Coaching in the classroom is one area where the two coaches had very different coaching styles. The coach from the West Orange County Consortium had a unique co-teaching coaching style. She would work with the teacher, discussing at the beginning of class different strategies to implement when possible, and stopping the teacher during the lesson to suggest and model different active participation strategies. She would meet with the teachers she had coached after class to discuss (debrief) the coaching session. In the session that the principal investigator observed the administrator also attended this debrief session as well as other teachers who had also observed the session. The coach from the Fresno Consortium reported that she had an observation/coaching format. The coach would come into class, observe the teacher, and meet with the teacher after the observation for a debrief session.
Teachers from both consortiums reported being very nervous before the coaching sessions, but ultimately, most found these sessions in the classroom to be very helpful.
During the West Orange County Consortium focus group, I had the opportunity to discuss the coaching experience in depth with a group of teachers. While most of the teachers reported that they really benefited from the coaching session, there were reports that some teachers found the co-teaching/coaching model and interruption of their teaching disconcerting, and did not find it helpful. In one case, the teachers had not been notified of what the coaching would entail, and were not prepared for the coaching experience and in another case, the teacher found it difficult to keep focused on the lesson objectives with the continual input of the coach. However, the majority of teachers found both coaching models very helpful, and there were many positive comments from all the focus group participants:
The outside coaching component was one aspect of ERIA that all participants agreed should not be eliminated as plans to scale up the ERIA project are made. The consensus from the participants is that the outside coaches provide accountability (if [outside coach] is coming, I better be ready), expertise (someone from the outside had more validity), and a less biased viewpoint (it is easier to hear it from the outside). The outside view and perspective assisted the specific schools in examining different options and possibilities as they planned and problem solved to get ERIA up and running.
While onsite coaching was part of the original model of ERIA, this was one aspect that did not seem as consistently developed as the other ERIA components. Some schools had literacy coaches while other schools did not. Some literacy coaches were an integral part of ERIA, while other literacy coaches were not. One school had identified an individual who had anticipated being trained as a coach, but at the end of the second year had not been trained. At one school, ERIA was situated in the special education department and these teachers were training other teachers to use the ERIA model. Some schools had identified an ERIA point person, but this person did not have any release time to work with the ERIA participants. One administrator had been identified as the ERIA point person.
What seemed to be the most productive outcome was that there was a school literacy coach already identified who worked with the administration to implement ERIA on campus. This individual worked closely with the outside coach and supported individual teachers by assisting with the assessment, procuring materials, training individual ERIA curricula as needed, and observing and coaching individual teachers in the classroom, sometimes along side the outside coach. Onsite coaches would also lead team and grade level meetings and help analyze the data. In the most positive reports this individual worked closely with supported the teachers to implement ERIA and became a link between teachers and the outside coach and administration.
"I have received requests from other districts that they would like to participate in ERIA. They have heard that the support, the coaching is there."
There were many identified strengths of the ERIA model. Attending the academy, the outside coaching and the flexibility of the model were reported most often as being the strongest aspects of the program. Although not identified as a strength by the participants, collaboration emerged as a powerful theme throughout the focus groups. Alignment with previous beliefs and practices, student outcomes and changing practices at the school sites were also frequently mentioned as strengths of the program. Participants also reported the value of learning more and becoming empowered as teachers and researchers as a positive outcome and strength of the program. Some of these themes have been previous discussed in this report, and are summarized again in this section.
The focus group participants consistently reported that attending the academy (sometimes referred to as listening to Kevin Feldman speak) with a large group of school faculty and administration was a key strength of the ERIA model. Many explained how inspired they were by Kevin's presentation and explanation of ERIA. When asked about what was most important to consider when scaling up the ERIA model, one participant responded, "I just wanted to add to the importance of the speaker, but it needs to be someone as knowledgeable and dynamic as Kevin Feldman." Attending the initial academy together inspired many participants and provided an opportunity for core group of faculty to have a shared vision and common focus.
Having the outside coach support the individual schools to implement the program provided accountability and expertise. Administrators, literacy coaches, other point personnel and the individual teachers all reported that the coaching/support is what made this program successful and unique from other professional development activities that had previously experienced.
Another essential aspect of the ERIA models developed was the flexibility of the model. Schools could incorporate the entire model, or schools could decide to implement only the elements of the model that were most salient to the needs of the students, the structure of the school and programs already in place, or possible due to the capacity of the school to implement the program. The coaches were very supportive and followed the schools lead in determining what ERIA would look like at their site, and offered guidance and alternative options to consider as schools implemented ERIA.
Another clearly identified strength of the program was the collaborative outcomes. Participants spoke of the opportunity to collaborate regionally, across districts, and within schools and grade levels. In some schools the ERIA project expanded the collaboration between general education and special education. The most valued components of the ERIA collaboration were a shared vision and a common perspective and language. The direct impact of the shared vision at the school level was the ability for the members of school communities to narrow their focus and develop a plan together to work toward their vision. This collaboration was supported by school and grade level meetings, sometimes facilitated by the outside coach and sometimes facilitated by school faculty. Trainings and coaching activities also supported the collaborative culture that developed in many schools. The impact at the district and regional level included less isolation of the administrators at individual schools, and a new understanding of the value of reaching out and working together on a common project. Whether regionally, at the district, school or grade level, the opportunity for a "broader, richer conversation" was often noted and valued as an outcome of ERIA.
"I don't think that we can give credit [for our success] only to the project, which is the direction we are already going in and fits perfectly with that."
Another strength of the ERIA model was how closely it aligned with the intervention activities already in place in most of the participating schools. Teachers and administrators alike reported that ERIA had a similar focus as the adopted district and school vision and goals. Many participants related ERIA to assessment and instructional practices already in place in their school. Other members of the focus groups discussed how well ERIA had met the needs of some of their most challenging students including English learners, groups of students reading at below and far below their grade level, and/or their special education students. Some members spoke about other programs and projects that they were involved in, including Best Practices, Springboard, Beginning Support and Teacher Assessment (BTSA) and Response to Intervention models (RTI) that shared similar practices and/or beliefs about supporting students, implementing new programs and changing practice in schools. There were references to previous workshops by Kevin Feldman, Anita Archer, Kate Kinsella and Frank Smith that had provided prior knowledge and experience with similar instructional programs. ERIA was appreciated and valued as the next step by many participants.
Other strengths included the impact on student reading skills, motivation, engagement and involvement. Teachers reported that they see "significant growth in reading" and that their students read "more and better." They are more successful in other content classes as well, and have more confidence in reading and sounding out new vocabulary words. The impact on student motivation and engagement was also favorably reported by many of the focus group participants. In some instances students monitored their own progress in the intervention programs, which many teachers felt was very empowering. The use of active participation strategies and the practice of reading aloud through consistent use of the Six Minute Solution curriculum changed the dynamics of some classroom cultures. One teacher reported that before implementing the Six Minute Solution, her students never volunteered to read, but now, they were all eagerly raising their hands, hoping that they would be the next one picked for this task. Teachers also reported how the use of active participation strategies throughout the curriculum had dramatically changed their practice, and allowed them to assess student understanding continually through the lesson. They reported that students seemed more motivated and engaged while actively participating and how quickly time flew by.
Another area of strength identified by the focus group participants was how practice had changed at their sites and in their classrooms. Changes reported included general education teachers understanding and valuing the intervention process and supporting the assessment and instructional changes needed to implement intervention at their sites. This began with a careful look at the assessment data that clearly articulated the students' ability levels. The availability of intervention programs and the structure and support to implement the programs contributed to this change in practice. Changes were also evident in the reports about staff meetings, a unified focus and common language, and the collaborative aspects of working in new and supportive intervention models. Inside the classrooms the reported changes focused the assessment process and the new instructional strategies the teachers had learned.
Teachers also reported that they valued the knowledge and skill that they have gained by their participation in the project. Often this focused on specific strategies, interventions or assessment practices, but there was also a realization of how the participation in ERIA has opened new avenues for exploration. One participant reported, "We are asking new questions." Another explained how her credential program had been subject specific, but ERIA had made her understand how she could infuse reading strategies not only to understand content, but to support the development of reading skills of struggling readers. Other participants reported how empowered the teachers felt by new knowledge and skills and the resulting buy-in from the teachers to support ERIA in their schools. One teacher boasted that she now no longer just gives her struggling readers easy books to read, but provides them with strategies and teaches them skills to improve their reading.
The participants also identified the challenges they encountered as they implemented the ERIA model at their school site. These included scheduling, teacher buy-in, capacity and time constraints, and the need for more training and strong leadership. Some of these themes have been discussed previously in the report and are summarized again in this section.
Scheduling was an ongoing challenge for many schools. There were two main themes in the discussions about scheduling. The first theme included the logistics of scheduling. Logistical struggles included with grouping issues, staffing issues, and meeting the needs of students who were not in intervention. Some schools infused strategies and intervention curriculum into content classes by taking a short amount of time during the period to complete intervention activities, incorporating active participation into every learning activity, and providing opportunities to practice decoding skills and fluency in the daily vocabulary and reading activities. Some schools infused intervention through elective "wheels" at middle school, after school programs and summer school. For many schools, however, students were grouped based on assessment data and a schedule was developed to provide intervention to the identified groups. This proved to be a challenging process.
The second theme that emerged in this area was the difficulty in maintaining access to core curriculum while students were engaged in intervention activities. Some schools did not make a distinction between intervention and the core curriculum so this did not emerge as an issue. The schools that struggled the most with this issue were schools that were identified as "Program Improvement" schools and had clear guidelines to follow in terms of utilizing state adopted core curriculum for the reading period.
Teacher-buy in was also reported as a challenge. Two main themes emerged from the participants' responses in regards to this challenge. First, teachers are concerned with workload. They feel overwhelmed by the current demands of their job, and do not understand how they are supposed to fit assessment and intervention into their already fully scheduled days. Second, in many schools a core group of teachers were ERIA participants, and while in most cases these core teams had embraced the model, this was not always been the case for the rest of the faculty. In some cases this is because the model had not yet been adequately shared with other faculty. In other cases the administrators and other teacher leaders had not promoted the model. In still other situations, the teachers were resistant to the model based on theoretical and philosophical beliefs about the teaching of reading and/or concerns with workload. In one school, the grade level team that attended the academy didn't completely buy in and continued to resist implementing the model.
The need for additional training was also identified as a challenge. Many participants wanted to learn more about assessment. Issues included clarifying the purpose of the ERIA assessments ("can they be used as progress monitoring or are they only screening tools?"), more instruction on how to interpret the assessments, and a desire for more enhanced training ("we just got the basics, I know there is a lot more to learn"). Other participants felt a need for more training on how to implement the curricula ("at our school it was really brief, a training and a video"). There were concerns that the faculty had not all been trained, and those who were trained wanted to be reassured that they are implementing the program correctly ("three days isn't really enough to really learn the programs"). One participant also mentioned the need for more site support and modeling in the classroom.
While issues of capacity and time are intertwined with many of the other challenges that have been reported, they were also frequently reported as one of the foremost challenges facing the schools in the implementation of ERIA. Time constraints included the time to sufficiently train faculty in the content of the programs as well as the actual time to implement the model. Teacher capacity issues were also identified. Some administrators were hesitant to ask their teachers to learn more or take on another program ("teachers are stretched to their limit") as well as within unique challenges that individual schools were currently facing. One school had just changed districts, one school had a number of new teachers, and another school had a new administrator. Participants also commented on the demands of implementing a new curriculum, the pressure of state mandates and the need to teach the core curriculum as barriers to fully implementing the ERIA program.
Three schools reported that the lack of leadership was a challenge to implementing ERIA. At one school the principal was not supportive. At another school it was the literacy coach who did not support the program, and at another school it was the curriculum coordinator. In these situations the teachers continued to implement the program without the support, but found it very difficult. At one school it was the special education teachers who had implemented ERIA with their students, assessing, grouping and implementing intervention specifically for special education students. At the second school, the ERIA point person continued to try to implement ERIA, but was becoming increasingly discouraged with the lack of interest and support from the administration. The third school continued to work within the grade level, without the support of the literacy coach to assist with assessment and implementation.
Other challenges that were less frequently mentioned by participants included the lack resources needed to purchase curriculum and fund an onsite coach and the fact that ERIA curriculum is not on the state adopted intervention materials list.
The ERIA models that were developed by the outside coaches were very similar and successfully supported most of the project participants in implementing ERIA in their schools. Assessment, decision making, grouping, scheduling, implementing curriculum were all in place based on the reports of the focus group participants and the survey respondents. One aspect that was not as well developed in these models was the identification and role of the onsite coach.
The two-day academy and the outside coaching provided by CalSTAT were deemed invaluable by the all of the focus group participants and most of the survey respondents. The two most powerful themes that emerged as strengths of the program were the collaborative aspects that emerged in the schools, districts and regions of the project participants, and how closely the model aligned with previously held beliefs and practices. The flexibility of the model, the positive impact on students, the changing practice of teachers, and the teachers' increased knowledge and skills were also identified as strengths of the ERIA project.
There were also challenges identified which also need to be considered as the project moves forward. Most notable was the issue of scheduling, which needs to be addressed through the initial training and ongoing coaching sessions. Another important aspect to emerge as a challenge is the issue of capacity and time constraints. Participants need to be informed participants, and able and willing to make the commitment that this project entails. Closely aligned to this is the need for strong leadership and commitment of the administrator and teacher leaders. Support to alleviate the challenges of teacher buy-in and burn out and the need for additional training should also be considered in the coaching models that develop to support future cohorts.
The second goal of this evaluation was to describe an ideal ERIA model that would help inform efforts to scale up this project in the coming years. Given the previous models that were evaluated, and the identified strengths and challenges of the models, the following are some suggestions to consider to extend and refine the existing model: