Effective Reading Intervention Academy (ERIA)
Program Overview and Summary Report 2008-09

The Effective Reading Intervention Academy (ERIA) supports schools in identifying struggling students and trains teachers in providing effective interventions to assist in improving specific student reading skills.

CalSTAT began working with local education agencies in 2004 to bring ERIA to school sites. Each cohort of approximately 10 schools receives training and ongoing support. Sites in West Orange County and Antelope Valley piloted ERIA starting in the 2004-05 school year (Cohort 1), followed by San Joaquin Valley in 2005-06 (Cohort 2) and San Diego County in 2006-07 (Cohort 3). Cohort 1 has conducted a series of districtwide scale-ups, increasing the size of the cohort to 28 sites. Cohort 2 and 3 expanded by roughly double in the 2008-09 school year, to 20 and 16 sites respectively.

Key Elements of ERIA

ERIA delivers differentiated instruction to students depending on need. Students who are falling behind or are at risk of falling behind in their reading levels are identified for intensified instruction in key areas to reach proficiency in English-language arts (ELA). As a program, ERIA seeks to embed key principles of evidence-based literacy education in schools throughout California.

The ERIA program includes four key elements:

• Frequent and comprehensive assessment of student reading skills.
• Delivery of literacy intervention programs as necessary.
• Review of placement following a response to intervention model.
• Systems change resulting in the sustainability of these best practices at sites.

Assessing Students

Assessment of student reading levels is key to making data-informed decisions about student placement and interventions. Assessment is a multifaceted activity which involves examining existing data sources, such as scores from the California Standards Testing (CST), generating new benchmark data at regular intervals with a variety of measures, and interpreting these data against specific criteria. Three major skill areas are identified for assessment at most ERIA sites:

• The San Diego Quick (SDQ), a grade-levelling vocabulary-based assessment, also described as a decoding tool in some parts of the ERIA manual.
• Oral Reading Fluency, a timed reading test used to measure incremental change.
• Comprehension, testing understanding of text by asking questions.

Student assessment is only the first step in implementation of ERIA, but many sites report that it has its own inherently beneficial effects in securing faculty interest and staff buy-in. Some sites have instituted weekly faculty meetings to review student progress, and many teachers have begun sharing assessment scores with students weekly. Organizational culture at ERIA sites appears to be shifting to make use of these newly-available sources of data, and sites are reporting that new attention to measurable outcomes has become a source of motivation at all levels.

Specific Reading Intervention Programs

Intervention delivers additional resources to students with additional needs, typically drawing from one or more research-based literacy education programs. Interventions are organized into three tiers, with a core tier serving all students and two additional tiers delivering additional help to students with additional needs. Student needs are matched to one of these intervention tiers through assessment data. Decisions about student placement are made on a student-by-student basis and are guided by specific criteria. Placement criteria are developed at the site-level, relating the needs of individual students to the reading level of the overall student population and the instruction provided in the core tier.

Tiers modulate the intensity of instruction and move towards more individualized intervention, providing struggling students with the resources and support necessary for them to succeed. Instruction is differentiated within each tier as well, allowing students to benefit from literacy programs targeting specific reading skills. Some literacy programs being delivered as interventions at ERIA sites include Read Naturally, Rewards, Language!, Read 180 and The Six Minute Solution. Many literacy interventions in use address oral reading fluency and grade-levelling, while others such as Soar to Success target reading comprehension skills.

Response to Intervention (RtI)

Frequent, comprehensive assessment and data-informed decision-making are ongoing activities at ERIA sites. This data informs regular collaborative review of intervention placements, and adjustments are made in the intervention placement of students as student needs change.

Systems Change

The ERIA program cultivates structures and expertise at the site level which support and sustain these best practices. Formation of a site-level ERIA program starts with professional development activities, delivered to administrators, teachers, and other educators. ERIA sites regularly access ongoing professional development for help with aspects of literacy education from data-informed decision-making to more active teaching styles that keep students engaged. Participants in these activities return to the school with these skills and proceed to build a site-level ERIA program.

Site teams have also found ways to meet challenges to full, schoolwide implementation of ERIA’s key elements. For many sites, implementation has been delayed by a lack of resources to conduct comprehensive assessments, too few teachers trained in interventions, and not enough time for collaborative review of student progress. Hiring substitute teachers or recruiting parent volunteers to conduct assessments are examples of how site-level innovation can create effective solutions.

Training and Ongoing Coaching

CalSTAT facilitates implementation of these elements through two primary forms of support:

• Training Events: Trainings with reading specialists are arranged for site teams from participating ERIA sites. These include comprehensive content trainings for new sites and booster trainings to provided additional and advanced content for sites who are already implementing ERIA. Selection of presenters and content is directed at the regional level and is responsive to needs within each cohort.
• Ongoing Coaching and Support: Regional coaches provide ongoing support for site teams throughout the year. These coaches also oversee each cohort and organize additional activities throughout the year and around trainings to facilitate an exchange of knowledge and successful strategies between site teams.

Implementation of ERIA

The Team Implementation Checklist (TIC) allows ERIA Site Teams to summarize site implementation of ERIA according to 26 explicit criteria. The average response to each item is summarized below. Additional information about the TIC is on page 3.

The TIC is a 26 item checklist (items summarized on page 2) through which sites self-report implementation of explicit ERIA program criteria. Responses to each item are given as “achieved,” “in progress” or “not started,” and interpreted as 100%, 50% and 0% implementation, respectively.

• An aggregated average of 80% or more is considered to be “fully implementing” ERIA (in Spring 2009, most items were in this category).
• An average of at least 50% but less than 80% is considered to be “partially implementing.”
• Below 50%, sites are considered to be “minimally implementing.”

In Spring 2009, 21 of 26 checklist items are, on average, being fully implemented across all of ERIA, and no item averaged below 50%.

• Item B2, “Staff have been trained in use of reading intervention programs,” was one item that is not being fully implemented, averaging only 78% across all sites.
• Four more items relating to systems change also are not being fully implemented, including coaching and observations around fidelity of implementation (D4 and D5), and schoolwide implementation and districtwide scale-up (D9 and D0).

Degree of Implementation by Site

The average of all 26 items within each site has been explored as an important measure of ERIA implementation at the site level, and is summarized below.

• As of Spring 2009, full implementation of ERIA advanced considerably (to 44 of 64 sites, 69%). New sites were much less likely to describe themselves as fully implementing ERIA, with only 7 of 20 sites (35%) compared to 37 of 44 continuing sites (84%).
• There did not appear to be a correlation between self-reported implementation through the TIC and student outcomes measured as improvement in Oral Reading Fluency or the San Diego Quick assessments for struggling students. These outcomes are discussed in greater detail in the second half of this report.

Implementation of Training Content

To monitor implementation of strategies presented at these trainings, email addresses were collected from participants in ERIA training events, and follow-up surveys were sent to these participants between 3 and 6 months after the training.

Rating their experiences on a five-point scale from “not at all” to “many times,” respondents described extensive implementation, effectiveness, and sharing of strategies.

This was further substantiated by participant comments, which included the following:

“I have shared the information with the three paraprofessionals working in the Reading Lab. It has improved all of our small group lessons in grades K-6, especially in 4th-6th. The main difference has been more progress monitoring. We can do these once a week while our students are on the computers. This helps us evaluate our weekly instruction.”

“We were able to bring the information back to our school as well as share with other schools in our county who are just beginning to start up RtI and may possibly be interested in ERIA."

“Our reading intervention for all students in grades 5 and 6 last spring is one reason we realized an 85-point increase in API [Academic Performance Index].”

“Due to ERIA’s support, our school was selected to be a site model for the county Department of Education’s RtI Network. We hosted 60 administrators, teachers, and district office administrators to come and see RtI in action. We shared how much the support of ERIA and the systems support helped us make huge gains in our RtI journey.”

ERIA Evaluation Tools

Data is generated by sites through two assessments—the San Diego Quick (SDQ) and oral reading fluency (ORF)—then forwarded to CalSTAT for evaluation purposes. All ERIA outcomes described here are from the 2008-09 school year.

These assessment scores allow sites to make data-based intervention decisions, delivering specific, targeted help to students based on their individual needs. To allow CalSTAT to aggregate data from nearly fifty schools, sites have been asked to make assessments based on Summary Reporting Criteria, which standardize assessments between sites.

Summary Reporting Criteria

• Must include matching fall and spring scores
• Student and passage grade levels are noted
• Fluency test passages must be at student grade level

Note: CalSTAT does not receive student-level data from sites that include any student identifiers. Student anonymity and privacy is preserved above the site level for all sites and students participating in ERIA.

Assessments represented in this report include only those meeting summary reporting criteria.

Many sites conducted a large number of student assessments not meeting summary reporting criteria. It is important to note that such measurements may still be valuable in improving student outcomes. While sites are asked to make and report measures to CalSTAT which can be aggregated with those of other sites, ERIA site teams are encouraged to continue making decisions about how best to assess students at their sites.

All three Cohorts recorded more student assessments in 2008-09, driven by the addition of 20 new sites and an expansion of testing at existing sites (11,512 students assessed, up from 8,091 in 2007-08), but a smaller percentage of these assessments identified struggling students (40% versus 55%). As a result, overall, a similar number of struggling students were identified in 2008-09 (4,587) and 2007-08 (4,498).

ERIA Student Outcomes Data

Oral Reading Fluency

Oral reading fluency assessment tracks incremental changes in the number of words read aloud in a minute from a passage of text specifically calibrated by grade level. Student data is only included if it met the summary reporting criteria and the student showed oral reading fluency scores below the 50th percentile in fall (struggling).

Change in Words Correct per Minute (WCPM) is calculated as a percentage of typical change in WCPM, based upon Hasbrouck and Tindal’s 2004 oral reading fluency study. Because struggling students are reading more slowly than the grade level median, greater-than-typical growth is needed to reach fluency. Struggling students making greater than typical growth are shown below in blue and green.

• A majority of struggling students identified in Cohort 1 showed greater-than-expected growth, helping them catch-up with grade level reading standards.
• New ERIA sites outperformed continuing sites in Cohorts 1, while new sites showed more moderate gains in test scores in Cohorts 2 and 3.

San Diego Quick

The San Diego Quick assessment requires students to identify words on a grade-level vocabulary list, with correct identification of 8 or more of the 10 words representing grade level skill. Student data is only included here if it meets summary reporting criteria and the student tests below grade level in fall SDQ assessment (“struggling”).

Students below the line are struggling, with some one grade level behind (yellow in the chart) and others two or more grade-levels behind (red). Of all of these struggling readers identified in the fall assessments, by the end of the year, roughly half were meeting (green) or exceeding (blue) grade level reading targets.

• Unlike oral reading fluency scores (page 6), San Diego Quick assessments suggest uniform, positive outcomes across all three cohorts for both new and continuing sites. (Due to these small differences, new and continuing sites are combined in the chart.)
• For Cohorts 1 and 2, these outcomes improve slightly on last year’s. For Cohort 3, the number of students meeting this goal almost doubled, from 27% to 50%. This may be a sign of advancing implementation at this, the youngest, cohort.

California Standards Testing (CST) Outcomes

Monitoring CST English Language Arts (ELA) proficiency scores has been a focus of ERIA evaluation, just as it is a focus for all California schools. The percent of students scoring proficient or above is given schoolwide and for students with disabilities schoolwide. The chart below summarizes the median percent proficient of sites within each cohort.

The chart on page 11 shows CST ELA trends for individual sites, as well as site-specific implementation data from the TIC.

• ERIA cohorts have typically tracked with statewide proficiency over time, often showing improvements above statewide scores.
• Proficiency for students with disabilities has been more volatile for schools in all cohorts, though there have been increases in the percent proficient overall.
• Some cohorts experienced a drop in proficiency in 2007, either for all students or the students with disabilities subgroup, though 2008 scores show a recovery.
• Overall, encouraging gains have been seen in all three cohorts, particularly for all students in Cohorts 1 and 2 and for students with disabilities in Cohorts 1 and 3.

Exploring the Data

There is a great deal of diversity in both implementation and outcomes across all three cohorts. The ERIA program has given individual sites a high level of autonomy, and that is apparent in the data. Overall, however, student outcomes have been very positive: roughly half of struggling readers assessed showed substantial growth in reading skills, and CST English-language Arts proficiency is generally advancing.

While a larger portion of continuing sites described themselves as fully implementing ERIA on the TIC, many new sites have already made considerable advances in establishing their own ERIA programs. There were also several high-performing new sites, most of whom had experience with response to intervention programs prior to involvement with ERIA.

There was no correlations between TIC responses from a site and the progress measured in CST English-language Arts proficiency or oral reading fluency measures. There were both sites with minimal implementation who showed large increases in proficiency and sites with advanced implementation who showed moderate declines. Site action plans (SAPs, a qualitative self-report on site-level implementation updated twice annually) also do not suggest thematic differences between high- and low-performing sites.

Sites by Cohort

Sites in Cohort 1 have consistently demonstrated positive student outcomes, even as they’ve expanded from an original core of 10 sites to the current 28 sites. Cohort 1 has a high level of district and school principal involvement which is supporting implementation of ERIA. Two districts that already contained continuing ERIA sites pursued scale-up of ERIA, bringing the program to two new sites who experienced very positive outcomes with the support and expertise of established ERIA veterans.

Cohort 2 is characterized by numerous, widely-dispersed school sites spanning many districts and counties. In 2008-09, the cohort added nine new sites which achieved a very high degree of implementation in a short period of time, and student outcomes relatively similar to continuing sites. Cohort 2’s median CST scores for all students also showed the largest increase of all three cohorts.

This year, Cohort 3’s continuing sites saw substantial gains in student outcomes over previous years. This was especially apparent in median CST scores for students with disabilities, jumping nearly 15 percentage points between 2008 and 2009. Otherwise, implementation and outcomes at new sites were more mixed.