skip to content | CalSTAT
California Department of Education, Special Education Division’s special project, California Services for Technical Assistance and Training (CalSTAT) is funded through a contract with the Napa County Office of Education. CalSTAT is partially funded from federal funds, State Grants #H027A080116A. Additional federal funds are provided from a federal competitively awarded State Personnel Development Grant to California (#H323A070011) provided from the U.S. Department of Education Part D of the Individuals with Disabilities Education act (IDEA). Opinions expressed herein are those of the authors and do not necessarily represent the position of the U. S. Department of Education.
The Effective Reading Intervention Academy (ERIA) supports schools in identifying struggling students and trains teachers in providing effective interventions to assist in improving specific student reading skills.
CalSTAT began working with local education agencies in 2004 to bring ERIA to school sites. Each cohort of approximately 10 schools receives training and ongoing support. Sites in West Orange County and Antelope Valley piloted ERIA starting in the 2004-05 school year (Cohort 1), followed by San Joaquin Valley in 2005-06 (Cohort 2) and San Diego County in 2006-07 (Cohort 3). Cohort 1 has conducted a series of districtwide scale-ups, increasing the size of the cohort to 28 sites. Cohort 2 and 3 expanded by roughly double in the 2008-09 school year, to 20 and 16 sites respectively.
ERIA delivers differentiated instruction to students depending on need. Students who are falling behind or are at risk of falling behind in their reading levels are identified for intensified instruction in key areas to reach proficiency in English-language arts (ELA). As a program, ERIA seeks to embed key principles of evidence-based literacy education in schools throughout California.
The ERIA program includes four key elements:
Assessment of student reading levels is key to making data-informed decisions about student placement and interventions. Assessment is a multifaceted activity which involves examining existing data sources, such as scores from the California Standards Testing (CST), generating new benchmark data at regular intervals with a variety of measures, and interpreting these data against specific criteria. Three major skill areas are identified for assessment at most ERIA sites:
Student assessment is only the first step in implementation of ERIA, but many sites report that it has its own inherently beneficial effects in securing faculty interest and staff buy-in. Some sites have instituted weekly faculty meetings to review student progress, and many teachers have begun sharing assessment scores with students weekly. Organizational culture at ERIA sites appears to be shifting to make use of these newly-available sources of data, and sites are reporting that new attention to measurable outcomes has become a source of motivation at all levels.
Intervention delivers additional resources to students with additional needs, typically drawing from one or more research-based literacy education programs. Interventions are organized into three tiers, with a core tier serving all students and two additional tiers delivering additional help to students with additional needs. Student needs are matched to one of these intervention tiers through assessment data. Decisions about student placement are made on a student-by-student basis and are guided by specific criteria. Placement criteria are developed at the site-level, relating the needs of individual students to the reading level of the overall student population and the instruction provided in the core tier.
Tiers modulate the intensity of instruction and move towards more individualized intervention, providing struggling students with the resources and support necessary for them to succeed. Instruction is differentiated within each tier as well, allowing students to benefit from literacy programs targeting specific reading skills. Some literacy programs being delivered as interventions at ERIA sites include Read Naturally, Rewards, Language!, Read 180 and The Six Minute Solution. Many literacy interventions in use address oral reading fluency and grade-levelling, while others such as Soar to Success target reading comprehension skills.
Frequent, comprehensive assessment and data-informed decision-making are ongoing activities at ERIA sites. This data informs regular collaborative review of intervention placements, and adjustments are made in the intervention placement of students as student needs change.
The ERIA program cultivates structures and expertise at the site level which support and sustain these best practices. Formation of a site-level ERIA program starts with professional development activities, delivered to administrators, teachers, and other educators. ERIA sites regularly access ongoing professional development for help with aspects of literacy education from data-informed decision-making to more active teaching styles that keep students engaged. Participants in these activities return to the school with these skills and proceed to build a site-level ERIA program.
Site teams have also found ways to meet challenges to full, schoolwide implementation of ERIA’s key elements. For many sites, implementation has been delayed by a lack of resources to conduct comprehensive assessments, too few teachers trained in interventions, and not enough time for collaborative review of student progress. Hiring substitute teachers or recruiting parent volunteers to conduct assessments are examples of how site-level innovation can create effective solutions.
CalSTAT facilitates implementation of these elements through two primary forms of support:
The Team Implementation Checklist (TIC) allows ERIA Site Teams to summarize site implementation of ERIA according to 26 explicit criteria. The average response to each item is summarized below. Additional information about the TIC is on page 3.
|Fall 2007||Spring 2008||Fall 2008||Spring 2009|
|Assessing Students||A1. CST ELA Proficiency of each student is examined||96%||98%||99%||99%|
|A2. Decoding skills of students less than Proficient are assessed||86%||92%||93%||98%|
|A3. Fluency skills of students less than Proficient are assessed||95%||96%||97%||99%|
|A4. Comprehension skills of students less than Proficient are assessed||76%||79%||74%||80%|
|A5. Specific criteria exist for reading intervention placement||81%||91%||89%||95%|
|A6. Specific-skill reading intervention needs have been determined for the school as a whole, based upon student assessments||75%||82%||79%||87%|
|Specific Intervention Programs||B1. Based upon the needs of the school, research-based specific-skill reading interventions have been purchased and are in place||76%||87%||83%||92%|
|B2. Staff have been trained in use of reading intervention programs||68%||78%||71%||78%|
|B3. Intervention placement criteria are used to match and exit students||68%||81%||77%||87%|
|B4. Students needing intervention(s) are receiving them regularly||74%||87%||77%||89%|
|B5. Reading intervention programs are being used with fidelity||69%||78%||73%||85%|
|B6. Periodic tests and/or measures from the intervention programs are being used to monitor student progress||68%||86%||76%||92%|
|Response to Intervention||C1. Initial assessment tests (decoding, fluency, etc.) are repeated regularly to inform the review of intervention placements||74%||94%||79%||92%|
|C2. Progress monitoring data are recorded and charted for ease of use||62%||82%||68%||85%|
|C3. Regular, collaborative review of individual student progress and intervention placements is occurring||63%||74%||69%||86%|
|C4. Multiple levels of interventions are provided ranging in intensity||69%||84%||78%||91%|
|Systems Change||D1. An ERIA Site Team is guiding implementation||65%||77%||76%||87%|
|D2. The school principal is active in leading ERIA implementation||72%||82%||79%||87%|
|D3. Site Team members communicate regularly (formally/informally)||66%||77%||78%||90%|
|D4. Coaching is provided to support implementation with fidelity||48%||56%||54%||65%|
|D5. Regular fidelity observations are done by administrators/coaches||32%||47%||46%||55%|
|D6. Ongoing professional development activities are taking place||60%||74%||68%||81%|
|D7. Time is provided for collaboration on a regular basis||70%||78%||73%||82%|
|D8. Schedule reflects required time to accommodate interventions||76%||88%||82%||89%|
|D9. The ERIA/RTI process is being implemented schoolwide||60%||70%||71%||77%|
|D0. The School Site ERIA program is part of a district-wide scale-up||59%||68%||60%||74%|
The TIC is a 26 item checklist (items summarized on page 2) through which sites self-report implementation of explicit ERIA program criteria. Responses to each item are given as “achieved,” “in progress” or “not started,” and interpreted as 100%, 50% and 0% implementation, respectively.
In Spring 2009, 21 of 26 checklist items are, on average, being fully implemented across all of ERIA, and no item averaged below 50%.
The average of all 26 items within each site has been explored as an important measure of ERIA implementation at the site level, and is summarized below.
|New ERIA Sites||Continuing Sites|
|Fall 2008||Spring 2009||Fall 2008||Spring 2009|
|Fully Implementing||4 schools||20%||7 schools||35%||24 schools||54%||37 schools||84%|
|Partially Implementing||8 schools||40%||7 schools||35%||17 schools||39%||5 schools||11%|
|Minimally Implementing||4 schools||20%||2 schools||10%||2 schools||5%||0 schools||0%|
|Did Not Submit Data||4 schools||20%||4 schools||20%||1 schools||2%||2 schools||5%|
To monitor implementation of strategies presented at these trainings, email addresses were collected from participants in ERIA training events, and follow-up surveys were sent to these participants between 3 and 6 months after the training.
Rating their experiences on a five-point scale from “not at all” to “many times,” respondents described extensive implementation, effectiveness, and sharing of strategies.
|Implemented Strategies||Strategies Worked Well||Shared Strategies with Others|
(a 4 or 5 on a 5-point scale)
(a 3 on a 5-point scale)
This was further substantiated by participant comments, which included the following:
“I have shared the information with the three paraprofessionals working in the Reading Lab. It has improved all of our small group lessons in grades K-6, especially in 4th-6th. The main difference has been more progress monitoring. We can do these once a week while our students are on the computers. This helps us evaluate our weekly instruction.”
“We were able to bring the information back to our school as well as share with other schools in our county who are just beginning to start up RtI and may possibly be interested in ERIA."
“Our reading intervention for all students in grades 5 and 6 last spring is one reason we realized an 85-point increase in API [Academic Performance Index].”
“Due to ERIA’s support, our school was selected to be a site model for the county Department of Education’s RtI Network. We hosted 60 administrators, teachers, and district office administrators to come and see RtI in action. We shared how much the support of ERIA and the systems support helped us make huge gains in our RtI journey.”
Data is generated by sites through two assessments—the San Diego Quick (SDQ) and oral reading fluency (ORF)—then forwarded to CalSTAT for evaluation purposes. All ERIA outcomes described here are from the 2008-09 school year.
These assessment scores allow sites to make data-based intervention decisions, delivering specific, targeted help to students based on their individual needs. To allow CalSTAT to aggregate data from nearly fifty schools, sites have been asked to make assessments based on Summary Reporting Criteria, which standardize assessments between sites.
Summary Reporting Criteria
- Must include matching fall and spring scores
- Student and passage grade levels are noted
- Fluency test passages must be at student grade level
Note: CalSTAT does not receive student-level data from sites that include any student identifiers. Student anonymity and privacy is preserved above the site level for all sites and students participating in ERIA.
Assessments represented in this report include only those meeting summary reporting criteria.
Many sites conducted a large number of student assessments not meeting summary reporting criteria. It is important to note that such measurements may still be valuable in improving student outcomes. While sites are asked to make and report measures to CalSTAT which can be aggregated with those of other sites, ERIA site teams are encouraged to continue making decisions about how best to assess students at their sites.
|Content of Data||Students Struggling with Decoding, Fluency, or Both||4,587||40%|
|Students Not Struggling in Decoding or Fluency||2,852||25%|
|Data Not Meeting Summary Criteria||4,073||35%|
All three Cohorts recorded more student assessments in 2008-09, driven by the addition of 20 new sites and an expansion of testing at existing sites (11,512 students assessed, up from 8,091 in 2007-08), but a smaller percentage of these assessments identified struggling students (40% versus 55%). As a result, overall, a similar number of struggling students were identified in 2008-09 (4,587) and 2007-08 (4,498).
Oral reading fluency assessment tracks incremental changes in the number of words read aloud in a minute from a passage of text specifically calibrated by grade level. Student data is only included if it met the summary reporting criteria and the student showed oral reading fluency scores below the 50th percentile in fall (struggling).
Change in Words Correct per Minute (WCPM) is calculated as a percentage of typical change in WCPM, based upon Hasbrouck and Tindal’s 2004 oral reading fluency study. Because struggling students are reading more slowly than the grade level median, greater-than-typical growth is needed to reach fluency. Struggling students making greater than typical growth are shown below in blue and green.
Orange County and Antelope Valley
2,773 students, 28 sites in 5 districts
Southern San Joaquin Valley
821 students, 20 sites in 9 districts
San Diego County
1,122 students, 16 sites in 7 districts
161 students, 2 sites
2,612 students, 26 sites
349 students, 9 sites
472 students, 11 sites
601 students, 9 sites
521 students, 7 sites
|Growth Above 150%||47 students||29%||803 students||31%||57 students||16%||74 students||16%||26 students||4%||152 students||29%|
|Growth Above 100% to 150%||51 students||32%||610 students||23%||61 students||17%||112 students||24%||77 students||13%||105 students||20%|
|Growth Above 50% to 100%||45 students||28%||608 students||23%||93 students||27%||149 students||32%||166 students||28%||129 students||25%|
|Growth of 50% or Less||18 students||11%||591 students||23%||138 students||40%||137 students||29%||332 students||55%||135 students||26%|
The San Diego Quick assessment requires students to identify words on a grade-level vocabulary list, with correct identification of 8 or more of the 10 words representing grade level skill. Student data is only included here if it meets summary reporting criteria and the student tests below grade level in fall SDQ assessment (“struggling”).
Students below the line are struggling, with some one grade level behind (yellow in the chart) and others two or more grade-levels behind (red). Of all of these struggling readers identified in the fall assessments, by the end of the year, roughly half were meeting (green) or exceeding (blue) grade level reading targets.
Orange County and Antelope Valley
2,137 students, 28 sites in 5 districts
Southern San Joaquin Valley
565 students, 20 sites in 9 districts
San Diego County
985 students, 16 sites in 7 districts
|Ahead 1 or More Grade Levels||0 students||0%||375 students||20%||0 students||0%||114 students||20%||0 students||0%||216 students||22%|
|At Grade Level||0 students||0%||610 students||29%||0 students||0%||154 students||27%||0 students||0%||276 students||28%|
|Behind 1 Grade Level||951 students||45%||668 students||31%||253 students||45%||25 students||25%||473 students||48%||305 students||31%|
|Behind More than 1 Grade Level||1,186 students||55%||484 students||23%||312 students||55%||157 students||28%||512 students||52%||188 students||19%|
Monitoring CST English Language Arts (ELA) proficiency scores has been a focus of ERIA evaluation, just as it is a focus for all California schools. The percent of students scoring proficient or above is given schoolwide and for students with disabilities schoolwide. The chart below summarizes the median percent proficient of sites within each cohort.
The chart on page 11 shows CST ELA trends for individual sites, as well as site-specific implementation data from the TIC.
|All Students||Cohort 1: Orange County and Antelope Valley||45.7||54.6||56.9||55.7||56.1||61.1|
|Cohort 2: Southern San Joaquin Valley||40.1||43.6||53.8||50.8||51.8||60.3|
|Cohort 3: San Diego County||35.2||36.5||37.5||43.7||46.2||51.1|
|Students with Disabilities||Cohort 1: Orange County and Antelope Valley||15.9||21.3||23.4||21.6||25.0||36.9|
|Cohort 2: Southern San Joaquin Valley||4.7||15.0||10.6||16.4||20.0||27.1|
|Cohort 3: San Diego County||9.2||10.5||18.9||21.8||22.9||37.5|
|AYP Target (approximate)||12||23||23||23||34||45|
|Implementation||Schoolwide Implementation||CST ELA: All Students||CST ELA: Students with Disabilities|
|Fall 08||Spr 09||Fall 08||Spr 09||Baseline||2009||Baseline||2009|
|Joined 2004-05||Demille Elementary||87%||88%||Achieved||Achieved||38||47.7||25||29.6|
|Ethel Dwyer Middle||79%||92%||Partial||Partial||58.2||74.2||10.2||47.2|
|Marine View Middle||79%||73%||Achieved||Achieved||59.2||72.3||20||31|
|Mesa View Middle||73%||83%||Achieved||Achieved||53.5||73||8.1||29.1|
|Quartz Hill Elementary||58%||75%||Partial||Partial||42.9||54.8||6.1||29.8|
|Spring View Middle||79%||81%||Partial||Achieved||46.8||64.8||7.1||43.9|
|Vista View Middle||98%||100%||Achieved||Achieved||39.4||61.2||3.4||23.3|
|Joined 2005-06||Alvina Elementary Charter||75%||88%||Achieved||Achieved||33.1||43.2||.||41.2|
|Herbert Hoover High||42%||58%||Minimal||Minimal||43.6||39.7||4.5||13.5|
|Sierra View Elementary||85%||81%||Partial||Partial||.||59.3||.||0|
|Joined 2006-07||Campo Elementary||0%||0%||Minimal||Minimal||35.6||50.9||22.2||39|
|Clover Flat Elementary||90%||0%||Achieved||Minimal||37.5||60.3||29.4||-1|
|Marshall (Thurgood) Elementary||88%||90%||Minimal||Minimal||61.9||72.7||23.3||43.6|
|Spring Valley Middle||88%||90%||Minimal||Minimal||46.4||51.1||11.4||18|
|Winter Gardens Elementary||77%||92%||Minimal||Minimal||33.3||37.6||13.3||23.5|
|Joined 2007-08||Anderson Elementary||81%||90%||Achieved||Achieved||48.6||54.3||14.3||32.1|
|Circle View Elementary||83%||85%||Achieved||Achieved||81.3||81.8||60.6||57.6|
|College View Elementary||88%||94%||Achieved||Achieved||48.9||60.9||33.3||34.6|
|Golden View Elementary||85%||85%||Achieved||Achieved||57.3||60.6||22.2||47.5|
|Harbour View Elementary||90%||90%||Partial||Partial||72||79.9||37.1||47.1|
|Hope View Elementary||90%||94%||Achieved||Achieved||75.8||83||46.9||66.7|
|Isaac L. Sowers Middle||48%||98%||Partial||Partial||72.9||75.1||31.1||52.4|
|Jessie Hayden Elementary||54%||69%||Partial||Partial||55.2||61.7||42.9||47.6|
|Lake View Elementary||100%||100%||Achieved||Achieved||49||51||21.2||46.3|
|Oak View Elementary||75%||100%||Achieved||Achieved||37||34.5||21.9||20|
|Star View Elementary||90%||94%||Achieved||Achieved||63.2||63.7||17.4||41.7|
|Sun View Elementary||60%||83%||Achieved||Achieved||58.8||48||37.9||29.8|
|Top of the World Elementary||77%||87%||Achieved||Achieved||76.9||78.8||.||48.8|
|Village View Elementary||88%||96%||Partial||Partial||71.3||71.5||44.7||53.5|
|Joined 2008-09||Bancroft Elementary||60%||69%||Partial||Partial||35.9||40.3||26.9||37.5|
|Edison-Bethune Charter Academy||46%||77%||Partial||Partial||27.4||35.7||9.5||14.3|
|El Morro Elementary||67%||75%||Partial||Partial||74.1||73.5||50||44.7|
|Joseph R. Perry Elementary||67%||77%||Minimal||Partial||51.9||49.8||20.5||35.1|
|Kempton Street Elementary||0%||0%||Minimal||Minimal||38||36||10.8||20.8|
|La Presa Elementary||0%||0%||Minimal||Minimal||31.3||38.8||12.5||16.4|
|Lone Star Elementary||73%||92%||Partial||Partial||51.8||55.2||41.7||21.7|
|Mariposa Special Education||23%||40%||Minimal||Minimal||.||.||.||.|
|Stone Ranch Elementary||0%||0%||Minimal||Minimal||81.8||85.7||37||45.9|
There is a great deal of diversity in both implementation and outcomes across all three cohorts. The ERIA program has given individual sites a high level of autonomy, and that is apparent in the data. Overall, however, student outcomes have been very positive: roughly half of struggling readers assessed showed substantial growth in reading skills, and CST English-language Arts proficiency is generally advancing.
While a larger portion of continuing sites described themselves as fully implementing ERIA on the TIC, many new sites have already made considerable advances in establishing their own ERIA programs. There were also several high-performing new sites, most of whom had experience with response to intervention programs prior to involvement with ERIA.
There was no correlations between TIC responses from a site and the progress measured in CST English-language Arts proficiency or oral reading fluency measures. There were both sites with minimal implementation who showed large increases in proficiency and sites with advanced implementation who showed moderate declines. Site action plans (SAPs, a qualitative self-report on site-level implementation updated twice annually) also do not suggest thematic differences between high- and low-performing sites.
Sites in Cohort 1 have consistently demonstrated positive student outcomes, even as they’ve expanded from an original core of 10 sites to the current 28 sites. Cohort 1 has a high level of district and school principal involvement which is supporting implementation of ERIA. Two districts that already contained continuing ERIA sites pursued scale-up of ERIA, bringing the program to two new sites who experienced very positive outcomes with the support and expertise of established ERIA veterans.
Cohort 2 is characterized by numerous, widely-dispersed school sites spanning many districts and counties. In 2008-09, the cohort added nine new sites which achieved a very high degree of implementation in a short period of time, and student outcomes relatively similar to continuing sites. Cohort 2’s median CST scores for all students also showed the largest increase of all three cohorts.
This year, Cohort 3’s continuing sites saw substantial gains in student outcomes over previous years. This was especially apparent in median CST scores for students with disabilities, jumping nearly 15 percentage points between 2008 and 2009. Otherwise, implementation and outcomes at new sites were more mixed.
This report was developed for CalSTAT by the SIG Evaluation Team of Cheryl “Li” Walter, PhD, and Alan Wood.
CalSTAT (California Services for Technical Assistance and Training), at Napa County Office of Education, is a special project of the California Department of Education, Special Education Division. Visit CalSTAT at http://www.calstat.org.
“CalSTAT is partially funded from federal funds awarded in Part B of Public Law 108-446, the Individuals with Disabilities Education Act (IDEA), as amended in 2004. Additional federal funds are provided from a federal competitively awarded State Program Improvement Grant to California (CFDA 84.323A) allowed in Part D of Public Law 108-447, the Individuals with Disabilities Education Act (IDEA), as amended in 2004. These dollars are considered local assistance funds. Both funds are to assist individuals serving children birth to 22 years of age and their families.”