California Department of Education, Special Education Division’s special project, California Services for Technical Assistance and Training (CalSTAT) is funded through a contract with the Napa County Office of Education. CalSTAT is partially funded from federal funds, State Grants #H027A080116A. Additional federal funds are provided from a federal competitively awarded State Personnel Development Grant to California (#H323A070011) provided from the U.S. Department of Education Part D of the Individuals with Disabilities Education act (IDEA). Opinions expressed herein are those of the authors and do not necessarily represent the position of the U. S. Department of Education.
The Effective Reading Intervention Academy (ERIA) supports schools in identifying struggling students, trains teachers in providing effective instruction and intervention to assist in improving specific student reading skills, and tools to monitor student progress throughout the year. CalSTAT began working with local education agencies in 2004 to bring ERIA to school sites. Each cohort receives training and ongoing support. Sites in West Orange County and Antelope Valley piloted ERIA starting in the 2004-05 school year (Cohort 1), followed by San Joaquin Valley in 2005-06 (Cohort 2) and San Diego County in 2006-07 (Cohort 3). A series of scale-ups in 2007 and 2008 expanded the size of these cohorts from an original 10 sites each to a peak of 64 sites in the 2008-09 school year.
ERIA did not train any new sites in the 2009-10 school year, and several sites previously trained in ERIA stopped participating for various reasons. This facilitated a transition for ERIA, shifting to focus a more intense concentration of resources on a smaller number of sites, emphasizing implementation in middle and high schools. In 2010-11, each cohort added new sites (17 total). Each cohort was also allowed to select a handful of sites previously trained for one final year of ERIA training to improve the sustainability of ERIA practices.
ERIA supports school sites in delivering differentiated instruction to students depending on need. Students who are falling behind or are at risk of falling behind in their reading levels are identified for intensified instruction in key areas to reach proficiency in English-language arts (ELA). As a program, ERIA seeks to embed key principles of evidence-based literacy education in schools throughout California.
The ERIA program includes four key elements:
In the 2010-11 school year, ERIA sites began implementing a revised version of these elements, which are described in more detail in the ERIA Program Guide, which is available online at http://calstat.org/effectivereading.html
Response to Instruction and Intervention (RtI2)
ERIA trainings outline foundational elements of RtI2, including enhanced instructional practices, use of assessment tools and data, implementation of intervention programs, and year-round data-informed decision-making. While true RtI2 is far more rigorous than most ERIA sites currently have the expertise or resources to implement, ERIA begins establishing these concepts and practices in schools and districts as a way of improving student outcomes. Additionally, many sites have already used ERIA as a foundation to pursue and achieve full RtI2 implementation.
Assessing Students
Assessment of student reading levels is key to making data-informed decisions about student placement and interventions. Data-informed decision-making involves examining existing data sources, such as scores from the California Standards Testing (CST), generating new benchmark data at regular intervals with a variety of measures, and interpreting this data against specific criteria. Two major skill areas are identified for assessment at most ERIA sites:
Student assessment is only the first step in the implementation of ERIA, but many sites report that it has its own inherently beneficial effects in securing faculty interest and staff buy-in, with some sites instituting regular faculty meetings to review student progress. Organizational culture at continuing ERIA sites appears to be shifting to make use of these newly-available sources of data, and sites are reporting that new attention to measurable outcomes has become a source of motivation at all levels.
Specific Reading Intervention Programs
Intervention delivers additional resources to students with additional needs drawing from one or more research-based literacy education programs. Once identified through assessment data, the specific skill needs of struggling readers are matched to an intervention program suited to teaching those needs. Decisions about student placement are made on a student-by-student basis and are guided by specific criteria which is developed at the site level, with guidance from literacy content experts.
Students who receive interventions continue to participate in the same core English-language Arts curriculum as students who aren't struggling with reading. Interventions are delivered as additional support in the context of an elective, pull-out, or after-school class, helping struggling readers catch up with and succeed in the core curriculum.
Systems Change
The ERIA program cultivates structures and expertise at the site level which support and sustain these best practices. Implementation of ERIA revolves around a site team. Site teams have also found ways to meet challenges to full, schoolwide implementation of ERIA's key elements. For some sites, implementation has been delayed by a lack of resources to conduct comprehensive assessments, too few teachers trained in interventions, and not enough time for collaborative review of student progress. The involvement and support of administration, both at the school and at the district level, is regularly cited as a key factor in the successes of sites who are implementing at the highest levels and seeing the most concrete benefits.
Training and Ongoing Coaching
CalSTAT facilitates implementation of ERIA through two primary forms of support:
In the 2010-11 school year, the last of the original 64 sites will "graduate" from ERIA, continuing to implement its practices without these external training or coaching. While ERIA was always intended to work with sites at the secondary level (middle and high schools), it delivered support to many elementary schools. Starting with the 2011-12 ERIA will focus on developing the foundational elements of RtI2 at the secondary level exclusively, though regional cohorts are expected to continue relying on the regional professional learning communities ERIA has fostered.
Implementation of ERIA was monitored informally by cohort coaches and formally through a Rubric of Implementation. The Rubric requires ERIA Site Teams to report their degree of implementation in ten areas at one of five levels, listing explicit criteria for each level. Level 3, "basic implementation," is the minimum target for sites, but two additional levels ("advanced" and "exemplary implementation") provide sites with guidance for how they can go beyond ERIA's basics, while providing an opportunity to document these instances of advanced implementation. The rubric is available for download online: http://calstat.org/pdf/ERIA_Rubric.pdf
The Rubric is also offered as an Excel file, to collect a site's responses twice per year, over three years. Automated reports, including charts, are also generated to help sites reflect on their implementation in a way that is more useful and easier to share with teachers, parents, and stakeholders: http://calstat.org/docs/ERIA_Rubric_blank.xls
Most-common Implementation Levels at ERIA Sitesas Reported in Spring 2011
Site teams complete the Rubric twice a year, once in Fall and once in Spring. Sites who completed and submitted the Rubric reported a high level of implementation in eight of the ten key areas. Sites tended to report partial or non-implementation on the last two items: improving content literacy instructional practices and improving fidelity of implementation through observations and coaching. Nine sites did not complete and submit the rubric, three newly trained sites in each of Orange and San Diego counties, and three graduating sites in San Diego county.
By averaging the 10 items, a single number between 1.0 and 5.0 can represent the overall implementation level of a given site. This is noted in tables on pages 9-10 and 14-15. New sites tended to implement at a lower level and show greater progress over the course of a year.
Monitoring English Language Arts (ELA) proficiency scores has been a focus of ERIA evaluation, just as it is a focus for all California schools. Proficiency is monitored annually through State Testing and Reporting (STAR), the California High School Exit Exam (CAHSEE), and other annual testing. The chart below summarizes the median percent proficient and above of All Students and Students with Disabilities, a subgroup defined and monitored under No Child Left Behind (NCLB). ERIA sites are grouped by when they started ERIA, with the median test score represented in the chart.
Test scores have been rising for schools generally statewide, as indicated by the statewide test scores. Schools are under pressure to meet Adequate Yearly Progress (AYP) targets, which began increasing annually following the 2007 round of testing.
AYP is monitored, in part, through test scores for all students and eleven subgroups, but only if the group is "numerically significant." Subgroups include eight ethnic groups, socioeconomically disadvantaged students (those receiving free/reduce-price lunch), English learners, and students with disabilities. Often, whether or not a given subgroup is numerically significant is a major factor in whether or not a school will be designated for program improvement (PI).
A group is "numerically significant" if it meets either of two criteria:
There are a number of factors which affect whether or not a subgroup is numerically significant, including enrollment. In 2011, most of the test scores used in AYP calculations came from CST and CMA assessments through grade 8. However, the key assessment used for higher grade level was the CAHSEE assessment, which is given in grade ten (with multiple make-up opportunities for students who fail). Because this reduces the number of test scores considerably, high schools are less likely to have numerically significant subgroups despite typically having a larger enrollment. The subgroups which are numerically significant at ERIA sites is both a reflection of which student populations are most visible in test scores and the wide variety of situations which individual sites are facing.
The enrollment (according to CA Longitudinal Pupil Achievement Data System), the number of students enrolled for testing in each of these subgroups as a percentage of all students enrolled for testing, the number of AYP criteria met and which a site is accountable for, and PI status of all ERIA sites is listed on page 11. Program Improvement employs 5 levels for schools (Year 1 through Year 5), with a school advancing deeper into PI if it misses AYP and exiting if it makes AYP two years in a row. This data-point is blank for many schools, usually because they aren't a Title I school and aren't subject to PI, though other reasons may exist.
Oral Reading Fluency (ORF) assesses a student's mastery of word-level reading skills, helping to target delivery of interventions and supports. Each student reads three passages for a minute each and an observer notes the number of words read ("words correct per minute," WCPM) and number of reading errors, both clues to skill needs.
At lower grade levels, building reading fluency is a key curriculum goal, but around sixth grade, the emphasis shifts to comprehension skills, increasing the importance of identifying students with weak fluency skills. This is reflected in the assessments below, in that only 10% of students in grades 1-3 demonstrate fluency in fall assessments, compared to 41% of students in grades 6-12. After an initial round of assessments to identify students who may not have their specific word-level reading skill needs met in the core curriculum, these "struggling readers" (120 WCPM or less) may receive intervention or other supports to build their skills.
ERIA focuses on assessment and intervention in the secondary level, which is reflected below in the greater number of assessments made with students in grades 6-12.
Depending on each ERIA site's needs and resources, some number of students were assessed at each participating school, from a few dozen students to the whole school. Because only a portion of students were assessed, this data should not be used to make judgments about the reading skills of students schoolwide.
As the first cohort with the most participating sites, Orange County made far more ORF assessments than the other cohorts, due mainly to participation of a handful of middle schools implementing at an advanced level (many initially trained in 2004-05). A late training schedule interfered with many new sites' (Year 1 of 3) ability to conduct assessments early in the 2010-11 school year, especially in Orange County, where only 20 struggling students were identified in Fall to be reassessed the following Spring. Due to differences in available resources and students needs, both between cohorts and between individual sites, some ERIA implementers opt for a more targeted assessment strategy that engages fewer students. Some sites have indicated that Oral Reading Fluency assessments may not be optimal for high school students, and are exploring alternatives with content experts for use in the 2011-12 school year.
Below and on page 15, the Fall, Winter, and Spring oral reading fluency assessment outcomes are summarized for each ERIA site. Unlike the summaries on page 13, these summaries are not limited to students who are identified as struggling with fluency skills in Fall or for whom a matching Spring assessment was recorded. In fact, students who are assessed as having mastery of fluency should not necessarily be reassessed later in the year. Oral reading fluency assessments measure specific, basic reading skill needs (such as decoding and fluency), rather than the comprehension skills which are focused on in higher grade-level statewide testing. The objective of these assessments is to identify students with basic skill needs so they can receive targeted intervention, enabling them to develop comprehension skills. These assessments don't reflect on the quality of any particular school.
With the focus on the secondary level and the limited number of assessments referenced in AYP scores, it may be worth exploring other ways of monitoring ERIA's effectiveness. The Academic Performance Index is a more comprehensive measure which includes several tests across all subject areas. Given that reading comprehension continues to be an issue in all subject areas, including history, science and math, perhaps this would be a more effective measure to track. This would also emphasise ERIA's focus on expanding explicit instruction in reading comprehension strategies to classrooms schoolwide, rather than just being taught in English classes and through interventions. Five years of API Growth scores are charted below for comparison with a similar chart using AYP data on page 7.
This document was developed for CalSTAT by the SPDG Evaluation Team of Cheryl "Li" Walter, PhD, and Alan Wood.