skip to content | CalSTAT

CalSTAT Technical Assistance and Training
CalSTAT (California Services for Technical Assistance and Training) is a special project of the California Department of Education, Special Education Division, located at Napa County Office of Education. It is funded through the Special Education Division and the California State Personnel Development Grant (SPDG). The SPDG, a federal grant, supports and develops partnerships with schools and families by providing training, technical assistance and resources to both special education and general education.

• Alan Wood, alan.wood@calstat.org or (707) 287-0054
• Cheryl “Li” Walter, li@sonic.net or (707) 293-3829

Janet Canning, California SIG Director, California Department of Education, Special Education Division

Anne Davin, CalSTAT Director

CalSTAT (California Services for Technical Assistance and Training), at Napa County Office of Education, is a special project of the California Department of Education, Special Education Division. Visit CalSTAT at http://www.calstat.org.

CalSTAT is partially funded from federal funds awarded in Part B of Public Law 108-446, the Individuals with Disabilities Education Act (IDEA), as amended in 2004. Additional federal funds are provided from a federal competitively awarded State Program Improvement Grant to California (CFDA 84.323A) allowed in Part D of Public Law 108-447, the Individuals with Disabilities Education Act (IDEA), as amended in 2004. These dollars are considered local assistance funds. Both funds are to assist individuals serving children birth to 22 years of age and their families.

Table of Contents

• Introduction
• Training and Technical Assistance (TA)
  • Collaborative Sites Survey
• Leadership Community
  • Statewide Institute
  • Leadership Site Award Program
  • Leadership Site Profiles and Cross-site Analysis
  • Regional Institutes
• Effective Reading Intervention Academy (ERIA)
  • Program Overview
  • Summary of Outcomes
• Building Effective Schools Together (BEST)
  • Program Overview
  • Summary of Outcomes
• Stakeholder Input and SPP
• Teacher Recruitment and Retention
• Publications
• Family Outreach and Partnerships
  • Family Participation Fund (FPF)
  • CAC Capacity Building and Recruitment
• Data Tools
  • CST Charting Program
  • Training Evaluation Database (TED)

California State Improvement Grant (SIG2)

In 1999, a five-year State Improvement Grant (SIG) was received from the federal Office of Special Education Programs (OSEP) to improve special education outcomes and services in the State of California. In 2005, a second SIG was granted, SIG2, continuing and expanding original SIG service delivery through June 2008. A new five-year grant, the State Personal Development Grant (SPDG/SIG3) was awarded in 2008 and will continue these SIG activities.

From February 2005 through June 2008, SIG2 implemented activities directed at personnel development, family education, and systems change. Those served by the grant activities included teachers and administrators from both general and special education, early intervention service providers, students and families, related agency personnel and service providers, university/college professors, and university students currently enrolled in teaching preparation programs.

A major focus of SIG2 was to communicate common messages to the field about selected topics. These common, or core messages, articulate critical research findings and essential components of effective practice. All these evidence-based core messages have been identified by experts in the field and have been approved by the California Department of Education, Special Education Division.

CalSTAT, through SIG2 funds, supported activities that reflect all of the following core messages:

• Reading or Literacy
• Positive Behavioral Supports (PBS)
• Special Education/General Education Collaboration
• Transition
• Family-School Partnerships
• Least Restrictive Environment (LRE)
• Individuals with Disabilities Education Act (IDEA)
• Core Components of a strong RtI Process

SIG2 was fully aligned with improvement strategies identified in the California Performance Plan (CA SPP) and is cited explicitly as a part of improvement activities under several key indicators. The CA SPP describes the role of SIG2 in improving special education outcomes in California in the following passage:

SIG2 will be used to improve special education services in California in several areas such as the quality and number of teachers and other personnel who work with students with disabilities, coordination of services for students with disabilities, behavioral supports available for students with disabilities, academic outcomes, especially in the area of literacy, participation of parents and family members, and in the collection and dissemination of data. The grant has a significant site-based component that will include an entire network of educators who have been trained through the first SIG to assist schools in implementing research-proven behavioral approaches.
State of California Part B State Performance Plan (SPP) for 2005-10, page 7

Some of CalSTAT’s many activities include:

• Funding for Training and Technical Assistance (TA) events delivering the core messages to schools and districts throughout California
• The competitive Leadership Site Award Program, identifying sites with best practices in core messages and supporting them in developing and sharing their models
• Building Effective Schools Together (BEST), a schoolwide positive behavioral supports program
• Effective Reading Intervention Academy (ERIA), a literacy intervention program
• Community Advisory Committee (CAC) Capacity Building and Recruitment and the Family Participation Fund (FPF), supporting parent involvement in education
• Teacher recruitment, original publications, and online activities

SIG2 Activities throughout California

Professional development delivered through the Leadership Sites, BEST, ERIA, and other Training/Technical Assistance (TA) activities included 565 events with 12,008 participants during SIG2

• Sites Receiving TA: over 1,200 sites in over 500 LEAs from 54 of California's 58 counties.
• Leadership Sites: 28 sites in 22 LEAs and 14 counties.
• BEST Sites: 105 sites in 30 LEAs and 9 counties.
• ERIA Sites: 47 sites in 18 LEAs and 7 counties.
• Community Advisory Committees (CACs): 4 committees serving 4 SELPAs.

Training and Technical Assistance (TA)

Training and TA are provided around research-based core messages from the eight Core Message Areas through:

• Field-requested TA
• Leadership Site-provided TA
• Regional Institute Training Follow-up

Leadership Site and Regional Institute TA are aspects of the Leadership Community activities, which are described in greater detail starting on page 14.

Over the course of SIG2 (not including BEST, ERIA, or Statewide Institute activities), there were 472 training and technical assistance (TA) events with 13,433 participants.

• While the primary focus of TA in SIG2 has been on SE/GE Collaboration, TA events have included all of the core messages as primary topics.
• To date, teams from 1,200 school/district sites have benefitted from TA, including 671 who received 3 or more days. (Sites are encouraged to access multiple days of training over time as a way to continuously build on past training and to maintain momentum.) These teams include individuals from schools, districts and COEs, along with parents and other stakeholders.

Event Topics and participant roles were tracked by CalSTAT during SIG2

• The majority of training and TA events were in the core message areas of General/Special Education Collaboration and Literacy.

• Most participants were teachers (58% including both general and special education).

• TA participants reported an average 45% increase in knowledge, from 2.9 to 4.2.
• TA events were rated highly by participants, with average usability and overall ratings of 4.3 on a five-point scale.

Follow-up surveys were emailed to recipients to measure the impact of TA between 3 and 6 months after events. Responses to follow-up surveys described implementation and sharing of strategies that worked well.

• The vast majority of participants implemented strategies (83%), found the strategies worked (90%) and shared strategies (87%).
• A majority implemented repeatedly (53%), found the strategies worked repeatedly (62%) and shared strategies repeatedly (62%).

Participants were given the opportunity, both in end-of-event and follow-up surveys, to comment on the TA they received. Responses include the following:

The training received has become a daily resource. Thanks to CalSTAT for providing the forum to have powerful learning and conversations around the latest research-based practices.

Perhaps what is most useful for us is being exposed to so many great practices and ideas then as a district team narrowing those down to some key practices that we implement.

The presentation was very effective because it was presented by teachers to teachers.

We have started to implement true professional learning communities; we are beginning with the math department. We have integrated clusters and are using conversation questions to look at our data more critically.

I enjoyed visiting the classrooms and seeing the strategies that have been put in place to address all students and students at risk. I enjoyed listening to how the district supports the schools and message that is given district-wide. I would enjoy visiting again!

Collaboration transformed our approach to our systems and structures. Teachers feel empowered as never before as they see immediate progress by students. We have shared with other schools and other districts multiple times! Thanks for the support.

CalSTAT has used California Standards Testing (CST) English-language Arts (ELA) proficiency as an indicator of academic outcomes at sites receiving TA. Not all TA events are directly focused on improving CST ELA scores, though most are and many others are intended to improve general academic outcomes, which include CST ELA scores.

• Most sites accessing TA have more students CST ELA proficient and above than California overall. Proficiency at these sites continued to grow throughout SIG2.
• A higher percentage of students with disabilities have become proficient at sites receiving TA than statewide.

AYP targets increased to approximately 34% in 2008, and will increase again in 2009. Both leadership sites and statewide scores are currently exceeding these targets for all students, while neither are meeting these higher targets for students with disabilities.

• The average CST ELA percent proficient and above for students with disabilities at sites receiving TA was roughly equal to AYP Targets in 2006 and 2007. Despite large increases in 2008, they were not enough to meet the higher AYP target for the year.

Collaborative Sites Survey

Special Education/General Education Collaboration has been a major focus of TA through SIG2, with 51% of all trainings delivered in this core message area. Many of the other core message areas include a collaborative component as well, helping at-risk students through delivery of interventions which often include both general and special education teachers in the process.

Sites receiving more than three days of TA in collaboration were identified by CalSTAT and contacted with a survey regarding site-level collaboration between general education and special education. Of the 349 sites meeting this criteria, 63 surveys were returned.

• The average degree of collaboration increased in all ten areas, with the largest increases in Assessment and Intervention.
• These areas, along with Core Curriculum and Teaming, were determined to be key areas of collaboration in an earlier Collaborative Sites Survey. All four indicators grew by 2.4 points or more, with individual sites seeing an average growth of 87%.
• Parent role was the least integrated area overall, though now more collaborative than separate with an average of 6.4. This area saw the smallest average increase for sites responding to the survey.
• On average, sites were more separate than collaborative prior to receiving TA in most areas, and more collaborative than separate after.

Sites responding to the collaborative sites survey were generous in taking the time to provide detailed comments describing their sites before and after receiving TA, their experiences along the way and their expectations for the future. These comments include the following.

Please describe your organization’s special education service delivery system prior to the training/TA:

Some pull out for specific core subjects, and some push into the classroom. In addition, both certificated teacher and aide ran groups include SE and Non-SE students. RSP [resource specialist program] teacher monitored all progress.

Prior to our district’s involvement with CalSTAT, we had a very traditional model for delivering services to students with IEPs [individualized education program]. SE teachers had caseloads and taught the students on those caseloads.

Students were referred to a student study team. There was not a systematic approach in providing intervention services in the GE setting prior to SST [student success team] referral. At the SST meeting the focus was usually on identifying the student for SE services. SE staff worked in isolation and developed their own curriculum and curricular modifications without a supportive structure that enables frequent communication or collaboration with GE staff. In addition, they did not always have access to a full range of core curricular materials.

The use of a formalized progress monitoring system was not being utilized. There was a disconnect/lack of structured communication regarding formalized progress between SE and GE in the area of literacy.

Please describe your organization’s special education service delivery system now:

SE and GE staff collaborates more effectively. School sites have GE interventions and most referrals for SE assessments happen after it is clear the interventions were not working. SDC [special day classes] students are more integrated in GE classes when appropriate. GE teachers and administrators are more engaged in the IEP process to determine needs, goals, and services.

Each district has a collaboration and intervention plan. New service delivery systems started in all districts. Training more focused on specific needs and interventions.

Teachers meet weekly to review data and discuss students. Students are served based on their needs as indicated by the shared data. Even students without an official label can access these services.

Now that over 36 school sites have attended the RtI trainings—supported by CalSTAT funding—for the past two years and the PEAK 3 Literacy series for one; [our county] now has more school districts looking to make an entire shift in all their schools to an RtI service delivery model. Systems change is happening in [the county].

While we still operate using a ‘blended services’ model, we have broadened our service delivery model so that it aligns more closely with the RtI model. As a result, we have far fewer ‘intensive’ intervention classes with the development and success of our approach. Additionally, we are identifying fewer students for SE under the SLD [specific learning disability] category. We have also been working on our strategic supports to prevent students from getting to that intensive level.

What challenges have you had to address in the process of working towards a more collaborative approach?

Main issue we have to deal with and we continue to deal with is time. We have increased instructional minutes and we are trying to implement a full collaboration model. When and how often do we meet to fully implement the principles of collaboration?

The greatest challenge has been the fundamental belief that not all students can meet the expectations and standards addressed in the core curriculum. Additionally, there is the belief that GE teachers may not have the skills to deliver curriculum to students with IEPs. “ I don’t know how to teach those students.” These two challenges are at the core of our ongoing professional development.

Lack of resources to provide more TA to support sustainability of systems change for making a shift to adopt collaborative models of services for special education students.

What has helped facilitate the process of working toward a more collaborative approach?

The statewide leadership institute has provided us with the exposure to best practices throughout the state and the opportunity to network with the practitioners who are implementing these best practices. Each year the institute has inspired a new group to think differently about serving students in the LEA.

Weekly collaboration time—school wide—strictly focused on curriculum and instruction. Training on how to use data to drive instruction. Building trust and relationships. Challenging the culture of the school. Celebrating successes. Setting yearly school-wide and department goals. Requiring objective, outcomes, and feedback for all lessons.

Now that the administrative team has focused knowledge of the goals and purposes of collaboration the support is much more effective and consistent in reaching the goals associated with collaboration

Once a site team gets excited and learns to facilitate the processes and has good data to measure progress, we have seen great movement.

What results have you seen since adopting these changes? In API scores? SE referrals? Behavioral referrals? Student/teacher attitude?

Students have more success in the GE setting and also we feel that the results of collaboration has also aided in the passing of the exit exam as well as completion of high school requirements.

One of the most dramatic results of our changing our service delivery is that we have decreased the number of students in SE by 1,100 over the last 5 years. Additionally, many regular education students are successfully served in collaborative classrooms that might otherwise be referred for SE. Several of our schools have seen a steady rise in test scores of their students with disabilities.

Since adopting the co-teaching model, our behavioral referrals have dropped drastically. The students are in with their peers and not isolated. We actually exited students who probably would not have been able to be exited in a non-collaborative model.

When we hear our teachers, parents, and students speak about collaboration they are passionate and enthusiastic and state they would never turn back to the way it used to be.

What future plans do you have for sustaining or expanding the move towards collaboration?

We hope to sustain the changes we have seen by creating the foundational systems that will support this work regardless of who is in charge or who is involved. Our teachers would never want to return to the old model of doing school. Our district is providing the resources we need to implement assessments and interventions with fidelity.

District is continuing to work with certificated staff and employees bargaining groups to expand collaboration. Additionally, the district is working on benchmark assessments, common formative and summative assessments, and “translating essential standards into student friendly language.”

SE is working more closely in the PLC’s [professional learning communities] to become part of the weekly rotation time. This is a time where the teachers identify students who need additional help in a particular skill set. The students divide into fluid, flexible groups among the grade level teachers to re-teach that particular skill to proficiency using differentiated instructional strategies. We hope to incorporate our SE students into this rotation.

Are there any areas where your site is in need of further training or technical assistance?

Absolutely-we need to be become more sophisticated in data management at multiple data entry points and translating that knowledge into practice with teachers to assist them in making shifts in their performance in the classroom.

We need to continue our work in looking at data and identifying what the data is really telling us. Also we need to further develop skills in peer reflection .

We look forward to gaining more training and information regarding research based intervention programs in the areas of math and reading.

Yes, we would support any future trainings that might become available to increase our skills base in helping teams communicate, address issues, and resolve obstacles that would interfere with the collaboration process.

SE/GE Collaboration at Leadership Sites

In addition to the 242 TA events in Special Education/General Education Collaboration, SIG2 also advanced this core message by making 16 of 28 leadership awards to schools with exemplary collaborative practices, strategies and programs. Detailed profiles of these sites are available online, offering a clear picture of Collaboration in action. Visit http://www.calstat.org/leadershipSites/index.html and click one of the links to any of the Collaboration site profiles including information about the site, the site’s story, and presentations and PowerPoints about their program.

The Leadership Site Award Program is detailed below.

The Leadership Community

SIG2 devotes a significant share of TA and resources to a “communities of practice” model throughout CalSTAT activities. This approach reflects effective, evidence-based practices in improving personnel knowledge and skills and involving collaborations between parents, administrators, and personnel representing all aspects of education, both within and between schools.

The Leadership Community expanded and proliferated these principles during SIG2. The community included three major components:

• Statewide Leadership Institute, a collaborative and professional development activity culminating in an annual three-day event.
• Leadership Site Awards, recognizing highly performing sites who were identified through a competitive process and renewed pending an annual review.
• Regional Institutes, locally sponsored professional development activities hosted by sites who are identified through a competitive process.

By building upon the support they received through the Statewide Leadership Institute and other resources throughout the year, these Leadership Sites and Regional Institutes delivered most SIG2 TA. Of the 472 TA events delivered during SIG2 (see page 6), 345 were based around a Leadership Site or Regional Institute host site (73%).

TA Delivery Within a Learning Community

In implementing SIG2, CalSTAT embraced the evidence-based principle of effective teacher training through learning communities. In successful programs, learning communities are effective in offering feedback to the community, answering questions, helping plan activities and set goals, discussing problems, and providing on-going support for these research-based activities. Peer-to-peer instruction in learning community settings utilizes personnel who have “been there, done that” and allows them to share their experience schoolwide.

Effective training programs also give participants the opportunity to reinforce what they have learned by testing, repeating and practicing. Over the course of a year, Leadership Sites and Regional Institute host sites continued to be a resource to sites receiving TA, sustaining lessons learned through repeated contact. Further, as members of a larger regional community of educators, sites who received Leadership Awards and hosted Regional Institutes often continued to serve as regional leaders, year to year.

While the formation of a thriving learning community was a key focus of the SIG2 Leadership Community, these principles were pervasive throughout all TA delivery and, often in modified forms, throughout all SIG2 activities. Behavior and literacy intervention activities, ERIA and BEST, were both sustained through statewide collaborative activities, including statewide meetings, conference calls, and online discussion forums. These activities further built on themes of collaboration and partnership in other contexts.

Statewide Leadership Institute

Each year during SIG2, CalSTAT hosted teams from leadership and regional institute host sites in a statewide institute. This three-day event is supported with wrap-around TA and online activities, presenting an opportunity for leadership sites to communicate with one another and further develop their programs.

Participants in the institute benefited from content training and discussions with experts in the core message areas. Site-hosted roundtables allowed sites to share learning and practice presentation skills. Participants were also given opportunities to network with other sites and meet with their own teams, to synthesise learning and plan for the future.

• Each site sends a diverse team of representatives from their site, including teachers and administrators, and often parents and other certificated professionals.
• Leadership sites were strongly encouraged to bring parents to the institute, leading to 8% of participants being parents or family members.

• Participants in the statewide institutes rated the event highly, with an average overall rating of 4.4 and a usability rating of 4.6.
• Self-reported knowledge prior to and after the event shows a 39% increase.

Leadership Site Awards

As part of SIG2, high performing sites were identified through a rigorous, competitive process each year. These sites articulated their own site-level successes through this initial evaluation and demonstrate a continuity of their award-winning practices and high standards through annual review. Leadership sites continued to develop their programs, as well, by receiving ongoing TA activities and participating in the Leadership Community.

Additionally, leadership sites distributed TA to other sites based on their first-hand experience developing successful strategies, by offering site tours, and through other peer-to-peer contacts. These sites received awards for best practices in the core messages of literacy, behavior, collaboration, transition and family partnerships, and each established themselves as regional leaders in their own core message area.

During SIG2, CalSTAT issued Leadership Site Awards to 28 sites representing 22 LEAs in 14 counties, including:

• GE/SE Collaboration, 16 sites
• Behavior, 5 sites
• Literacy, 3 sites
• Transition, 3 sites
• Family Partnerships, 1 site

Sites who received awards committed to:

• Providing three professional development presentations at Regional Institutes, local training events, and/or site visits
• Participating in the CalSTAT Statewide Leadership Institute annual face-to-face meeting and online wrap-around services
• Continuously improving their model project by receiving training through ongoing professional development activities
• Developing a web page describing their program to be posted online

During SIG2, 28 Leadership Sites were awarded, 23 to schools and 5 to school districts and county offices of education. Leadership site awards were made in five areas:

• Sixteen awards were made in the area of General/Special Education Collaboration, an area of systems change which emphasizes working together in new ways to conduct assessments and interventions to improve outcomes both for students with disabilities and for all students. Reflecting the SIG2 focus on this core message, the majority of awards have been given for Collaboration.
• Five awards were made in the area of Positive Behavior Supports, creating a school culture based on safety and respect that fosters positive education outcomes.
• Three awards were made in the area of Literacy; while improving literacy outcomes has been central to other leadership sites, these sites focused on reading programs in particular.
• Three awards were made in the area of Transition to Adult Living, supporting students with disabilities in having post-school successes. Because this leadership site is awarded based on post-school rather than academic outcomes, these sites have been dis-aggregated from measures of leadership site academic success.
• One award was made in the area of Family-School Partnerships based on systemic efforts to gain family support and solicit parent feedback.

Integration with other SIG2/CalSTAT Activities

Leadership sites serve California by delivering much of SIG2 TA. Additionally, some of the sites participating in BEST and ERIA have also accessed these TA events, both to reinforce and expand upon the benefits of these programs and to support the systems change necessary to fully implement research-based practices with fidelity.

• Leadership sites participated in 272 TA events during SIG2, including 230 events where leadership sites were delivering TA to other schools and 42 events where leadership sites received TA to continue improving their models.
• All three Literacy leadership sites are members of CalSTAT’s ERIA literacy program (page 22). Leaders at these sites have been key in effecting district-level scale-ups of ERIA in Orange County.
• Two of the five behavior leadership sites are also members of CalSTAT’s BEST positive behavior support (PBS) program (page 36).

CST ELA Scores

CalSTAT has used California Standards Testing (CST) English-language Arts (ELA) proficiency as an indicator of academic outcomes at sites with Leadership Site awards.

Improving ELA scores have been an objective of the 25 leadership sites with academic goals. In addition to the 3 literacy sites, all 16 collaboration sites target reading education with collaborative activities. For the 5 behavior sites and 1 family participation site, improving academic outcomes is a perceived benefit of the program, though not the focus of the leadership site award. However, the 3 transition sites have specific goals which do not include academic outcomes and are not included here.

• The average CST ELA percent proficient and above at leadership sites has remained over 10 percentage points higher than the percentage of all students statewide.
• Leadership sites have had average CST ELA scores for all students at least six percentage points higher than the average for sites receiving TA (see page 9), demonstrating how leadership sites have led the way to improved education outcomes.
• The average CST ELA percent proficient and above for students with disabilities at leadership sites has been much closer to that of students with disabilities statewide, while edging slightly ahead over the last five years.

Leadership Site Profiles and Leadership Cross Site Analysis

Profiles for each leadership site at www.calstat.org offer program overviews, PowerPoint presentations, contact and other information to visitors who want to learn more about innovations and successes in the core message areas. These profiles are available online at http://www.calstat.org/leadershipSites/index.html and have been used as a training tool in a number of contexts throughout SIG2.

Additionally, during SIG2, interviews were conducted with staff from the Leadership Sites to learn more about their systems change processes. Questions focused on what led the site to develop its award-winning program, programmatic details, stages of implementation, and outcomes. Out of these interviews, stories have been developed describing each site’s unique experience and approach. These stories are included in the site profile for each leadership site.

These site profiles and an cross-site accompanying analysis describe “inclusive collaboration,” a term for the common characteristics that emerged from these site’s efforts to improve. The product of this cross-site analysis, They Are All Our Kids, summarizes these common elements and is available on the CalSTAT website at http://www.calstat.org/leadershipSiteAward.html#eval.

Stages Leading to “Inclusive Collaboration” at Leadership Sites

While there were a number of variations from one leadership site to another, a distinct progression of implementation was identified across all seventeen SIG2 leadership sites who were interviewed. For example, many sites pursued research-based literacy interventions as part of a data-driven effort to improve academic outcomes, and leadership sites universally experienced resistance by teaching staff to several changes. Leadership sites adopted inclusive collaboration by moving through four phases:

• Impetus for Change, a prelude to implementing inclusive collaboration which impacted both general and special education, disrupting “business as usual.”
• Early Implementation, a phase with small-scale efforts implemented on a voluntary basis, piloting the changes that would later develop schoolwide.
• Later Implementation, where each site capitalized on successful pilot programs with additional resources and increased formalization. Despite continued voluntary teacher participation at this phase, traditional special education delivery became an exception rather than a rule at this phase.
• Replication and Expansion of successful strategies.

Successes and Results: Most sites reported experiencing positive changes in test scores, grades, student behavior, and school culture. Many sites also reported a reduction in the number of students receiving special education services, both due to fewer student entrances and an increased rate of exits. Additionally, almost all sites reported that inclusive collaboration created a sense of accountability for all students, transcending traditional general and special education divisions. The emergent sense that “they are all our kids” was attributed to the joint effort in teaming, blending classrooms, and creating interventions to assist all students in succeeding.

Regional Institutes

During SIG2, CalSTAT supported a series of Regional Institutes to create an ongoing learning community, sustain and expand systems change efforts, encourage and support meaningful family involvement, expand the capacity of attending school-site teams, and identify and share successful practices.

These institutes were locally sponsored by regional host sites who participated in a competitive award process to received CalSTAT support. Unlike the competitive Leadership Award process, which continued support of sites who maintained high academic standards, Regional Institutes were newly distributed each year. These sites received TA and coaching from CalSTAT on how to effectively deliver professional development activities regionally and locally and were participants at the Statewide Leadership Institute.

During SIG2, CalSTAT funded 21 Regional Institute Host Sites who hosted 49 events. These Regional Host Sites included 10 SELPAs, 10 LEAs, and 1 school representing 15 counties. Many regional Host Sites provided TA in multiple core message areas.

During SIG2, there were 49 Regional Institutes. The professional development provided in these multi-day events were expanded upon and sustained by follow-up TA delivered to institute participants at 73 additional events. The 49 actual Regional Institutes are described below while institute-related TA is summarized with other TA.

• The majority of regional institutes were in the core message area of SE/GE Collaboration, and most participants were SE or GE teachers.

• Regional Institute participants reported an average 45% increase in knowledge, from 2.9 to 4.2.
• Regional Institute events were rated highly, with average usability and overall ratings of 4.4.

Effective Reading Intervention Academy (ERIA)

The Effective Reading Intervention Academy (ERIA) supports schools in identifying struggling students and trains teachers in providing effective interventions to assist in improving specific student reading skills.

CalSTAT began working with local education agencies in 2004 to bring ERIA to school sites. Each cohort of approximately 10 schools receives training and ongoing support. Sites in West Orange County and Antelope Valley piloted ERIA starting in the 2004-05 school year (Cohort 1), followed by San Joaquin Valley in 2005-06 (Cohort 2) and San Diego County in 2006-07 (Cohort 3). Cohort 1 has conducted a series of district-wide scale-ups, increasing the size of the cohort to 27 sites. Cohort 2 and 3 will expand in the 2008-09 school year.

In the 2007-08 school year, there were 47 active ERIA sites in 18 LEAs and 7 counties, including:

• Cohort 1: West Orange COunty and Anteleope Valley (27 sites)
• Cohort 2: Southern San Joaquin Valley (11 sites)
• Cohort 3: San Diego County (9 sites)

During the 2007-08 school year, implementation of ERIA advanced considerably and student outcomes measurements showed gains.

• Team Implementation Checklist (TIC) responses indicate that 28 sites (60%) are fully implementing ERIA’s key elements, and 20 of these sites are (43%) fully implementing these elements schoolwide.
• Improvements in literacy outcomes have grown compared with last year in both grade leveling and oral reading fluency measures.

Of note: Wildfires forced evacuations of the schools and communities around many Cohort 3 ERIA sites last October, interrupting interventions and impacting outcomes.

Key Elements of ERIA

ERIA delivers differentiated instruction to students depending on need. Students who are falling behind or are at risk of falling behind in their reading levels are identified for intensified instruction in key areas to reach proficiency in English-language arts (ELA). As a program, ERIA seeks to embed key principles of evidence-based literacy education in schools throughout California.

The ERIA program includes four key elements:

• Frequent and comprehensive assessment of student reading skills.
• Delivery of literacy intervention programs as necessary.
• Review of placement following a response to intervention model.
• Systems change resulting in the sustainability of these best practices at sites.

Assessing Students

Assessment of student reading levels is key to making data-informed decisions about student placement and interventions. Assessment is a multifaceted activity which involves examining existing data sources, such as scores from the California Standards Testing (CST), generating new benchmark data at regular intervals with a variety of measures, and interpreting these data against specific criteria. Three major skill areas are identified for assessment at most ERIA sites:

Three major skill areas are identified for assessment at most ERIA sites:

• The San Diego Quick (SDQ), a grade-levelling vocabulary-based assessment, also described as a decoding tool in some parts of the ERIA manual.
• Oral Reading Fluency, a timed reading test used to measure incremental change.
• Comprehension, testing understanding of text by asking questions.

Assessment in all three areas is encouraged, but sites are only required to turn in scores from the SDQ and oral reading fluency as part of annual program evaluation. Most of these assessments involve one-on-one contact between test administrators and students.

Frequent and comprehensive assessment requires large time commitments; this issue has been mitigated by sites in a number of ways: Many sites use CST data and curriculum-embedded assessments as a primary assessment tool, referring students at risk in these measures for additional testing. One school which applies comprehensive assessments to its entire student population hires a team of substitute teachers to conduct testing, easing the burden on teachers and instruction schedules. Other sites have made use of parent volunteers in a similar role.

Student assessment is only the first step in implementation of ERIA, but many sites report that it has its own inherently beneficial effects in securing faculty interest and staff buy-in. Some sites have instituted weekly faculty meetings to review student progress, and many teachers have begun sharing assessment scores with students weekly. Organizational culture at ERIA sites appears to be shifting to make use of these newly-available sources of data, and sites are reporting that new attention to measurable outcomes has become a source of motivation at all levels.

Specific Reading Intervention Programs

Intervention delivers additional resources to students with additional needs, typically drawing from one or more research-based literacy education programs. Interventions are organized into three tiers, with a core tier serving all students and two additional tiers delivering additional help to students with additional needs. Tiers modulate the intensity of instruction and move towards more individualized intervention, providing struggling students with the resources and support necessary for them to succeed.

Student needs are matched to one of these intervention tiers through assessment data. Decisions about student placement are typically made on a student-by-student basis and are guided by specific criteria. These placement criteria are developed at the site-level, relating the needs of individual students to the reading level of the overall student population and the instruction provided in the core tier.

Instruction is differentiated within each tier as well, allowing students to benefit from literacy programs targeting specific reading skills. Some literacy programs being delivered as interventions at ERIA sites include Read Naturally, Rewards, Language!, Read 180 and The Six Minute Solution. Many literacy interventions in use address oral reading fluency and grade-levelling, while others such as Soar to Success target reading comprehension skills.

Response to Intervention (RtI)

Frequent, comprehensive assessment and data-informed decision-making are ongoing activities at ERIA sites. This data informs regular collaborative review of intervention placements, and adjustments are made in the intervention placement of students as student needs change.

As a program evaluation requirement, ERIA sites turn in student scores on the SDQ and oral reading fluency assessments representing fall and spring outcomes, but sites make many assessments in addition to these. Many sites “benchmark” their students with three annual comprehensive assessments, adding a mid-year testing phase to the schedule required by CalSTAT. Frequent formal and informal assessments are made between these testing periods, often through intervention-embedded assessments.

Like assessment, review of student placements happens through formal and informal processes throughout the year. Collaboration between teachers, administrators, and often parents and students is a key aspect of effectively monitoring and responding to student progress. Data, including recent assessment scores and classroom observations, are recorded and available for review in a variety of formats which vary by site, ranging from basic spreadsheets to sophisticated charting programs. Most sites support weekly or monthly staff meetings to review student outcomes. Changes in student placement can be made at any time during the year.

Systems Change

The ERIA program cultivates structures and expertise at the site level which support and sustain these best practices. Formation of a site-level ERIA program starts with professional development activities, delivered to administrators, teachers, and other educators. ERIA sites regularly access ongoing professional development for help with aspects of literacy education from data-informed decision-making to more active teaching styles that keep students engaged. Participants in these activities return to the school with these skills and proceed to build a site-level ERIA program.

The key component of ERIA at every site is the ERIA Site Team. Site teams are comprised of administrators and teachers who guide implementation, train and advise teachers, and find ways to leverage local resources and initiative to improve outcomes. Most site teams are led by a site team coach (a member of the team with additional training, expertise, and responsibilities) who works in association with cohort coaches (literacy specialists coaching many schools in many LEAs), district staff, and reading specialists including Anita Archer, Jan Hasbrouck, and Kevin Feldman.

Site teams have a high level of autonomy and are encouraged to adapt the core ERIA program into a unique site-level ERIA program. Site teams work closely with cohort coaches and district officials to adapt key elements in a way that works at the local level. Changes site teams have influenced include integrating time for assessments into the master schedule, reserving the necessary time in instruction to accommodate interventions, and establishing mechanisms for collaborative review. Experimentation and innovation in site team planning is both welcome and encouraged in ERIA.

Site teams have also found ways to meet challenges to full, schoolwide implementation of ERIA’s key elements. For many sites, implementation has been delayed by a lack of resources to conduct comprehensive assessments, too few teachers trained in interventions, and not enough time for collaborative review of student progress. Hiring substitute teachers or recruiting parent volunteers to conduct assessments are examples of how site-level innovation can create effective solutions.

Evaluation and Planning Tools

In planning the implementation of ERIA, site teams develop a Site Action Plan (SAP). This SAP is an electronic form which asks the site team to detail their site-level ERIA program by responding to 24 open-ended questions. The SAP is then revisited by the site team at least twice per year to facilitate reevaluation of strategies and progress, and allowing updates to be made as necessary. The SAP is being accessed as an evaluation instrument as well, creating a coordinated picture of how ERIA’s core elements have been adapted and implemented across all ERIA sites.

Another instrument, the Team Implementation Checklist (TIC), is a companion piece to the SAP collecting implementation data according to 26 specific items and helps the site team evaluate its progress in implementing the four key elements of ERIA.

ERIA in Practice

The 2007-08 school year was the first to see use of the Team Implementation Checklist (TIC). This tool allows ERIA Site Teams to summarize site implementation of ERIA according to 26 explicit criteria. The TIC is complementary to the Site Action Plan (SAP); these forms reinforce one another and are completed at the same time, at the beginning and end of the school year, reflecting changes in both ERIA planning and practice.

The TIC is distributed and completed by sites in an electronic form called the TIC Charting Program. This Excel-based form automatically generates longitudinal, graphic charts as site teams add data, so they can see and use this information in their planning processes without waiting for evaluator feedback.

The average of all 26 items at each site has been explored as an important measure of ERIA implementation. Responses to each item are given as “achieved,” “in progress” or “not started,” and interpreted as 100%, 50% and 0% implementation, respectively.

• An aggregated average of 80% or more is considered to be “fully implementing” ERIA (in Spring 2008, most items were in this category).
• An average of at least 50% but less than 80% is considered to be “partially implementing.”
• Below 50%, sites are considered to be “minimally implementing.”

As of Spring 2008, a majority of sites were fully implementing ERIA.

• Implementation of ERIA advanced considerably during the 2007-08 school year, with the percent of sites fully implementing doubling to 60% from fall to spring.

The table below explores implementation of each of the 26 checklist items as averaged across all ERIA sites.

Patterns of Implementation at the Site Level

Among all active ERIA sites, 20 are fully implementing ERIA schoolwide (43%). The 27 sites which have not yet reached full implementation schoolwide have typically followed one of two strategies:

• 8 sites (17%) are fully implementing ERIA’s key elements, but have not yet achieved a schoolwide implementation of their ERIA program. This has been one strategy ERIA sites have followed, meeting the challenge of limited resources by serving only a portion of the full school population (such as one or two grade levels). At these sites, the existing program can be used as the basis of a school-wide scale-up.
• 3 sites (6%) are implementing ERIA schoolwide, though they have not yet achieved full implementation of key elements. These sites are serving the schoolwide population with some aspects of the ERIA program (such as assessment or intervention), but not others (such as frequent reassessment or collaborative review). These sites plan to establish additional ERIA elements each year, building to full implementation over time.
• 16 sites (34%) are still working to implement ERIA’s key elements and building towards schoolwide implementation. These sites, which tended to have joined ERIA more recently, made substantial gains which were observed between fall and spring.

Average Implementation of ERIA Elements

In ERIA’s key element areas of assessment, intervention, and response to intervention, these gains resulted in the full implementation of 12 of 16 TIC items as an average across all ERIA sites.

• Implementation of assessment-related items is pervasive, with average implementation above 90% for examining student CST ELA scores (item A1), assessment of decoding and fluency skills (A2 and A3), and frequent use of measures and assessments to monitor progress (B6 and C1).
• Implementation of interventions is advanced as well, with interventions in place (B1), being used regularly (B4) and being accommodated within the schedule (D8).
• While assessment data is being collected and used to guide intervention placements, a process of collaborative review has not yet been fully developed at many sites (item C3).

Implementation of systems change is advancing more slowly than other key elements of ERIA. Administrator and structural supports appear to be the most advanced aspect of this area (D1 and D8), with principals and site teams providing active leadership. However, most TIC items in this area are not yet being fully implemented.

• Coaching and administrator awareness of issues related to the fidelity of implementation were emerging as of spring 2008. Increased attention to the fidelity of implementation is a goal for the 2008-09 school year.

ERIA Continues to Grow

CalSTAT has supported specific schools in cohorts across California in developing a literacy intervention program through teaching best practices, providing cohort coaches, and funding associated with the ERIA program. ERIA is prepared to expand further, as well. The 2008-09 school year will see additional scale-up activities grow the size of Cohorts 2 and 3 by roughly double; Cohort 1 will likely expand again as well.

District-wide Scale-up

Districts have taken a leading role in promoting implementation of ERIA. District administrators have motivated sites to participate in ERIA, worked with cohort coaches to provide content trainings to sites and supported site administrators in implementing ERIA at the school level.

Once the four key elements of ERIA are fully implemented in a portion of the schools within a district, the districts participating in ERIA have initiated scale-up activities, spreading these practices beyond the original ERIA sites to other schools in the district.

23 site teams (49%) noted achieving the checklist item “The School Site ERIA program is part of a district-wide scale-up” on the TIC, and another 32% said that it was in progress.

ERIA Sites Received CalSTAT Leadership Site Awards

Another CalSTAT program, the Leadership Site Award competition, identifies sites implementing effective, evidence-based best practices. These sites are given additional resources to develop their programs, give demonstrations to other schools, and serve as regional model sites. Three of the first ERIA sites applied for and received Leadership Site Awards in the core message area of literacy, all in the Ocean View School District: Marine View Middle, Mesa View Middle, and Vista View Middle.

• During the 2007-08 school year, these three sites provided 32 days of technical assistance to other schools in the region.
• As Leadership Sites modeling research- and evidence-based best practices, these three schools have spread ERIA’s key elements beyond the three ERIA cohorts.

Data Collection for Summary Reporting

Data generated by sites around two assessments—the SDQ and oral reading fluency—is also forwarded to CalSTAT for evaluation purposes, once in the fall and once in the spring. All ERIA outcomes described here are from the 2007-08 school year.

These assessment scores allow sites to make data-based intervention decisions, delivering specific, targeted help to students based on their individual needs. To allow CalSTAT to aggregate data from nearly fifty schools, sites have been asked to make assessments based on Summary Reporting Criteria, which standardize assessments between sites. 84% of student data received for the 2007-08 school year met these criteria, up from 57% in 2006-07.

Summary Reporting Criteria

• Must include matching fall and spring scores
• Student and passage grade levels are noted
• Fluency test passages must be at student grade level

Note: CalSTAT does not received student-level data from sites including any student identifiers. Student anonymity and privacy is preserved above the site level for all sites and students participating in ERIA.

Assessments represented in this report include only those meeting summary reporting criteria.

Many sites conducted a large number of student assessments that were incompatible with summary reporting criteria. It is important to note that such measurements may still be valuable in improving student outcomes. While sites are asked to make and report measures to CalSTAT which can be aggregated with those of other sites, ERIA site teams are encouraged to continue making decisions about how best to assess students at their sites.

Additionally, many sites assess only a portion of their students with the SDQ or oral reading fluency measures. Because of time pressures on faculty implementing these tests, existing data is often used to identify students who may be at risk before additional testing is conducted. Consequently, the number of student data reports received by CalSTAT may not be fully indicative of the number of students being monitored for possible intervention across all ERIA sites.

San Diego Quick

The San Diego Quick assessment requires students to identify words on a grade-levelled vocabulary list, with correct identification of 8 or more of the 10 words representing grade-level skill. Student data is only included here if it meets summary reporting criteria and the student is testing below grade level in fall assessments (struggling).

Students who are only one grade-level behind (yellow in the chart) are students at risk, but not necessarily high risk. ERIA sites have shown progress helping these students grow, as well as helping students at greater risk (more than 1 grade level behind) move closer to grade-level literacy.

The student outcomes of individual ERIA sites are identified above and below.

Sites with fewer than 20 struggling students are not identified individually in this report to protect student confidentiality.

Cohort 3 outcomes were suppressed this year compared to last year. This may be due, in part, to wildfires in October that disrupted ERIA implementation. Additionally, a large portion (85%) of data generated in Cohort 3 came from a single site, Spring Valley Middle School, which has implemented schoolwide decoding assessment but does not yet describe the ERIA’s four key elements being either fully implemented or implemented schoolwide.

Oral Reading Fluency

Oral reading fluency assessment tracks incremental changes in the number of words read aloud in a minute from a passage of text specifically calibrated by grade level. Student data is only included if it met the summary reporting criteria and the student showed oral reading fluency scores below the 50th percentile in fall (struggling).

In order to monitor whether ERIA has helped enable accelerated learning at sites, change in WCPM values was determined for all struggling students (spring WCPM - fall WCPM). This was further calculated as a percentage of typical change in WCPM, based upon Hasbrouck and Tindal’s 2004 oral reading fluency study.

Because struggling students are reading more slowly than the grade-level median, greater-than-typical growth is needed to reach the fluency level of other students. Growth above that typical for other students is shown below in blue and green.

• 56% of struggling students in Cohort 1 showed increases in fluency ahead of the typical rate, including 32% growing by more than 150% of what was expected.
• In this chart, the drop-shadow shows student outcomes from last year, demonstrating that growth was much stronger for Cohorts 2 and 3 this year. Cohort 2 has been implementing ERIA for one year less than Cohort 1, and is now achieving similar student outcomes. Outcomes at Cohort 3 are also advancing compared to last year.

California Standards Test (CST) Proficiency

Monitoring CST English Language Arts (ELA) proficiency scores has been a focus of ERIA evaluation, just as it is a focus for all California schools. The following chart averages the percent of students proficient and above schoolwide, with all sites weighted equally.

For each cohort, CST ELA proficiency prior to involvement with ERIA is noted with a dotted line. Cohort 1 is divided between the original ERIA sites (Cohort 1) and those who joined in the 2007-08 school year (Cohort 1 Expansion).

• ERIA cohorts have typically tracked with statewide proficiency over time, often showing improvements above statewide scores.
• The average percentage of all students proficient and above at each cohort is above statewide levels and has increased steadily for five years.
• Proficiency for students with disabilities has been more volatile for schools in all cohorts, though there have been increases in the percent proficient overall.
• Some cohorts experienced a drop in proficiency in 2007, either for all students or the students with disabilities subgroup, though 2008 scores show a recovery.

The chart below shows CST ELA trends for individual sites, as well as site-specific implementation data from the TIC.

Building Effective Schools Together (BEST)

BEST is a program based on positive behavioral supports (PBS) that helps schools develop and implement positive school rules, rule teaching, and positive reinforcement systems schoolwide. To date, over 450 sites and almost 5,000 people have been trained in PBS through the BEST program, including ongoing trainings that have reached over 100 sites and 800 people through the second California State Improvement Grant (SIG2, February 2005 to June 2008), administered by CalSTAT on behalf of the California Department of Education, Special Education Division.

Nearing the end of SIG2, a total of 105 BEST sites remain active. Of these sites, 54 were newly trained in the 2006-07 school year. The other 51 sites were initially trained earlier in SIG2 and have continued to implement PBS and receive booster trainings.

Each BEST school site is served by one of seven BEST regional cadres of trainers which are comprised of local program specialists, administrators, parents and others. Personnel from Institute on Violence and Destructive Behavior (IVDB) at the University of Oregon provided trainings and support to the BEST cadre of trainers and sites in California. These cadres of trainers then provide the initial and follow-up training for school sites as well as assisting each site in implementing BEST. The cadres of trainers are distributed statewide and serve a diverse range of California schools: both rural and urban, coastal and central, in Northern and Southern California.

California's SIG2 BEST/PBS Cadres and Sites

• Los Angeles Cadre (31 sites)
• San Francisco Cadre (18 sites)
• Redding Cadre (14 sites)
• Sonoma Cadre (14 sites)
• Butte Cadre (11 sites)
• Placer Cadre (9 sites)
• Napa Cadre (8 sites)

Implementation of BEST

BEST functions through school-level “site teams”—a collection of administrators and faculty, often including parents and district counselors. These site teams are formally trained in BEST and in turn guide and monitor implementation of the program at their school.

Site teams adapt standard PBS elements and develop new ones as necessary to make sure the program is functional and effective at the local level. This process is driven by site level dialogue and guided by attention to data. Because of the innovations of BEST site teams, there is a great deal of diversity in how BEST is implemented across California.

Common elements of BEST throughout California include:

• Clear and concise behavioral expectations.
  Many sites avoid creating complicated lists of rules for students to follow, instead focusing on formalizing the underlying values from which behavior expectations are derived. The “Three Be’s” are one example which many sites have adopted, consisting of three straightforward guidelines: be safe, be respectful, and be responsible. Many sites modify or add to these standard pieces, such as one site’s fourth Be—”be a leader.” Global, abbreviated standards like the Three Be’s allow behavior lessons to be direct, explicit, and to extend beyond the school.
• Teaching positive behaviors directly.
  BEST sites teach and model expected behaviors directly in a number of different ways. One site produced a series of skits with faculty members acting out positive and negative behaviors, while another held a special “Be Day” event, spending the school day moving students through a circuit of twelve behavior workshops instead of regular classes. Many sites used video cameras in these lessons to allow for subsequent review and reinforcement throughout the year.

“All of the teachers have formulated their classroom rules around the concepts of Be Safe, Be Respectful, and Be Responsible. We have trained the students on cafeteria behavior, playground behavior, assembly behavior, etc. Rules have been explicitly taught and practiced. We have a schedule for reteaching the rules as well. We are off to a great start with BEST!”
–General Education Administrator

• Acknowledging and rewarding positive behaviors.
  Sites often formalize this acknowledgement of positive behaviors by having teachers or staff give out some form of award, ticket or token when they catch a student in the act. These rewards are typically coordinated with a broader theme of good behavior (such as calling them “drops” to reference the book How Full is Your Bucket?), school pride (“good acorn” at a school with “The Mighty Oak” mascot), or another theme emphasizing strong character. These tokens of formal acknowledgement typically serve a double-purpose as raffle tickets or as credits toward prizes, which then further reward the positive behavior.

We implemented the new reward system, and it is making a big difference. The kids love getting them and are trying their best to receive one.
–Parent

“We also have a ‘Caught Being Good’ program, and the kids get tags to hang from their backpack. They can also add perfect attendance tags and academic achievement tags.”
–General Education Teacher

• Attention to the reasons why.
  Many sites participating in BEST approach both teaching positive behaviors and correcting negative ones with a strong focus on the underlying reasons. Conscious and explicit attention to behavior expectations ensure that staff can answer students’ questions without resorting to “I told you so.” Attention to the reasons behind bad behavior is viewed as important as well. One site recognized that their students like and feed “drama,” turning small problems into “passion plays;” they responded by making a rule that staff spend at least two minutes talking to a student as part of any correction, both giving the student a chance to explain the problem and to learn perspective and proportion. Another site’s staff adopted the practice of adding a “because” to every correction (“please pick up your trash so the playground will be clean,” for example), defusing potential confrontations by helping students recognize the reasons for rules instead of just enforcing the rules.

It has been helpful to have a common language that all staff members can use with the students!
–General Education Teacher

• Attention to data and continual reassessment.
  In addition to biannual implementation data and annual behavioral incidence data collected with the support of a CalSTAT data stipend, BEST site teams engage in ongoing self-assessment to build a more effective and responsive PBS program. In one district including a large number of BEST schools, an internal student database was modified to track behavior data alongside academic data, specifically to help BEST site teams. Less formally, another site developed a rubric to guide responses to specific behaviors, only to quickly realize that the behavior problems they were having were completely different than what they expected. The rubric was then updated to reflect these revelations.

Starting in Spring 2007, evaluation of BEST has included use of the PBS Team Implementation Checklist (TIC) developed by Rob Horner and George Sugai to monitor implementation of the program. This instrument asks sites to self-assess degree of implementation according to 17 items relating to activities such as staff buy-in, defining expectations, and teaching behavior expectations.

Responses to each item are given as “achieved,” “in progress” or “not started,” and interpreted as 100%, 50% and 0% implementation, respectively, for the purposes of aggregated analysis.

If all 17 checklist items average to 80% or more, a site is considered to be fully implementing BEST. Most sites averaged at least 50% but less than 80%, and are considered to be partially implementing. Below 50%, sites are considered to be minimally implementing.

• The degree of implementation is advancing at BEST sites. Comparing Spring to Fall 2007, the number of sites fully implementing nearly doubled, from 15 sites to 27.
• Additionally, 11 of the 19 sites who reported minimal implementation in Spring 2007 increased implementation at their sites enough to move to the partial category in the subsequent Fall. (This may not be readily apparent in the pie charts because 6 of the sites who had “not reported” data in Spring 2007 moved to “minimal implementation” in Fall 2007.)
• By Fall 2007, 66 of the 105 (63%) sites reported having partially or fully implemented BEST.

High staff transiency was described as a major hurdle to implementation in some areas, dissolving newly-created site teams year after year. Having time to meet and plan for implementation was another significant hurdle reported.

Implementation by Cadre

BEST is organized around the seven regional trainer cadres presented on page one. Sites within these regional cadres showed varying levels of participation in the Fall 2007 TIC data collection, as well as varying degrees of implementation.

• Four cadres, all with full or nearly-full data reporting, demonstrated high levels of implementation among their sites. These were San Francisco, Los Angeles, Butte, and Redding.
• A total of 25 sites (mostly from Napa, Sonoma, and Placer cadres) did not submit TICs in Fall 2007. Most sites who did not provide a TIC also did not report behavior incidence data (page 9), making it difficult to gauge outcomes in these cadres.

Implementation of Individual BEST Elements

The chart on page seven shows the average degrees of implementation for each of the 17 TIC checklist items by all sites who completed a TIC in Fall 2007. Average responses from Spring 2007 are included as well for comparison purposes (as the gray shadow columns). Of the 17 items, 14 showed increases in implementation.

Items in the TIC are grouped into six categories which relate to different aspects of the BEST program. These and the individual items are explored here and on the next page as averages across all BEST sites. BEST empowers site teams to make implementation decisions at the site level, resulting in a diversity of approaches, so it is important to recognize that an individual school may deviate from these averages.

Exploring average implementation of these items individually, BEST sites overall have demonstrated their commitment and active use of schoolwide positive behavior supports. These elements are seen mostly in the TIC categories of establishing commitment, establishing and maintaining the site team, and establishing school-wide expectations.

• Three items that were fully in place were administrator support and active involvement (item 1), establishment of a representative team (item 3), and defining three to five behavior expectations (item 9).
• Two other items were almost fully implemented, creating a system to acknowledge and reward expectations (item 13) and identifying and involving personnel with behavioral expertise (item 16).

Strong responses to these five items in the TIC demonstrate key aspects of the BEST program overall. Site teams, with support from administrators and staff with relevant expertise, appear to be in place in most BEST sites. BEST sites are also widely implementing use of a concise list of behavioral expectations which they reinforce with a formal reward system. Both of these latter two elements are described on pages two and three of this report.

Meanwhile, the use and integration of data in day-to-day implementation of BEST has seen more moderate progress across BEST sites overall, despite strong implementation and successes in many individual sites. These checklist items are seen primarily in the categories of self-assessment and establishing an information system.

• A number of sites have been slow to complete an audit for efficient integration of the BEST team with other behavioral teams and initiatives (item 5). Note that the formal audit described in this checklist item does not preclude the use of data in less formal contexts.

Finally, many sites have also been slow to develop a plan to identify and establish systems for teacher support, functional assessment, and development and implementation of a support plan (item 17). It appears that many BEST sites would benefit from continued support in cultivating behavioral expertise and fostering the ongoing development of their programs through time.

Ongoing Support for Positive Behavior

BEST’s successes are attributable to the partnerships and teamwork of the agencies and personnel working together to build safer, more productive schools. A theme of partnerships is pervasive in all levels of the program, with coordinated efforts between CalSTAT, the Institute on Violence and Destructive Behavior (IVDB), and schools and districts throughout California. Nowhere have these collaborations been more successful than between the cadres of trainers, school faculty and administrators, parents, and even students.

The positive outcomes of BEST are expected to continue for many schools through the ongoing work of many of these partners and with two additional sources of support.

CalSTAT administers many other education resources, and these have been open to BEST sites who have been interested in pursuing them. Interactions between BEST sites and CalSTAT’s Leadership Site Award program have been particularly productive.

• Paradise Unified School District received a Leadership Site Award for its program of GE/SE collaboration, providing demonstrations and trainings to other schools in the area. Many of Paradise USD’s schools sought out PBS training as well and became BEST sites, to combine the benefits of these different programs.
• Further, two BEST sites went on to receive their own awards as Leadership Sites in the core message area of behavior. These schools, Bidwell Junior High (Chico USD) and Walter Reed Middle (Los Angeles USD) provided additional PBS trainings to neighboring schools with funding from the Leadership Site Award Program.

In Los Angeles Unified School District, the school board recently approved a district-wide PBS program by passing the Discipline Foundation Policy in March 2007.

BEST sites in the district modeled many of the policies that will be advanced under this policy and will continue to receive PBS program support from LAUSD. Additionally, members of the Los Angeles cadre of trainers have become key members in the policy’s Implementation Task Force.

“Every student, pre-school through adult, has the right to be educated in a safe, respectful and welcoming environment. Every educator has the right to teach in an atmosphere free from disruption and obstacles that impede learning. This will be achieved through the adoption and implementation of a consistent school-wide positive behavior support and discipline plan for every school in LAUSD.”
Discipline Foundation Policy
Policy Bulletin, March 27, 2007

Stronger Communities, Safer Schools

The most pervasive outcome reported by the teachers, administrators, and parents involved was a cultural shift at schools implementing BEST. By emphasizing the teaching and support of positive behaviors among students, discipline became a proactive enterprise at many BEST sites, with faculty working with students to create a safer and more congenial learning environment.

BEST faculty members related many situations in which students took initiative in promoting this better environment, several of which have been described earlier in this report.

In another site, the Three Be’s were rebranded as “Cougar Pride,” after the school mascot. The site rewards good students by recruiting them to read morning PA announcements. These students adopted, without prompting, the practice of reciting the Cougar Pride tenets every morning at the conclusion of the morning announcements.

Another community-building effort enlisted faculty and staff in a “valet” service each morning, helping students out of cars and with their backpacks. By greeting students and parents each morning, the school created a safer and more friendly environment, starting off every school day on a high note.

One site has created a “village” environment by setting up “special friend” relationships between students and faulty members. These staff-student pairings are created so that faculty members (who intentionally do not have a professional relationship with the student) take an interest in the student, checking up on them and getting to know them. This creates a “trickle” effect where students see teachers caring outside of their roles as teachers, building a community environment.

Many sites have also found ways of enlisting student support in modeling and rewarding positive behaviors. For example, one school recruits students from third and fourth grade to serve as “play leaders” for first and second graders. These leaders apply for the position not unlike applying for a job, receive training and serve roles similar to faculty members in modeling and rewarding positive behaviors. Play leaders become part of the playground and classroom communities of these younger students, learning their names and working with them over time.

“We have started a BEST Club, where students help with how to follow the rules. They also run a table once a week that give the kids a chance to give drops [rewards] to students and faculty.”
–General Education Teacher

Measurable, Behavior-related Outcomes

Changes in the use and application of office discipline referrals (ODRs) varied considerably, with about half of the sites decreasing their use of ODRs, while the other BEST sites saw increases, including 39% whose use of ODRs doubled or more.

These measures are not based on negative behaviors themselves, but on the response of school officials to negative behavior. Increases may be due to changing school policies and practices, such as more consistent enforcement of school rules or enhanced awareness of negative behavior and its impact on a school’s academic mission. Consequently, the increase or decrease of ODR (or suspension) rates at a given site may or may not be evidence of increased incidence of negative behaviors. Clearly, data collected with these measures suggest a wide range of outcomes.

One of the measures showing the largest decreases was of the rate of students receiving more than a single day of suspension over the school year. For 44% of sites who reported a baseline for this statistic, this rate decreased by more than half. This indicates a sharp decline in the number of students suspended for major infractions or with continued, serious behavior problems.

The PBS program taught by BEST utilizes an RTI (response to intervention) approach. In RTI, students are identified for one of three levels of support according to need (support for all students, for at-risk students, or for high-risk students). Students receiving suspensions would be identified for additional support in this RTI-based behavior model, which may be contributing to the decreases seen in students receiving more than one day of suspension at these BEST sites.

The rate of suspensions and ODRs at fully and partially implementing BEST sites for whom baseline data was available are presented on the following page. These sites represent considerable diversity both in student populations and in how they chose to pursue a schoolwide PBS program. These 38 sites suggest the following:

• Almost two-thirds (25 of 38) of BEST sites had a suspension rate below the statewide rate in the 2006-07 school year.
• Over 60% (23 of 38) saw either a reduction in suspension rate or increases smaller than those of California as a whole.
• Some sites saw decreases in suspensions but an increase in ODRs (and a few saw the reverse). This may be an example of how changing school policies and practices can impact statistics in seemingly contradictory ways.

It is important to note that these rates can change for a number of reasons. Changes in school policy, which many BEST site teams chose to implement, could have had a large impact. Examples of this include greater reliance on non-suspension responses to misbehavior or expanding use of ODRs to respond to a wider spectrum of negative behaviors. Increased awareness of behavior issues and attention to accurate record keeping may also be responsible for increases or decreases in these statistics.

Many sites described positive outcomes as a response to increased attention to ODR and suspension data. Indicative stories include the following:

• One site team realized that African American and Latino students were receiving far more ODRs and suspensions than Caucasians. The revelation resulted in a number of uncomfortable faculty conferences, but the process has prompted this site to begin addressing previously unrecognized disproportionality.
• In another site, analysis linked a large portion of their office discipline referrals to the recess after lunch. By moving lunch from before to after the midday recess, many of these behavioral problems (and referrals) were eliminated.

One cadre of trainers has used teleconferencing software to address data collection and interpretation, having all site teams complete surveys and compile data at the same time. This has resulted in 100% data collection for CalSTAT’s purposes, as well as helping sites share information and reinforce each other’s efforts.

Establishing a system for effectively recording, tracking, and accessing data at the site level was noted as a challenge for many sites. Many sites have used a service called SWIS but expressed concern about its cost. Development of less expensive (or free), usable and accessible data systems could be helpful for BEST sites and other schools which are already trying to track and access data for decision-making.

Helping us think about our data collection process was very helpful. We are well on our way and have incorporated it into our student information system.
–General Education Teacher

Stakeholder Input and the SPP

Improving Special Education Services (ISES) is a collaborative stakeholder group convened by the California Department of Education (CDE), Special Education Division. The California State Performance Plan (CA SPP), Annual Performance Report (APR), and SIG2/SPDG are guided by this meeting in addressing issues such as personnel development, improving statewide assessment outcomes, and progress monitoring through a unified planning process. ISES is the unification of what were previously the Partnership Committee on Special Education (PCSE) and the Key Performance Indicator (KPI) group. The committee consists of a broadly diverse and representative group of individuals involved in, or concerned with, the education of children with disabilities.

ISES has met twice annually since the group was created in January 2007. ISES will continue to meet, following this schedule, to guide SIG3/SPDG.

Each ISES meeting includes stakeholders supported by CDE workgroup facilitators and CalSTAT staff. Participants include both new and returning ISES members.

• ISES meeting attendees represent broad diversity. Predominant participant roles include Special Education Administrators, staff from CDE and other agencies, and Parent Leaders.

The CA SPP is developed by the California Department of Education Special Education Division, with input from broad stakeholder groups including ISES. The CA SPP describes how the state will meet the requirements of the Individuals with Disabilities Education Act (IDEA), as amended in 2004.

To provide input to the SPP, ISES committee members divide themselves into workgroups, each of which addresses 1-4 indicators described in the CA SPP. Experts are recruited by CalSTAT to facilitate discussion and planning, and attending stakeholders participate in the workgroup of their choosing. Workgroups generate recommendations around the workgroup content area, CA SPP indicators, and related SIG/SPDG improvement activities which are then reported to all ISES participants at the end of the event and posted online. This process has informed the writing of the most recent version of the CA SPP, updated in February 2008.

In end of event evaluations, stakeholders who participated in ISES meetings rated their experience highly.

• Overall, the four meetings were rated with an average of 4.1 on a 5-point scale.Participants responded to the question, “Was this meeting of value in advancing the goals/vision of the SPP?” with an average of 4.2.
• Participants rated the workgroups and their own ability to contribute even more highly, with an average response of 4.3 to both questions.

The survey also asked participants, “What were the most positive aspects of this meeting, and why?” Responses include the following:

The variety of participants and their perspectives.

The small groups work and report out of their groups—it’s wonderful to see so many minds actually engaged in problem solving.

All stakeholders made contributions! We walked out with a product!

The diversity of participants was so helpful in getting all of the perspectives and stakeholders involved in the process.

Revisiting earlier stakeholder recommendations and revising as needed to reflect current trends and accomplishments.

Shared understanding—the networking and collaboration, better understanding of the SPP/APR process.

Teacher Recruitment and Retention

The Institutions of Higher Education (IHE) Leadership Project, funded by CDE, is focused on helping to address the continuing teacher shortage in California by delivering information resources to potential teachers through publishing activities and the convenient interface of www.teachcalifornia.org.

This website was established during SIG1 to directly deliver information about teaching requirements, opportunities, and benefits to the public. During SIG2, the website has seen more than 1.4 million visits, from over 600,000 visitors, viewing over 4.5 million pages, and continues to be a resource attracting an average of over 1,000 visits per day.

A pop-up survey collected over 33,000 responses from website visitors during SIG2. Responding visitors represented a broad diversity of age and ethnicity. Additionally, responses indicate the website is being accessed by its target audience.

• Of the visitors to the TEACH California website, over three-quarters of respondents reported thinking about becoming a teacher. Over 3,000 visitors also reported thinking of becoming a special education teacher specifically.
• Of the over 26,000 respondents thinking of becoming a teacher, roughly two-thirds reported already taking steps to become a teacher.

Many of the visitors who responded “N/A” to the survey may be university professors and counselors who are accessing resources to aid in teacher recruitment. In addition to delivering information directly to potential teachers, the TEACH California website offers online ordering for recruitment materials, including videos and brochures; many orders placed are for multiple copies of these resources.

TEACH California provides a number of materials to individuals, schools, and organizations helping to bring new teachers into California classrooms. These pamphlets, videos, and other materials can be ordered or downloaded for free through teachcalifornia.org. Items can be ordered or downloaded separately, or a “toolkit” of nine related items can be ordered as a single unit.

• The documents most frequently shipped were the Becoming a Special Education Teacher in California and Pathways to Teaching pamphlets, both introduced in 2004.
• Several pamphlets introduced in 2006 have been at least as popular, including Teach Science, Teach Math, Service Learning, and Paraeducator.
• The general education-focused Teach Math and Teach Science pamphlets were the most-shipped items in 2007 and, along with Becoming a Special Education Teacher in California.
• The special education-focused pamphlets Becoming a Special Education Teacher in California and Paraeducator were more frequently downloaded.

Generation and shipping of these resources is administered in concert with CalSTAT’s other publishing activities, detailed below.

Publications

CalSTAT administers a number of publishing activities as a resource for stakeholders at all levels of general and special education. These activities involve both print media and electronic resources, and include the following:

The Special EDge Newsletter

The Special EDge is published three times a year and delivered to approximately 50,000 print and email subscribers, including over 8,000 parents. It is designed to inform and support parents, educators, and other service providers on special education topics, focusing on research-based practices, legislation, technical support, and current resources. Current and past issues are available online in English and Spanish.

Some of the most-recent issues published during SIG2 include:

• Schoolwide Behavioral Supports at the Secondary Level
• Special Education and High School Reform
• Inclusive Classroom: One Road to Accessibility

Transition to Adult Living: An Information & Resource Guide

This comprehensive handbook is written for students, parents, and teachers. It offers practical guidance and resources in support of transition efforts for students with disabilities as they move from their junior high and high school years into the world of adulthood and/or independent living.

Frequently updated, approximately 60,000 copies of the Transition Guide were printed and shipped during SIG2.

Podcasts

Podcasts are the newest addition to the CalSTAT website. Users can hear recorded conversations with experts from the field on a variety of topics featured in The Special EDge and CalSTAT activities. Eleven podcasts are currently available, with titles including What is Disproportionate Representation? and the multi-part series, Communities of Practice.

The RiSE Library

CalSTAT also manages the RiSE (Resources in Special Education) Library, which makes available documents that offer research-proven strategies and information recommended by educational stakeholders. Residents of California can check out these resources free of charge by ordering through phone, email, U.S. mail, fax, or by visiting the library in person. The RiSE Library is located at Parents Helping Parents of Santa Clara, a federally funded Parent Training and Information Center.

During SIG2, over 2,000 items have been checked out through the RiSE Library.

Family Outreach and Partnerships

The parents, guardians, and family members of children with disabilities play a significant role in their children’s lives. Building collaborative, responsive environments and relationships between family members and schools enables a greater number of family members to become active participants — and when family members are engaged, it can enhance the system’s capacity to address the needs of a diverse student population.

The California Department of Education has prioritized parents as key stakeholders and partners in the professional development of education leaders and staff. As a project of the Special Education Division, CalSTAT has funded several parent outreach activities and identified families as a key team member in the learning communities they support.

SIG2 activities have included elements of family outreach and partnership as well. Family-school partnership is one of the eight core messages which have guided SIG2 activities, and SIG2 activities have included parents wherever possible. Some of these other SIG2 activities which have contributed to the goal of family outreach and partnerships include:

• 15 TA events with Family and Professional Partnerships as the primary topic. These events included over 400 participants and had an average overall rating on end-of-event evaluations of 4.5.
• Almost 300 parents and family attending TA events in any core message area.
• 60 leadership site teams including at least one parent member attending the three statewide institutes.
• Over 8,000 parent subscribers to The Special EDge newsletter.
• Other publishing activities such as the Transition Guide and RiSE Library serving the needs of parents and family members of students with disabilities.

Family Outreach and Partnerships as a Primary Focus

During SIG2, CalSTAT addressed this core message by providing resources directly to parents and parent organizations, as well. These activities include the following:

• Family Participation Fund (FPF), which offers financial support to the parents and family members of students with disabilities who participate in decision-making meetings around education.
• Community Advisory Committee (CAC) Capacity and Recruitment Project, which partnered CACs with regional Family Empowerment Centers (FECs) to help build the membership and capacity of these community advocacy groups.

Both the Family Participation Fund (FPF) and the Community Advisory Committee (CAC) Capacity and Recruitment Project engaged family members, targeting those from diverse backgrounds, and supporting their participation within the education system while also building their capacity as advocates for their children and other students with disabilities. These projects are discussed in the following pages.

Family Participation Fund (FPF)

The Family Participation Fund (FPF) was developed to meet the need for fiscal resources to help encourage and support meaningful family involvement in local, regional, or statewide educational decision-making activities, events, and groups. The intent of this activity is to build partnerships that reflect the diversity of our population and that are aligned with state reform mandates and initiatives. During SIG2, nearly 5,000 requests for financial support were funded.

The FPF is administered through the California Association of Family Empowerment Centers (CAFEC), which works as a center of information, technical assistance, and systems change advocacy for the statewide network of local FECs. FECs work directly with families of children with disabilities to provide family education and empowerment, regularly referring families to the FPF when they show interest in participating as a member of a decision-making body.

Ethnic minorities and low-income families are often underrepresented in the membership of decision-making bodies. Surveys completed by stipend recipients demonstrate that a majority of FPF resources have served these communities.

• Roughly half of recipients were African American (49%).
• The FPF successfully facilitated the participation of traditionally underrepresented families, with 80% of recipients identifying themselves as ethnic minorities.

• The FPF was also successful in reaching out to low-income families. Of recipients who provided income information, 82% (over 3,700) reported earning less than $30,000 annually, with 57% earning under $10,000 per year.

• The recipients responded with high ratings regarding their experience at the meetings they participated in, with a 4.4 (on a 5-point scale) or higher in every category.
• When asked whether they felt they were an effective participant at the meeting, the average response was a 4.5 on a 5-point scale.

Another goal of the FPF is to increase awareness and build collaborative partnerships between special education (SE) and general education (GE) by encouraging family members of children with disabilities to join decision-making bodies focused on GE issues.

Comments from FPF recipients include:

At this meeting we as parents were welcomed, and our input into policies affecting our children’s education was valued and utilized. Our concerns were responded to, and we saw results.

There is a lot of information to learn and share!

This meeting presented an opportunity for parents to speak directly to the School Board members and for them to hear and understand a parent’s perspective on the current state of affairs.

This stipend is a great help!

To determine their impact on the system, 38 active FPF recipients (family members who received three or more stipends between January 2005 and February 2006) participated in an in-depth telephone interview process.

Each family member indicated a main area of focus in their participation. These areas of focus fall into five general categories:

• The empowerment of other parents and students to become advocates within the education system and the systems change process.
• Holding schools, districts, and the state accountable and responsible for providing an appropriate education for students with disabilities.
• Being an advocate for the needs of students with disabilities.
• Increasing the participation of other parents within the education system.
• Increasing awareness within the general education community of the needs and issues that affect students with disabilities.

Within this small sample, it appears that, as family members continue to be engaged in the education system, they progressively move their focus from the school level to the district level to the state level. For family members with 4 or fewer years of participation, typical focus was on the school level, such as school board meetings. For those with 4 to 10 years, typical focus was on the district or regional level, such as CAC meetings. For those with 10 or more years, typical focus was on the state level, such as stakeholder meetings. Family members typically described their contributions as valuable at all levels of focus.

• A majority of the family members feel as though they are influencing discussions (84%) and decisions (79%) at the meetings they attend.
• A majority of the family members also report taking action to influence systems change (71%).

Interviewed family members reported that within all levels of the education system, they were supporting efforts to keep the system accountable, providing ideas and feedback based on their experiences, and making change happen.

Holding the System Accountable through Monitoring What’s Happening

• Held administration accountable for reaching mandated benchmarks and served as independent monitors and evaluators.
• Became more educated on the laws related to special needs children to hold districts and local schools accountable.
• Challenged school reports that “glossed over” or misrepresented actual lack of progress, then worked to help accomplish real progress.
• Learned to understand complex district budgets, tracked spending, and spoke up when this was not being handled fairly and/or effectively.

Contributing Ideas and Giving Feedback in Planning Processes

• Interacted with and influenced superintendents, directors, district personnel, teachers, experts, etc.
• Participated in the revision of special education training manual IEP guidelines.
• Contributed ideas to staff for how to make transitions easier for both students and parents: elementary to middle school; middle school to high school; high school to adult life.
• Brought ideas and materials back from meetings, workshops, and conferences from outside of the district and shared with staff, administration, and other parents.
• Advocated for more staffing in classrooms and for afterschool activities.
• Participated in summits with other parents, school staff members, experts, etc., working to generate new ideas and sharing success stories so others might try these ideas in their districts.
• Helped “educate” teachers about special needs children to better understand their needs.

Actually Got Changes Made

• Compiled data and experiences to effectively counter arguments to cut or remove needed speech services, occupational therapy, teacher assistants, and transportation to and from school for special needs children.
• Prompted one school serving special needs children without any medical staff on site to hire a nurse, first for a few days a week and then every day.
• Initiated installation of ramps throughout a school campus so that students in wheelchairs could reach classrooms (and be counted in attendance statistics).

CAC Capacity Building and Recruitment

It is the mission of all CACs to act as collaborative partners with Special Education Local Plan Areas (SELPAs) to ensure that students with disabilities receive free and appropriate public education and have equal access to all services. CACs serve as a liaison between the SELPA, COE, local districts, families, and community members. Therefore, it is important that the membership of a CAC be representative of its community—economically, ethnically, and by disability.

As a pilot project, in 2005 five Community Advisory Committees (CACs) were partnered with Family Empowerment Centers (FECs) in their areas to build the capacity of the CACs as advocates for students with disabilities, to develop support networks for parents, and to increase the membership of underrepresented groups. Collaborative and individual surveys have allowed CAC members to describe their recent experience, forming the basis of this evaluation.

Participating CACs included:

• Trinity CAC
• San Fancisco CAC
• Santa Clara CAC
• Santa Ana CAC
• Frenso CAC (participated briefly in first year, but chose not to continue)

The CACs involved in this project were selected because they had identified the need to recruit a greater number of members who were representative of the community they served. These five CACs serve diverse communities with over 230,000 students, over 20,000 of whom are in special education. Each CAC has a distinct student population:

• Two CACs have no majority ethnic group
• The percent of students who are English learners ranges from 1% in Trinity County to 56% in Santa Ana
• Free/Reduced-price Lunch enrollment ranges from 45% to 60%

Between 2005 and 2008, this partnership has allowed the members of the CACs to benefit from the resources and experience of FEC personnel. This is in alignment with the general mission of local FECs to provide family education, empowerment, and parent-professional collaborative activities for families of children with disabilities.

Much of this support comes in the form of general assistance with procedure, budgeting, and communication with school officials. Material support has included providing advertising, transportation, and translation services.

Membership Recruitment

Official CAC members must be appointed by the local SELPA director, but many more parents participate in CACs unofficially. Each CAC recently reported that their membership (both official and unofficial) is anchored by a handful of participants (between 5 and 16) who attend regularly and actively. A larger body of participants attended trainings, events, and occasional meetings. As a central goal, these CACs continue to encourage parents to become more-frequent participants.

Since this program started in 2005, efforts to increase membership have been successful, increasing the overall number of regularly-attending members by 52% (from 27 participants in 2005 to 41 in 2007 across the four participating sites).

• Increases in the number of low-income and minority members surpassed growth targets of 30%, with increases of 46% and 50% respectively.
• With 10 or fewer regularly attending members at three of the four CACs, there remains a need for aggressive recruitment efforts, despite recent gains.

Activities and Successes

Recruitment continues to be a primary activity of the CACs. Successful, ongoing efforts include:

• Potlucks, open-houses, and community meetings to attract newcomers
• Working with schools to distribute fliers (with meeting dates and other information) in back-to-school packets
• Generating resources (in print and on the web) that reach new parents

All four CACs distributed literature to parents. These activities included sending fliers and brochures home with students, distributing newsletters and developing websites. The goal of these communications varied from posting CAC meeting times to providing information about disabilities and special education.

• Two CACs are publishing newsletters (with circulations of 10,000 and 6,000 respectively), including editions in English, Spanish, Chinese and Vietnamese
• Two CACs established websites to post resources and meeting information
• Many CACs actively distribute other resources such as brochures, resource lists, handbooks, and information about the Family Participation Fund

Another function of the CACs has been to help parents take a larger role in the decisions effecting their children. This was achieved through a variety of strategies including:

• Distributing information, including resources developed by the CAC itself
• Encouraging parents to participate in the CAC

CACs have provided parents within their community the opportunity to attend trainings on topics including assistive technology, alternative dispute resolutions, leadership and advocacy. Many of these trainings were given by outside agencies hosted by or in partnership with the CAC. All four CACs hosted at least one training or workshop during SIG2.

Additionally, CACs have a structural role as a point of contact between parents and school officials in the special education community. Ideally, the community will use the CAC as a forum to ask questions and provide input while school officials use the CAC as a collaborative partner and venue to disseminate information to the community. While this relationship is strained in some CAC locations, these contacts are still reported as successful.

Beyond activism and advocacy, CACs successfully function as an anchor point for their community. CACs help create social networks and provide emotional support for the parents and families of students with disabilities. One FEC facilitator described this in a survey response:

Our goal is to empower parents and the community with the importance of knowing, understanding, and utilizing their rights and benefits through their consistent and dedicated involvement with the CAC as a CAC member.

Critical Elements

When asked to identify what facilitated these successes, the respondents indicated the following three elements are critical for success:

• Providing skill development training for CAC members and support for networking among family members
• Discretionary budget and/or administrative support for use toward meeting and training announcements, information dissemination, and recruiting and retaining members
• Collaborative working relationships and responsive communication between CACs and special education administrators or SELPA directors

Each of these three elements were found to have contributed to the successes reported at the sites. However, these elements are only partially in place at the CACs.

Barriers to Success

All four FEC representatives were very clear that they see the CACs being blocked from achieving their stated goals of representing community viewpoints and delivering input to education policy.

According to FEC representatives, parents who want to be involved in decision-making are often discouraged by the perception of an adversarial relationship with school officials. These parents suggest that some school officials want the CACs to function as a rubber stamp and have accused them of using their power to achieve this outcome—by refusing to appoint new CAC members, by restricting access to parents through student take-home packets and fliers, or by cutting CAC participants out of the process.

Frustration with this environment is named by many FEC representatives as the leading reason parents cite when they stop attending meetings and participating in CAC activities. Many parents, especially those with small incomes, are described as refusing to invest any more of their own valuable time in the CACs when they doubt they will ever be able to actually influence education policy or outcomes.

FEC representatives made a priority of improving relationships between CAC members and school officials over the last two years. While describing successes both incremental and broad, many CACs continue to operate in an uncertain environment. Streamlining the appointment process for new CAC members and reinforcing the critical elements of CAC success are seen by many FEC representatives as essential to building a functional network of CACs.

One FEC facilitator described how these barriers have created opportunities for growth:

One person, in particular, was threatening to file a complaint and her director caved in and elected her; she has become a leader in her community. She has worked hard and positively. Once she was elected, she wrote back to her director saying ‘thank you so much, I know we can get more done together than separately.’ She said she would look forward, not backwards, and I thought that was excellent.

Data Tools

SIG evaluators have used the evaluation of SIG activities as an opportunity to help build a culture of data use across California’s educational community. SIG evaluators committed to timely development of evaluation materials that will be relevant and accessible to site-level decision-makers. During SIG2, data tools were developed which allowed sites to track and use data at the local level, with instantaneously generated visual representations of data.

The two tools with the widest application are detailed in the following pages, including:

• CST Charting Program: A chart-generating Excel template allowing sites to create visual representations of California Standards Test (CST) proficiency level data.
• TED: A sophisticated Filemaker Pro database allowing sites to track event data and automatically create reports based on event evaluations.

Other Evaluation Tools

Additionally, evaluation instruments were created for the BEST and ERIA programs using electronic formats allowing instantaneous, longitudinal site-level charting. The centerpiece of these efforts has been the TIC Charting Program, which includes separate versions for BEST, a schoolwide positive behavior supports program, and ERIA, a literacy intervention program.

The TIC, or Team Implementation Checklist, is a an instrument which site teams review and complete twice per school year. As the implementation of program elements advances at the site, additional checklist items can be recorded as completed. In both the BEST and ERIA program evaluations, the TIC is one of several instruments which together create a full picture of the degree of implementation.

The TIC Charting Program operates in Microsoft Excel. It includes data entry sections within the workbook and a number of automatically updating charts which create visual representation of evaluation data. As site teams complete the TIC using the TIC Charting program, they are instantly rewarded with a graphic demonstrating their progress. The TIC Charting program is configured to collect up to four years of data, allowing sites to monitor their progress through time.

Other electronic instruments have been integrated in the ERIA assessment. These include the use of word processing files to build a comprehensive, longitudinal Site Action plan (SAP) and a modified version of the CST Charting program which allows sites to access CST ELA scores more directly in their planning.

CST Charting Program

The CST Charting Program was designed to enable schools and districts to create a single page graphical chart showing the percentage of all students compared with students with disabilities as a subgroup performing at each of the five levels of proficiency on the California Standards Test (CST) for English-language arts (ELA). The chart shows the achievement gap, and that most students have a long way to go in achieving grade-level proficiency. Using the same tables of numbers that parents, teachers, and administrators have had access to for years, the CST Charting Program presents this information with colors and proportions and makes it simple for most people to grasp where students are at a glance.

Over 750 copies of a CD-ROM (or an emailed version) containing the CST Charting Program have been distributed to personnel and parents. Also included are reporting templates as well as a series of brief instructional movies for how to use the program.

The CST Charting Program is brilliant and invaluable. Simple to use, the program produces an elegant graphic that shows clearly whether we are closing the gap between groups/subgroups. Thanks to ERIA, we are doing just that!
–School District Administrator

The CST Charting program is available to download online at http://www.calstat.org/CSTProficiencyCharting.htm

TED: Training Evaluation Database

The Training Evaluation Database (TED) was originally developed during the first State Improvement Grant to track the professional development training events funded by CalSTAT and to evaluate the success of these events. As part of the second State Improvement Grant this project was expanded and TED was further developed to be shared with SELPAs, COEs, and LEAs that were also providing professional development within their local area. The objective was to assist selected sites in effectively tracking the trainings provided while also offering easy-to-use evaluation functions that determine the success of events and inform future training decisions.

The TED Program

TED was developed to hold both basic and advanced event tracking information so that each site can customize their level of use. The following is a brief list of some of these functions:

• Track events across multiple funding sources
• Participant registration
• Participant payment tracking
• Printable event roster and name tags
• Mailing labels
• Email lists by type of participant
• Event certificate of completion
• Professional development hours
• Presenter database
• Event evaluation form
• Evaluation entry
• Reports at the click of a button

Sites that are interested in using TED receive both initial and ongoing user support to ensure a smooth transition.

• Initial support includes assistance in determining the in-office set-up and use-protocols, installation of the program, and in-person training for all staff that will be interfacing with TED and the generated reports.
• Ongoing support includes refresher emails with tips and tricks on the use of TED’s many functions and support via phone and email with any questions users may have.

During SIG2 several sites also elected to participate in an advanced training with the SIG Evaluator that looked at the training data they had collected in conjunction with student test scores. The intent of this advanced training was to determine how the site might tailor future professional development training to support the areas in which they excelled and to intensify training in areas of deficit.

TED can provide essential training provision information and assist sites in answering a number of questions that might help them more effectively manage their professional development and training program. For example:

• How many training events have been provided in each core message area?
• How many participants have received training in each core message area?
• Who is attending these events by role and school site?
• To what degree do participants indicate that their knowledge has increased due to the information provided at the event?
• Advance users of TED can also use the follow-up survey function and easily find out whether participants of trainings have implemented and shared the information learned and whether they found the information to be useful.

Users of TED who piloted the software during SIG2 described some of the benefits of using TED. Comments include the following:

“One of the most important features of TED (for us) is the ability to run reports for our Human Resources Department to show what trainings a particular employee has had. This is also useful in due process hearings to show what trainings a particular employee has had to demonstrate their qualifications for a particular position. Also, when participants call in asking for proof of their attendance (for CEU purposes), we are able to locate the events they attended easily.”

“I have worked with TED for two years. The program has been a great tool. As with any new program, there have been growing pains but [the SIG2 evaluation team] has always stepped right in and given immediate assistance. I am amazed at how many of our requests and needs have been implemented into the program since I first started using this program.”

“We love TED. We use it for all of our staff development workshops. It really makes tracking things over multiple years easy.”

“We provide copies of event summary reports to our special education directors to review. We also provide them with copies of the evaluation summaries and have found this a really helpful way to provide them with feedback on events.”

A demo version of TED which does not require Filemaker Pro to operate was developed and used as a marketing tool. Sites received this tool as a CD-ROM or by downloading it from the CalSTAT website, using it to determine if they wanted to purchase Filemaker Pro and pursue installation of the full version.

This demo version is available to download at http://www.calstat.org/ted.html.

TED Implementation During SIG2

During SIG2 the installation and ongoing support of TED was made available to SELPAs and LEAs for free. These sites became part of a pilot project, during which they used TED at their site and provided regular feedback on the usefulness of the available functions and made suggestions on additional functions that might be added to meet their tracking and evaluating needs. The objective was to include 20 diverse sites in the pilot project. During the course of SIG2, 19 sites elected to have TED installed and their staff trained in its use. Of these 19 sites, 15 have been consistent users and have provided feedback and suggestions that have contributed to the further development and refinement of the TED program. The 15 actively participating sites include 6 SELPAs, 6 districts, 1 COE, and 2 partnering organizations from northern, central, and southern California.

When asked to describe their experience, 12 sites responded and rated the TED program highly on a 5-point scale. Respondents were asked whether TED was user friendly (4.7 average rating), saved their staff time in tracking training events (4.5 average rating), and whether the reports provided useful information (4.5 average rating). Overall, respondents indicated that TED was effective at tracking all components of the professional development their site provided and made preparing for events easier. Of the functions provided in TED, each site indicating using a different assortment and thereby customizing their use to meet their need.

The TED program has also assisted CalSTAT in tracking SIG2-funded events. Currently the TED program holds information on 570 training events, 8,257 unique participants, and 12,136 training evaluations. A number of the evaluation data-points presented throughout this report were generated by the automated reporting function in TED. The capacity of this database to hold this level of information has allowed CalSTAT to track the large number of training events it has supported over the course of SIG2 and to easily summarize the information by multiple criteria.

Looking to the Future

With the close of SIG2, TED is a user-friendly training and evaluation tracking database that will continue to be made available to sites in the state of California to support their efforts in making data-informed decisions regarding the professional development they offer in their local area.

In the future, the TED program will continue to be a component of the State’s effort to support data-informed decision making. As part of the State Personnel Development Grant (SPDG) beginning in 2008, sites interested in using TED will continue to receive installation and ongoing user support. In addition, the SPDG Evaluation Team will be focused on developing a TED package that will sustain this effort beyond the SPDG. This package will likely include automated installers, installation instructions for possible set-up styles, a user manual, and a movie demonstration.