Building Effective Schools Together (BEST)
Summary Report 2008-09

BEST is a program based on positive behavioral supports (PBS) that helps schools develop and implement positive school rules, rule teaching, and positive reinforcement systems schoolwide. Over two State Improvement Grants (SIG1 and SIG2) from 1999 to 2008, over 450 sites and almost 5,000 people have been trained in PBS through the BEST program. These and ongoing trainings have been administered by CalSTAT on behalf of the California Department of Education, Special Education Division.

Each BEST school site is served by a regional cadre of trainers which is comprised of local program specialists, administrators, parents and others. Personnel from the Institute on Violence and Destructive Behavior (IVDB) at the University of Oregon provide trainings and support to the BEST trainers and sites in California. These trainers then provide follow-up training and ongoing support for school sites in implementing BEST.

Starting in the 2008-09 school year, BEST focused on Los Angeles Unified School District (LAUSD) only, initiating a new model of training and support delivery to sites through a three-year, three-phase plan. In the 2008-09 school year, 30 sites participated: fifteen in Phase One and fifteen in Phase Two. Phase Three begins in 2009-10.

BEST: Four Elements in Three Phases

BEST delivers four elements of training and support to sites implementing PBS. The four elements are detailed as follows:

• Content Training: A comprehensive two-day training with PBS specialists from the Institute on Violence and Destructive Behavior (IVDB) at the University of Oregon and from the regional cadre of BEST trainers.
• Booster Training: A one-day training with IVDB specialists and a regional cadre of BEST trainers, focusing on delivering tools and support for school site teams in the process of implementing a schoolwide PBS program.
• Ongoing Coaching and Support: IVDB specialists and a regional cadre of BEST trainers monitor and advise participating sites throughout the year.
• Data Collection and Monitoring: Sites record and submit a variety of measures, including behavior incidence data and a team implementation checklist.

Each year of participation moves a BEST site through one phase of the program, detailed as follows:

• Phase One: New sites are identified and given initial Content Trainings in the spring. New sites participate in data collection, and the Phase One school year is considered a baseline for implementation of PBS, behavior incidence data, and academic outcomes.
• Phase Two: Sites who received a Content Training the previous spring begin implementing schoolwide PBS in the new school year. These sites will receive ongoing coaching and support throughout the year and receive a Booster Training in fall/early winter. Data is collected during Phase Two which can be compared to the Phase One baseline.
• Phase Three: These sites continue to receive ongoing coaching and support, but no additional trainings, as they continue to advance implementation of their now-mature schoolwide PBS program. These sites continue to collect and report data throughout the remainder of the Phase Three school year.
• Exit from BEST: After three years of participation in BEST, each school site is expected to be fully implementing a sustainable schoolwide PBS program. These sites are graduated from BEST after the third year, allowing resources to be shifted to new sites.

Content and Booster Trainings in the 2008-09 School Year

In the 2008-09 school year, participants from 30 schools in Los Angeles Unified School District (LAUSD) participated in a Phase One Content Training and Phase Two Booster Training. Participants were invited by the LAUSD regional cadre of trainers based on readiness to begin implementation of PBS.

Participants at both the Content and Booster Trainings represented a similar diversity of education providers and stakeholders:

• Teachers formed the largest group of participants, with 41% spanning both general and special education.
• Administrator involvement was high, with 30% of participants from both general and special education.

Participants also rated the quality of the training on end-of-event evaluations. Both trainings were rated similarly and highly, with participants reporting substantial increases in knowledge.

PBS Team Implementation Checklist (TIC)

The TIC is an 18 item checklist (items summarized on page 5) allowing sites to monitor implementation of explicit PBS program criteria. It was developed and is under copyright by George Sugai, Rob Horner, and Teri Lewis-Palmer. Responses to each item are given as “achieved,” “in progress” or “not started,” and interpreted as 100%, 50% and 0% implementation, respectively. The average of all 18 items within each site has been explored as an important measure of BEST implementation at the site level.

• An aggregated average of 80% or more is considered to be “fully implementing.”
• An average of at least 50% but less than 80% is considered to be “partially implementing.” Most sites are in this category.
• Below 50%, sites are considered to be “minimally implementing.”

BEST sites complete the TIC in Spring of every year. Phase Two sites have TIC responses from the previous year available for comparison (when they were Phase One sites), while Phase One sites do not.

• TIC responses indicate that a substantial portion of BEST sites are fully implementing PBS (33% and 40% of Phases One and Two respectively).
• There are few differences, overall, between Phase One and Phase Two sites in TIC responses, and few between Phase Two’s 2008 and 2009 responses. This suggests that, while checklist items may continue to be adapted and institutionalized, Phase Two implementation is largely based on elements established in Phase One.

Average responses to individual PBS TIC items are summarized on the following page. Overall, sites appear very successful in establishing key elements of PBS, but implementation of underlying support elements is not yet in place.

• Establishing a site team and receiving administrator support were fully in place at most BEST sites.
• Positive behavior supports, such as articulating behavior expectations and systems that reward students, were also in place at most sites.
• Underlying support for schoolwide PBS was not as advanced, particularly around use of data and establishing systems for teacher support.

Eight Key Features of Implementation

In addition to the Team Implementation Checklist, Los Angeles Unified has adapted a Rubric of Implementation for coaches and trainers to use in evaluating degree of implementation of schoolwide PBS (SWPBS) during site visits. The rubric identifies eight key features of SWPBS and defines four levels of implementation for each feature, described below and on page 7. Only Phase Two sites were assessed with the rubric.

• Administrative Leadership and Support, Behavior Expectations Defined, and Behavior Expectations Taught show particularly advanced implementation overall.
• Features with lagging implementation Parent and Community Collaboration, Team Based Implementation, and Data Based Decision Making.

Rubric Response Key

• Four (full implementation)
• Three
• Two
• One (minimal implementation)

Administrative Leadership and Support

Level 3: At least one school administrator is a member of the SWPBS team. SWPBS is on the agenda at some faculty meetings. SWPBS is addressed in some staff and parent newsletters.

At level 4, SWPBS was on the agenda at all faculty meetings and addressed in all staff and parent newsletters.

Team Based Implementation

Level 2: The SWPBS team is established and meets at least two times per school year.

At level 3, the SWPBS team is representative of all stakeholders (including parents and students) and has monthly meetings. At level 4, sites clearly established the role and responsibility of each team member and memorialize meeting agendas, minutes, etc.

Behavior Expectations Defined

Level 4: 3-6 positively stated expectations are established, defined and clearly visible (posted) in all of the common areas. When asked, students, staff and families know the 3-6 expectations.

At level 3, expectations are defined for all common areas, but only posted in most of them. At level 2, expectations are only established for some common areas.

Behavior Expectations Taught

Level 3: There is a documented system for annually teaching the behavioral expectations to all students.

At level 4, a review of expectations was conducted on a weekly to monthly basis, and strategies were developed to involve families and community in teaching expectations.

Acknowledge and Reinforce Appropriate Behavior

Level 3: The documented acknowledgment/reinforcement system (ticket, token, etc) guidelines and procedures are implemented throughout the school by 75% of staff.

At level 4, acknowledgment/reinforcement implementation was 90% schoolwide and established a ratio of four positives to one negative acknowledgment for students, staff, and families. At level 2, acknowledgment (but not reinforcement) was in place, but with only 50% of staff. At level 1, sites did not have a consistent acknowledgment/reinforcement system in place.

Monitor and Correct Behavioral Errors

Level 3: Problem behaviors are clearly defined, agreed upon by at least 75% of school staff and documented. At least 75% of school staff can clearly articulate the process for behavior handled in the classroom and behavior referred out (to dean, counselor, etc).

At level 4, these thresholds were 90% and evidence showed that consequences for “behavioral errors” are consistent, progressive, and communicated to all stakeholders. At level 2, only 50% of staff agreed on problem behaviors and an inconsistent process was in place. At level 1, problem behaviors were not yet clearly defined and response was inconsistent.

Data Based Decision Making

Level 3: A system is used to keep track of discipline data. The SWPBS team uses data to make decisions in designing, implementing, and revising schoolwide efforts at least two times per school year. Data is shared with school staff at least two times per year.

At level 4, the frequency of using and sharing data schoolwide was at least three times per year, and data was reviewed at monthly team meetings. At level 2, data collection was underway but was not yet a part of the decision-making process.

Family and Community Collaboration

Level 2: A family/community member is inconsistently part of the SWPBS team. Updates on SWPBS efforts are communicated through newsletters, brochures, open house, parent meetings (etc) at least twice per school year.

At level 3, at least one family/community member was an active member of the SWPBS team and SWPBS efforts were communicated at least five times per year. At level 4, SWPBS efforts were communicated at least eight times per year. At level 1, there was no family/community involvement in the SWPBS system.

Suspensions at BEST Sites

As a response to a serious “behavior error,” suspension events represent one way of quantifying behavior outcomes at a school site. Suspension removes a student from the classroom for one or more days, creating a barrier to learning. It also represents a serious disruption of the learning environment, with potential impacts for other students. Reduction of the behaviors which prompt suspension, and eventually reduction of suspensions themselves, are a goal of BEST and schoolwide PBS.

However, suspensions data is an imperfect measurement tool. Please note:

• Schools have considerably leeway and independence to define consequences for specific behaviors, so the same behavior visible through a suspension at one site would be invisible in this metric at a site which doesn’t respond with suspension.
• As schools begin explicitly defining and monitoring behaviors and criteria in implementing SWPBS, an increase in suspension rates could signal improvement, with negative behaviors now being recognized and responded to consistently.

• The reason for most suspensions was “caused, attempted to cause, or threatened to cause physical injury to another person,” with 51% of all suspensions.
• Suspensions for students with disabilities were slightly more likely to be for this reason (59% to 51%), but were not particularly different overall.

Suspension data demonstrates that, overall, special education students are suspended at a higher rate than students overall, representing 26% of suspensions at BEST sites, compared to 11% of districtwide enrollment.

• Most students with disabilities receiving suspensions were in a Resource Specialist Program (RSP, 41%) or Specific Learning Disability (SLD, 30%) program.
• Of all students with disabilities suspended at BEST sites, 8% had Emotionally Disturbed (ED) IEPs, compared with ED enrollment at 4% of all disabilities districtwide.

Among the 30 sites implementing BEST, there was considerable diversity in suspensions, which does not appear related to phase or degree of implementation. This is further shown in the specific suspension data for the fifteen Phase Two sites who recorded baseline data in the previous year, which is summarized beginning on page 10.

Suspension of students with disabilities showed great variability, as well, ranging from no suspensions at some of the sites with higher suspension rates to being a substantial portion of suspensions at sites with lower suspension rates.

In addition to teaching and reinforcing positive behaviors, schools implementing PBS deliver additional support to at risk students, intending to avert additional disruptive incidents. Of the 338 students who were suspended at BEST sites in the 2008-09 school year, 83% did not receive a second suspension.

BEST Sites in Phase Two

While baseline suspension data for Phase One sites was collected over their first year, baselines for Phase Two sites were previously collected and can now be compared to suspensions data from the 2008-09 school year. Additionally, because Phase Two sites are establishing schoolwide PBS programs, they were evaluated with the Rubric of Implementation, allowing suspension data to be cross-referenced with implementation.

Remarkable sites are highlighted and described on page 11.

BEST Sites in Phase Two During the 2008-09 School Year

Site 2 was found to have the most-advanced schoolwide PBS program, ranking level 4 on all rubric features except Team Based Implementation. This site also received the single largest suspension event increase, as well as increases in all of the other suspension statistics described on page 10.

In addition to Site 2, most of the sites with the most advanced implementation had increases in the number of suspension events, many of them substantial. This may be due to the closer attention being paid to unacceptable behavior and application of consistent consequences. It will be interesting to see if these increases prompt a change in how schoolwide PBS is implemented at these sites, such as greater attention to working with at risk students, and whether that may lead to corresponding changes in suspension rates in Phase Three.

There was a notable increase in suspension events for students with disabilities at a majority of the Phase Two BEST sites. Like increases in suspension events for all students, these increases were concentrated among sites implementing schoolwide PBS at a more advanced level.

BEST trainers describe administrative support as being a critical feature in connecting schoolwide PBS implementation with positive student outcomes. The Rubric of Implementation describes a high level of administrative involvement at all BEST sites (participation in SWPBS team, etc). However, the quality of administrative support is not documented; some of the sites with the least positive student oucomes saw midyear changes in principals and a lack of administrator expertise with PBS. BEST trainers are continuing to stress the need for administrators to participate in PBS trainings, engaging 44 administrators from 30 sites at 2008-09 trainings.

Change in Suspensions Data Overall

Changes in suspensions data overall were primarily driven by large shifts at a handful of sites. It should be noted, again, that suspensions data is an imprecise and indirect measure of student behavior outcomes. Additionally, with only fifteen sites, these statistics are particularly sensitive to factors which may have little or nothing to do with overall program outcomes.

In aggregate, while enrollment decreased slightly, the total number of suspension events and students receiving multiple suspensions edged up. Increases were larger, proportionally, for students with disabilities.

A handful of sites are responsible for many of these aggregated changes, in particular:

• Suspension events were driven by Sites 2 and 7.
• Events for students with disabilities were more global, led by Sites 1 and 8.

CST ELA and BEST Sites in Phase Two

Recognizing that many factors influence student academic outcomes at the sites, it is possible that positive changes in the school environment due to BEST/PBS may have contributed to improvements in student academic performance. An examination of the percent of students scoring proficient and above in California Standards Test (CST) English Language Arts (ELA) may suggest academic outcomes.

The chart below averages the percent proficient and above at all sites in Phase Two during the 2008-09 school year (starting BEST during the 2007-08 school year). The red dotted line indicates the AMO benchmarks which are used in AYP criteria. While BEST sites have historically trailed statewide scores, recently gains have been made.

BEST sites started a bit behind statewide ELA proficiency, but did see improvement in 2008-09, the first full year of schoolwide PBS implementation.

• After gaining only 1.3 percentage points in 2007-08, BEST sites gained 4.6 percentage points during Phase Two’s 2008-09 school year.
• A slight decline of 0.6 percentage points in 2007-08 was nearly reversed with a 0.5 percentage point gain in 2008-09 among students with disabilities.