CalSTAT Technical Assistance and Training

(g) Quality of the project evaluation

g.1.  Thorough, feasible, appropriate methods to evaluate goals, outcomes, and objectives.

The following plan details the SPDG evaluation management structure and methods that will produce quantitative and qualitative data, provide performance feedback and periodic assessment of progress toward achieving the intended outcomes, and ultimately enable examination of the effectiveness of project implementation strategies. California’s SPDG Logic Model can be found in the Appendix.

Evaluation management structure: The existing network of evaluation personnel, resources, and the Evaluation Task Force, which has conducted and guided the evaluation for SIG and SIG2, will continue in the SPDG.

(1) SPDG Evaluation Task Force:To ensure the ongoing involvement of stakeholders in monitoring progress toward the objectives and outcomes, an Evaluation Task Force meets for a full day twice a year. Comprised of a broad range of ISES stakeholders, this 15-member group provides guidance and feedback on instruments and evaluation methods, reviews formative and summative data and progress reports, and makes recommendations based on the findings, which are presented to the CDE and ISES twice annually. (A Task Force roster is in the Appendix.)

Improving Special Education Services (ISES). ISES is a broadly representative stakeholder group of approximately 100 participants which meets twice annually to monitor progress and guide the implementation of both the SPP and the SPDG. SPDG progress reports, along with Evaluation Task Force recommendations, are presented to ISES for consideration and further refinement or inclusion as recommendations to CDE related to the implementation of the SPDG. All SPDG Evaluation Task Force members are also members of ISES.  

            SPDG participants and the public. SPDG evaluation reports will also be shared widely with personnel participating in SPDG activities, at annual State Institutes, and ERIA and BEST trainings and are posted on the SPDG website.

SPDG lead evaluator.The SPDG evaluation effort is being directed by Cheryl "Li" Walter, Ph.D., who has been the SIG Evaluator for over seven years. Her role is to gather information and draft evaluation plans; design program evaluations for the activities of the grant subcontractors; develop evaluation instruments and methods; draft progress and summative reports; facilitate meetings of the SPDG Evaluation Task Force; present to ISES; develop data tools; and teach stakeholders to use these tools in the process of making data-informed decisions. She is an active participant in the OSEP SPDG Evaluator’s Planning Workgroup.

The SPDG Evaluator will work closely with Janet Canning, CDE Designated SPDG Director and SPDG Project Manager Anne Davin, Ph.D. who is responsible for day-to-day program operations and implementation. Two full-time internal SPDG evaluation staff will manage the gathering and entry of data and generate SPDG activity reports; their work is supervised by the SPDG Evaluator. This structure will effectively address the SPDG intention to build the capacity of the system to generate and use data to inform decision-making processes.

Evaluation Methods:  The evaluation will provide thorough, appropriate formative and summative assessments, the data reports from which will be shared widely with SPDG partners and other stakeholders.

Formative evaluation: The formative evaluation will target the following continuous improvement issues: What is the number, nature, and quality of project activities actually implemented to date? What aspects need improvement and/or change? What problems are anticipated in implementing the next phase, and how will these be overcome? 

Summative evaluation. The summative evaluation will use the performance measures to establish the degree to which the objectives and outcomes have been met, including the quality of process and products, the degree of program implementation with fidelity, and the impact of SPDG activities on the intended audiences. Summative evaluations will be conducted annually.

The table below outlines the activities and the instruments, data sources, and methods that will be used to evaluate the performance measures as they are detailed in section c.1, pp. 40-43. The performance measures were constructed to address each of Guskey’s five levels of professional development evaluation (Gusky, 2000). Following the table is a narrative description of the methods of evaluation being employed for each of the SPDG activities, including the key evaluation questions and focus, instruments and data sources, data collection timelines, and how data will be shared with program implementers.    

Approval for Use with Human Subjects. Most of the proposed evaluation activities have been approved for use with human subjects; new activities will receive approval prior to being implemented. Reapplication will take place as necessary to maintain up-to-date approvals.     

            1) Evaluation Methods for: All Training and Technical Assistance eventsincluding those provided as part of ERIA, BEST, Regional Institutes, Leadership Site TA, and Centralized TA. The primary evaluation questions include: Who is receiving training and technical assistance? Was the training and TA identified as being of high quality and useful by participants? Do participants report having increased their knowledge/skill in the content of the training? Do participants report having implemented what they learned? These questions will be answered using the following instruments, data sources, and methods:

1.1 Training Evaluation Database (TED). Over the course of SIG and SIG2, California has developed a Filemaker Pro-based Training Evaluation Database (TED). Information on each training and technical assistance (T/TA) event provided is entered into TED, which enables tracking of the Core Message scientific- or evidence-based instructional and behavioral practices targeted in the training (in most cases, events must focus on these practices to meet Core Message criteria for SPDG funding approval), as well as who is providing and attending the trainings. For each event, individual and team sign-in sheet information, including names, roles, schools/districts, and email addresses, is entered into TED. TED also has event evaluation entry, follow-up survey, and automated reporting capabilities.

1.2 End-of-Event Evaluation: How Was the Training? Anonymous end-of-event evaluations are collected from participants. (A sample of this tool is located in the Appendix.) The response rate is 77%; completed participant evaluations are required to be submitted along with presenter invoices to receive payment. Using a 5-point Likert scale from “Low” to “High,” participants are asked to rate their level of knowledge/skill in the training focus area prior to the event and after, assign an overall rating to the event, and provide qualitative feedback on how the training could be improved. The rating difference between prior and current knowledge/skill divided by prior generates the percent change, e.g., a 1 point increase, from a rating of 3 to 4 represents a 33% increase. Once entered into TED, the database is capable of generating automated reports by event, trainer, core message area, and time period. (A sample TED Event Summary is in the Appendix.) Over 50,000 training and TA participants have completed How Was the Training? thus establishing expected patterns of reported knowledge/skill gains (1.0 point) and overall event ratings (4.5 rating). Throughout the year, program staff monitors the quality ratings and written comments for events and specific trainers by comparing the targeted data to these norms. Reports on each event are emailed to event organizers and presenters, and the data is used by the Evaluation Task Force to generate recommendations for continuous improvement.

1.3 Follow-up Evaluations. Follow-up evaluation surveys are conducted in the form of emails with links to a brief web survey; these are sent to participants approximately 3 months after each training to determine whether they report having implemented what they learned at the training. Participants who do not respond receive up to two reminder emails. The response rate on valid email addresses is approximately 35%. Items are rated using a 5 point Likert scale from “Not At All” to “Many Times.” Qualitative data on what worked well and why, and what barriers they might have encountered implementing their learning in practice are also elicited. (A sample follow-up survey is in the Appendix.)  Follow-up evaluations have proved particularly effective in providing an understanding of barriers to implementation and enabling adjustment of the content of future trainings to address those barriers. Completed web survey data are automatically uploaded into TED, which has the capacity to generate automated reports (responses are reported only in the aggregate), that are then distributed and used in the same ways as the end-of-event evaluations in the continuous feedback and improvement cycle.

            1.4 Dissemination of TED. During SIG2, TED was re-programmed and distributed for adoption by school districts, County Offices of Education, SELPAs, and other SPDG partners, enabling immediate use of evaluation data for data-informed decision-making at the local level. During the SPDG, TED will continue to be updated to add functional capabilities and report capacity requested by local level users. TED installation and training will be available to approximately 7 new sites per year. Follow-up training in how to use the advanced data reporting features will be provided to all new and continuing TED users upon request. 

2. Evaluation Methods for: ERIA and BEST.  While the content focuses of these two major SPDG activities are different (literacy and positive behavioral supports), the manner in which they are being delivered and evaluated is quite similar. In addition to the training and TA event evaluations of quality, initial knowledge/skill, and implementation described above, the primary evaluation questions for ERIA and BEST include: Are the sustaining activities offered being accessed by personnel and school sites? Are school sites and personnel implementing the key elements of instructional and behavioral practices with fidelity? Are the school sites which are implementing with fidelity seeing positive changes in student outcomes? Are localized concentrations of sites in districts/counties scaling up implementation of these practices? These questions will be answered using the following methods:

            2.1 Review of records. Tracking of the utilization of sustaining activities will occur through examining records from three sources: TED, Coach’s reports, and web tracking statistics. These sources include data on the participation of personnel and site teams in the coaching, training for site team leaders, booster trainings, Leadership Site TA, data tools, teleconferences, and the online community activities available to sites on an ongoing basis following completion of their initial ERIA or BEST training. For the purposes of the performance measure, “accessing supporting or sustaining activities” means participating in three or more activities by the end of their second year in the program, and annually thereafter.

2.2 Team Implementation Checklist (TIC). Site teams from both ERIA and BEST will complete periodic self-assessments of degree of implementation using a Team Implementation Checklist (TIC), beginning at their first training session and continuing 2-3 times per year until they have achieved implementation with fidelity, and once per year thereafter. Site teams, which include teachers and administrative staff, complete the checklist together, either at a team meeting on-site or at a training event. The TIC provides a 3 point rating scale (Not Started=0, In Progress=1, and Achieved=2) of degree of implementation for the key elements of the practices, also serving to keep teams focused on the nature of those key elements.

The BEST/PBS TIC consists of 23 items in 7 areas for self-assessment of site level implementation, which provides an overall implementation score, as well as subscale scores in each of the key areas. The criterion for “implementation with fidelity” is an overall score of 80% plus 80% on the “expectations taught” item. It is expected that the majority of sites will take two years to achieve implementation with fidelity. Examples of items include: whether behavioral expectations have been defined and taught, whether rewards and consequences are in place, and whether data is gathered and used to monitor progress. The PBS/TIC is available through the work of the OSEP TA Center on Positive Behavioral Interventions and Supports (Sugai, Horner & Lewis-Palmer, 2001) and IVDB. (A sample TIC is in the Appendix.) For data entry and reporting, www.pbssurveys.org will be used. This web tool (developed and supported by the OSEP Center for PBIS as well) will enable sites to enter their TIC online and then see the results charted immediately. BEST cadre coaches will have passwords to access the reports on their schools sites, and thus use TIC results in the coaching process. SPDG evaluators will subscribe to www.pbseval.org which gives access to summary data on every school site. For each of their first two years of development, sites will also be encouraged to conduct staff surveys using a more extensive 50+ item Self-Assessment Survey, (Sugai, Horner & Todd, 2000), which reports what items are in place and which are priorities. This survey can be conducted and/or entered and charted on www.pbssurvey.org. (A sample Self-Assessment Survey is in the Appendix.)

The TIC for ERIA is under development and is being designed mirroring the PBS/TIC approach of rating items within key elements of practice implementation, drawing upon the Sopris West training manual used in ERIA, and in collaboration with ERIA Trainers and Coordinating Coaches and the Cohort 1 ERIA sites. Examples of items to be included are whether sites are conducting specific literacy skill assessments in decoding and fluency with struggling readers, whether students are placed into specific skill intervention programs based upon those assessments, whether reading intervention programs are being implemented with fidelity, and whether data on response to interventions is being used to determine next steps. Hard copies of completed ERIA/TICs will be submitted by sites to their coaches who will provide copies to SPDG evaluators. Eventually, a web-based tool will be developed.

2.3 Schoolwide Evaluation Tool (SET).  Fidelity of implementation will also be assessed by external coaches and SPDG evaluators using a Schoolwide Evaluation Tool (SET). In conjunction with site team self-assessment ratings through the TIC, SET provides for a multi-dimensional assessment which is facilitated by the SET items and scoring scheme paralleling those of the TIC. The SET is completed on-site through review of school data and documents, observation, and interviews. SETs will be completed annually for at least 20% of sites that finish the initial year of training and proceed to begin implementation. The correlation of externally conducted SET scores and internally generated TIC scores will be examined.

The PBIS/SET is an established research-validated instrument (Sugai, Lewis-Palmer, Todd, & Horner, 2001). (A sample SET is in the Appendix.) BEST SETs will be conducted by the sites’ cadre coaches and/or SPDG evaluators. As with scores from the TIC, the SET data can be entered into the www.pbssurvey.org web tool for charting and sharing of results with sites. SPDG evaluators gain access to the results through www.pbseval.org.

The SET for ERIA is under development and is being designed to mirror the BEST SET in terms of methodology and alignment with the TIC, while building in the elements unique to the literacy focus of ERIA. One important difference in the ERIA SET is that it will involve school principal and/or external coach observation of classroom teacher fidelity of implementation of the literacy interventions and pedagogy.

2.4 Student Outcomes Progress Monitoring.  Progress monitoring of ERIA site student English language arts (ELA) proficiency outcomes and BEST site office discipline referral and suspension/expulsion outcomes will be conducted by sites, coaches and the SPDG Evaluators. Outcomes data will be used as a formative assessment vehicle for feedback on progress and quality and will serve to continually focus personnel on the goal of improving student outcomes. For sites that have implemented with fidelity, outcomes data will also be used as a summative assessment of expected changed in performance measures.

The CST ELA will be a key measure for ERIA literacy outcomes monitoring for formative and summative assessments. Students are tested each spring, with test scores available the following fall. For the purposes of meeting the AMO benchmark of percent of students proficient, the critical factor in determining whether a school has made Annual Yearly Progress (AYP) under NCLB, students are considered “proficient” if they score in the Proficient or Advanced categories. The focus for ERIA sites is on narrowing the achievement gap between the ELA proficiency of students with disabilities and ELA proficiency for all students, while increasing the proficiency of the entire school population. An important additional progress monitoring data point with be tracking of potential decreases in the percent of students with disabilities scoring in the categories below Basic. For performance measurement, comparisons will be made between average CST scores from each school’s baseline year to scores from their second year of having fully implemented ERIA with fidelity. Annual performance measurement will take place in subsequent years to determine if their performance continues to improve.

CST ELA data are available from the CDE website in tables and research files. In addition, each school receives from the test contractors site level summary data and detailed data on each student, including cluster scores of reading comprehension and vocabulary items, which are used by sites as part of the ERIA individual student specific reading skill assessment process.

2.5 A CST Charting Program was designed by the SPDG Evaluation Team during SIG2 to enable school sites to create a single-page graphical chart in Excel showing the percent of all students compared with students with disabilities as a subgroup performing at the five levels of proficiency. While using the same numbers that parents, teachers, and administrators have been seeing for years, the program exhibits data with colors and proportions that makes it simple to grasp student progress with a glance and is particularly revealing of the achievement gap. A copy of the type of chart produced by the program is located in the Appendix.  ERIA sites (and all other SPDG sites, along with the CDE consultants) will continue to receive copies of the charting program, instructional movies on CD-Rom, and training in its use at statewide meetings.

Currently under development is a Site Academic Measures Metafile (SAMM), which is an extremely large database in which the SPDG Evaluation Team is compiling longitudinal CST, AYP and other demographic and academic performance data from a wide range of CDE databases and research files for all schools and districts in the state. This data is being integrated with TA utilization data from the SPDG. Pilot use of SAMM has enabled a preliminary look at the academic outcomes of sites that have accessed 3 or more days of Literacy or Collaboration TA as reported elsewhere in this application. The goal is to move toward a programmed format in which a several-page longitudinal data profile summary for a site could be produced by entering that site’s unique identifier code, as well as being able to do further outcomes-based research on the sites served through the SPDG, particularly with ERIA sites.

There is also a data tool being used by each ERIA site for tracking individual student assessment, placement, and reading intervention data, which facilitates site tracking of response to intervention and working closely with their coordinating coach in examining site-level data. (A sample ERIA individual student tracking tool is in the Appendix.) The possibility of setting up access to a web-based tool for tracking those data is being explored.

The BEST sites will be tracking Behavioral Incidence Data (BID), specifically office discipline referrals (ODR), suspensions, and expulsions. Each site develops its own tracking system or uses the approach mandated by their district. A few use the School-Wide Information System (SWIS), a web-based information system designed by the University of Oregon to help school personnel to use office referral data to design school-wide and individual student interventions. A summary BID form is submitted by each site to their cadre coach annually, and the data are forwarded to the SPDG Evaluators. (A summary BID form is in the Appendix.) For performance measurement, comparisons will be made between BID from each school’s baseline year and their second year of having fully implemented BEST with fidelity, with an expected decrease of 20% in suspensions/expulsions.

Supplemental information summarizing BID for the students with disabilities at each site will be obtained from the CDE California Special Education Management Information System (CASEMIS). In addition, the SPDG Evaluators will examine BEST site ELA data using SAMM to ascertain whether academic outcomes correlate with changes in behavioral outcomes at sites implementing BEST with fidelity. Again, all data is analyzed for use in continuous improvement.

ERIA and BEST outcomes data will be fed back to sites and reflected upon with sites at annual booster training and site facilitator meetings and teleconferences. Additionally, in consultation with external or cadre coaches, the data will be incorporated into a sustaining activity that assists sites in focusing on the progress they’re making in changing student outcomes as they implement scientific- or evidence-based instructional/behavioral practices. Summary reports for ERIA and BEST will be shared with the Evaluation Task Force, ISES, at Institute meetings, and through the CalSTAT website.

2.6 Evaluation of Coaches.  An online evaluation will be conducted annually to elicit feedback from the ERIA and BEST site cohorts on the work with their coaches to assess what’s working well, additional needs, what could be improved, and how it could be improved. SPDG evaluators will provide each coach with summary feedback to promote continuous improvement.

3. Evaluation Methods for the Leadership Community.  The SPDG training event evaluation process described earlier in this evaluation plan applies to all of the Regional Institutes and follow-up TA as well as Leadership Site TA, much of which supports and sustains ERIA and BEST practices. The Leadership Community itself is comprised of Leadership and Regional Host site team members who 1) participate in the annual State Leadership Institute, and 2) provide TA to one another and others to support systems change processes. For this group, the evaluation focus is quite different than in other activity areas. Here, the focus is on the development of a “knowledge strategy” for the professional development learning community (Wenger, 2004). The primary evaluation questions are: Are learning community members involved in translating their learning into useful practices for the community? What knowledge has the community produced in terms of lessons learned and best practices? Has that learning been “harvested” by the community and embodied in tools and other documents that enable it to share that knowledge within the community and with others? These questions will be explored using the following methods:

3.1 Development of a knowledge strategy and products.  During SIG2, a new development in the knowledge strategy of the Leadership Community of practice began to emerge. Case studies of the seventeen Leadership Sites were conducted, focusing on two areas. Interviews with each of the site teams focused on what and how systems change had occurred at their sites, with particular attention to special and general education collaboration. In another round, telephone interviews were conducted with a key informant from each site focused on what and how shifts in the use of resources occurred to make systems change possible.

The products of those interviews were individual site profiles which are now linked to the web pages for each Leadership Site. In this way, prospective TA recipients can get a sense of whether the challenges and changes experienced by that Leadership Site would be a good fit for the school/district seeking TA support for their own change process. In addition, cross site analyses were done for each set of data, articulating key commonalities and variations, and identifying critical factors and lessons learned. (A sample site profile and a report entitled, “They Are All Our Kids” based on the cross site analysesare located in the Appendix.) These profiles and reports were then fed back to the community online and face-to-face at the February 2007 State Leadership Institute. The discussion then centered on SPDG planning and “next steps,” meaning: 1) how these findings could be translated into a series of tools and documents, 2) how those tools and documents could then be refined, piloted, and eventually used and disseminated by the Learning Community in their ongoing transformations and 3) how the data could be used in their work supporting and sustaining other sites in making the systems change needed to successfully implement scientific or evidence-based practices. Potential tools and documents discussed included: a Readiness for Change Self-Assessment, a School Climate Survey, and a Collaborative Systems Change Self-Assessment. During SPDG, success in advancing the development of the Leadership Community knowledge strategy will be measured by the development, dissemination, and use of 2 or more new products that emerge from the knowledge base of the community. The tool and document development and refinement process will be documented; dissemination through hard copies and web downloads will be tracked; and use will be evaluated through user feedback forms and Institute roundtable discussions.

3.2 SAMM. The Site Academic Measures Metafile provides Leadership Sites and the Evaluator with longitudinal data and data profiles for measuring school site progress in student achievement. SAMM site profiles will be used by reviewers in the Leadership Site new and continuation application processes. In addition, summary data for each of the Leadership Sites will be presented in a single document at the State Leadership Institute so sites can assess their areas of weakness and arrange to receive TA from a site that is strong in that area.

4. Evaluation Methods for Parent/Family Involvement.  The SPDG will measure progress in facilitating parent/family involvement with schools and the broader educational system. The primary evaluation questions include: Are Family Participation Fund stipends being received by low income, under-represented ethnic minority parents/family members? Are they participating in both special- and general education-focused activities? Has their participation made a difference? Are they participating in the Leadership Community as team members? Do they have a significant role in deciding how the voices of parents are heard? These questions will be answered using the following instruments, data sources, and methods:

4.1 Family Participation Fund (FPF) Invoices/After Event Evaluations. Increases in the participation of low-income parents/family members, in parents attending general and special education decision-making committees and events, and in parents/family members reporting that their participation has made a difference will be measured using the FPF Invoice/After-Event forms. Parents/family members submit these forms following participation in a committee meeting or event in order to receive reimbursement; see sample in the Appendix. The form inquires about 1) income level and ethnicity to enable monitoring for receipt of stipends by the intended population of low-income, under-represented parents/family members; 2) the nature of the work of the committee (SE or GE), and 3) their effect through their presence and contributions (self-rated). Annually, FPF after-event evaluations will be forwarded by the FPF contractor to SPDG for data entry and summarizing into tables and graphs that will be shared with FPF staff and SPDG managers to monitor the use of the funds. These data will be summarized and included in the SPDG annual reports, and discussed at the Evaluation Task Force and ISES meetings.

            4.2 TED Event Registration Forms and Sign-In Sheets. The participation of parents/ family members as site team members at State Leadership Institute events will be measured using Institute team registration forms and sign-in sheets which ask participants to specify their role. This data is entered into TED. As event registrations come in, teams without parent members will be monitored, contacted, and requested to continue to seek a parent member who is able to attend. At the end of each annual event, summary reports will document the percent of State Institute teams with parent members. This data will be shared with the Evaluation Task Force, ISES, and included in the SIG annual report which is posted on the website.

4.3 ISES Task Force development of Parent-School Involvement Facilitation Survey. An ISES Task Force will be formed to assist and advise CDE in the development of data collection and interpretation processes for SPP Indicator #8 Parent Involvement, primarily through adaptation of the parent-school involvement facilitation survey instrument produced by the National Center for Special Education Accountability Monitoring (NCSEAM) for use in California. FPF funds will be used, in part, to support participation in this Task Force. The Task Force roster and agendas will be reviewed, and end-of-event evaluations will be tracked and fed back to facilitators. The products and processed developed will be documented as well.

5. Evaluation Methods for Teacher Recruitment, Preparation, and Retention. The evaluation questions include: Are more potential teachers accessing the www.teachcalifornia.org website? Are potential teachers taking advantage of online registration for tracking and support of their progress through the credentialing process? Is the enrollment of the CSULA Special Education Internship Program increasing? Are graduates of the CSULA program still teaching in special education three years after completing the program? These evaluation questions will be answered using the following data sources and methods:

5.1 Web usage tracking and online registration data. Software that enables website usage tracking will be sourced to examine the annual number of web visits, visitors, repeat visitors, page views, and downloads, as well as referral sources, and most accessed pages. Data from a web site user survey that appears as a pop-up will be analyzed to learn more about the characteristics of web visitors, including: age, gender, ethnicity, current status/interest in teaching, and areas of interest in teaching (special education, math, science, etc.). Analysis of data on the progress of online registrants through the teacher credentialing process will be conducted to better understand who is accessing the site and their progress toward credentialing. Annual increases in web traffic and user registrations will be expected following the first year of outreach. Use of the database by LEAs will be monitored. Data reports will be fed back to www.teachcalifornia.org outreach staff, IHE and Evaluation Task Force members, and ISES.

5.2 CSULA internship program enrollment, Individual Education Enrichment Plans, and graduate follow-up tracking. CSULA Special Education Internship Program enrollment figures will be tracked annually, with expected performance measure increases to begin after the initial year of outreach. Once in the program, interns complete and monitor, in collaboration with their support provider, an Individual Education Enrichment Plan (IEEP) in which they identify a professional goal to be achieved every 10 weeks throughout the school year. Support providers also conduct one formal observation of the intern every 10 weeks and complete an observation form that they share with the intern. Program graduates are contacted annually for three years following the two year program to assess retention in special education employment.

g.2. Evaluation examines the effectiveness of project implementation strategies. 

As shown in g.1 above, the implementation of SPDG strategies will be assessed for effectiveness through the monitoring of quantitative and qualitative data and against the performance measures. Issues around implementation of the activities will be documented by event evaluations, the 3-month follow-up surveys, team checklists, student outcomes analysis, interviews and observations. These will be addressed at the formative evaluation meetings.

g.3. Objective performance measures producing quantitative/qualitative data.  As shown in the narrative in g.1, the performance measures and proposed methods clearly produce both quantitative and qualitative data on performance pertaining directly to project objectives. The analysis plan for the evaluation is yet to be finalized and is dependent on the final structure of the data. In general, descriptive procedures and non-parametric analytical processes will be used, and where appropriate, parametric processes will be used. Quantitative data will be managed and analyzed using Filemaker Pro, Excel, and SPSS software. Qualitative data will be managed and analyzed using Atlas ti software.

g.4. Methods provide performance feedback and periodic assessment of progress.  The SPDG formative evaluation provides performance feedback at the activity level, to summarize data-to-date in terms of progress toward achieving outcomes, and to permit implementation adjustment based on findings twice within the project year with the Evaluation Task Force and ISES. The annual summative evaluation will then report on both project performance in completing the activities and progress toward achieving the intended outcomes. Formative and summative data and reports will be reviewed, with resultant recommendations made by the Evaluation Task Force and ISES.

            As California educators, we continue to search for ways to improve the system, to involve the community, and to better serve the children. In our efforts toward systemic change it is essential that we focus on how those changes affect the individual student, the unique child with disabilities. California’s Superintendent of Public Instruction Jack O’Connell in his State of Education address on February 5, 2007 stated: "Too often, the struggles of the African American student, the English learner, the learning disabled student were hidden by overall school achievement gains. That day is past. Today we are holding ourselves accountable for the results of all children. And when we see significant groups of students falling far short of the goal of proficiency that we hold for all students we must act. Today, equipped with specific knowledge of those gaps, we must focus as never before on solutions."  It is our most sincere desire that, in creating a better system, the SPDG will enable all children to succeed.