Evaluation structure: The evaluation will provide thorough, appropriate formative and summative assessments of service quantity and quality and the impact of these services on the intended audiences as described below.
Formative evaluation: The formative evaluation sessions will be held with the PCSE SIG Evaluation Task Force in March and October of each year. The timing of these sessions was designed to provide findings and recommendations for the Partnership's annual strategic planning session. CIHS handles the scheduling and logistics; attendees include the Task Force members, CDE Designated Project Director, Co-Managers, Evaluator, and other key staff. CIHS, working with the Department of Education, will supply relevant data available to date.
Formative evaluations will target the following continuous improvement issues: What is the number, nature, and quality of project activities actually implemented to date? What aspects need improvement and/or change? What approaches are working very well and might warrant expansion? What problems are anticipated in implementing the next phase, and how will these be overcome? Participants will discuss the available data on process objectives and outcomes, and develop recommendations for improvement that can be implemented immediately. The process thus creates a feed-back loop which repeats and generates continuous improvement.
Summative evaluation: The summative evaluation will establish the degree to which the objectives and outcomes have been met, including the quality of process and products, and the continued significance of the Project's work after the award period. The summative evaluation will be conducted annually and will contribute to the Partnership's annual strategic planning session. The annual summative evaluations will be forwarded to OSEP.
Evaluation management: The existing network of evaluation personnel, resources, and PCSE Evaluation Task Force, which have conducted/guided the evaluation for the initial SIG and supplement, will continue in SIG2.
SIG Evaluation Task Force: To ensure the ongoing involvement of stakeholders in monitoring progress toward the objectives, a SIG Evaluation Task Force meets twice a year. Comprised of a broad range of stakeholders, this 15-member group gives feedback on instruments and evaluation methods as they are developed, reviews data and progress reports, and makes recommendations based on the findings, which are presented to the PCSE annually.
Partnership Committee on Special Education (PCSE). The SIG Evaluation Task Force recommendations are considered, adjusted as needed, and accepted by the PCSE prior to recommendation to CDE Special Education Division. This process takes place annually.
SIG Evaluator. The SIG evaluation effort is being directed by CIHS employee, Cheryl "Li" Walter, Ph.D. Her role is to gather information and draft evaluation plans; develop evaluation instruments and methods, as needed; facilitate meetings of the SIG Evaluation Task Force and its subcommittees; draft progress reports and develop other avenues for presenting the findings; and design program evaluations for the activities of the grant subcontractors.
Additional key personnel: The SIG Evaluator works closely with Janet Canning, State Designated SIG Director from the CDE Special Education Division, and with the SIG Co- Managers. Kelly Riedel acts as the internal evaluation staff, managing the gathering and entry of data and generating the SIG activity reports; her work is supervised by the SIG Evaluator. This structure was established in order to address the SIG goal of building the capacity of the system to generate and use data to inform decision-making processes.
Evaluation design and methodology: Outlined below are the specific outcomes, objectives, and indicators developed for SIG2, along with the evaluation methodologies planned to assess progress toward meeting those outcomes/objectives.
Outcome 1: Improved Quality of Personnel Working with Students with Disabilities
Objective 1. To increase the quality of personnel through the provision of core message, research-based, in-service professional development through technical assistance (TA) and Regional Institutes, as demonstrated by:
Objective 1 Evaluation Methodology: Assessments will be conducted using end-of-event and follow-up evaluation questionnaires with all TA and Regional Institute participants.
1.1. End-of-event evaluation questionnaires. Increases in knowledge and overall ratings of TA events will be measured using an end-of-event questionnaire to be filled out by all participants; see a sample in Appendix C.1. Using a 5-point Likert scale from "Low" to "High," participants are asked to rate their level of useable skills/knowledge of the TA topic prior to the TA and after and assign an overall rating to the TA, among other items. Completed end-of-event evaluations are collected and submitted by TA providers along with the event invoice and a cover sheet specifying the core message areas addressed in the TA. Over the past four years, 72% of TA participants have completed evaluations.
Information on the core message addressed in each TA session and the data from the completed evaluations are entered into the SIG's searchable database. The database is capable of generating automated reports by event, trainer, core message area, and time period. Nearly 40,000 TA participants have completed end-of-event evaluation forms, thus establishing expected patterns of skill/knowledge gains and overall event ratings. Throughout the year, program staff can monitor the quality ratings for individual events and trainers, the TA being offered in specific core message areas, and the TA provided in given timeframes by comparing the targeted data to these norms.
Summary reports are generated annually showing the average of TA and Regional Institute participant ratings of increased knowledge/skills, quality of training, and intent for future action. The rating difference between prior skills/knowledge divided by the 5 points of the Likert scale generates the percent change, e.g., a 0.8 point gain in skills/knowledge represents a 16% increase. Aggregated levels of skills/knowledge gains are included in annual reports. Knowledge gains and overall ratings are generated for each core message area for use by SIG staff and the PCSE Evaluation Task Force in monitoring the quality of the TA and making adjustments through daily management and the formative evaluation process to ensure continuous improvement.
Follow-up Evaluations. Follow up evaluations are used to measure the degree of implementation and use of the knowledge/skills targeted in the TA and Regional Institutes and the degree to which the participants shared the core message knowledge/skills with other appropriate individuals or groups, e.g., parents, GE and/or SE educators, support personnel, health/social service agencies, etc. The follow up evaluation is conducted electronically using a follow-up evaluation questionnaire; see a sample in Appendix C.2.
The follow up evaluation questionnaire is sent to all participants ~3 months after the event. Participants receive an email that states the event title and date and requests their cooperation in completing the brief form. A link in the email takes participants to a website where they are asked to rate their degree of implementation and use of the knowledge/skills taught and the degree of sharing of their knowledge/skills from the TA on a 5-point Likert scale from "Not at All" to "Many Times."
Participant email addresses are obtained from event sign-in sheets and entered into the SIG database. Participants receive up to three reminder emails to complete the follow-up evaluation. Their responses are reported only in the aggregate, never with a link to the participant. In the past year, valid, functional email addresses were reported by ~31% of participants. Of those, the follow-up response rate was 35%. This amounted to approximately 11% of the T/TA participants allowing for a 95% confidence level of +or - 2.24%. Similar tests of confidence level and confidence interval will be conducted to ensure that future response rates are sufficient to generate valid findings.
When data is entered on the website, it is automatically added to the SIG database which has the capability of generating automated reports by event, trainer, core message area, and summaries for periods of time. Throughout the year, program staff use these reports to monitor California State Improvement Grant, p. 81 the degree of learning in the specific core message areas and the degree to which the learning in specific core message areas is being implemented and further disseminated through sharing. The data-to-date is also reviewed by the PCSE SIG Task Force in the formative evaluation.
At the end of each year, summary reports are generated showing the average TA and Regional Institute participant ratings of degree of implementation/use and sharing of their knowledge/skills. Responses of 4 or 5 on the 5-point Likert scale are recorded as the participant having repeatedly implemented or shared the knowledge/skills. The percent of participants at follow-up reporting having repeatedly implemented or shared their knowledge/skills is calculated and included in annual reports.
Outcome 2: Improved Educational Service Coordination for Students with Disabilities
Objective 2. To improve collaboration between special education (SE) and general education (GE) by:
Objective 2 Evaluation Methodology: Assessments will be conducted using the Collaborative Sites Survey with all sites receiving collaborative Site-to-Site TA or participating the Core Message Institutes. In addition, an in-depth research project focusing on the relationship between collaborative service delivery models and student outcomes will be conducted.
2.1. Collaborative Sites Survey. Increases in degree of collaboration in the areas of assessment, intervention, presenting the core curriculum, and teaming will be measured using a Collaborative Sites Survey. This survey is completed by the contact person at each of the California State Improvement Grant, p. 82 Collaboration Learning Sites and at all school/district sites receiving 3 or more days of Site-to- Site TA on Collaboration.
Site contacts are asked to rate the degree of collaboration at their site prior to receiving TA and after the TA on a 10-point Likert scale from "Totally Separate" to "Fully Collaborative" for each area of interaction in which collaboration might be taking place. In addition there are six open-ended questions inquiring about specific changes in the service delivery system, challenges faced in collaboration, factors that have helped facilitate the change process, results seen since adopting the changes, and future plans for sustaining or expanding the move toward collaboration. See Attachment C.3 for a sample of the survey.
The areas of collaboration examined in this survey were identified from two sources:
A survey pilot in 2003 examined sites in California that had received collaborative TA over the past four years. Of the 100 sites contacted that had received TA, nearly half completed the pilot survey. The results are reported in section b.2.(b), pp. 23-37.
The Collaborative Sites Survey will be conducted in the Fall of 2006 and will include all Collaboration Learning Sites and other district/schools that have received 3 or more days of TA California State Improvement Grant, p. 83 on collaboration and have been involved in these SIG activities for at least one year. Thus the time involvement of each site will vary, and this data will also be collected and considered in the outcomes evaluation.
Site contacts will receive emails announcing that the Collaborative Sites Survey is being sent and reminding sites of their commitment to complete the Survey and its purpose. Sites not returning completed surveys by the initial request date will receive reminder emails and a new deadline. Sites not responding by the deadline will receive phone calls and an offer to complete the survey through a phone interview. This methodology has been successful in the past in generating a relatively high rate of return.
Analysis of the quantitative data will focus on comparing the average degree of collaboration "Prior to TA" and "Now" for each of the areas of collaboration as reported by the sites to ascertain what degree of change occurred, if any. In addition, trend analysis will be used to look at the relationship between change in the degree of collaboration and the amount of TA received. Analysis of the qualitative data will focus on identifying the types of changes that were made in the service delivery systems of the sites, unique experiences, and common themes among the challenges faced, facilitating factors, results, and future plans reported.
The evaluator will develop a brief report that describes the quantitative and qualitative findings in both narrative format and through graphs easily understandable by lay persons. This report will be shared with SIG managers and TA coordinators to foster discussion of how the findings could be used in assessing needs and planning future activities. In addition, the findings will be shared with site teams at the Statewide Institute and with all survey participants, along with suggestions on how they might use the findings as a tool to initiate discussion of their site's process of moving toward collaboration. The report will also be used by the PCSE at their annual California State Improvement Grant, p. 84 strategic planning meeting. Finally, the report will be posted on the SIG website to afford easy access and sharing with the broader community.
2.2. Desert/Mountain SELPA Research Project. Collaboration between SE and GE is related in the research and literature to positive outcomes in other SIG objectives, specifically objectives 2--service coordination, 3--student achievement, and 6--use of data. To examine the relationship and bridge these objectives, the SIG plans an in-depth research project looking at the effects of a collaborative service model on student achievement.
The project will be conducted by the SIG Evaluator, in partnership with CalSTAT and the Desert/Mountain (D/M) SELPA. The D/M SELPA has been involved in the Statewide Collaborative Leadership Institute for several years and has sponsored 3 Regional Collaborative Institutes in the past 3 years. Several schools within the SELPA have received additional TA from the initial SIG and ongoing professional development and support from the D/M SELPA.
The Desert/Mountain SELPA is a consortium of 15 school districts, 5 charter schools, and the San Bernardino County Superintendent of Schools. Currently there are 2 collaborative model sites (Cottonwood Elementary and Big Bear Middle School) and 1 collaborative model district (Hesperia Unified School District) within the D/M SELPA region. The research project will allow the D/M SELPA to conduct an investigation of the changes in service delivery for students with disabilities and how the service delivery model relates to student achievement. The research project will be a 3-year undertaking, with the first year focusing on the effects of a collaborative service delivery model at the elementary level and subsequent years looking at the effects of a collaborative service delivery model at the middle school level.
The first year of the research project will be a 2-tiered endeavor, with one tier focusing on school-wide student achievement and the other on the academic achievement of students with California State Improvement Grant, p. 85 disabilities. The research will focus on the 51 elementary schools within the SELPA region. Schools will be divided into 2 groups: those schools operating a collaborative service delivery model and those operating a traditional, pull-out service delivery model.
Student achievement data from the 2001 California Standards Test (CST) in English/Language Arts will be compared with CST results from 2004 to determine whether a collaborative service delivery model is related to student achievement in the general curriculum. These data will be analyzed using chi square analysis, multiple regression, and analysis of variance (ANOVA).
The research will also target related areas. Data will be analyzed to determine whether a collaborative service delivery model is related to number of referrals to and/or placements in SE. In addition, the project will examine the amount of collaboration TA received by the site in relation to student achievement. Data will also be collected on the basic reading program used, as well as the intervention program(s) in effect during the study period, to analyze the relationship between reading curriculum and student achievement. On an individual student level, CST data will be analyzed to determine whether increased time in the GE classroom is related to improved student achievement for students with mild disabilities.
During the second year, the research will extend to middle and junior high schools in the D/M SELPA region, using the same type of measures: the relationship between collaborative service delivery model and student achievement, as measured by the CST E/LA; the relationship between amount of TA and student achievement; the relationship of the type of model and the number of referrals to and placements in SE; and the relationship between the reading curriculum and E/LA achievement. As in year 1, all data will be examined in relation to both school-wide E/LA achievement and E/LA achievement for students with disabilities.
During the third year, the research project will target schools in the D/M SELPA, K-8th grade and wrap up the analyses from the previous years by analyzing the data in an attempt to answer the question, "What kind of collaboration and what level of TA/support leads to improved student achievement at the elementary/middle school level?" Staff will also develop profiles of schools that have experienced systemic change and research which specific components of TA/support are most successful. The hope is that this model can then be replicated in other SELPAs and districts across the State.
This project will specifically analyze the effects of a collaborative service delivery model but secondary questions will deal with the effects of TA and ongoing support from the SIG and the SELPA, as well as literacy instruction and professional development at the elementary and middle school levels. The findings will then be widely disseminated through the SIG channels.
Outcome 3: Improved Academic Outcomes for Students with Disabilities
Objective 3. To increase the academic performance of students with disabilities as demonstrated by:
Objective 3 Evaluation Methodology: Assessments will be conducted using California's CST and API data reports for the Core Message Model and Learning sites and all school/district sites meeting the requirements stated in the outcome measures.
3.1. California Standards Test (CST) data reports. Increases in reading proficiency will be measured based on data reports of student performance on the CST E/LA. California's students are tested annually in the spring, with test scores by school and district available the following fall. As previously discussed, district, school, and student sub-group scores for reading proficiency are reported by percent in five categories: Advanced, Proficient, Basic, Below Basic, and Far Below Basic.
CST data is available from the CDE website CST Research Files. Where site level numbers of students with disabilities fall under the threshold for reporting on the website, data will be obtained either directly from the school or purchased from a specialized consulting firm. The evaluator will construct and regularly up-date a relational database, by site, for the CST school-wide and sub-group data, as well as the number of reading TA days received.
The percent of students scoring Proficient or Advanced and the percent of students scoring Far Below Basic will be proportionally averaged for all grades at each school/district in the database for the baseline year 2004 scores. Each year when CST scores become available, summary graphs will be generated comparing the baseline percentages to the current year percentages for each site and for the average of all sites. Progress toward the target percentage point increases and decreases will be highlighted. Site level data will be shared with each site and the SIG managers, while the project wide data will be shared with the Evaluation Task Force, at Institute meetings, through the CalSTAT website, at the PCSE annual strategic planning session, and in annual reports. A final report will provide results comparing the baseline year to the final year to produce the grant project period outcomes.
2.2 Academic Performance Indicator (API) Data Reports. Increases in overall academic literacy will be measured through API scores. The API score is a single number California State Improvement Grant, p. 89 between 200 and 1000 that synthesizes weighted CST and CAT 6 reading and math scores, along with High School Exit Exam results for secondary schools; this number is then translated to a decile rating of 1-10. Schools are considered to be performing adequately at 800+ for schoolwide and sub-groups; however, less than 5% of middle/high school sites currently score in that range. Schools also receive a Similar Schools Ranking on a scale of 1-10 in comparison with 100 other schools in California with similar parent, student, and teacher demographic characteristics.
API data will be obtained from the California Department of Education's website API Research Files. An up-dated database of the schools/districts participating as Core Message Model and Learning sites and sites receiving Site-To Site TA will include the amount of TA received, the API scores for all students and students with disabilities as a subgroup, and API scores for Similar Schools rankings. Each year when API scores become available, summary graphs will be generated comparing the baseline API and Similar Schools rankings to the current year APIs for each site, school-wide and for the disabilities sub-group. Site level data will be shared with each site and the SIG Managers, while the project wide data will be shared with the Evaluation Task Force, at Institute meetings, through the CalSTAT website, with the PCSE in the strategic planning meeting, and in annual reports. A final grant report will provide results comparing the baseline year to the final year to produce the grant project period outcomes.
Outcome 4: Improved Behavioral Supports and Outcomes for Students with Disabilities
Objective 4. To improve behavior supports and outcomes for students with disabilities, as demonstrated by:
Objective 4 Evaluation Methodology: Assessments will be conducted using a variety of BEST assessment tools by site teams and BEST cadre trainers with the assistance of the IVBD.
4.1. Combined Self-Assessment Survey and the System-wide Evaluation Tool (SET). Increases in the number of schools that have fully implemented BEST will be measured using the BEST Combined Self-Assessment Survey and the System-wide Evaluation Tool (SET), which assess intervention fidelity; see the sample in Appendix C.4 and C.5. Completed twice in the start-up year and annually thereafter, SET consists of 2 distinct tools to be completed by all school staff at each site: the Assessing Behavioral Support in the Schools Survey (EBS--Sugai et al, 1998) and the Oregon School Safety Survey (OSS--Sprague et al, 1999). This data is sent to the University of Oregon Institute on Violence and Destructive Behavior (IVDB) for analysis. A BEST Cadre member completes the SET annually, making observations and conducting interviews at the school sites to determine the level of program implementation and then forwarding the data to the IVDB for analysis. An implementation level of 80% or above on all measures is considered to indicate high intervention fidelity; IVDB expects sites to take at least two years to achieve this level.
IVDB analyzes and summarizes the data in graphs with text which are provided to BEST Cadre trainers to present to the school-level BEST Teams for discussion and use in refining their BEST implementation. In addition, IVDB summarizes information for all the sites and provides annual reports on baseline data and progress to SIG, specifically identifying the number of school sites that have fully implemented BEST. This information will be share widely with the PCSE and the field through the SIG annual report and on the website.
4.2. School Profile Reports. Decreases in office discipline referrals and discipline suspensions will be measured through School Profile Reports (SPR) submitted annually by each BEST site team; see the sample in Appendix C.6. In SPR, the schools provide demographic attendance, grades, and achievement score data, along with discipline office referral and suspension data for the previous year. The SPR is given to the site's BEST Cadre trainers who forward it to IVDB for analysis. The analysis, data sharing for continuous improvement, and dissemination of findings then proceeds as described above for SET.
BEST is a school-wide system change approach and so the focus of the objectives has been on improvements in behavioral supports and outcomes for all students. Nevertheless, there is a keen interest in ensuring that students with disabilities are directly benefiting from this change, beyond the improved school atmosphere and opportunity of the school's students and teachers to focus on learning. Data on discipline suspensions specific to student with disabilities at the school site level has recently begun to be collected through the California Special Education Management Information System (CASEMIS). By the end of the year, site level data on suspensions specific to students with disabilities will be available. Because of the small group size, this data will need to be totaled for all schools that have fully implemented BEST, with comparisons then drawn between baseline and end of the grant outcomes.
Outcome 5: Improved Participation of Parents/Family Members of Students with Disabilities
Objective 5. To increase the participation of parents/family members of children with disabilities in the systems change process by:
Objective 5 Evaluation Methodology: Assessments will be conducted using event registration forms and sign-in sheets, CAC progress reports, Family Participation Fund invoices, End-of-Event Evaluations, and follow-up phone interviews of parents/family members.
5.1. Event Registration Forms and Sign-In Sheets. The project will measure increases in the participation of parents/family members at TA and Regional Institute events and as site team members at Statewide Leadership Institute events using event sign-in sheets and registration forms which ask participants to specify their role; see a sample sign-in sheet in Appendix C.7. This data is entered into a SIG database, along with event invoices and evaluations. The database is capable of generating automated reports on the number/percent of parent/family member participation by event and by core message area, and by time period. Throughout the year, staff can access the database to monitor parent participation in individual events, Regional Institutes, and the TA being offered in specific core message areas. At the end of each year, summary reports will show pie charts of participating in TA and Regional Institutes by role and the percent of Statewide Institute teams with parent members. This data will be shared with the PCSE, posted on the website, and included in the SIG annual report.
5.2. Community Advisory Committee (CAC) Project Progress Reports. Increases in the active participation of low income parents/family members in Community Advisory Councils (CACs) in high poverty areas will be measured through semi-annual progress reports submitted by the California Association of Family Empowerment Centers (CAFEC). CAFEC administers the Family Participation Fund, linking potential low-income participants in 5 highpoverty CACs with training at the Family Empowerment Centers and the resources of the Family Participation Fund. Semi-annual progress reports on the CAC Project will provide details on the composition of the CACs in terms of the number of members, the degree to which they reflect the demographics of their community, support services provided to both the CAC and recruited parent representatives, and additional needs for future support. These reports will be shared and disseminated as discussed above.
5.3. Family Participation Fund (FPF) Invoices/End-of-Event Evaluations. Increases in the participation of low-income parents/family members in non-special education decision-making committees and in parents/family members reporting that their participation has made a difference will be measured using the FPF Invoice/End-of-Event forms. Parents/family members submit these forms following participation in a committee meeting in order to receive reimbursement; see a sample in Appendix C.8. The form inquires about income level and ethnicity to enable monitoring for receipt by the intended population of low-income, underrepresented minority parents/family members. The new forms will add questions about the nature of the committee (SE or GE) and ask participants to rate their effect through their presence and contributions on a 5-point Likert scale from "Not At All" to "Many Times." Responses of 4 or 5 will be counted as their having made a difference. The percent of participants having made a difference will be calculated and cited in annual reports.
Annually, FPF end-of-event evaluations will be forwarded from the FPF to SIG for data entry and summarizing into tables and graphs that will be shared with FPF staff and SIG managers to monitor utilization and effect of the funds. These data will be summarized and included in the SIG2 annual reports, at the annual strategic planning, and on the website.
5.4. Follow-up Phone Interviews. To generate some in-depth data on degree of difference parent/family members are making through their participation and how they might be better supported in enhancing their effectiveness, follow-up phone interviews will be conducted annually with selected FPF support recipients. Structured discussions focusing on the defined topic areas outlined above will be conducted by an experienced qualitative phone interviewer with a purposeful sample of up to 50 FPF recipients--including at least 15 each of FPF recipients who attended SE meetings and those who attended GE meetings--until the point of saturation/redundancy is reached. Each interview will last for approximately 30 minutes.
Phone interviewees will be recruited through a separate Consent Form attached to the end-of event evaluation. The form will discuss the questions to be asked, the time commitment involved, the nature of the sampling process, the $50 stipend for participation, and use of an outside consultant as interviewer. Those willing to be interviewed will submit the Consent Form along with their invoice and end-of-event evaluation. Identifying information will not in any way be connected with their interviewee responses in the recording, use, or reporting of the data. Prior to this phone interview process being conducted, the study protocol will be reviewed and approved by the Institutional Review Board at Sonoma State University.
The qualitative data obtained through the phone interviews will be analyzed for themes and unique experiences, specific kinds of effects parents report having had through their participation, and needs for additional training/support. These findings will be reported along with illustrative quotes from interviewees and recommendations. This report will be shared with the Association administering the FPF as well as CalSTAT managers to foster data-informed planning and more broadly to the PCSE, in the annual SIG2 report, and on the website.
Outcome 6: Improved Data Collection/Use of Data in Addressing the Needs of Students with Disabilities
Objective 6. To improve data collection and the use of data by local educational agencies in identifying and addressing the need for T/TA to support the continuous improvement of outcomes for students with disabilities by:
Objective 6: Evaluation Methodology. Increases in the number of SELPAs and COEs using TED to monitor data on their TA will be measured by the number of sites that complete the TED training, have TED installed at their site, enter TA data into TED, and submit their data to SIG electronically via TED. Sites that use TED data, along with school/district student outcomes, to directly inform decisions on future TA offerings to the continuous improvement of the outcomes of students with disabilities in their service areas will be identified and monitored.
The idea for TED emerged from a sustainability planning process carried out collaboratively by the SIG Evaluator, SIG Managers, CDE Designated Director/Contract Monitor, and PCSE SIG Evaluation Task Force in the final year of the initial SIG. The rationale was as follows:
TED fills the need of SELPAs, COEs, and districts to determine how much TA is being provided in each core message area, who is attending by role and school site, what evidence exists of change in the core message areas targeted, at what level of TA concentration does change begin/optimize, and who must be involved in the TA for maximum effect? Data-based answers to these questions will allow effective targeting of TA to areas of need (e.g., core message areas, districts/schools, personnel/parent roles); better choice of which TA to scale-up; clearer communication with GE about what works; and stronger applications for enhanced professional development funding.
It was concluded that the SIG electronic evaluation system and database developed as a management tool could be modified for broader application, making it a cost-effective instrument for tracking, and eventually evaluating, the use of SE-funded TA in the State as well as at the local level. Broadening efforts in this area is consistent with the State's intent of the SIG evaluation, which is to build capacity--at every level of the system--to gather, access, and use data to inform decision-making processes.
As currently developed, TED will allow SELPAs, districts, and COEs to track possible training effects and target training resources to areas of need, and most importantly to make reports based on that data as easy as possible to access in a useable form for decision-making processes at the local level; see sample screens from TED as well as samples of the automated reports TED is capable of generating in Appendix C.9.
TED will be made available for adoption on a voluntary basis by SELPAs and COEs. Currently TED is scheduled to be beta tested by two SELPAs in the spring of 2004 and will be ready for a broader pilot beginning in the summer. Under the SIG2, TED will be piloted in an additional 7-10 sites each year for the 3 years of the grant. Pilot sites will initially be recruited from among the SELPAs and COEs that have been actively involved in the SIG TA effort, due to their familiarity with the components of the SIG evaluation process and forms. Initial pilot sites will receive the TED database program and training in its use, with stipends to cover time/travel costs for 2 personnel from each site. During the year TA will be provided as needed to the pilot sites by SIG and the TED programmer. At the end of the year, a second meeting with the pilot sites will focus on how to generate and use summary reports from TED in conjunction with student outcomes data in planning and decision-making processes.
Feedback on TED will be elicited from the pilot sites at each stage of the process through brief reports and discussion at training meetings and will be used to refine the database program and add needed reporting capabilities. To broaden the piloting of TED, it will be demonstrated at Statewide Core Message Leadership Institute meetings and other statewide committee meetings such as those of the Comprehensive System of Personnel Development Advisory Committee (CSPDAC) and potentially the annual SELPA Director's meeting. This will help recruit additional pilot sites in years 2-3, as well as foster the use of data for decision-making around the use of TA resources by example.
Examination of implementation strategies and performance is key to an assessment of the SIG Supplemental project. As shown in G.1 above, the objective activities will be assessed for effectiveness through collection of qualitative and quantitative data. Issues around implementation of the activities will clearly be documented by the post-TA evaluation surveys, the 3-month follow up surveys, test data analysis, and interviews. These will be addressed at the formative evaluation meetings.
As shown in the narrative in G.1, the objective performance measures clearly produce both quantitative and qualitative data about the activities related directly to the project outcomes.
The Partnership has designed the SIG2 formative evaluation to provide performance feedback at the activity level, to summarize data-to-date in terms of progress toward achieving outcomes, and to permit implementation adjustment based on findings twice within the project year with the PCSE Evaluation Task Force. The annual summative evaluation will then review both project performance in completing the activities and to assess progress toward achieving outcomes during the last year.
As California educators, we continue to search for ways to improve the system, to involve the community, and to better serve the children. But in our efforts toward systemic improvement, it is essential that we focus on how those changes affect the individual student, the unique child with disabilities. John F. Kennedy said that we must "think of education as the means of developing our greatest abilities, because in each of us there is a private hope and dream which, fulfilled, can be translated into benefit for everyone and greater strength for our nation." It is our most sincere desire that, in creating a better system, the SIG2 will enable each child to realize their private hopes and dreams.