Eddie L. Locklear, Ed.D.
Department Extension Leader
North Carolina State University
R. David Mustian, Ph.D.
Extension Program Evaluation Leader
North Carolina State University
Since 1988, the Cooperative Extension Service has devoted significant
resources to helping improve the quality of school-age care programs.
This study was conducted to determine if Extension-supported school-age
care programs benefitted youth. The quasi-experimental design
involved three North Carolina schools: K-2, K-8, and middle school.
The results were mixed. Principal, teachers, and parents indicated
few significant changes in the behavior of youth involved in the
school-age care programs. However, school-age care staff indicated
several academic and social behavioral changes of the youth. Recommendations
include more work to design specific evaluation instruments for
specific respondents, only involve respondents who have significant
interaction with the youth, and a longitudinal study longer than
one year. With GPRA (Government Performance Results Act, 1993)
standards mandated by Congress, it is imperative that Extension
identify school-age care programmatic impacts that can be measured
to justify the resources devoted to this important program.
Data for this research study were collected at public schools
serving K-2, K-8, and middle school youth in three North Carolina
counties. With a quasi-experimental design, students in the 4-H
school-age care programs served as the experimental group and
were matched with students in the schools with similar demographic
and performance factors to serve as a control group.
Data on academic performance, attendance, and tardiness for the first and third school periods (nine weeks in a period and four periods in the school year) were provided by the school principal from student school files.
Structured pre and post questionnaires were used to obtain data from school-age child care providers, teachers, parents, and the principal, with respect to both program and control group students' attitudes toward school, learning behaviors, and character education. Character education in this study was defined as decision-making skills, leadership skills, citizenship, responsibility, community service, and other behaviors which reflect the character of the young person.
Differences in pre and post scores between the experimental (program youth) and control groups were analyzed with the student t-test or z scores. An .05 level of significance was used throughout the study.
The research design was executed as planned with completed
questionnaires from parents, teachers, principals, and SACC providers
for both the program and control groups. Data from the K-2 and
middle schools were not complete and were not used in this analysis.
Therefore, only data from the K-8 school were used in this study.
Data on academic performance were available for the first and
third school quarters.
Data from the questionnaires were summarized by computing means for each scale item. Respondents reported their perceptions on a ten-point scale. In reviewing the mean responses from parents in both the program and control groups, it is noted that changes in the scale items were not significant for either group or in comparing the two groups. It is interesting to note that parents perceived changes in the predicted direction for the most part, i.e., the mean response for "talking about what the student was learning in school with parents" rose from 7.52 to 7.68. Similar changes were reported for "cooperate with parents", "ask to participate", "participate on teams and in the neighborhood", "do homework on own", "takes responsibility", "develops interests in new topics","join in group activities", "demonstrates self discipline", handles anger by talking", and "shows good judgment". Equally important were decreases in mean scores in behaviors such as: "express anger by 'hitting'", "get into trouble", and "associate with people with negative behavior".
Parents in the control group tended to give higher ratings for their children at the beginning of the school year, but reported lower scores on the post questionnaires. This decrease many represent an overstatement of behaviors at the first data collection point.
Questionnaire results from teachers were similar to those from parents. Direction of predicted changes were observed for responses of "cooperation with adults and teachers", "show high level of interest in learning", "show high level of interest in school", "join group activities", "share with others", "demonstrate self discipline", "show respect", and "show good judgment". Significant changes were reported for program participants in "handling anger by talking" and "doing homework on their own". Data from the principal mirror the results from teachers. The Principal reported significant changes in "handling anger by talking" and "doing homework on their own" for program youth.
School-age child care providers reported more significant changes in program participants (Table 1). Significant changes that the providers reported included: "talking about what the students were learning in school", "youth cooperating with provider", "youth cooperating with others', "showing a high level of interest in learning", "handling anger by talking", "participating in team and neighborhood activities", "doing homework on their own", "showing a high level of interest in school work", "developing interest in new topics", joining in group activities", "showing responsibility", and "sharing with students". Providers reported significantly decreased cases of "associating with friends with negative behavior".
Table 1: Pre and Post Feedback From SACC Providers
|Talk about learning in school||6.17||7.88*|
|Cooperate with you||7.12||9.00*|
|Cooperate with others||7.25||9.04*|
|Show high level of interest in learning||6.33||8.62*|
|Express anger by hitting||2.82||2.13|
|Handle anger by talking||6.71||8.83*|
|Get in trouble||1.83||1.71|
|Do homework on own||6.94||9.53*|
|Show high level of interest in other school work||6.61||8.39*|
|Associate with friends with negative behavior||3.67||2.29*|
|Develop interests in new topics||6.25||8.08*|
|Join in group activities||6.67||8.42*|
|Share with students||6.54||9.29*|
|Show good judgement||9.88||9.12|
On the pre-questionnaire to parents of youth in both the program
and control groups, parents reported scores that were higher at
the beginning of the school year than at the last data collection
instrument. The lower scores at the end of the year may reflect
a better measurement of where the youth were actually in their
behaviors. While there were no changes in the parents' scores
of their children, changes were in the predicted direction, i.e.,
parents gave higher scores on the post-questionnaire items such
as "talking about what the youth were learning in school"
and lower scores to items such as "handle their anger by
For teachers and the principal, the same observations are drawn about their perceptions of the young people. However, for these respondents, there were two significant differences in scores. Both teachers and principal reported significant positive changes in the items of "handle their anger by talking" and "doing their homework on their own". There were a number of significant changes in the scores reported by the SAC providers between the pre and post questionnaires. There were significant changes in school-related behaviors such as "talking about what they were learning in their schools", "handling their anger by talking", and "doing their homework on their own". The providers also reported significant changes in the youth's character items such as increased "cooperation with others", "joining in more activities", "sharing with others", and "showing responsibility".
This research project was plagued with many of the same problems associated with social research. The original design involved working with three school principals to conduct the research with a wide range of school-age children. The three schools included children in: K-2, middle school, and K-8 grades. Although an incentive was provided to each principal, a computer for their after school program, only the K-8 school principal followed through on the research. The K-2 school principal had a decrease of interest in the after school program and eventually closed the program. The middle school principal failed to distribute and collect the evaluation instruments in a timely manner.
The project was also on a tight time line which may have prevented enough time lapse to obtain significant change in youth behavior. The evaluation project was funded with a grant. Since the grant was for one year, the project had to fit within the 12 month time period. Given that contract negotiations and processing of necessary forms took several weeks to complete, the evaluation occurred over a seven month period. Perhaps using a similar design over a period of two or three years would show more conclusive evidence of change.
Individuals completing the evaluation surveys may have limited the results of the study. Respondents were the principal, school teachers, parents, and school-age care providers. Based on feedback from the principal, it became clear that principals are not in a position to provide feedback on individual children's behavior. Unless the children were sent to the principal's office for negative behavior, the principal may never have any interaction with the children in the after school program. One of the objectives of the after school programs was to prevent children from engaging in negative behavior. Therefore, principals would have limited interaction with these children.
Another group, parents, provided conflicting information. In some cases, parents reported worse scores for their children on the post test than they did on the pretest. Parents' responses made it difficult to ascertain if the program had a negative impact or if parents did not pay close attention to any changes which may have occurred during the study period. Any small changes in children's homework habits, behavior, or other factors included in the study may have been missed by parents. Since parents are using the services of an after school program, perhaps their work and family schedules are so hectic that small changes were missed.
Teachers, another group of respondents, tended to have stronger indications of children's behavior. But some of the indicators that measured behavior during non-school hours may have been missed by teachers. An instrument dealing with academic behavior may be more appropriate for teachers to use to give feedback on after school programs designed to improve academic achievement.
One group, the school-age care providers, provided good information concerning behavioral changes. This situation may be attributable to the amount of time providers spend with the school-age children. On the average, the provider spends about three hours per day, five days a week with the children. This level of involvement may have allowed the providers to see a change in children, even during a short period of seven months.
Several lessons were learned from the study. First, a generic
instrument which tries to capture all pertinent information concerning
the impact of after school programs is not appropriate for different
respondents. Instruments should be designed around specific behavior
which can be observed by specific audiences. Two primary instruments
used to evaluate school-age care programs, the School-Age Care
Environmental Rating Scale (SACERS) and the standards used by
the National School-Age Care Alliance (NSACA), are designed to
evaluate the school-age care environment rather than the impact
the program has directly on children. Additional work is needed
to design an instrument which will help measure the impact of
quality school-age care programs on children's behavior.
Another important lesson is that principals may not have enough information about individual children to provide adequate feedback. With principals, an instrument to evaluate the group of children may be more appropriate. The quasi-experimental design used in this study may be too difficult to implement with principals as a major point for distribution and collection. Given the busy schedules of principals, perhaps another contact in the school will provide more attention to the details needed to manage the experimental design.
Previous research efforts on studying school-age child care have employed, for the most part, an observational design where observers rate on a checklist of cultural and environmental factors and children's behavior during the observational period. Checklists have been long and observational periods short. This study sought to try an innovative approach. The primary tool was a structured rating scale where four groups of respondents rated the child's behavior.
With these points, it is important to note that programs did produce changes in perceived youth behavior among the program participants. The SACC providers, who perhaps saw more of the identified behaviors in the youth whom they worked with daily, reported significant changes among the program youth. The impact and success of this program design require further testing, for these preliminary results are indicative of a program that can make a difference in the lives of young people who are at risk in this ever-changing society. Furthermore, with GPRA (Government Performance Results Act, 1993) requiring Cooperative Extension to be more accountable, it is imperative that Extension identify methods to measure the impact of school-age care programs supported by Cooperative Extension resources.
Further studies may consider collecting data from children, themselves. It will be important also for data to be collected on academic performance where a quality curriculum can theoretically and practically be articulated to quantified measures of performance.
Allen, M., Brown, P., & Finlay, B. (1994) Helping children
by strengthening families. Washington, DC: Children's Defense
Locklear, E.L., Riley, D., Steinberg, J., Todd, C., Junge, S., & McClain, I. (1994). Preventing problem behaviors and raising academic performance in North Carolina children: The impacts of school age child care programs supported by the University Extension Service. Raleigh, NC: North Carolina Cooperative Extension Service.
Locklear, E.L. (1992). The impact of the 4-H system manager training on child care provider's perceptions of quality school-age child care. Unpublished doctoral dissertation, North Carolina State University, Raleigh, NC.
National Association of Elementary School Principals. (1993). Standards for quality school-age child care. Alexandria, VA: National Association of Elementary School Principals.
Contact Us | Non-discrimination Statement and Information Disclosures | © Iowa State University, 2002 | Last update: 8/3/06