Eddie L. Locklear, Ed.D.
Department Extension Leader
North Carolina State University
Raleigh NC
elocklea@amaroq.ces.ncsu.edu
R. David Mustian, Ph.D.
Extension Program Evaluation Leader
North Carolina State University
Raleigh NC
rmustian@amaroq.ces.ncsu.edu
Copyright/Access Information
Since 1988, the Cooperative Extension Service has devoted significant
resources to helping improve the quality of school-age care programs.
This study was conducted to determine if Extension-supported school-age
care programs benefitted youth. The quasi-experimental design
involved three North Carolina schools: K-2, K-8, and middle school.
The results were mixed. Principal, teachers, and parents indicated
few significant changes in the behavior of youth involved in the
school-age care programs. However, school-age care staff indicated
several academic and social behavioral changes of the youth. Recommendations
include more work to design specific evaluation instruments for
specific respondents, only involve respondents who have significant
interaction with the youth, and a longitudinal study longer than
one year. With GPRA (Government Performance Results Act, 1993)
standards mandated by Congress, it is imperative that Extension
identify school-age care programmatic impacts that can be measured
to justify the resources devoted to this important program.
Data for this research study were collected at public schools
serving K-2, K-8, and middle school youth in three North Carolina
counties. With a quasi-experimental design, students in the 4-H
school-age care programs served as the experimental group and
were matched with students in the schools with similar demographic
and performance factors to serve as a control group.
Data on academic performance, attendance, and tardiness for the
first and third school periods (nine weeks in a period and four
periods in the school year) were provided by the school principal
from student school files.
Structured pre and post questionnaires were used to obtain data
from school-age child care providers, teachers, parents, and the
principal, with respect to both program and control group students'
attitudes toward school, learning behaviors, and character education.
Character education in this study was defined as decision-making
skills, leadership skills, citizenship, responsibility, community
service, and other behaviors which reflect the character of the
young person.
Differences in pre and post scores between the experimental (program
youth) and control groups were analyzed with the student t-test
or z scores. An .05 level of significance was used throughout
the study.
The research design was executed as planned with completed
questionnaires from parents, teachers, principals, and SACC providers
for both the program and control groups. Data from the K-2 and
middle schools were not complete and were not used in this analysis.
Therefore, only data from the K-8 school were used in this study.
Data on academic performance were available for the first and
third school quarters.
Data from the questionnaires were summarized by computing means
for each scale item. Respondents reported their perceptions on
a ten-point scale. In reviewing the mean responses from parents
in both the program and control groups, it is noted that changes
in the scale items were not significant for either group or in
comparing the two groups. It is interesting to note that parents
perceived changes in the predicted direction for the most part,
i.e., the mean response for "talking about what the student
was learning in school with parents" rose from 7.52 to 7.68.
Similar changes were reported for "cooperate with parents",
"ask to participate", "participate on teams and
in the neighborhood", "do homework on own", "takes
responsibility", "develops interests in new topics","join
in group activities", "demonstrates self discipline",
handles anger by talking", and "shows good judgment".
Equally important were decreases in mean scores in behaviors such
as: "express anger by 'hitting'", "get into trouble",
and "associate with people with negative behavior".
Parents in the control group tended to give higher ratings for
their children at the beginning of the school year, but reported
lower scores on the post questionnaires. This decrease many represent
an overstatement of behaviors at the first data collection point.
Questionnaire results from teachers were similar to those from
parents. Direction of predicted changes were observed for responses
of "cooperation with adults and teachers", "show
high level of interest in learning", "show high level
of interest in school", "join group activities",
"share with others", "demonstrate self discipline",
"show respect", and "show good judgment".
Significant changes were reported for program participants in
"handling anger by talking" and "doing homework
on their own". Data from the principal mirror the results
from teachers. The Principal reported significant changes in "handling
anger by talking" and "doing homework on their own"
for program youth.
School-age child care providers reported more significant changes
in program participants (Table 1). Significant changes that the
providers reported included: "talking about what the students
were learning in school", "youth cooperating with provider",
"youth cooperating with others', "showing a high level
of interest in learning", "handling anger by talking",
"participating in team and neighborhood activities",
"doing homework on their own", "showing a high
level of interest in school work", "developing interest
in new topics", joining in group activities", "showing
responsibility", and "sharing with students". Providers
reported significantly decreased cases of "associating with
friends with negative behavior".
Table 1: Pre and Post Feedback From SACC Providers
| Pre | Post | |
| Talk about learning in school | 6.17 | 7.88* |
| Cooperate with you | 7.12 | 9.00* |
| Cooperate with others | 7.25 | 9.04* |
| Show high level of interest in learning | 6.33 | 8.62* |
| Express anger by hitting | 2.82 | 2.13 |
| Handle anger by talking | 6.71 | 8.83* |
| Participate team/neighborhood | 6.67 | 8.92* |
| Get in trouble | 1.83 | 1.71 |
| Do homework on own | 6.94 | 9.53* |
| Show high level of interest in other school work | 6.61 | 8.39* |
| Miss school | 1.00 | 1.00 |
| Associate with friends with negative behavior | 3.67 | 2.29* |
| Develop interests in new topics | 6.25 | 8.08* |
| Join in group activities | 6.67 | 8.42* |
| Show responsibility | 6.71 | 8.83* |
| Share with students | 6.54 | 9.29* |
| Demonstrate self-discipline | 9.92 | 8.92 |
| Show respect/concern | 9.88 | 9.12 |
| Show good judgement | 9.88 | 9.12 |
On the pre-questionnaire to parents of youth in both the program
and control groups, parents reported scores that were higher at
the beginning of the school year than at the last data collection
instrument. The lower scores at the end of the year may reflect
a better measurement of where the youth were actually in their
behaviors. While there were no changes in the parents' scores
of their children, changes were in the predicted direction, i.e.,
parents gave higher scores on the post-questionnaire items such
as "talking about what the youth were learning in school"
and lower scores to items such as "handle their anger by
hitting',.
For teachers and the principal, the same observations are drawn
about their perceptions of the young people. However, for these
respondents, there were two significant differences in scores.
Both teachers and principal reported significant positive changes
in the items of "handle their anger by talking" and
"doing their homework on their own". There were a number
of significant changes in the scores reported by the SAC providers
between the pre and post questionnaires. There were significant
changes in school-related behaviors such as "talking about
what they were learning in their schools", "handling
their anger by talking", and "doing their homework on
their own". The providers also reported significant changes
in the youth's character items such as increased "cooperation
with others", "joining in more activities", "sharing
with others", and "showing responsibility".
This research project was plagued with many of the same problems
associated with social research. The original design involved
working with three school principals to conduct the research with
a wide range of school-age children. The three schools included
children in: K-2, middle school, and K-8 grades. Although an incentive
was provided to each principal, a computer for their after school
program, only the K-8 school principal followed through on the
research. The K-2 school principal had a decrease of interest
in the after school program and eventually closed the program.
The middle school principal failed to distribute and collect the
evaluation instruments in a timely manner.
The project was also on a tight time line which may have prevented
enough time lapse to obtain significant change in youth behavior.
The evaluation project was funded with a grant. Since the grant
was for one year, the project had to fit within the 12 month time
period. Given that contract negotiations and processing of necessary
forms took several weeks to complete, the evaluation occurred
over a seven month period. Perhaps using a similar design over
a period of two or three years would show more conclusive evidence
of change.
Individuals completing the evaluation surveys may have limited
the results of the study. Respondents were the principal, school
teachers, parents, and school-age care providers. Based on feedback
from the principal, it became clear that principals are not in
a position to provide feedback on individual children's behavior.
Unless the children were sent to the principal's office for negative
behavior, the principal may never have any interaction with the
children in the after school program. One of the objectives of
the after school programs was to prevent children from engaging
in negative behavior. Therefore, principals would have limited
interaction with these children.
Another group, parents, provided conflicting information. In some
cases, parents reported worse scores for their children on the
post test than they did on the pretest. Parents' responses made
it difficult to ascertain if the program had a negative impact
or if parents did not pay close attention to any changes which
may have occurred during the study period. Any small changes in
children's homework habits, behavior, or other factors included
in the study may have been missed by parents. Since parents are
using the services of an after school program, perhaps their work
and family schedules are so hectic that small changes were missed.
Teachers, another group of respondents, tended to have stronger
indications of children's behavior. But some of the indicators
that measured behavior during non-school hours may have been missed
by teachers. An instrument dealing with academic behavior may
be more appropriate for teachers to use to give feedback on after
school programs designed to improve academic achievement.
One group, the school-age care providers, provided good information
concerning behavioral changes. This situation may be attributable
to the amount of time providers spend with the school-age children.
On the average, the provider spends about three hours per day,
five days a week with the children. This level of involvement
may have allowed the providers to see a change in children, even
during a short period of seven months.
Several lessons were learned from the study. First, a generic
instrument which tries to capture all pertinent information concerning
the impact of after school programs is not appropriate for different
respondents. Instruments should be designed around specific behavior
which can be observed by specific audiences. Two primary instruments
used to evaluate school-age care programs, the School-Age Care
Environmental Rating Scale (SACERS) and the standards used by
the National School-Age Care Alliance (NSACA), are designed to
evaluate the school-age care environment rather than the impact
the program has directly on children. Additional work is needed
to design an instrument which will help measure the impact of
quality school-age care programs on children's behavior.
Another important lesson is that principals may not have enough
information about individual children to provide adequate feedback.
With principals, an instrument to evaluate the group of children
may be more appropriate. The quasi-experimental design used in
this study may be too difficult to implement with principals as
a major point for distribution and collection. Given the busy
schedules of principals, perhaps another contact in the school
will provide more attention to the details needed to manage the
experimental design.
Previous research efforts on studying school-age child care have
employed, for the most part, an observational design where observers
rate on a checklist of cultural and environmental factors and
children's behavior during the observational period. Checklists
have been long and observational periods short. This study sought
to try an innovative approach. The primary tool was a structured
rating scale where four groups of respondents rated the child's
behavior.
With these points, it is important to note that programs did produce
changes in perceived youth behavior among the program participants.
The SACC providers, who perhaps saw more of the identified behaviors
in the youth whom they worked with daily, reported significant
changes among the program youth. The impact and success of this
program design require further testing, for these preliminary
results are indicative of a program that can make a difference
in the lives of young people who are at risk in this ever-changing
society. Furthermore, with GPRA (Government Performance Results
Act, 1993) requiring Cooperative Extension to be more accountable,
it is imperative that Extension identify methods to measure the
impact of school-age care programs supported by Cooperative Extension
resources.
Further studies may consider collecting data from children, themselves.
It will be important also for data to be collected on academic
performance where a quality curriculum can theoretically and practically
be articulated to quantified measures of performance.
Allen, M., Brown, P., & Finlay, B. (1994) Helping children
by strengthening families. Washington, DC: Children's Defense
Fund.
Locklear, E.L., Riley, D., Steinberg, J., Todd, C., Junge, S.,
& McClain, I. (1994). Preventing problem behaviors and raising
academic performance in North Carolina children: The impacts of
school age child care programs supported by the University Extension
Service. Raleigh, NC: North Carolina Cooperative Extension Service.
Locklear, E.L. (1992). The impact of the 4-H system manager training
on child care provider's perceptions of quality school-age child
care. Unpublished doctoral dissertation, North Carolina State
University, Raleigh, NC.
National Association of Elementary School Principals. (1993).
Standards for quality school-age child care. Alexandria, VA: National
Association of Elementary School Principals.
Contact Us | Non-discrimination Statement and Information Disclosures | © Iowa State University, 2002 | Last update: 8/3/06