national network for child carenavigation bar Home Newsletters E-mail Forum Articles & Resources About National Network Search

IMPACTS OF SACC PROGRAMS

Preventing Problem Behaviors and Raising Academic Performance in the Nation's Youth: The Impacts of 64 School Age Child Care Programs in 15 States


Supported by the Cooperative Extension Service Youth-at-Risk Initiative
Dave Riley
University of Wisconsin
Sharon Junge
University of California
 Jill Steinberg
University of Wisconsin
Ina McClain
University of Missouri
 Chris Todd
University of Illinois
 

Copyright/Access Information

ACKNOWLEDGMENTS

EXECUTIVE SUMMARY

...Big impacts, both in lives made better and in public monies saved...

As part of its Youth-at-Risk initiative, the Cooperative Extension Service has started or assisted school-age child care (SACC) programs in high-risk communities across the nation. The intent of these programs is not just to keep children out of trouble, but to actively promote their positive development.

To determine if these programs are having their desired impacts, we surveyed the SACC staff from 64 programs serving 2,664 children in 15 states. Classroom teachers and school principals associated with these children were also surveyed, providing three independent estimates of program-related changes in children.

The results confirm that Extension - assisted SACC programs are having multiple, positive impacts on children, their families, and their schools. These impacts range from improved social skills, to reduced problem behaviors, to increased academic performance. For example:

Both principals and classroom teachers say that the SACC program has caused 34% of the children to become more cooperative with adults.

Classroom teachers say that over one-third (37%) of SACC program children "have learned to handle conflicts by talking or negotiating more often, instead of just hitting or fighting."

Over one-third of principals (35%) reported that the SACC program had "led to decreased vandalism in the school."

One-third of the SACC program children were getting better grades due to their participation in the program, according to classroom teachers (33%) and principals (36%).

Principals and teachers were in agreement that about 16% of the SACC program children had "avoided being retained in grade for next year because of their involvement in the program." This created a savings of over one million dollars for the school districts.

These are big impacts, both in lives made better and in public monies saved.


OUR TASK

A study of program impacts in 15 states

Unsupervised children with little to do after school have been a concern of educators and policy makers for more than a decade. When a lack of supervision combines with other risk factors (such as substance abuse at home, low self-esteem, learning difficulties, etc.) we know that the likelihood of poor developmental outcomes can increase greatly.

In an effort to prevent these problems, the Cooperative Extension Service has helped establish or support high-quality programs, providing attention from caring adults after school in high-risk communities across the nation. Compared to traditional 4-H clubs, children have roughly ten times as many contact hours in school-age child care (SACC) programs, so the potential for Extension to positively affect the development of youth is greatly multiplied by operating through SACC programs.

Have these programs had positive impacts on children, impacts that are dramatic enough for third-party observers such as classroom teachers and school principals to notice? To determine if they have, we simultaneously surveyed staff of school-age child care (SACC) programs receiving funding through the Extension Service's Youth-at-Risk national initiative, as well as the classroom teachers and principals of the elementary schools serving children in these SACC programs.


The Evaluation Study

This study taps the expertise of professionals who observe the children daily.


The impacts of 64 Extensionsupported SACC programs upon 2,525 children (ages 4 to 15) was investigated by use of a "key informant" survey of three kinds of observers: the SACC lead teachers, the classroom teachers associated with those children, and the school principals. The response rates to our questionnaire survey were 91% for SACC staff, 45% for teachers, and 51% for principals. The distribution of the sample across 15 states is shown in Table 1.

By surveying Principals and Teachers, this study taps the expertise of professionals who observe the children daily, and who can see how they have changed over the course of a year as compared to children not in the SACC programs. This method is also efficient, with each respondent reporting on several children. But this method also has weaknesses. While 43 principals returned our questionnaires, 42 did not, and we need to worry, therefore, if the nonrespondents might have painted a different picture of the program's impacts. For reasons such as this, we must exercise caution and remain tentative in stating the conclusions of the study. (See "Appendix: Data Quality" at the end of this report.)

The SACC programs are part of Extension's federally funded "YouthatRisk" initiative. The communities in which these SACC sites are located were selected by the state Extension Services as especially likely to benefit from the programs. Of the 43 elementary schools for which we have data from principals, 79% received Chapter 1 funding.


TABLE 1: SAMPLE

Number of Respondents

 

 SACC Staff

 School Principals

 Classroom Teachers
 California

  26

 13

 150

 Connecticut

 4

 1

 21

 Georgia

 2

 4

 30

 Illinois

 3

 3

 13

 Iowa

 1

 1

 5

 Kentucky

 3

 2

 18

 Mississippi

 1

 1

 2

 Missouri

 1

 1

 1

 New Hampshire

 2

 2

 14

 New Jersey

 1

 1

 17

 New York

 2

 2

 0

 North Carolina

 5

 3

 36

 Ohio

 1

 3

 8

 Oklahoma

 5

 4

 19

 South Dakota

 1

 2

 8

 Overall number

 58

 43

 342

 Response rate

 91%

 51%

 45%




How many of the children in these SACC programs might be considered "at risk" in their development? The school principals told us that 39% of the program children were "working below grade level in terms of their academic skills at the beginning of the year." The SACC staff gave a smaller estimate (28%), while the classroom teachers offered a harsher assessment, saying that 42% were behind academically. In the area of social skills, principals again gave estimates midway between those of SACC staff and classroom teachers. They reported 35% of children were performing below average, with SACC staff reporting 29% and classroom teachers 41%. These estimates are not too different, and suggest that about a third of the SACC program children were having difficulties in school, academically or in social behavior or both. See Table 2.


TABLE 2

Percentage of SACC program children judged below grade level, according to 3 observers

 

 SACC Staff

 School Principals

 Classroom Teachers
 Below grade level in ACADEMIC skills

 28

 39

 42
 Below average in SOCIAL skills

 29

 35

 41


All respondents - the SACC Staff, the Classroom Teachers, and the Principals - were asked about changes in the children which they could say were specifically due to the effects of the SACC program. Some respondents wrote that they found this attribution of cause and effect to be difficult: Since changes in children are usually due to many factors working together, it was difficult to give clear credit to the SACC program. Based on these comments, we suggest our method elicits conservative estimates of actual program impacts.

Where it was logical to do so, the staff, teachers, and principals were asked the same questions, but some questions were asked only of those respondents who were in a position to answer accurately. For example, we asked school teachers, but not SACC staff, about changes in children's grades and homework.

Were the SACC programs having clear, positive impacts upon the children? According to all three types of informants, the answer was "yes." We will summarize the impacts in three areas: increasing pro-social behaviors, decreasing problem behaviors, and raising academic performance. Following that, we will describe the characteristics of the programs themselves, and their connections to the Extension Service.


INCREASED PRO-SOCIAL BEHAVIORS

Rejection by peers in childhood predicts lifelong negative consequence.

We expected these programs might have special benefits for children who were low in social skills. Like a good 4-H program, a good SACC program offers opportunities for recreation-based, multi-age activities in the company of a caring adult. SACC programs, however, have about 10 times more contact hours with children than traditional 4-H programs, so the potential impacts are much greater.


SHY AND REJECTED CHILDREN

Two areas of social skill were of special concern: shy children and rejected children. Rejection by peers in childhood is important, not just in current heartache, but also because it predicts lifelong negative consequences in social relations and personality development.

When asked whether any children "who were shy have become more outgoing, more skilled at joining group activities" because of the SACC program, 97% of SACC staff answered yes, and they named 20% of the children in this category. But since SACC staff were reporting on their own programs, we might reasonably expect a tendency to exaggerate the impacts of the program. That is why the school teachers were asked the identical question. The classroom teachers reported an even greater percentage of children, 33%, who had become less shy and more socially skilled "because of their involvement in the SACC program."


TABLE 3: INCREASED PRO-SOCIAL BEHAVIORS

Percentage of children increasing their Pro-Social Behaviors, according to 3 observers

 

 SACC Staff

 School Principal

 Classroom Teachers
 1. Shy child became more outgoing.

 20

 *

 33
 2. Rejected child learned to make friends.

 17

 *

 23
 3. More cooperative with adults.

 25

 34

 34
 4. Increased responsibility for planning SACC program.

 22

 *

 *
 5. Developed new interests.

 36

 *

 46


*Note: Question not asked. Respondents were asked about only those child behavior they were likely to have directly observed.


Here are three examples of such a child, described by SACC staff and classroom teachers:

"...When she came to the program, I observed that she was shy and withdrawn. She didn't like to participate. I visited her school to talk to her teacher to find the best way to help her. She was also withdrawn at school. In order to help her, I had to build her confidence in herself. Once her self-esteem was restored, she was able to become more outgoing."

SACC Staff in New Jersey


"...One child was shy and a little backward. She enjoyed the attention she received in the program. It was a positive influence in her life."

Classroom Teacher in Ohio


"I would say socially, both children have had positive changes throughout the school year. Both children were quiet and withdrawn from the other students in class. I feel the SACC program gave these two children an opportunity for more socialization with other children of varied ages, and as a result, they both became well-liked in class, participated more in class instruction, and became more confident in themselves."

Classroom Teacher in Connecticut


Did the SACC programs provide opportunities for rejected children to learn how to make friends? Most SACC staff saw this happening, with positive impacts on 17% of the children they served. The classroom teachers confirmed this estimate, giving the SACC programs credit for positive impacts in this area on 23% of their students from the SACC program.

"I am very glad [student] is in a program like [this one]. It may not show right away, but the skills he learns there will help him in the future, I'm positive. I've seen a willingness to help others, which is a great help. It gets him more accepted by others who would otherwise not like him due to his behavior problems."

Classroom Teacher in New Hampshire

We have a young boy in our program that has very poor social skills. He does not interact positively with other children. We have been able to work on this child's social skills through game playing. Through collaboration with his teacher and parents as well as SACC staff we have tried to bang out the positive qualities in this child. With a gradual increase in self-esteem levels, confidence, and morale he now seems a little happier!"

SACC Staff in Connecticut

"She has gained friendship, learning to be a friend, and have a friend. She has learned to mature into a nice young lady and learned to deal with not always being a winner. She does a lot more smiling and less frowning or crying. "

Classroom Teacher in Oklahoma


RESPECTING THE AUTHORITY OF ADULTS

Learning to work well in a group of one's peers is one aspect of pro-social behavior. Another which we asked about is learning to get along with the adults in the program. SACC programs, we suspected, have the potential to teach young people to live within rules, and to respect the reasonable authority of adults. This kind of learning, if it has not taken place already within the home, is crucial in the years of middle childhood, before young people reach the wider world of adolescence.

When asked if any children "have become more cooperative with adults now, more willing to follow the directions and rules of adults" because of the SACC program, all of the SACC staff answered "yes" and named 25% of the children. But did this improvement generalize across settings such that teachers and principals could notice a change as well? The answer is "yes." The classroom teachers reported this improvement in 34% of the children, and the principals in 34%, identical estimates.

"Have seen positive changes in student due to child being able to have supervision after school. The child has to answer to adults and is forced to keep "better company" while being supervised. He has actually picked up a few good habits."

Classroom Teacher in Kentucky


"One of my students really needed some guidance. His attitude towards school and his teachers has really changed."

Classroom Teacher in Georgia


"There is a 10-year old boy who has a short fuse and thinks he does not have to follow the rules. What we did that was effective was to set boundaries for him and stick by them. Very few adults have been consistent with him. He responded with greater participation in the activities and with significantly fewer out-of-control rages."

SACC Staff in California


Leadership is a form of pro-social behavior upon which 4-H programs have traditionally focused. In this study, we assessed leadership by asking SACC staff if any of their children had "learned to take more responsibility for planning and running the program and its activities." Nearly all of the SACC programs saw this, and they reported that 22% of the children had grown in this way.


HELPING CHILDREN TO EXPLORE NEW INTERESTS

Besides learning to work well with peers and with adults, a third area of pro-social behavior we asked about was the development of new interests by children. The middle years of childhood are a period in which children like to do real things for themselves, in which their horizons widen greatly, and in which they develop new and sometimes lasting interests. A good program for preadolescent children will expose them to a wide variety of activities, and provide the opportunity for exploration in depth of those activities which capture their interests. At its best, new interests developed during this period can form the basis for lifelong vocational or avocational pursuits. Did children in these SACC programs develop "interests they would not otherwise have, in new topics or activities"? Yes. The SACC staff reported that 36% of the children had developed new interests. The classroom teachers verified this by reporting an even greater percentage of children (46%) with new interests gained in the SACC programs.

"The arts and crafts program is very limited at our school and even more so for the 5th and 6th graders. The after-school program gave these kids an opportunity to explore their artistic and creative abilities."

Classroom Teacher in Oklahoma


"The particular student I had this year opened up many new interests. He became particularly fond of gardening based on a classroom project as well as a 4-H project. Since this particular student would have gone home to an empty home, it was nice he could have assistance in the 4-H program with his homework!

Classroom Teacher in California


"For one child in particular, the SACC program has offered an opportunity for choices and time to pursue interests - not always possible in the classroom. This child still has problems, but has evidenced 'dramatic positive changes' in social and academic skills."

Classroom Teacher in California


REDUCED PROBLEM BEHAVIORS

Reducing the violence and aggression that have become so common in America's playgrounds ought to be a high priority.

We generally expect that children who have positive social skills will have relatively few behavior problems. A program that teaches children to work well with peers and cooperate with adults (the outcomes reviewed above) should thereby reduce behavior problems among its children. Rather than assume this to be true, however, we asked the SACC staff, school teachers, and principals directly about behavior problems, and the possible effect of the SACC programs in reducing them.

When asked if "some of these children have begun to demonstrate fewer behavior problems" because of their involvement in the SACC program, school principals and classroom teachers reported improvement in 26% and 29% of their children, respectively, while SACC staff reported these gains for 19%. Here are several examples:

"We had one child that was labeled by the state guidelines as Behavior Disordered. He received wonderful structure, but in a non-threatening manner. The SACC program became a positive place for this little guy. He looked forward to being in the program."

School Principal in Illinois


"A new sixth grade boy came to our school and was having a lot of discipline problems related to not fitting in. Once he started in the [SACC] program, he made a dramatic improvement which lasted the remainder of the year."

School Principal in Kentucky


"One of my boys has been doing...much better with a structured after-school atmosphere, rather than just being 'free' to do as he pleases.."

Classroom Teacher in South Dakota


"The child benefiting most dramatically is one of the older children in the SACC program. Attending the program has resulted in positive changes in classroom behavior and academic progress. It has also improved behavior at home and in the community. This association with young children has increased his maturity and leadership skills. He has also become more helpful and considerate of others."

School Principal in North Carolina


Increased after-school supervision can have an impact on the wider school community as well. Indeed, one-third of the principals (35%) reported that because of the program, there had been a decrease in the amount of vandalism in and around the school.

Reducing the violence and aggression that have become so common in America's playgrounds and classrooms ought to be a high priority of any youth-serving program. One of the ways that a good program can reduce aggression is by teaching more acceptable ways to solve problems. Both the SACC staff and the school personnel agreed that the SACC programs are doing this. The SACC staff reported 25% of the children "have learned to handle conflicts by talking or negotiating more often, instead of just hitting or fighting." Principals saw this behavioral change in 23% of children, and the classroom teachers in 37% of the SACC children in their classes.

"Through consistent supervision, students have been provided a structure that has reduced the number of office referrals. Students manage conflicts in a more positive way - less fighting and more dialogue."

School Principal in California


"One student in particular has learned to resolve conflicts by telling the adults in charge. He has also learned to control his temper. Prior to the [SACC] program this student exhibited aggressive, immature behavior."

Classroom Teacher in Connecticut



TABLE 4: REDUCTION IN PROBLEM BEHAVIORS


Percentage of children decreasing their Problem Behaviors, according to 3 observers

 

 SACC Staff

 School Principal

 Classroom Teachers
 1. Fewer behavior problems.

 19

 26

 29
 2. Decreased vandalism in school.

 *

 35**

 *
 3. Learned to handle conflicts by talking rather than hitting.

 25

 23

 37


*Note: Question not asked. Respondents were asked about only those child behaviors they were likely to have directly observed.

**Note: This one number represents percentage of principals (and schools), rather than percentage of children.


IMPROVED ACADEMIC BEHAVIORS.

Did SACC programs in targeted neighborhoods improve children's academic performance?

Social skills and behavior problems are not the only domains in which children might benefit from a high quality SACC program. We were also interested in the possibility that SACC programs in targeted neighborhoods might improve children's academic performance.

The classroom teachers suggested that this was true. They reported that 33% of the SACC program children had developed an interest in recreational reading, 21% had improved their school attendance, and 33% were turning in more or better quality homework. Not surprisingly, this had led to better grades for 34% of the children. It had even led, in the teachers' estimation, to 17% of the children avoiding being held back in grade, and 13% avoiding placement in special education. The principals confirmed these estimates with their responses (see Table 5). In each case, the classroom teachers and principals reported that these improvements were specifically "because of their involvement in the SACC program."

"The five children in my class who attend have definitely done better in their homework skills - this was a very important highlight for me because they weren't getting it done at home. Also - the Books-Across-America program - the 4-H kids ended up being the only ones who participated and they all won awards."

Classroom Teacher in California


"I worked with L. one-on-one sometimes. I helped him understand things and put things in a simple light. He loved to draw so I encouraged that in him, but with the drawing that he did, he had to count the things that he drew. So when he drew, he counted too..."

SACC Staff in Mississippi


"I have a student who entered the after-school program 5 weeks ago. Since then I've noticed her behavior has settled down. She is completing assignments in class, is not as distracted as before - seems more focused. Her homework is passed in each day. Before we were lucky to see one assignment in a month. Her social skills have improved too. The after school program has been a positive change in this student. Wish she had enrolled sooner."

Classroom Teacher in New Hampshire


"S. has shown improvement in the quality of her work. She participates in class discussions. She is more ready to give an answer to a question."

Classroom Teacher in New Jersey

These are big impacts not only for the children, but also in monetary terms. In these school districts, the average cost to repeat a year of school was $4,318. We therefore estimate that the SACC program saved taxpayers over one million dollars last year just by preventing retention in grade of the 253 children named by the principals.


TABLE 5: IMPROVED ACADEMIC BEHAVIORS

Percentage of children improving their Academic Behaviors, according to 3 observers
 

 SACC Staff

 School Principals

 Classroom Teachers
 1. Developed interest in reading

 15

 *

 33
 2. Improved school attendance

 *

 21

 21
 3. Better homework

 *

 36

 33
 4. Improved grades

 *

 *

 34
 5. Avoided retention in grade

 *

 16

 17
 6. Avoided placement in special education

 *

 11

 13

*Note: Question not asked. Respondents were asked about only those child behaviors they were likely to have directly observed.


EXTENSION'S ROLE

"...Going to where the client is...."

What is the connection of these SACC programs to the Extension service? Most (91%) of the SACC programs we evaluated are supported by Extension Youth-at-Risk (federal) funds, and 88% are operated by Extension. Two-thirds (67%) have 4-H clubs and activities as part of their programs, and 82% have received staff training from Extension agents. The SACC staff from these 64 programs met with the local Extension agents between O and 150 times in the previous year (with 15 meetings as the median response). The SACC staff who responded reported they clearly gained from the training and consultation provided by the Extension agents.

"They sent me to training school and had Telenet that we were required to listen to. We also had staff meetings to discuss problems or questions that arose, which was helpful "

SACC Staff in Illinois


"[The Extension Agent] provided constant support, resources, and help in any area of need. Has been personally invested in the success of the program. Most helpful - experience and knowledge!"

SACC Staff in Kentucky


"[Extension Agents] provided organized county-wide 4-H activities which were opened for my group. Youth were able to participate in these activities which they otherwise would not have been able to do."

SACC Staff in New Hampshire


"I think it's wonderful that finally the after school programs can now incorporate 4-H. Most parents in my program love the 4-H aspect of it and the children do too! (I hope they continue to provide Resource manuals, because they are priceless."

SACC Staff in California


These SACC programs represent an innovative response by the Extension Service to the changing demography of American family life and work life. Most children today are being raised by employed parents, whether in a single-parent household or a 2-parent 2-earner household. SACC programs represent a safe and enriching alternative to the "latchkey child" situation, in which children at young ages are left without adult supervision.

By "going to where the client is," in this case a school-age child care program, the Extension System has adapted its traditional programs for youth development to these recent societal changes. Because children are in SACC programs for many more hours each week than in a traditional 4-H club, Extension's flexible approach to delivery of its programs has led to a greatly expanded potential for effecting children in positive ways.




TABLE 6: THE EXTENSION SERVICE'S ROLE IN THE SACC PROGRAM

 1. Received federal Youth-at-Risk Initiative funds.

 91%

 2. Extension helped start the SACC program.

 74%

 3. Program is currently operated by Extension.

  88%

 4. 4-H Clubs operate within the SACC setting.

  67%

 5. SACC staff received training from Extension.

 82%



CONCLUSION.

...multiple, positive impacts...


The SACC staff, classroom teachers, and school principals showed considerable agreement in their questionnaire responses. This agreement supports the validity of the results reported here. Surprisingly, in many cases the principals and classroom teachers saw even greater gains due to the SACC programs than did the staff of those programs.

We suggest that the SACC programs operated by the Cooperative Extension Service appear to be having multiple, positive impacts on the children, their families, and their schools. These impacts range from social skills, to reduced problem behaviors, to increased academic achievement, and are evident not only to the staff of the SACC program, but also to local child development experts such as school principals and classroom teachers.

Based on these findings, we believe Extension is well justified in helping to establish additional school-age child care programs in communities which lack them. In addition, Extension's proven capability in training adults to work with youth should be shared far more broadly, with the vast number of existing SACC programs, which have little or no contact with Extension at present. These non-Extension SACC programs represent a crucial non-school, non-family environment for millions of American youth today. Extension can greatly multiply its positive impact on youth development by working with and through these SACC programs.

How solid are these data?


APPENDIX: DATA QUALITY

Before conclusions can be believed, we must consider the quality of the evidence. Two areas of clear concern for this study are the generalizeability of the findings (given the response rates), and the validity of the data (given their origin in self-report questionnaires).


GENERALIZEABILITY

Only about half the recruited principals and classroom teachers actually responded with completed questionnaires. We should ask whether the non-respondents would have painted a different picture of program impacts. It is not unreasonable to suggest that those with more favorable responses are more likely to respond. If so, then our estimates of impact are over-estimates.

To test this possibility, the 17 programs with the highest return rates were analyzed separately. All 17 programs had at least 50% response rates from all three types of respondents. The overall response rates in this sub-sample were 100% for SACC staff, 88% for principals, and 66% for classroom teachers.

Were the estimates of program impact substantially different in this high-response-rate sub-sample? The answer was no. Considering the percentages reported in Tables 3, 4, and 5 of this report, in nearly half the cases (44%) the corresponding number in the sub-sample was within two percentage points. The mean difference across all pair-wise comparisons was 3.5%. Thus we can discern little difference between the high-response-rate sub-sample and the overall sample used for this report. We suggest, then, that the results of this study are likely to generalize to other SACC programs operated in high risk communities by the Extension Service.


VALIDITY OF THE DATA

If one wished to conduct the best possible evaluation of the impact of SACC programs, using a method in which the validity of the conclusions would be as unassailable as possible, one would pick an experimental design utilizing direct observations or tests of children's attitudes and abilities. As with many studies using human subjects, however, this research design is difficult to apply here. At a minimum, to create an experimental design would require waiting lists for each of the programs, and the use of random assignment to the program from the waiting lists. In practice, few of these programs have extensive waiting lists. Moreover, there are ethical concerns with denying needed services to children.

Conducting longitudinal research using non-program children (i.e. latchkey children or parent-supervised children) as a comparison group is also problematic. Prior research has shown that children in SACC programs are systematically different from non-program children right from the start of the year. Estimating change from pretest to posttest can be misleading if children in the two groups (treatment and control) are on different developmental trajectories, so establishing true equivalence of the comparison group is crucial. Unfortunately, forming equivalent groups for comparison, while not impossible, is very difficult for this topic.

The method selected for this evaluation has a weaker claim to validity than an experiment, but nonetheless has strengths of its own. The strengths of our method were (1) the use of expert informants, (2) three independent observers for each child, (3) direct attribution of causality, written into the questionnaire items, and (4) the use of open-ended comments about specific children to verify that real changes did take place. These strengths will be explained below. But a general threat to the validity of the data remains: the data are not based on objective tests or observations, but on a self-report questionnaire survey.

The threats to the validity of self-report data are many. We focus on the two most important: the social desirability bias and the attribution bias.

The social desirability bias derives from respondents having a psychological motive to paint a desirable picture of themselves, or to otherwise provide responses they believe will be pleasing to the investigator. In the current study, we need to be most concerned with the responses of SACC staff, since they are reporting on the impacts of their own programs. To counter-balance the potential bias of this method, we asked identical questions of school principals and classroom teachers. We expected that in many cases their natural inclination will have been, not to give credit to the SACC programs, but rather to see the school as responsible for positive changes in the children. Indeed, some principals and teachers wrote comments exactly to this effect. The strategy of using multiple observers of each child, therefore, should have mitigated much of the impact of the social desirability bias.

The fairly high level of correspondence between the responses of SACC staff, school principals, and classroom teachers, (shown in Tables 3, 4, and 5), all reporting on the same children, provides some reassurance that the particular viewpoint of any one observer was not strongly biasing the final results. To display this convergence between multiple sources, we have graphed in Figure 1 the responses to the four items which were asked of all three respondents. As can be seen, the first three items, which all index problem behaviors, ranged from 19% to 37%, while the item indexing a change in parental participation (a very different outcome) had no estimates in this range (varying from 10% to 17%). The mean difference between these two groups of items was 14%, while the mean difference of estimates within each item was 7.3% for the problem behavior items, and a 4.7% for the parental involvement item. Thus the difference was greater between groups than within, supporting the claim that the different observers were converging in their estimates of particular item content, and discriminating different item contents.

The attribution bias refers to the tendency among humans to seek order even in chaos, and in particular to believe that cause-and-effect relationships exist even when they do not. Thus, even if children are changing randomly (or, for example, are experiencing different timing and pace of biological maturation), most respondents will develop personal theories about what is causing the changes.

The attribution bias does not specify that the SACC program, in particular, will be the beneficiary of the bias. We therefore avoided wording questions in such a way as to encourage the spread of the causal attribution bias to the SACC program. Specifically, we avoided wording the questionnaire items so that the SACC program could be given some small and partial credit for changes in children (i.e. we avoided questions like "Did the program contribute to..."). Instead, the questionnaires asked for a firmer commitment to cause and effect: "Because of their participation in the [SACC] program," did any children change in each way.




FIGURE 1: CONVERGENT VALIDITY CHECK USING THREE DATA SOURCES

Percentage of Children or Parents reported changing, according to 3 observers

 

 SACC Staff

 School Principals

 Classroom Teachers

 Mean Discrepancy Between Sources
1. Child became more cooperative with adults.

 25%

 34%

 34%

 6.0
2. Child had fewer problem behaviors

 19%

 26%

 29%

 6.7
3. Child learned to talk rather than hit

 25%

 23%

 37%

 9.3
4. Parents increased involvement in program or school

 10%

 15%

 17%

 4.7


As evidence that this strategy may have worked, many respondents wrote notes on their questionnaires reminding us that many forces contribute to a child's development, not just the program we were evaluating. Here are two examples:

"We have other programs, so it would be difficult to give credit to any particular program."

School Principal in Oklahoma

"It's very hard to determine who or what makes a difference in the behavior changes of the children at this age. A lot of behavior in these two children reflects the home situations - maturation, the school programs and expectancies, and the adjustment to a new school."

Classroom Teacher in California

A separate form of evidence for the validity of the data comes from the quotations which have been included in this report. The quotes are not representative in any sense; they were included not as evidence, but rather to exemplify and put a human face on the numerical findings. Still, by the fact that they provide so coherent and convincing a picture of the same changes in children as in the numerical data, they lend credibility to the numbers. In other words, the testimony of SACC staff, classroom teachers and school principals about particular children who made great gains assures us that at least some children were benefiting in the ways suggested by the numerical data.

By thoughtful attention to our methods of investigation, we have attempted to minimize the effects of non-random recruitment, social desirability biases, and causal attribution biases. We cannot claim to have excluded these biases altogether, but we believe a claim can be supported that these biases have been minimized. The results of this study do not obviate the need for randomized field experiments, but they provide the most convincing evidence to date that Extension-supported SACC programs are causing significant, positive changes in the lives of children.




DOCUMENT USE/COPYRIGHT
National Network for Child Care - NNCC. Part of CYFERNET, the National Extension Service Children Youth and Family Research Network. Permission is granted to reproduce these materials in whole or in part for educational purposes only (not-for-profit beyond cost of reproduction) provided that the author and Network receive acknowledgment and this notice is included:

Reprinted with permission from the National Network for Child Care - NNCC. Riley, D. (1994). *Preventing problem behaviors and raising academic performance in the nation's youth: The impacts of 64 school age child care programs in 15 states.* Madison, WI: University of Wisconsin Cooperative Extension Service.

Any additions or changes to these materials must be preapproved by the author.

AVAILABLE FROM::
Dave Riley
Cooperative Extension Service
University of Wisconsin - Madison
120 Family Resources Building
1300 Linden Drive
Madison, WI 53706
Phone::608-262-3314
FAX::608-262-5335
INTERNET::riley@wisplan.uwex.wisc.edu

COPYRIGHT PERMISSION ACCESS::
Dave Riley
Cooperative Extension Service
University of Wisconsin - Madison
120 Family Resources Building
1300 Linden Drive
Madison, WI 53706
Phone::608-262-3314
FAX::608-262-5335
INTERNET::riley@wisplan.uwex.wisc.edu

FORMAT AVAILABLE:: Internet
DOCUMENT REVIEW:: Level 3 - National Peer Review
DOCUMENT SIZE:: 64K or 18 pages
ENTRY DATE:: August 1996

Contact Us | Non-discrimination Statement and Information Disclosures | © Iowa State University, 2002 | Last update: 8/3/06