Reflections on the challenges of program evaluation
Carla A. Peterson
Wagner, Spiker, and Linn’s work (this issue) challenges us to examine the design of large-scale program evaluations with simultaneous consideration of evaluation findings. Current evaluation results of the Parents as Teachers (PAT) program revealed that the effects on low income parents and children were both modest and inconsistent, similar to findings from other evaluations of two-generation programs, especially home visiting programs. This large-scale evaluation employed a technically solid design that was executed as well as possible given the constraints of attrition of both a program and participating families. I offer these comments to expand our thinking on the challenges of evaluation, but they are not meant to detract from the demanding work of Wagner and her colleagues.
As these authors state, policy directives have called for evaluations of programs designed to serve families parenting children who are especially at risk (e.g., PAT; Gomby, Culross, & Behrman, 1999). There is little doubt that this and similar evaluations in the past have provided valuable information regarding the potential effects, as well as limitations, of a variety of programs. Unfortunately, what many of these large-scale evaluations fail to adequately provide is the kind of information needed to help practitioners and researchers understand the intricacies of the programs and the participants or their interactions with each other. Without these details, we are often left trying to interpret the evaluation data feeling “removed from” the program.
If we are to meet the challenge of enhancing developmental outcomes and educational achievement (Forgione, 1999) for all young children, program evaluation work must move beyond simply measuring intervention outcomes to the equally important task of helping us understand the complex interactions between family and child characteristics, intervention goals addressed, processes employed by programs, and outcomes. Questions that need systematic attention include the following:
1. What intervention goals are established for specific programs and/or for individual participants?
2. What intervention should be delivered to address those goals?
3. How can the intervention be delivered most effectively?
Consider the theory of change (Weiss, 1995) as a guide to design, implement, and evaluate an intervention program.
THEORY OF CHANGE
Today, many large-scale intervention programs employ comprehensive intervention strategies to address a wide range of child and family risk factors. The rationale for this two-generation program approach is that targeting parent variables will increase parental competence and confidence and, therefore, have a greater likelihood of offering long-term benefits than interventions aimed only at children (Smith & Zaslow, 1995). Like many two-generation programs, the PAT program targets goals directly related to child outcomes, such as knowledge of child development and a variety of parenting behaviors. However, the PAT program also seeks to enhance child outcomes through more indirect, parent-focused, psychological routes, such as assisting parents in building confidence and developing home–school–community partnerships. Other two-generation programs target even more indirect routes to influence child outcomes, such as enhancing parental educational attainment and/or psychological well-being.
Within any intervention program, it is important to identify a theory of how and why an initiative works (Weiss, 1995). This theory of change could guide intervention implementation by challenging program designers to identify the pathways by which intervention services are expected to enable intended outcomes and directing resources to support those activities. This type of careful planning is especially important for home visiting programs like the PAT that provide very individualized services. In turn, clear articulation of program goals and pathways for change can guide systematic selection of measures for assessing program implementation integrity and thus enable coherent study of the links between intervention activities, contexts, and outcomes (Connell & Kubish, 1999). Evaluation activities that account clearly for intervention implementation capture the spirit of second-generation research efforts suggested by Guralnick (1997) when he challenged the field of early intervention to move from asking, “Does this intervention work?” to examining how specific program features are associated with child and family outcomes. Guralnick argued that this type of information could inform practice and provide a research framework for evaluating innovative approaches by advancing our understanding of an intervention’s operational mechanisms. Attention to the questions outlined above could move the field toward conducting second-generation research.
Establishing Intervention Goals
Established goals for the PAT program are quite well articulated; however, similar to other two-generation programs (e.g., Early Head Start, Healthy Families America), the established goals are broad and do not prescribe particular intervention strategies. Broad goals are consistent with family systems theory, which supports the notion that the entire family should be targeted for intervention (Turnbull & Turnbull, 2001), as well as ecological theory that holds that child development is best facilitated when all family members have the instrumental and psychological supports necessary to function in a healthy manner (Bronfenbrenner, 1977). Lack of preprescribed intervention strategies is consistent with a family-centered approach to early intervention services, which holds that practitioners and family members are equal partners in intervention and should collaborate to identify goals and strategies most likely to capitalize effectively on family strengths and support growth in prioritized areas (Dunst, Trivette, & Deal, 1994; McWilliam, Winton, & Crais, 1996).
The notion that the PAT program will be individualized to meet participants’ needs seems apparent. First, participation is generally voluntary (as it was for participants in the evaluation reported in this issue). Curricular materials have been adapted to better address the needs of parents who differ in such characteristics as being teenagers, parenting children of certain ages, or parenting children enrolled in childcare centers. The current evaluation report does not provide sufficient information to allow understanding of why individual families chose to participate in the PAT program, how they prioritized goals in their efforts to enhance their parenting competence, or their satisfaction with how well the PAT Program assisted them in achieving established goals.
Intervention Strategies to Address Identified Goals
Establishing goals is the first step in any intervention effort, but this step rarely dictates only one strategy to address those goals. For example, two parents working to learn appropriate ways to foster their children’s growth and learning could participate in very different intervention activities. One parent might need child development information that an interventionist could share via media presentation, conversation, or written information. Another parent might have sufficient knowledge about child development but find interacting with her child awkward, suggesting the interventionist could coach her through parent–child interactions that allow her to practice new skills with support.
Again, it seems apparent that the PAT program has responded to the need for individualized intervention services. The PAT program has developed materials, a recommended implementation framework, and training supports. Although a certain level and pattern of services is recommended (e.g., monthly home visits, periodic screenings), the assumption that intervention is intended to be individualized seems clear. For example, home visitors are directed to schedule visits at the parents’ convenience, involve parents in developmentally appropriate activities, and respond to parents’ questions; one goal of parent group meetings is to build informal support networks; and families are referred to needed community services. Such flexibility is a potential strength of the PAT program but hinders clear understanding of intervention goals and strategies essential to evaluate program outcomes.
Delivering Intervention Effectively
Intervention strategies designed to address identified goals can often be delivered in a variety of service models and/or settings, including home visits, which may be an ideal setting for implementation of a variety of intervention strategies. Home visiting allows interventionists the unique opportunity to understand the individual needs of young children and their families within the contexts of their natural environments and tailor services to address families’ identified needs in direct and immediate ways (McWilliam et al., 1996; Powell, 1993). Contrary to current discussion of home visiting interventions, home visits must be seen as an intervention setting rather than the intervention itself.
This broadened perspective will allow the field to think about implementing intervention services designed to address multiple goals delivered via a variety of intervention strategies and within the context of numerous settings. In turn, evaluation efforts can be targeted appropriately to examine how various intervention components contribute to overall intervention outcomes. Much work remains if we are to use a theory of change (Weiss, 1995) to guide intervention and evaluation activities and conduct second-generation research (Guralnick, 1997).
Intervention and Evaluation Contexts
Most, if not all, interventions and accompanying evaluation efforts occur within competing social and political contexts. Large numbers of children in our country live in poverty, and abundant evidence shows that such circumstances put these children at significant risk for poor developmental outcomes (see Duncan & Brooks-Gunn, 1997; Wagner et al., this issue, for more information). This compels our society to find the means to ameliorate the effects of growing up in such challenging, and potentially compromising, circumstances. The need is broad, the challenge is great, and responses have been varied.
Both state and federal government policies support children indirectly by assisting their families in providing for their care. Government policies mandate family leave benefits, at least for some employers; provide tax relief to families raising children, especially families who remain poor despite working; provide support for basic needs and childcare services; and provide support designed to enhance parents’ abilities to participate in the workforce (Knitzer, 2001). Other programs designed to provide developmental support for children, via education and support to parents through home visits or caring for children directly, have proliferated with investment of both public and private dollars (Gomby et al., 1999). Many of these efforts have been accompanied by mandates to evaluate program effects, and evaluation findings have been disseminated broadly and scrutinized carefully.
The PAT programs participating in the current evaluation illustrate some of the implementation and evaluation challenges faced by both practitioners and researchers working within these contexts. One program dropped out before evaluation started, but more pertinent is that large numbers of families left their programs prior to their children’s third birthdays. The current evaluation, like many from the past, does not provide sufficient information to help us understand if these families left because the program failed to address goals they deemed important, the intervention strategies employed seemed ineffective, or something about the service delivery itself (e.g., activity schedule, intervention setting[s]) made them uncomfortable. Clearly, many families voted with their feet and discontinued participation. Although this compromises evaluation efforts, it highlights how specific information about the relationships between intervention processes and outcomes could assist practitioners by focusing on matching key intervention components, strategies, and settings with individual family strengths and needs, thus increasing the likelihood of intervention effectiveness.
Current evaluation efforts focused on examining how programs that use home visiting to deliver services (e.g., Wagner et al., this issue) reflect the current political context in a number of ways outlined by Weiss (1975). The programs reflect recognition of the need to support children’s early development. Evaluation findings are reported in political, as well as scientific, arenas and have been influential in the origin of some programs (e.g., Early Head Start) and the systematic refinement of others (e.g., Nurse Home Visitation Program; Olds et al., 1999). Both political and scientific scrutiny of efforts to evaluate home visiting programs seem apparent in questioning the efficacy of home visiting programs to affect the positive development of children (Gomby et al., 1999) and accompanying recent recommendations for direct services to children (e.g., childcare and preschool programs), in addition to family support (Ramey & Ramey, 1998). The evaluation of the PAT program reported in this issue could certainly be seen as adding further support to those recommendations. However, this approach may be avoiding the difficult task of helping parents interact with their children in ways that positively influence their development. And, without high-quality childcare programs available to all families (Phillips & Adams, 2001), we may still be disappointed with outcomes for children. It is highly likely that many families may need support from an array of programs equipped to implement interventions across multiple settings to facilitate their children’s healthy development.
The challenge of supporting the optimal development of all children is huge; we need to capitalize on the effectiveness of all available intervention strategies and programs. Designers, implementers, and evaluators of all intervention programs should think more carefully about the specific theory of change guiding the program, operationalizing that theory of change by developing effective intervention strategies to target identified goals, delivering the intervention with fidelity, and doing careful analyses of what actually occurred during the intervention process. Only then will evaluation efforts yield results that can effectively guide implementation efforts, clarify research directions, and inform policy on behalf of children and families.
REFERENCES
Bronfenbrenner, U. (1977). Toward an experimental ecology of human development. American Psychologist, 32, 515-531.
Connell, J. P., & Kubish, A. C. (1999). Applying a theories of change approach to the evaluation of comprehensive community initiatives: Progress, prospects, and problems. In K. Fulbright-Anderson, A. C. Kubisch, & J. P. Connell feds.), New approaches to evaluating community initiatives: Theory, measurement, and analysis. Washington, DC: Aspen.
Duncan, G. J., & Brooks-Gunn, J. feds.). (1997). Consequences of growing up poor. New York: Russell Sage Foundation.
Dunst, C. J., Trivette, C. M., & Deal, A. G. (Eds.). (1994). Supporting and strengthening families: Methods, strategies, and practices. Cambridge, MA: Brookline Books.
Forgione, P. (1999). Achievement in the United States: Are students performing better? Committee on Education and the Workforce, U.S. House of Representatives. Retrieved from http://www.nces.ed.gov/ Pressrelease/housetest2.html
Gomby, D. S., Culross, P. L., & Behrman, R. E. (1999). Home visiting: Recent program evaluations–analysis and recommendations. The Future of Children, 9, 4-27.
Guralnick, M. (1997). The effectiveness of early intervention. Baltimore: Brookes.
Knitzer, J. (2001). Federal and state efforts to improve care for infants and toddlers. The Future of Children, 11, 79-98.
McWilliam, P. J., Winton, P. J., & Crais, E. R. (1996). Practical strategies for family-centered intervention. San Diego, CA: Singular.
Olds, D. L., Henderson, C. R., Kitzman, H. J., Eckenrode, J. J., Cole, R. E., & Tatelbaum, R. C. (1999). Prenatal and infancy home visitation by nurses: Recent findings. The Future of Children, 9, 44-65.
Phillips, D., & Adams, G. (2001). Child care and our youngest children. The Future of Children, 11, 35-52.
Powell, D. (1993). Inside home visiting programs. The Future of Children 3, 23-38.
Ramey, C. T., & Ramey, S. L. (1998). Early intervention and early experience. American Psychologist, 53, 109-120.
Smith, S., & Zaslow, M. (1995). Rationale and policy context for two-generation interventions. In S. Smith (Ed.), Two generation programs for families in poverty: A new intervention strategy. Norwood NJ: Ablex.
Turnbull, A. P., & Turnbull, H. R. (2001). Families, professionals, and exceptionality: Collaborating for empowerment (4th ed.). Upper Saddle River, NJ: Prentice Hall.
Weiss, C. H. (1975). Evaluation research in the political context. In E. L. Struening & M. Guttentag (Eds.), Handbook of evaluation research (Vol. 1). Beverly Hills, CA: Sage.
Weiss, C. H. (1995). Nothing as practical as good theory: Exploring theory-based evaluation for comprehensive community initiatives for children and families. In J. Connell (Ed.), New approaches to evaluating community initiatives: Concepts, methods, and contexts. Washington, DC: Aspen.
Address: Carla A. Peterson, Dept. of Human Dev. & Family Studies, 58 LeBaron Hall, Iowa State University. Ames, IA 50011.
Carla A. Peterson
Iowa State University
COPYRIGHT 2002 Pro-Ed
COPYRIGHT 2003 Gale Group
