Utilization-focused evaluation of a family preservation program
Smith, Mieko Kotake
THE FEDERAL ADOPTION ASSISTANCE and Child Welfare Act of 1980 (Public Law 96-272) requires states to maintain at-risk children in the least restrictive environment possible. The act provides financial support for short-term, home-based, intensive-intervention child welfare services (Cimmarusti, 1992). Congress based this act on widely acknowledged values: protection of the parent–child bond from unnecessary state intervention (Goldstein, Freud, & Solnit, 1973), the significance of the biological tie to the child’s identity (Laird, 1979), and the provision of intervention in the client’s environment (Whittaker, Schinke, & Gilchrist, 1986). Practitioners implementing this policy endeavor to keep children with their families, using out-of-home placement as a last resort. Cole and Duva (1990) state three goals of family preservation services:
(1) to keep the family safe; (2) to avoid unnecessary placement of children in substitute care and the consequent high human and fiscal cost; and (3) to improve family functioning so that the behavior that led to the crisis will be less likely to occur (p. 1).
Most family preservation services are delivered in a family-centered and home-based fashion. For example, Frankel (1988) reported that at least 180 home-based family preservation programs exist nationwide. Hodges and Blythe (1992) listed the advantages of home-based service delivery to high-risk families compared with office-based services. First, home-based services tend to produce more accurate and sensitive assessments, allowing practitioners to interact with family members in their natural setting. Second, home-based treatment improves the efficacy of outreach work. Third, home-based treatment facilitates any necessary modifications of the environment and behavioral adjustments in family members.
Child and family welfare professionals and practitioners, however, continue to debate the effectiveness of family preservation programs. Although initial evaluations of various programs have reported some positive results (Barth & Berry, 1987; Frankel, 1988; Pecora, Fraser, & Haapala, 1992), critical questions about their effectiveness remain. For example, reviewing three experimental studies in intensive family preservation services, Wells and Biegel (1992) concluded that, although the programs can delay out-of-home placement of at-risk children, the effects of intensive family preservation services are relatively short lived. Wells and Biegel also suggested that family preservation programs must be targeted only to certain types of families and underscored the importance of the functional interaction of children and families and effects of the family preservation service workers on them. They further emphasized the need for additional empirical data on the assumptions underlying the programs as well as on the long-term effects of such interventions.
Earlier, Whittaker and Tracy (1988) questioned the value of using the simple criterion of absence of placement at 60 or 90 days subsequent to the end of service as an adequate measure of success. They recommended that more standardized measures of family and child functioning in different settings be developed. For example, Feldman (1992) found that parental attitude at termination predicts future placement of a child. Another factor associated with positive outcomes is the degree of goal attainment by parents (Lewis, 1992). Another study reported that home-based services may be more effective with families with adolescents because teens tend to feel more relaxed in their home environment (Hodges Blythe, 1992). A recent evaluation of a family preservation program (Berry, 1992) provided comprehensive information about families and children, including placement outcomes, time spent with a family, specific services provided, environments conducive to child neglect, and program cost. The study concluded that short-term but intensive family preservation services appear to achieve a lasting improvement in families, suggesting the significance of parenting-skills improvements in avoiding out-of-home placement.
Despite the debates over definitions and criteria for appraising success, everyone agrees that the primary goal of home-based services is to prevent out-of-home placements. However, reported rates of avoiding placement vary greatly. The Homebuilders Program reported that 90% of participating families avoided outside placement at the end of one 16-month program (Kinney, Madsen, Flemming, & Haapala, 1977), and in another program, 88% of participating families remained intact after one year (Behavioral Sciences Institute, 1986). Nationally, the rate of preventing placement varies from 40% to 95% (Haapala & Kinney, 1988; Hinckley & Ellis, 1985; Pecora et al., 1992; Callister, Mitchell, & Tolley, 1986).
This article describes a pilot family preservation intervention undertaken by a county child welfare agency as part of its efforts to prevent repeated child neglect or abuse and presents the findings with regard to the effects of the program. The article identifies some criteria for successful family preservation programs as well as describes how a utilization-focused approach was used for evaluating the pilot family preservation program.
Family Preservation Program
The program under study sought to prevent out-of-home placements of children by preserving the family unit and by improving the family’s coping ability and parenting skills. Without the families’ participation in the program, removal of children would have been imminent. Accordingly, the program provided the families with children at risk for placement with clearly defined, concentrated services for a maximum period of 90 days. During these 90 days, a social worker managing two or three families maintained daily contact with each family, provided family counseling, and imparted family-functioning skills. The family’s level of functioning in various predefined behaviors was assessed at the time of admission to the program and at its conclusion. The social workers contacted all the families every month for three months afterward to determine whether the children were still at home.
The program’s target population included families with children ranging from newborn to 18 years who were identified as being at risk of placement within one month. More specifically, the program targeted families in crisis because of significant mental health, medical, or behavior problems of a family member. Families with mental health problems affecting a parent’s coping abilities entered the program if a parent was actively involved in a treatment program or expressed a willingness to seek assistance. The staff remained alert to parents with significant medical problems and inadequate resources, situations known to encourage child abuse, neglect, or substance-dependency situations. Moreover, children with significant medical problems often create stress for caretakers, which may in turn lead to abuse or neglect. Drug or alcohol dependence or chronic conflict often leads to an inability to cope and thus to abuse or neglect within the family. When staff found a family member with a drug or alcohol problem, they required the individual to enter a treatment program.
The agency considered families of the following three types to be inappropriate for program participation: (1) families in which a parent had killed or maimed a child or in which the parents’ rights had been terminated on a previous referral and whose circumstances had not altered, (2) families with a long history of chronic, ongoing problems, and (3) families previously involved in an intensive home-based program that had produced no discernable change. The agency considered these families in need of comprehensive protective services rather than preventive intervention. When referrals to the family preservation program exceeded the number of openings, the agency applied the following four admission criteria in the order listed: (1) urgency, based on how quickly children were scheduled for removal from the setting (24, 48, or 72 hours); (2) degree of risk, based on the agency’s risk-assessment procedures; (3) age of the child–the younger the child, the greater the threat; and (4) the family’s acknowledgment of problems and its willingness to receive services. To families showing a great urgency and risk but no recognition of the problems or an unwillingness to receive services, workers explained that the only alternative would be out-of-home placement. When families confronted this alternative, they generally agreed to enter the family preservation program.
The major intervention components included case management, casework counseling, family therapy, and an educational agenda. Each service was delivered at the family’s home by a social worker who became the family’s friend, teacher, and link with the agency and other community organizations. In the course of daily visits and frequent telephone conversations, the families developed a strong rapport with their social worker. During home visits, the social workers taught adults how to maintain a safe home environment for children and such homemaking skills as budgeting, careful shopping, meal planning, meal preparation, and housekeeping. The social workers also inculcated general child-development principles and effective discipline techniques. A worker interacted with the entire family, encouraging its members to participate in activities together and to take pride in meeting individual responsibilities. The family members also learned alternative methods for resolving conflicts and skills to improve their interactions. Because intensive services like these require highly developed professional skills, only those social workers possessing the requisite skills were assigned to this unit. They had an average of 6.8 years experience in family intervention.
Study Procedures
Following their first meetings with each family, the social workers completed a family preservation unit assessment sheet, a preliminary evaluation of the overall level of family functioning. At the completion of 90 days of services, the workers completed the same assessment form again. Although measurement errors may have arisen as a result of bias, in that social workers providing services also rated “their” families, the usefulness of this evaluation component as part of the social work intervention appeared to outweigh this liability. The agency based this judgment on the appropriateness, utility, practicality, credibility, and relevance of utilization-focused evaluation (Patton, 1990).
The family preservation unit assessment sheet addressed a constellation of objectives. First, the staff members listed and ranked areas of concern about family functioning and parenting skills. Second, the staff and researcher developed questions designed to assess levels of achievement in those areas. Third, the staff composed its instrument to capture the level of critical aspects of functioning commensurate with a safe and healthy environment for children. Accordingly, the assessment sheet broached such areas as money management; marital relationship; family communication; use of support systems; relationship building; home management; and child supervision, nurturance, and expectations.
The staff members assessed money management by comparing the amount and sources of income with itemized expenses. Marital relationship involved the nature and frequency of marital conflict. They measured family communication by reported frequencies of verbal communication among family members. Questions concerning family-support systems included whether the family members had friends they could call upon if needed as well as whether the family was involved in church or social groups. Ratings on relationship building were based on evident affectionate physical contact and participation in household duties among family members as observed by the social worker. Home management was also measured by the social worker’s observation of the condition of the home and meal-preparation procedures. The social workers based assessment of the parents’ child supervision and nurturance on answers to items such as “Do parents know where their children are and “Do parents display affection toward children?”
The agency considered it essential to involve staff members in the development of the evaluation instrument. This involvement increased staff members’ belief in the data (Patton, 1990). Also, staff members’ understanding of the meaning of their evaluations benefited from their deciding what data to collect and how to collect, analyze, and apply the data. Patton (1990) believes that the value of an evaluation study such as this lies in its extent of utilization, stating that “utilization occurs when there is an immediate, concrete, and observable effect on specific decisions and program activities resulting directly from evaluation findings” (p. 30). Thus, in this utilization-focused evaluation, face validity holds great value, although an instrument’s predictive validity, concurrent validity, and construct validity are unquestionably critical. This process is also vulnerable to bias from workers’ desire to rate a family’s functioning more favorably after intervention, a bias that could affect the reliability of the measure. The agency chose, however, to accept this risk in order to obtain more extensive staff involvement in the project and great use of its findings for program refinement.
Findings
Description of Participants
Twenty-six families participated during the three-month pilot period. Of those 26 families, 18 were white, 7 were black, and one family racially mixed. The children’s ages ranged from newborns to 14 years, with an average age of 3 years. The number of children per family ranged from one to four, with an average of 2.3. The age of the parents ranged from 16 to 45, with an average age of 26.3. Twenty families had two parents; 14 had only a mother. In eight mother-only families the mother’s boyfriend was a household member.
In the majority of the white families (56%), both parents were present. The majority of the black families (71%) included the mother and her boyfriend.
Money Management
Of the 26 families, 17 (65%) were receiving Aid to Families with Dependent Children; 16 of the 17 also received food stamps. Only five families had income from jobs; their average monthly incomes amounted to $1,437. The average monthly income of all the families was $915, an amount that translates into a yearly income of $10,980. Their monthly expenses for such essentials as housing, utilities, food, clothing, and transportation averaged $571. Thus, most of the families participating in the program lived in poverty; no significant changes in income or expenses occurred during the three-month intervention period. The number of income sources did, however, increase slightly during this period, probably because the social workers helped some families secure additional public assistance or child support. For example, whereas only four families were receiving Supplemental Security Income at pretest, six families received this benefit at posttest.
Marital Relations
Both male and female adults were present in 20 families. Of those, the majority (n = 15, 75%) reported fighting between mates at both pre- and posttests (Table 1). (Table 1 omitted) Most couples reported that they had been fighting for as long as they had been together, mostly over their relationship and their children. But despite this acknowledgment of persistent marital conflict, they all reported that they loved each other, and when asked what their fights “looked like,” their responses revealed differences in the nature of their fights before and after the intervention. Six families reported fighting between mates daily, and five couples reported weekly fights at pretest. At posttest, the frequency of marital fights shifted to weekly fights (n = 11) with no daily fights reported. One might infer, therefore, that the social workers helped the couples learn and use alternative methods of conflict resolution in the family, such as pausing before reacting and writing down feelings for later discussion.
When asked at pretest what the couples “do together,” eight couples reported that they did nothing together. At posttest, only two couples reported that they did nothing together (Table 2). (Table 2 omitted) The intervention included encouragement for couples to go out for dinner or movies, and at posttest more couples reported outings as one of the ways they spent time together. In addition, reports of other activities such as housework, talking on the phone, and sexual contact increased at posttest. Thus, the program appeared to have helped the couples change their behavior from confrontational modes to cooperative interaction with increased communication.
Family Communication
The program’s intensive coaching encouraged the family members to ask one another daily how their day had gone. The amount of communication significantly increased during the intervention, as measured by frequency of communication and number of subjects discussed. At pretest, only 14 families (54%) indicated that family members talked to one another. At posttest, all but one family reported that members talked to one another. At pretest, only 14 families communicated daily. That number increased to 21 families (81%) at posttest. (The subjects of discussion within the family at pretest and posttest appear in Table 3.) (Table 3 omitted) Thus, it appears that the intervention helped to increase family communication overall.
Support System
The number of families reporting that they had friends to call upon for support increased from 17 at pretest to 23 at posttest. Although this increase was not statistically significant, the families had evidently responded well to encouragement to join groups such as churches, young mothers’ groups, and Alcoholics Anonymous. Three parents reported that they had joined such school-related groups as a parent-teachers association at posttest, whereas no families had reported participation in such groups at pretest.
Relationship Building
To promote relationship building, social workers coached the families in relaxation exercises, including physical touching and listening skills. The family members received instructions to chart the chores and responsibilities assigned to each and to discuss their progress among themselves. As Table 4 shows, significant behavior changes occurred. (Table 4 omitted)
Home Management
The staff also worked with the families to improve housecleaning, minor maintenance, and shopping skills. In fact, social workers actually engaged in these home-management activities with the families, and cleanliness and general conditions in the home improved from pretest to posttest (Table 5). (Table 5 omitted) For example, the number of homes rated as good or excellent on cleanliness increased from 11 at pretest to 15 at posttest. Fourteen households at pretest and 16 at posttest were in good or excellent condition. In rating the general condition of their homes, social workers paid special attention to the safety of stairs and the condition of the plumbing, electrical wiring, and windows. Although the changes were not statistically significant, the program appeared to encourage families to improve their home environment. Moreover, significant changes did occur in meal preparation and food supplies.
Supervision of Children
Most observers associate adequate supervision of children with the prevention of child neglect or abuse. Consequently, social workers spent a lot of time teaching adults principles of child development, discipline techniques, and parental responsibilities. They explained how much independence and dependence one can expect from children of various ages and what the appropriate disciplining techniques are, including “time-out” and withdrawal of privileges. The social workers also helped their families organize their schedules and use a calendar to chart medical appointments and other family activities. Such intensive in-home coaching assumed a certain intellectual capacity and openness to behavioral change in the families; as Table 6 shows, coaching produced favorable responses. (Table 6 omitted)
Follow-up
Of 26 families that entered the program, 25 completed it. One family moved out of town before the 90-day period ended. A child in one family that completed the program was nonetheless placed in a foster home. Four families (18%) were referred to the Protective Services Department for further support services at the conclusion of the program. After 90 days, however, 24 families (96%) still had their children at home. At two months after the program’s conclusion, a child of one family was removed and another family was referred for protective services. Three months after the program, the children of the four families receiving protective services remained with their parents. At that time, 23 families (92%) were still intact. Without the intervention, most children would have been removed from home for temporary foster care, interrupting family-child relations. This figure (92% at follow-up) is in the upper range of success rates (40%-96%) nationwide for family preservation programs. Unfortunately, we have no comparative data on placement among families that have not received such intensive intervention.
Thus, the family preservation program enjoyed success in providing basic knowledge and essential skills conducive to a nurturing environment for the children of the families referred to the agency for child neglect or abuse. e families exhibited significant positive changes in all catalogued areas of family functioning and parenting behavior. These improvements appeared to contribute to the preservation of the family unit.
Conclusion and Discussion
The two major components of the family preservation program included intensive interaction with the same social worker and an ongoing educational regimen. The social workers appealed to the families’ cognitive capacity to learn the stages of child development and effective principles of family communication and child rearing. They coached the family members on how to change their communication patterns and child-rearing behavior on the basis of their cognitive learning. The workers physically demonstrated how to maintain a clean and safe environment and helped their families in six areas:
* Marital relationships improved, with a reduction in frequency of marital fights in general and physical aggression in particular.
* Family communication improved, with an increase in communication frequency and number of topics of conversation discussed among family members.
* The community support network gained in influence, with families increasing their participation in social groups such as parent-teacher organizations and churches.
* Intrafamily relationships improved, with closer physical contact, more frequent verbal communication, and the sharing of household chores.
* Home-management skills improved, as indicated by cleaner homes, better meal preparation, and improved shopping skills.
* Child-care behavior greatly improved, with parents showing more appropriate expectations and discipline techniques as well as increased attentiveness to the needs of their children.
One may attribute the apparent success of the program to three factors. First, the agency applied purposeful criteria for admitting families to this intensive family preservation program. Although the urgency and degree of risk for children appeared to be compelling, child neglect or abuse in most of the 26 families studied had been brought to the agency’s attention for the first time. Only families that recognized their problems, indicating cognitive readiness, were admitted. They also expressed willingness to receive services from a social worker. As Wells and Biegel (1992) emphasize, one should always first consider functioning of children and families to determine their potential for achieving success in a family preservation program. They, however, offer no specific characteristics of families that may benefit from intensive family preservation interventions. In light of the lack of confirmed, specific family characteristics, the admission criteria used in this program suggest appropriate characteristics of families that might benefit from family preservation programs. Such intensive interventions as in this program may be suited for families that have been brought into the child welfare system for the first time, that somehow recognize the gravity of child neglect or abuse, and that are willing to work with the assigned social worker, even if their acceptance of assistance occurs under the threat of a child’s removal. Further studies with comparison groups of families evidencing different characteristics may confirm the usefulness of the criteria described here and strengthen the list of criteria still further.
Second, this program involved intensive intervention by social workers with proven skills in working with families. Again, Wells and Biegel (1992) state that the roles of family preservation workers should be carefully planned for successful family preservation programs. Wells and Biegel did not, however, indicate how such programs should be planned. In this program, the social workers skillfully helped the families to learn interpersonal communication skills, use of community resources, and home-management and parenting skills, with a heavy emphasis on the family’s understanding of its importance as a unit. Ideally, what the families learned should promote continuing improvements in the physical and social environments of their children. Thus, the findings of this study suggest that to implement successful family preservation programs, social workers must have a wide range of skills, including case management, and skills to facilitate cognitive and behavioral learning. Undergraduate and graduate social work programs must teach such skills.
Third, the program included follow-up on families by the same social workers who worked with the families. Moreover, the program provided a sense of the supportive social network available to the families after leaving the program. The families contacted their social workers when they realized their tolerance was about to reach its threshold or when they wished to invite the social worker to a child’s graduation or birthday party. The agency continues to interpret “program termination” loosely as part of its goal to prevent the recurrence of child neglect or abuse. These findings offer a challenge to definitions of “open” and “closed” cases in the child welfare system, because such definitions carry financial implications. One might debate whether a social worker’s contact with officially “closed” families constitutes billable services.
Although 92% of the families still had their children six months after the initial referral to child protective services, a long-term follow-up is needed to evaluate the program’s lasting effects in preserving families. Nonetheless, the results of this pilot program suggest that families with certain characteristics can gain short-term benefits from an intensive family preservation program and that preventing out-of-home placement of children in such families with a labor-intensive program may be a cost-effective strategy. This study’s findings may be useful to child welfare practitioners in refining their family preservation programs. Also, child and family services researchers are encouraged to use these findings to pursue carefully designed studies of family preservation programs in order to further define the most effective approaches for families with different characteristics.
Besides the positive outcomes observed in this study, another positive effect of the evaluation process emerged. Through the processes of instrument development and data collection, the agency’s social workers became increasingly aware of the objectives of the program and the need for objective methods of measuring its success. The instrument provided the social workers with a tool to maximize their objectivity and consistency in conducting ongoing assessments of family functioning in predetermined areas.
Thus, this study demonstrated the value of a utilization-focused approach to evaluation, suggesting ways to strengthen the functions of evaluation methods in monitoring social work practice with the maximum participation of practitioners. The study emphasized the need for the collaboration of social work practitioners and researchers to maximize the effect of utilization-focused evaluation.
References
Barth, R., & Berry, M. (1987). Outcomes of child welfare services under permanency planning. Social Service Review, 61, 71-87.
Behavioral Sciences Institute. (1986). Homebuilders cost effectiveness with various client populations, 1974-1985. Federal Way, WA: Behavioral Sciences Institute.
Berry, M. (1992). An evaluation of family preservation services: Fitting agency services to family needs. Social Work, 37, 314-321.
Callister, J. P., Mitchell, L., & Tolley, G (1986). Profiling family preservation efforts in Utah. Children Today, 15(6), 2-25.
Cimmarusti, R. A. (1992). Family preservation practice based upon a multisystems approach. Child Welfare, 71, 241-255.
Cole, E., & Duva, J. (1990). Family Preservation. Washington, DC: Child Welfare League of America.
Feldman, L. H. (1992). Assessing the effectiveness of family Preservation services in New Jersey within an ecological context. Unpublished doctoral diss., Department of Social Work, Rutgers University.
Frankel, H. (1988). Family-centered, home-based services in child protection: A review of the research. Social Service Review, 62, 137-157.
Goldstein, J., Freud, A., & Solnit, A. (1973). Beyond the best interest of the child. New York: Free Press.
Haapala, D. A., & Kinney. J. M. (1988). Avoiding out-of-home placement of high-risk status offenders through the use of intensive home-based family preservation services. Criminal Justice and Behavior, 15, 334-348.
Hinkley, E. C., & Ellis, W. F. (1985). An effective alternative to residential placement: Home-based services. Journal of Clinical Child Psychology, 14, 209-213.
Hodges, V. G., & Blythe, B. J. (1992). Improving service delivery to high-risk families: Home-based practice. Families in Society, 73, 259-265.
Kinney, J. M., Madsen, B., Flemming, T., & Haapala, D. A. (1977). Homebuilders: Keeping families together. Journal of Consulting and Clinical Psychology, 45, 667-673.
Laird, J. (1979). An ecological approach to child welfare: Issues of family identity and continuity. In C. B. Germain (Ed.), Social work practice: People and environments (pp. 174-209). New York; Columbia University Press.
Lewis, R. E. (1992). Service-related correlates of treatment success in intensive family preservation services for child welfare. Unpublished doctoral diss., Department of Social Work, University of Utah.
Patton, M. Q. (1990). Utilization-focused evaluation (2nd ed.). Newbury Park, CA: Sage Publications.
Pecora, P. J., Fraser, M. W., & Haapala, D. A. (1992). Intensive home-based family preservation services: An update from the FIT project. Child Welfare, 71, 177-188.
Wells, K., Biegel, D. E. (1992). Intensive family preservation services research: Current status and future agenda. Social Work Research and Abstracts, 28(1), 21-27.
Whittaker, J. K., Schinke, S. P., Gilchrist, L. D. (1986). The ecological paradigm in child, youth and family services: Implications for policy and practice. Social Service Review, 60, 483-503.
Whittaker, J. K., & Tracy, E. M. (1988). Family preservation services and education for social work practice: Stimulus and response. In J. K. Whittaker, J. Kinney, E. M. Tracy, & C. Booth (Eds.), Improving practice technology for work with high-risk families: Lessons from “Homebuilders” social work education project. Seattle, WA: Center for Social Welfare Research, School of Social Work, University of Washington.
Mieko Kotake Smith is Associate Professor, Department of Social Work, Cleveland State University, Cleveland, Ohio. The author thanks Joseph White, Thomas P. Holland, and the Family Preservation Program staff members.
Copyright Family Service America Jan 1995
Provided by ProQuest Information and Learning Company. All rights Reserved.
