The Prevention Services Clearinghouse defines a “study” as one research investigation of a defined subject sample, and the interventions, measures, and statistical analyses applied to that sample. To be eligible for review for the Prevention Services Clearinghouse, studies must meet all of the eligibility criteria described below.
4.1.1 Date of Publication
Studies must be published or prepared in or after 1990. For studies whose results are reported in multiple documents, the earliest available document must be published or prepared in or after 1990.
4.1.2 Source of Publication
Studies must be publicly available and published in peer-reviewed journals or in reports prepared or commissioned by federal, state, or local government agencies or departments, research institutes, research firms, foundations or other funding entities, or other similar organizations. Dissertations, theses, and conference papers are not eligible.
4.1.3 Language of Publication
Studies must be available in English.
4.1.4 Study Design
Studies must use a randomized or quasi-experimental group design with at least one intervention condition and at least one comparison condition. Intervention and comparison conditions may be formed through either randomized or non-randomized procedures and the unit of assignment to conditions may be either individuals or groups of individuals (e.g., families, providers, centers). Eligible intervention and comparison conditions are defined as follows:
- Intervention Condition. The intervention group(s) must receive a program or service that is essentially the same for all of the participants in the group (i.e., there may be variation across individuals in what they receive but distinctly different interventions should not be applied to different subsamples that are aggregated into a single study sample).
- In a study with multiple intervention groups, reviewers determine the eligibility of each intervention based on the Program or Service Eligibility Criteria (Section 2.1). If all intervention groups are eligible, they can be reviewed and compared to the same comparison group.
- Comparison Condition. Comparison groups must be “no or minimal intervention” or “treatment as usual” groups. Minimal intervention group members may receive handouts, referrals to available services, or similar nominal interventions. “Treatment as usual” group members may receive services, but those services must be clearly described as the usual or typical services available for that population in the study. Studies that compare one intervention to a second intervention are not eligible for review, even if the second intervention is not eligible under the Program or Service Eligibility Criteria (Section 2.1).
- In studies with multiple comparison groups, reviewers select one comparison instead of comparing the same intervention group to multiple comparison groups. Selection of comparison group is based on the group that receives the least intensive services in order to maximize the treatment contrast.
4.1.5 Target Outcomes
Studies must measure and report program or service impacts on at least one eligible target outcome. Eligible target outcomes differ by program or service area and are defined as follows:
Eligible Outcomes for Mental Health, Substance Abuse Prevention and Treatment, and In-Home Parent Skill-Based Programs and Services
- Child Safety. Child safety refers to a current condition within a home or family and considers whether or not there is an immediate threat of danger to a child. A threat of danger refers to a specific family situation that is out of control, imminent, and likely to have severe physical, psychological, and/or developmental effects on a child. Eligible indicators of child safety for the Prevention Services Clearinghouse pertain to both child maltreatment and risk of maltreatment and include:
- Evidence of substantiated or unsubstantiated child maltreatment from administrative records.
- Injuries or ingestions taken from medical records of encounters with health care providers.
- Measures that assess neglectful, psychologically aggressive, or abusive parenting behavior.
- Child Permanency. Child permanency refers to the permanency and stability of a child’s living situation (in-home or in foster care) and includes the continuity and preservation of family relationships and connections. Eligible indicators of child permanency for the Prevention Services Clearinghouse include:
- Length of placements, placement disruptions, stability or permanency of placements, reunification, and use of kinship care.
- Eligible sources of this information may be reports from child welfare, juvenile justice, or similar administrative databases, including Child and Family Services Reviews. Studies may also obtain placement information from therapist, provider, or parent/caregiver reports.
- Child Well-being. Child well-being is a multi-faceted construct that broadly refers to the skills and capacities that enable young people to understand and navigate their world in healthy, positive ways. It is an umbrella term that includes child and youth development in behavioral, social, emotional, physical, and cognitive domains. The Prevention Services Clearinghouse reviews the following domains of child well-being, the specific nature of which may vary with age:
- Behavioral and Emotional Functioning. Characteristics and behaviors relating to the ability to realize one’s potential, cope with daily activities, and work and play productively and fruitfully. Both strengths-based and deficit-based indicators are eligible. Examples include measures of externalizing behaviors (e.g., aggressive behavior, disruptiveness, impulsive behavior), internalizing behaviors (e.g., depression, anxiety, mood or thought problems), mental/behavioral health diagnoses, positive behavior, resilience, self-regulation or self-control, and emotional adjustment.
- Social Functioning. Skills and capabilities relating to the ability to develop, maintain, and manage interpersonal relationships (e.g., social skills, assertiveness, cooperation, empathy, social adjustment, peer relations, rebelliousness, defiance, and other similar characteristics related to interpersonal interactions and relationships).
- Cognitive Functions and Abilities. Abilities related to reasoning, knowledge, problem-solving, mental processing, executive functioning, and the like. Eligible measures include intelligence tests, developmental assessments, measures of visual or spatial processing, and other indicators of cognitive functions and abilities.
- Educational Achievement and Attainment. Educational achievement refers to the extent to which students master academic content. Eligible measures include composite or subject-specific (e.g., reading, mathematics) standardized achievement test scores or overall grade point averages. Educational attainment refers to student progress through school or the completion of a degree, certificate, or program. Eligible measures of attainment include grade promotion, high school graduation or dropout rates, certificate or degree completion rates, and other indicators for educational attainment.
- Physical Development and Health. Characteristics related to the healthy functioning of the body may include indicators of physical health (e.g., Body Mass Index), physical capabilities (e.g., motor skills), normative indicators of healthy development (e.g., height), and any other measure relating to healthy (or unhealthy) physical development.
- Substance Use or Misuse. Measures of substance use or misuse may involve any substances and may be self- or other-reported, clinical tests such as urinalysis, or any other measure that provides an assessment of the participants’ substance use behavior. Measures must describe actual use or misuse, such as frequency or quantity of use, type of use, use/no use, time since last use, etc. Substance use diagnoses (e.g., from a clinical interview or DSM criteria) are considered eligible outcomes in this domain. Measures that do not directly index substance use or misuse (e.g., drug-related criminal or delinquency activity such as selling drugs, drug knowledge, behavioral intentions to use or not, etc.) are not eligible in this domain, but may meet the requirements for other outcome domains.
- Delinquent Behavior. Delinquent behavior refers to behavior chargeable under applicable laws, whether or not apprehension occurs or charges are brought. Chargeable offenses also include “status” offenses (e.g., runaway, truancy, curfew violations).
- Adult Well-being. Adult well-being refers to the specific skills and capabilities adults need to navigate their world in healthy, positive ways and provide for themselves and their children’s needs. Well-being is an umbrella term that includes outcomes in a range of individual and interpersonal domains. The Prevention Services Clearinghouse reviews the following domains of adult well-being:
- Parenting Practices. Parenting practices include a range of practices and behaviors focused on developing strong, functional relations between parents or caregivers and children and the parents or caregivers’ abilities to successfully manage child socialization and support child development, health, and well-being in an effective and constructive manner. Measures may include items about basic elements of caregiving, such as feeding and physical care; communication and listening; nurturing, loving, or supportive behavior; rules and consequences; setting boundaries; warmth; scaffolding children’s behavior to develop self-discipline; parent-child relationships, and the like. Measures may index either positive parenting practices or negative parenting practices.
- Parent/Caregiver Mental or Emotional Health. Mental or emotional health refers to a parent’s/caregiver’s ability to cope with daily activities, realize his or her potential, and interact productively in the world. Both strengths-based and deficit-based indicators are eligible. Examples include measures of externalizing behaviors (e.g., aggressive behavior), internalizing behaviors (e.g., depression, anxiety, mood or thought problems), mental/behavioral health diagnoses, parent/caregiver stress, relationship stress, positive behavior, resilience, and emotional adjustment.
- Parent/Caregiver Substance Use or Misuse. Measures of substance use or misuse may involve any substances and may be self- or other-reported, clinical tests such as urinalysis, or any other measure that provides an assessment of the participants’ substance use or misuse. Measures must describe actual use or misuse, such as frequency or quantity of use, type of use or misuse, use/no use, time since last use, etc. Substance use diagnoses (e.g., from a clinical interview or DSM criteria) are considered eligible in this domain. Measures that do not directly index substance use or misuse (e.g., drug-related criminal or delinquency activity such as selling drugs, drug knowledge, behavioral intentions, etc.) are not eligible in this domain, but may meet the requirements for other outcome domains.
- Parent/Caregiver Criminal Behavior. Criminal behavior refers to behavior chargeable under applicable laws, whether or not apprehension occurs or charges are brought.
- Family Functioning. Family functioning refers to the capacity or lack of capacity of a family to meet the needs of its members and includes physical care and maintenance of family members; socialization and education of children; and, economic and financial support of the family.
- Physical Health. Refers to the physical health of parents or caregivers and can include a variety of indicators including blood pressure; weight, obesity, or body mass index (BMI); chronic conditions such as asthma or diabetes; and, healthy lifestyle behaviors such as diet and exercise.
- Economic and Housing Stability. Economic and housing stability includes indicators of financial or economic stability (e.g., level of income, employment/unemployment, financial assistance) and/or housing stability (e.g., number of moves, quality of housing, homelessness).
Eligible Outcomes for Kinship Navigator Programs
- Child Safety (defined as above).
- Child Permanency (defined as above).
- Child Well-Being (defined as above).
- Adult Well-Being (defined as above).
- Access to Services. Access to services refers to a parent, caregiver, or family’s knowledge of and ability to access, or utilization of services to support the family’s financial, legal, social, educational, and/or health needs such as medical care, financial assistance, and social services. Parent/caregiver self-reports, informed collateral reports (e.g., from therapists or case managers), or administrative records are eligible indicators for Prevention Services Clearinghouse reviews.
- Referral to Services. Referral to services may include referrals to any needed financial, legal, social, educational, or health services. Measures may be obtained from parent/caregiver self-reports, therapist or provider reports or records, or administrative records. Examples include the presence or absence of referrals or counts/frequencies of referrals.
- Satisfaction with Programs and Services. Satisfaction with programs and services refers to parent or caregiver satisfaction with the programs and services to which they are referred or which they receive as part of a kinship navigator program.
4.1.6 Program Adaptations
When multiple formal versions of a program or service are available, the Prevention Services Clearinghouse selects just one version for review at a time and reviews eligible studies only of the version selected. Only studies of the version selected will be eligible for review for that program or service. Other versions may be eligible for review as separate programs or services. Multiple formal versions may be reviewed for the Prevention Services Clearinghouse in the same round of review or in later rounds of review.
To be eligible for review, studies of a program or service must all represent similar implementations of the program under review; that is, programs or services may not be substantially modified or adapted from the manual or version of the program or service selected for review. Adaptations or modifications to processes, such as accelerating program delivery (e.g., from two times/week to three times/week) over a shorter period, are acceptable. But, adaptations or modifications to content (such as adding a new component to an established program or service) or modality (such as changing from in-person to online) are not considered the same for purposes of Prevention Services Clearinghouse reviews (though such programs or services may be eligible themselves for review as separate programs). Reviewers document all adaptations that are reported in studies when screening them for eligibility. Senior content experts on the Prevention Services Clearinghouse staff will be consulted to develop a final decision on whether a particular adaptation is acceptable or not. Exhibit 4.1 provides examples of eligible adaptations as well as adaptations that are considered a different program or service for purposes of review.
Exhibit 4.1. Examples of Program and Service Adaptations within a Study for the Purpose of Study Review
|Eligible Adaptations||Adaptations that Result in Different Program or Service|
The Prevention Services Clearinghouse will review all eligible studies.
- If a program or service has less than 15 eligible studies, all studies are reviewed using the design and execution standards described in Chapter 5 and assessed for risk of harm, as described in Section 6.2.
- If a program or service has more than 15 eligible studies, all eligible studies will be assessed for risk of harm. Study review prioritization criteria (see below) will be used to determine the order of eligible studies reviewed using the design and execution standards. Once ordered, the first 15 eligible studies will be reviewed using the design and execution standards. If, after review of 15 eligible studies, a program or service has not achieved a rating of well-supported, additional studies will continue to be reviewed in order until the program or service has achieved a rating of well-supported or all eligible studies have been reviewed.
Study review prioritization criteria. As noted above, for programs and services with more than 15 eligible studies, a point system will be used to determine the order of studies reviewed. When a study is determined to be eligible for review using the above-described criteria, reviewers assign points to studies as follows:
- Design. 3 points for randomized controlled trials (RCTs), 2 points for quasi-experimental designs (QEDs).
- Sample Size. 1 point for a total sample size of 250 or more participants.
- Duration of Sustained Effects Examined. 2 points for sustained effects of 12 months or more; 1 point for sustained effects between 6 and 12 months.
- Number of Different Outcome Domains Examined. 1 point for each different outcome domain examined in the study (maximum of 3 points for Child Safety, Child Permanency, Child Well-Being, or Adult Well-Being).
- Pre-Registered Study Designs. 3 points for studies that were pre-registered in a trial registry, such as clinicaltrials.gov, or that have published study protocols.
Points are totaled for each study (maximum of 12 points). Studies are then sorted by the summed point total and reviewed in that order.