Program and Service Ratings

This chapter describes the process of translating design and execution ratings from one or more studies of a program or service into ratings for that program or service. As described at the beginning of Chapter 5, Prevention Services Clearinghouse reviewers rate study contrasts, rather than entire studies. To determine program and service ratings, the Prevention Services Clearinghouse combines design and execution ratings from multiple contrasts and (if available) contrasts from multiple studies. To determine the rating for a program or service, all contrasts for each eligible program or service that meet moderate or high evidence standards are examined.

Using the qualifying contrasts, reviewers assign one of four ratings to each program or service to characterize the extent of evidence for a particular program or service:

  • Well-supported. A program or service is rated as a well-supported practice if it has at least two contrasts with non-overlapping samples in studies carried out in usual care or practice settings (see Section 6.2.2) that achieve a rating of moderate or high on design and execution and demonstrate favorable effects in a target outcome domain. At least one of the contrasts must demonstrate a sustained favorable effect of at least 12 months beyond the end of treatment (see Section 6.2.3) on at least one target outcome.
  • Supported. A program or service is rated as a supported practice if it has at least one contrast in a study carried out in a usual care or practice setting that achieves a rating of moderate or high on design and execution and demonstrates a sustained favorable effect of at least 6 months beyond the end of treatment on at least one target outcome.
  • Promising. A program or service is designated as a promising practice if it has at least one contrast in a study that achieves a rating of moderate or high on study design and execution and demonstrates a favorable effect on a target outcome.
  • Does not currently meet criteria. A program or service that has been reviewed and does not achieve a rating of well-supported, supported, or promising is deemed ‘does not currently meet criteria.’ This includes (a) programs and services for which all eligible contrasts with moderate or high design and execution ratings have no statistically significant favorable effects and (b) programs and services that do not have any eligible contrasts with moderate or high design and execution ratings.

6.2.1 Risk of Harm

A program or service cannot be not rated as well-supported, supported, or promising if there is an empirical basis, as evidenced by the presence of an unfavorable effect(s) on target or non-target outcomes that suggest that the overall weight of evidence does not support the benefits of the program or service. To be considered, unfavorable effects must be reflected in contrasts that receive a moderate or high rating according to the design and execution standards. To determine whether there is risk of harm, all statistically significant unfavorable impacts on any outcome (whether an eligible target outcome or not) from any studies with contrasts receiving high or moderate evidence ratings are identified. If there is sufficient evidence of risk of harm based on statistically significant unfavorable findings, the program may be deemed ‘does not currently meet criteria’ by the Prevention Services Clearinghouse. Additionally, programs or services may not be designated as well-supported, supported, or promising if case data suggests a risk of harm that was probably caused by the treatment and was severe or frequent.

6.2.2 Usual Care or Practice Settings

To receive a rating of supported or well-supported, the favorable evidence for a program or service must have been obtained from research conducted in a usual care or practice setting. A usual care or practice setting is defined as an existing service agency or provider that delivers mental health services, substance use prevention or treatment services, in-home parent skill-based programs, and/or kinship navigator programs as part of its typical operations.

A usual care setting may use routine personnel who already work for the agency or it may employ outside staff (e.g., researchers, graduate students) if the services themselves are those that would typically be delivered by agency personnel in the absence of a research study. Ad hoc clinics set up expressly for the purposes of research do not constitute usual care or practice settings, even if staffed by personnel who might typically work in a usual care setting.

6.2.3 Beyond the End of Treatment

To receive a rating of supported or well-supported, programs and services must have sustained favorable effects beyond the end of treatment. The end of treatment is defined as the stated end of treatment by the study or program documentation. If a clear end of treatment is not defined, if treatment extends indefinitely or varies across participants, or if services are staggered, the Prevention Services Clearinghouse selects a time point that corresponds to when the majority of a clearly defined set of services were stated to have been delivered. If that information is not available, but studies provide information about the average or range of service delivery, reviewers will use the longest program duration (or estimate it from the data provided) as the end of treatment and determine the length of follow-up from that point.

If a study gives the time between pre-test and post-test, but not the time between the end of treatment and measurement of the post-test, reviewers subtract the stated intended duration of treatment from the pre/post interval to estimate the number of months beyond the end of treatment that measurement occurred.