A selected bibliography of research conducted in the CHP and related programs.

 


 

Cole, C.P. (2008) “Academic Interventions Designed to Improve Mathematic Performance among Middle School Students: Specific and Generalization effects in a Randomized Trial.” USC Doctoral Dissertation.

Terry, J. T., Waldrop, D., and Smith, B.H. (2008). Effects of self-efficacy enhancing feedback upon peer assisted learning strategies. Poster presented at the Annual Meeting of the Association for Behavioral and Cognitive Therapies, Orlando, FL.

Langberg, J., Epstein, J., Urbanowitz, C., Simon, J., & Graham, A. (2008). Efficacy of an Organization Skills Intervention to Improve the Academic Functioning of Students With Attention-Deficit/Hyperactivity Disorder. School Psychology Quarterly, 23, 407-417.

Molina, B.S.G., Flory, K. Bukstein, O.G., Greiner, A. R., Baker, J. L., Krug V., and Evans, S. W. (2008). Feasibility and Preliminary Efficacy of an After-School Program for Middle Schoolers With ADHD: A Randomized Trial in a Large Public Middle School. Journal of Attention Disorders, 12, 207-217.

 

Bogle, K.E. (2007). "Evaluation of a brief group parent training intervention in the context of an after-school program for middle-school students." USC Doctoral Dissertation

Langberg, J. M., Smith, B. H., Bogle, K., Schmidt, J., Cole, W., & Pender, C. (2006). After-School Programming to Improve the Behavioral and Academic Functioning of Middle School Students: A Randomized Efficacy Trial. Journal of Applied School Psychology, 23, 31-58.

Evans, S. W., Green, A. L., & Serpell, Z. N. (2005). Community participation in the treatment development process using community development teams. Journal of Clinical Child and Adolescent Psychology, 34(4), 765-771. Evans, S. W.,

Langberg, J. M., Raggi, V., Allen, J., & Buvinger, E. (2005). Development of a school-based treatment program for middle school youth with ADHD. Journal of Attention Disorders, 9, 343-353.

Evans, S. W., Axelrod, J., & Langberg, J. M. (2004). Efficacy of a school-based treatment program for middle school youth with ADHD: Pilot Data. Behavior Modification, 28, 528-547.

 


 

The Challenging Horizons Program (CHP)

 

This is an updated version of the CHP application to the National Registry of Evidence-Based Programs and Practices (NREPP). The emphasis of the CHP has expanded in recent years to students who do not have ADHD, with a particular emphasis on students from low-income families with low test scores. Research on this broader target population is just starting to be published.

 

The Challenging Horizons Program (CHP) is an innovative, multimodal approach to working with middle-school students who have learning or behavior problems, especially those with Attention Deficit/Hyperactivity Disorder (ADHD). The CHP has it origins in the summer treatment program (STP), which is recognized as a model multimodal treatment program for youth with ADHD (Pelham & Hoza, 1996). Four studies published in peer-reviewed journals indicate that the CHP can achieve medium to large effects on parent and teacher ratings of behavior and impairment, and small to moderate effects on school grades. Moreover, many of the core components of the CHP have empirical support for youth with ADHD (Pelham & Fabiano, 2008). The core features of the CHP are listed below along with references to empirical support.


  • A low student to staff ratio, thus allowing for a great deal of individual attention.
  • Staffing the program primarily with university service-learning students, that making low staff to student ration feasible.
  • A behavioral milieu rich in social reinforcement, tangible rewards, and clear and consistently enforced rules.
  • Training in enabling skills, such as organization.
  • Training in specific educational skills, such as notetaking.
  • An emphasis on improving social skills.
  • Goal setting and improving self-regulation of behavior.
  • Recreational activities to improve group cooperation, fitness, and sports skills.
  • Parent involvement through group parent training and weekly reports.
  • Quality control assisted with manuals, pre-service training, on-site supervision and feedback using fidelity forms, and empirical evaluation of CHP activities.

     

    In addition to the core components listed above, some key considerations in the CHP development process include a focus on provider acceptability, finding new approaches to service-delivery (e.g., after-school programs staffed by service-learning students), and an emphasis on continued innovation and testing (B. H. Smith, McQuillin, & Shapiro, 2008). This summary is designed to support our application for including the CHP in the National Registry of Evidence-based Programs and Practices. The summary describes the need for developing interventions such as the CHP, the accumulated CHP evidence-base (so far), and deployment considerations that have guided the development of the CHP.

     

    The Need for Programs Like the CHP

     

    There is an abundance of research on elementary-aged children with Attention-Deficit/Hyperactivity Disorder (AD/HD), but relatively little research on middle- or high-school aged youth. Whereas some reviews of children reported finding hundreds of studies, a review of treatment studies published prior to 1999 found only 29 studies that provided efficacy data on adolescents with AD/HD (B.H. Smith, Waschbusch, Willoughby, & Evans, 2000). Almost all of these studies of adolescents focused on pharmacological treatments, with 17 studies of stimulant medication, 3 studies of tricyclic antidepressants, and 3 studies of other medications. Only 6 studies examined non-pharmacological interventions. These studies showed promising preliminary support for note taking training, behavioral interventions in the classroom, and family-based interventions. Mean effect sizes of these interventions were in the medium to large range (i.e., .66 to .74); however, when using the APA criteria for evidence-based treatments (Chambless, et al., 1996), the only well-established intervention based on the literature published prior to 1999 was stimulant treatment with methylphenidate (B.H. Smith, et al., 2000).

     

    Stimulant treatment is controversial and suffers from problems with acceptability and feasibility, especially with adolescents (B.H. Smith, et al., 2000). Even when there is clear evidence to document a child’s positive response to medication through controlled clinical experimentation, there is a steady decline in medication use as children get older (Jensen, et al., 2007). Furthermore, there is a major deterioration of adjustment when students with ADHD enter middle school (J. M. Langberg, et al., 2008).

     

    The age-related decline in stimulant medication use among adolescents is just one of several reasons why there needs to be non-pharmacological options for treating adolescents who have ADHD (Pelham, Wheeler, & Chronis, 1998). First, it is generally accepted that about 30% of youth do not benefit from stimulant medication or can not tolerate this treatment (B.H. Smith, Barkley, & Shapiro, 2006). Second, stimulants interfere with sleep and therefore cannot be taken in the evening. This creates daily gaps in stimulant treatment during high stress periods (e.g., getting ready for school or for bed). These gaps need to be addressed by other interventions. Third, although there is some debate on this issue, there is growing consensus that pharmacological treatments most effective when combined with psychosocial or educational interventions ((Pelham, et al., 1998; B.H. Smith, et al., 2006). For these and other reasons, it is extremely important to develop effective, safe, and practical psychosocial interventions for middle and high school aged youth with AD/HD.

     

    Unfortunately, progress on identifying well-established treatments for adolescents with ADHD has been slow and continues to lag behind treatments for children with ADHD. For instance, in an update of a comprehensive review of

    evidence-based psychosocial treatments for children with ADHD (Pelham, et al., 1998), reviewers found 46 new studies (Pelham & Fabiano, 2008). Only 5 of these studies (11%) focus on treatment of adolescents. Two of these studies provide support for the effects of parenting interventions, but the authors of both of these studies identify limitations of parenting and state that the best approach to dealing with ADHD involves using multimodal interventions, such as after-school programs (Pelham & Fabiano, 2008).

     

    Efficacy of the CHP

     

    Two of the studies on adolescents reviewed by Pelham and Fabiano (2008) provide support for the CHP (Evans, Axelrod, & Langberg, 2004; Evans, Langberg, Raggi, Allen, & Buvinger, 2005). Furthermore, there are two randomized studies that were not included in the Pelham and Fabiano review. One is a randomized trial that was just recently accepted for publication (Molina, et al., in press). The other is the largest randomized study of the CHP, which shows that the CHP can be effective with youth who do not have ADHD (J.M. Langberg, et al., 2006). These studies are reviewed below.

     

    Evans, Axelrod, and Langberg (2004) report treatment group only pre- and post-test pilot data on seven white youth aged 11 to 13 (5 males, 2 females) diagnosed with ADHD. These youth attended the CHP two hours and 15 minutes a day, three days a week for at least three months. During the program, students met with primary counselors to work on goals and participated in interpersonal skills training, therapeutic recreation, and academic skills groups. Primary counselors met weekly with teachers. Parent participated in monthly groups, plus received frequent reports home from the program. Outcomes were measured using school grades and ratings on ADHD rating scale and the Impairment Scale completed by parents and teachers. Significant pre- to post-test (p < .05) improvements were found for teachers’ ratings of inattentive behavior. Large effect sizes (i.e., Cohen’s d > .80) were noted for parent and teacher ratings of academic functioning and teacher ratings of classroom behavior. Medium effect sizes were found for parent and teacher ratings of behaviors with siblings and peers, respectively. Small effect sizes were noted for grades, parent and teacher ratings of adult-child relationships, and parent ratings of ADHD symptoms. It noteworthy that the slope of grades shifted from negative to positive while students were participating in the CHP (Evans, et al., 2004).

     

    In an extension and replication of their pilot work, Evans and colleagues conducted a quasi-experimental evaluation of the CHP in which 13 Caucasian youth aged 11 to 14 (2 female, 9 on stimulant medication) participated in the CHP using very similar procedures and measures that were used in the pilot study (Evans, et al., 2005). Pre- to post-changes in the treatment group were compared to 14 "community control" youth aged 11 to 14 (4 female, 13 on stimulant medication) who attended a school where the CHP was not offered. Within group effect sizes were in the large range for parent ratings of peer interactions, academic performance, and overall functioning in the treatment group. This compares to small to negative effect sizes for parent ratings of youth in the control group. Effect-size comparisons of teacher ratings was inconclusive. There was a trend for beneficial effects of the CHP on grades (p = .071). However, rather than improving grades in the CHP group, the CHP appears to have stopped a decline in grades illustrated by the fact that grades in the community comparison group declined while those in the CHP group were unchanged. The authors note that low statistical power, missing data, and concern about the validity of teacher reports hampered the interpretation of these data.

     

    Molina and colleagues (in press) conducted a small, randomized study of the CHP. A total of 23 youth were recruited to the study, with 12 randomized to the CHP and 11 to control. Half of the youth were from ethnic minorities, most were from two parent households. The groups were similar on most characteristics, but more parents in the control group had earned a bachelor’s degree (44% vs 27%) and more youth in the control group were taking medication (36% vs. 25%). Subsequent attrition after a 10-week, two hour a day, twice a week program resulted in pre- and post-test data from 11 and 9 youth, respectively. CHP procedures were similar to the Evans et al. studies with the exception that a behavioral level system was introduced and parents attended three, two-hour parent meetings designed to review CHP content and improve management of behavior at home. The authors stated emphasis of this study was to demonstrate that is was feasible to replicate the CHP in the context of a randomized study. Nevertheless, the study did generate some effect size data indicative of efficacy.

     

    Molina et al. (2008) reported treatment in the CHP was associated with medium-sized improvements (d = .55 to .59) in parent-rated internalizing problems and adolescent-rated school maladjustment and emotional symptoms. This compares to no change on these measures in the control group. Similar to the findings of Evans et al. (2005), there was no change in grades in the CHP group, but there was a decline in grades in the control group (d = .34 to .46). Despite the very low statistical power, there were some significant group by time interactions indicative of positive effects of the CHP on parent-rated internalizing symptoms (p < .05), delinquency or conduct problems (p <.10), and school maladjustment (p < .10).

     

    Langberg et al. (2006) conducted the largest study of the CHP to date. However, the version of the CHP tested by Langberg et al. was modified to fit the needs of a school district supported after-school program for youth who had performed poorly on state standardized tests(J.M. Langberg & Smith, 2006). Key differences from Evans et al. and Molina et al. were that the CHP did not focus exclusively on youth with ADHD, the program was provided four days a week for two hours per day, and staff were trained in a specialist or "small group" model (e.g., leading small specialty groups such as organization and goals) as opposed to the generalist "primary counselor" in which staff provided many services to the youth individually. Also, the Langberg et al. study did not have a social skills group but social skills were emphasized throughout the program and specifically reinforced by counselors’ praise and the behavioral level system.

     

    Youth in the Langberg et al study were recruited if they had scored "below basic" on the state standardized test but were excluded if their Full Scale IQ was lower than 70. A total of 48 youth were randomly assigned and t-tests found no differences on key variables such as age, socioeconomic status, grade, or Full Scale IQ. Randomization was conducted at the beginning of the fall semester and youth in the CHP group started the program immediately. Those assigned to the wait-list control group were scheduled to begin the CHP in the spring. Importantly, 85% of the youth in the control group participated in the after-school program that met 1 to 3 days a week to provide academic remediation.

     

    Results of the Langberg et al. study demonstrate the flexibility and efficacy of the CHP. There were large effects on parent ratings of academic progress, in this case the group by time interaction was significant (p < .05). The within subject d was .74 for the CHP group and -.24 for the control group. There were also statistically significant (p < .05) changes favoring the CHP group in terms of parent-rated self-esteem, overall severity of problems, organization and homework completion, and the Conners Global Index (CGI). The latter is noteworthy because the CGI is a measure of ADHD symptoms (REF), thus suggesting the modified CHP is still effective in reducing ADHD related problems. Teacher ratings were limited by missing data in this study and none of the teacher ratings on the IRS showed a statistically significant group by time interaction. However, there was a significant effect of the CHP on mean grades for science and trend toward improvement with small to moderate effect sizes in history, math, and English language arts. Finally, youth in the control group were served detention or were suspended almost twice as often as those in the CHP (i.e., 10 vs. 6 detentions and 10 vs. 5 suspensions).

     

    To summarize, the studies on the general efficacy of the CHP have spanned a broad range of research designs and populations in three US cities. Parents of youth in the CHP have consistently reported positive effects of the program, and many of these findings reached statistical significance (p < .05) despite low statistical power. Another finding across the studies was the tendency for the control group to show deteriorating academic performance, while youth in the CHP maintained steady performance or showed improvement. There were few instances of teacher ratings showing significant change related to the CHP; however, this may be due to limitations of teacher ratings rather than a lack of efficacy of the CHP. Overall, it appears that 10 to 12 weeks in the CHP is sufficient to achieve benefits that are noticeable to parents and impact ecologically valid measures like grades.

     

    CHP Operations and Dissemination

     

    The CHP has been developed in accordance with the Deployment Focused Intervention Development Model (DFM) (Weisz, Jensen, & McLeod, 2004). The first step of the DFM involves collaborating with community providers to create and refine procedures that are perceived as acceptable, feasible, and likely to be effective. Day-to-day leadership in the CHP is expected to be provided by educational professionals or community leaders (e.g., teachers, school counselors, or program coordinators), many of whom will not have any graduate training. These program leaders are expected direct the activities of service-learning students, who will provide most of the direct services in the CHP. Graduate students or university faculty help provide consultation, training, recruitment, and logistical support.

     

    Over the past several years, the CHP program developers have acted as consultants who help to (a) summarize and clarify the perceived needs of the providers, (b) to identify feasible methods for dealing with problems, and (c) selecting mutually agreeable criteria for evaluating the effectiveness of the intervention (J.M. Langberg & Smith, 2006). The major products of step one of the DFM have been treatment manuals, training protocols, and support materials (available at www.scstudentexcellence.org). CHP procedures have been grounded in relevant theory and empirical research has been presented in a manner that is accessible and engaging to new trainees. Feedback from experienced providers and clients has guided the content and style of the program materials, which have been developed in an iterative manner. Indeed, flexibility and the opportunity to innovate may be key considerations in the acceptability of interventions to providers (Sanders, 1999).

     

    With regard to the cost of the program, the CHP can be done relatively inexpensively thanks to student labor. If schools provide teachers, transportation, supplies, and space for after-school program (which they often do), the CHP incurs minimal additional cost to the school when the CHP is provided as a service-learning class in a local university. All together the cost of the CHP is less then $50 per week per student, which is less than what many parents pay for after-school care. Youth from low-income families may be eligible for participation in the CHP through 21st Century Community Learning Center Grants and the provision of Supplemental Education Services made possible by Title 1 funding.

     

    The CHP is regarded as a model service-learning program by the University of South Carolina (see http://www.sc.edu/servicelearning). In the South Carolina CHP, prior to working with youth, USC undergraduates complete 8 to 12 hours of training in how to work effectively with high-risk school students. This involves learning a detailed manual, behavioral rehearsal of program activities, and tests of knowledge of program policies and procedures. After successfully completing training, university students in the CHP get closely supervised, hands-on experience with learning theory, adolescent development, social and emotional intelligence, self-regulatory theory, research methods, and many other areas of psychology and education. Students reflect on this didactic content in journals completed each visit to the CHP site. These observations are discussed in weekly meetings and as part of summary presentations of reflections at the end of the semester. Importantly the CHP has a "training ladder" such that returning staff can take progressively higher levels of responsibility with training and supervision of junior staff. This often results in upper-level undergraduates or graduate students designing, implementing, and evaluating innovative enhancements to the program under the supervision of Dr. Smith.

     

    In conclusion, the success and sustainability of the CHP is based on a low student to staff ratio made possible by service-learning students and community volunteers. In the CHP, the student to staff ratio varies across activities. An experienced teacher can provide group instruction at a 12:1 ratio. Other activities require more intensive staffing, such as study time, reading, and writing. In these activities, many students need quite a bit of individual attention and ratios should be 4:1 or less. Another staffing goal in the CHP is for all students to get at least an hour of individual, one-on-one attention each week. This staffing pattern makes the CHP a uniquely individualized after-school program that has produced measurable results that have been published in peer-reviewed scientific journals and recognized for innovation by the After-School Alliance.

     

    References

     

    Chambless, D. L., Sanderson, W. C., Shoham, V., Johnson, S. B., Pope, K. S., Crits-Christoph, P., et al. (1996). An update on empirically validated treatments. Clinical Psychologist, 49, 5-18.

     

    Evans, S. W., Axelrod, J., & Langberg, J. M. (2004). Efficacy of a school-based treatment program for middle school youth with ADHD: Pilot Data. Behavior Modification, 28, 528-547.


    Evans, S. W., Langberg, J. M., Raggi, V., Allen, J., & Buvinger, E. (2005). Development of a school-based treatment program for middle school youth with ADHD. Journal of Attention Disorders, 9, 343-353.


    Jensen, P. S., Arnold, L. E., Swanson, J. M., Vitiello, B., Abikoff, H. B., Greenhill, L. L., et al. (2007). 3-year follow-up of the NIMH MTA study. Journal of the American Academy of Child and Adolescent Psychiatry, 46(8), 989-1002.


    Langberg, J. M., Epstein, J. N., Altaye, M., Molina, B. S. G., Arnold, L. E., & Vitiello, B. (2008). The transition to middle school is associated with changes in the developmental trajectory of ADHD symptomatology in young adolescents with ADHD.
    Journal of Clinical Child and Adolescent Psychology, 37(3), 651-663.

     

    Langberg, J. M., & Smith, B. H. (2006). Developing evidence-based interventions for deployment in school settings: A case example highlighting key issues of efficacy and effectiveness. Journal of Evaluation and Program Planning, 29, 323-334.

     

    Langberg, J. M., Smith, B. H., Bogle, K., Schmidt, J., Cole, W., & Pender, C. (2006). After-School Programming to Improve the Behavioral and Academic Functioning of Middle School Students: A Randomized Efficacy Trial. Journal of Applied School Psychology, 23, 31-58.

     

    Molina, B., Flory, K., Bukstein, O., Greiner, A. R., Baker, J. L., Krug, V., et al. (in press). Feasibility and preliminary efficacy of an after-school program for middle-schoolers with ADHD: A randomized trial in a large public middle school. Journal of Attention Disorders.

     

    Pelham, W. E., & Fabiano, G. A. (2008). Evidence-based psychosocial treatments for attention-deficit/hyperactivity disorder. Journal of Clinical Child and Adolescent Psychology, 37(1), 184-214.

     

    Pelham, W. E., & Hoza, B. (1996). Intensive treatment: A summer treatment program for children with ADHD. In E. Hibbs & P. Jensen (Eds.), Psychosocial treatments for child and adolescent disorders: Empirically based strategies for clinical practice (pp. 311-340). New York: APA Press.

     

    Pelham, W. E., Wheeler, T., & Chronis, A. (1998). Empirically Supported Psychosoical Treatments for Attention Deficit Hyperactivity Disorder. Journal of Clinical Child Psychology, 27(2), 190-205.

     

    Sanders, M. (1999). Triple-P positive parenting program: Towards an empirically validated multi-level parenting and family support strategy for the prevention of behavioral and emotional problems in children. Clinical Child and Family Psychology Review, 2, 71-90.

     

    Smith, B. H., Barkley, R. A., & Shapiro, C. J. (2006). Attention-Deficit/Hyperactivity Disorder Treatment of Childhood Disorders. New York: The Guilford Press.

     

    Smith, B. H., McQuillin, S., & Shapiro, C. J. (2008). An installation-adaptation-diffusion model of university-school partnerships. The Community Psychologist, 41, 3.

     

    Smith, B. H., Waschbusch, D., Willoughby, M., & Evans, S. (2000). The efficacy, safety, and practicality of treatments for adolescents with Attention-Deficit/Hyperactivity Disorder. Clinical Child and Family Therapy Review, 3(4), 243-267.

     

    Weisz, J. R., Jensen, A. L., & McLeod, B. D. (2004). Development and Dissemination of child and adolescent therapies: Milestones, methods, and a new deployment-focused model. In E. H. a. P. S. Jensen (Ed.), Psychosocial treatments for child and adolescent disorders: Empirically-based approaches (Second Edition). Washington, DC: American Psychological Association.