Evidence-based assessment/ROC Party/Ready to ROC

This set of pages provides a practical introduction to Receiver Operating Characteristic (ROC) analyses, including sample code and examples generated with multiple popular statistical programs.

Example ROC Party page

 * Example ROC Party page

ROC sample code

 * Example ROC sample code (includes R, SPSS, SAS)

ROC Paper examples
The following papers are a selection that use ROC in their analyses, or illustrate application of DLRs to cases and decision-making, or both.

In press

 * Cohen, J.R., Adams, Z.W., Menon, S.V., Youngstrom, E.A., Bunnell, B.E., Acierno, R., Ruggiero, K.J., & Danielson, C.K. (in press). How should we assess for depression following a natural disaster? An ROC approach to post-disaster assessments in adolescents and adults. Journal of Affective Disorders.
 * Danielson, C.K. Cohen, J., Adams, Z., Youngstrom, E.A., Soltis, K., Amstadter, A., & Ruggiero, K. (in press). Clinical decision making following disasters: efficient identification of PTSD risk in adolescents. Journal of Abnormal Child Psychology.
 * Jarrett, M., Van Meter, A.R., Hilton, D., Youngstrom, E.A., & Ollendick, T. (in press). Evidence-based assessment of ADHD in youth using a Receiver Operating Characteristic (ROC) approach. Journal of Clinical Child and Adolescent Psychology.
 * Jo, B., Findling, R.L., Hastie, T., Youngstrom, E.A., Wang, C.P., Arnold, L.E., Fristad, M.A., Frazier, T.W., Birmaher, B., Gill, M.K., & Horwitz, S.M. (in press). Construction of longitudinal prediction targets using semi-supervised learning. Statistical Methods in Medical Research.
 * Mesman, E., Youngstrom, E.A., Juliana, N.K., Nolen, W.A., & Hillegers, M.H.J. (in press). Validation of the Seven Up Seven Down (7U7D) in bipolar offspring: screening and prediction of mood disorders. Findings from the Dutch Bipolar Offspring Study. Journal of Affective Disorders.
 * Ong, M. L., Youngstrom, E. A., **Chua, J. J. X., *Halverson, T. F., Horwitz, S. M., Storfer-Isser, A., . . . Findling, R. L. (2016). Comparing the CASI-4R and the PGBI-10M for differentiating bipolar spectrum disorders from other outpatient diagnoses in youth. Journal of Abnormal Child Psychology. doi:10.1007/s10802-016-0182-4
 * Ransom, D.M., Burns, A.R., Youngstrom, E.A., Vaughan, C.G., Sady, M.D., & Gioia, G.A. (in press). Applying an evidence-based assessment model to identify students at risk for perceived academic problems following concussion. Journal of the International Neuropsychological Society.
 * Salcedo, S., Fristad, M., Birmaher, B., Kowatch, R.A., Gadow, K.D., Phillips, M.L., Frazier, T.W., **Chen, Y-L., Arnold, L.E., Horwitz, S.M., Findling, R.L., & The LAMS Team (in press). Diagnostic efficiency of the Child and Adolescent Symptom Inventory (CASI-4R) depressive subscale for youth mood disorders. Journal of Clinical Child and Adolescent Psychology. doi: 10.1080/15374416.2017.1280807.
 * You, D.S., Youngstrom, E.A., Feeny, N.C., Youngstrom, J.K., & Findling, R.L. (in press). Comparing the diagnostic accuracy of five instruments for detecting posttraumatic stress disorder in youth. Journal of Clinical Child and Adolescent Psychology. doi:10.1080/15374416.2015.1030754

2016

 * Algorta, G. P., Dodd, A. L., Stringaris, A., & Youngstrom, E. A. (2016). Diagnostic efficiency of the SDQ for parents to identify ADHD in the UK: a ROC analysis. European child & adolescent psychiatry, 25(9), 949–957. https://doi.org/10.1007/s00787-015-0815-0
 * Youngstrom, E. A., & Van Meter, A. (2016). Empirically supported assessment of children and adolescents. Clinical Psychology: Science and Practice, 23(4), 327–347. https://doi.org/10.1037/h0101738
 * Frazier TW, Klingemier EW, Beukemann M, Speer L, Markowitz L, Parikh S, Wexberg S, Giuliano K, Schulte E, Delahunty C, Ahuja V, Eng C, Manos MJ, Hardan AY, Youngstrom EA, & Strauss MS (2016). Development of an objective autism risk index using remote eye tracking. Journal of the American Academy of Child and Adolescent Psychiatry, 55, 301-309.
 * Jenkins, M.M., & Youngstrom, E.A. (2016). A randomized controlled trial of cognitive debiasing improves assessment and treatment selection for pediatric bipolar disorder. Journal of Consulting and Clinical Psychology, 84, 323-333.
 * Van Meter, A. R., You, D. S., *Halverson, T., Youngstrom, E. A., Birmaher, B., Fristad, M. A., Kowatch, R.A., Storfer-Isser, A., Horwitz, S.M., Frazier, T.W., Arnold, L.E., Findling, R.L., The LAMS Group. (2016). Diagnostic Efficiency of Caregiver Report on the SCARED for Identifying Youth Anxiety Disorders in Outpatient Settings. Journal of Clinical Child & Adolescent Psychology, 1-15. doi:10.1080/15374416.2016.1188698

2015

 * Youngstrom, E.A., Hameed, A., Mitchell, M., Van Meter, A.R., Freeman, A.J., Perez Algorta, G., White, A.M., Clayton, P., Gelenberg, A., & Meyer, R.E. (2015). Direct comparison of the psychometric properties of multiple interview and patient-rated assessments of suicidal ideation and behavior in an adult psychiatric inpatient sample. Journal of Clinical Psychiatry, 76, 1676-82. doi:10.4088/JCP.14m09353
 * Youngstrom, E.A., & De Los Reyes, A. (2015). Moving towards cost-effectiveness in using psychophysiological measures in clinical assessment: Validity, decision-making, and adding value. Journal of Clinical Child and Adolescent Psychology, 44, 352-361. doi:10.1080/15374416.2014.913252
 * Youngstrom, E.A., *Choukas-Bradley, S., *Calhoun, C.D., & Jensen-Doss, A. (2015). Clinical guide to the evidence-based assessment approach to diagnosis and treatment. Cognitive and Behavioral Practice, 22, 20-35. doi: 10.1016/j.cbpra.2013.12.005
 * Lindhiem, O., Yu, L., Grasso, D.J., Kolko, D.J., & Youngstrom, E.A. (2015). Adapting the Posterior Probability of Diagnosis (PPOD) index to enhance evidence-based screening: An application to ADHD in primary care. Assessment, 22, 198-207. doi: 10.1177/1073191114540748
 * Prieto, M.L., Youngstrom, E.A., Ozerdem, A., Altinbas, K., Quiroz, D., Aydemir, O., Yalin, N., Geske, J.R., Feeder, S.E., Angst, J., & Frye, M.A. (2015). Different patterns of manic/hypomanic symptoms in depression: A pilot modification of the Hypomania Checklist - 32 to assess mixed depression. Journal of Affective Disorders, 172, 355-360. doi: 10.1016/j.jad.2014.09.047

2014

 * Youngstrom, E.A. (2014). A primer on Receiver Operating Characteristic analysis and diagnostic efficiency statistics for pediatric psychology: We are ready to ROC. Journal of Pediatric Psychology, 39, 204-221. Special Issue: Quantitative Methodology (Guest Editors: Bryan Karaszia & Kristoffer Berlin); doi: 10.1093/jpepsy/jst062
 * Frazier, T., Youngstrom, E.A., Fristad, M., Demeter, C., Birmaher, B., Kowatch, R., . . . Findling, R. (2014). Improving clinical prediction of bipolar spectrum disorders in youth. Journal of Clinical Medicine, 3, 218-232. doi: 10.3390/jcm3010218
 * Lee, H.J., Joo, Y., Youngstrom, E.A., Yum, S.Y., Findling, R.L., & Kim, H.W. (2014). Diagnostic validity and reliability of a Korean version of the Parent and Adolescent General Behavior Inventories. Comprehensive Psychiatry, 55, 1730-1737. doi:10.1016/j.comppsych.2014.05.008
 * Pendergast, L. L., Youngstrom, E. A., Merkitch, K. G., Moore, K. A., Black, C. L., Abramson, L. Y., & Alloy, L. B. (2014). Differentiating bipolar disorder from unipolar depression and ADHD: the utility of the General Behavior Inventory. Psychological Assessment, 26, 195-206. doi: 10.1037/a0035138
 * Van Meter, A. R., Youngstrom, E. A., Youngstrom, J. K., Ollendick, T., Demeter, C., & Findling, R. L. (2014). Clinical decision-making about child and adolescent anxiety disorders using the Achenbach System of Empirically Based Assessment. Journal of Clinical Child & Adolescent Psychology.

2013

 * Youngstrom, E.A. (2013). Future directions in psychological assessment: Combining evidence based medicine innovations with psychology’s historical strengths to enhance utility. Journal of Clinical Child and Adolescent Psychology, 42, 139-159. doi: 10.1080/15374416.2012.736358.
 * Algorta, G.P., Youngstrom, E.A., Phelps, J., *Jenkins, M.M., Youngstrom, J.K., & Findling, R.L. (2013). An inexpensive family index of risk for mood issues improves identification of pediatric bipolar disorder. Psychological Assessment, 25, 12-22. doi: 10.1037/a0029225

2012

 * Freeman, A.J., Youngstrom, E.A., Demeter, C., Youngstrom, J.K., & Findling, R.L. (2012). Portability of a screener for pediatric bipolar disorder to a diverse setting. Psychological Assessment, 24, 341-351. doi: 10.1037/a0025617


 * Jenkins, M.M., Youngstrom, E.A., Youngstrom, J.K., Feeny, N.C. & Findling, R.L. (2012). Generalizability of evidence-based assessment recommendations for pediatric bipolar disorder. Psychological Assessment, 24, 269-281. doi: 10.1037/a0025775

2011

 * Jenkins, M.M., Youngstrom, E.A., Washburn, J.J., & Youngstrom, J.K. (2011). Evidence-based strategies improve assessment of pediatric bipolar disorder by community practitioners. Professional Psychology: Research and Practice, 42, 121-129. doi: 10.1037/a0022506

2010

 * Horwitz, S.M., Demeter, C., Pagano, M.E., Youngstrom, E.A., Fristad, M.A., Arnold, L.E., Birmaher, B., Gill, M.K., Axelson, D., Kowatch, R.A. & Findling, R.L. (2010). Identification of a childhood bipolar phenotype-the Longitudinal Assessment of Symptoms of Mania Study: Background, design and initial screening. Journal of Clinical Psychiatry, 71, 1511-1517. doi: 10.4088/JCP.09m05835yel2009
 * Youngstrom, E.A., & Kendall, P.C. (2009). Psychological science and bipolar disorder. Clinical Psychology: Science and Practice, 16, 93-97. doi: 10.1111/j.1468-2850.2009.01149.x
 * Youngstrom, E.A., *Freeman, A.J., & *Jenkins, M.M. (2009). The assessment of bipolar disorder in children and adolescents. Child and Adolescent Psychiatric Clinics of North America, 18, 353-390. doi: 10.1016/j.chc.2008.12.002
 * Meyer, S. E., Carlson, G. A., Youngstrom, E., Ronsaville, D. S., Martinez, P. E., Gold, P. W., Hakak, R., & Radke-Yarrow, M. (2009). Long-term outcomes of youth who manifested the CBCL-Pediatric Bipolar Disorder phenotype during childhood and/or adolescence. Journal of Affective Disorders, 113, 227-235. doi: 10.1016/j.jad.2008.05.024

2008

 * Youngstrom, E.A., *Frazier, T.W., Demeter, C., Calabrese, J.R., & Findling, R.L. (2008). Developing a ten-item mania scale from the Parent General Behavior Inventory for children and adolescents. Journal of Clinical Psychiatry, 69, 831-839. doi:10.4088/JCP.v69n0517
 * Youngstrom, E.A., *Greene, J., & *Joseph, M. (2008). Comparing the psychometric properties of multiple teacher report instruments as predictors of bipolar disorder in children and adolescents. Journal of Clinical Psychology, 64, 382-401. doi: 10.1002/jclp.20462


 * Henry, D., Pavuluri, M., Youngstrom, E.A., & Birmaher, B. (2008). Accuracy of brief and full forms of a mania rating scale. Journal of Clinical Psychology, 64, 368-381. doi: 10.1002/jclp.20464
 * Meyers, O.I., & Youngstrom, E.A. (2008). A Parent General Behavior Inventory subscale to measure sleep disturbance in pediatric bipolar disorder. Journal of Clinical Psychiatry, 69, 840-843. doi: 10.1016/j.jad.20009.07.020

2007

 * Patel, N.C., Patrick, D.M., Youngstrom, E.A., Strakowski, S.M., & DelBello, M.P. (2007). Response and remission in adolescent mania: Signal detection analyses of the Young Mania Rating Scale. Journal of the American Academy of Child and Adolescent Psychiatry, 46, 628-635. doi: 10.1097/chi.0b013e3180335ae4

2006

 * Youngstrom, E.A., *Meyers, O.I., Youngstrom, J.K., Calabrese, J.R., & Findling, R.L. (2006). Comparing the effects of sampling designs on the diagnostic accuracy of eight promising screening algorithms for pediatric bipolar disorder. Biological Psychiatry, 60, 1013-1019. doi: 10.1016/j.biopsych.2006.06.023.


 * Frazier, T.W., & Youngstrom, E.A. (2006). Evidence based assessment of attention-deficit/hyperactivity disorder: Using multiple sources of information. Journal of the American Academy of Child and Adolescent Psychiatry, 45, 614-620. doi: 10.1097/01.chi.0000196597.09103.25

2005

 * Youngstrom, E. A., *Meyers, O. I., Demeter, C., Kogos Youngstrom, J., Morello, L., Piiparinen, R., Feeny, N. C., Findling, R. L., & Calabrese, J. R. (2005). Comparing diagnostic checklists for pediatric bipolar disorder in academic and community mental health settings. Bipolar Disorders, 7, 507-517. doi: j.1399-5618.2005.00269.x


 * Youngstrom, E. A., Findling, R. L., Youngstrom, J. K., & Calabrese, J. R., (2005). Towards an evidence-based assessment of pediatric bipolar disorder. Journal of Clinical Child and Adolescent Psychology, 34, 433-448. Special Issue: Evidence-Based Assessment. doi: 10.1207/s15374424jccp3403_4
 * Youngstrom, E. A., & *Duax, J. (2005). Evidence Based Assessment of Pediatric Bipolar Disorder, Part 1: Base Rate and Family History. Journal of the American Academy of Child and Adolescent Psychiatry, 44, 712-717. doi: 10.1097/01.chi.0000162581.87710.bd


 * Youngstrom, E. A., & Kogos Youngstrom, J. (2005). Evidence Based Assessment of Pediatric Bipolar Disorder, Part 2: Incorporating Information from Behavior Checklists. Journal of the American Academy of Child and Adolescent Psychiatry, 44(8), 823-828. doi: 10.1097/01.chi.0000164589.10200.a4

2004

 * Youngstrom, E. A., Findling, R. L., Calabrese, J. R., Gracious, B. L., Demeter, C., DelPorto Bedoya, D., & *Price, M.E. (2004). Comparing the diagnostic accuracy of six potential screening instruments for bipolar disorder in youths aged 5 to 17 years. Journal of the American Academy of Child & Adolescent Psychiatry, 43, 847-858. doi: 10.1097/01.chi.0000125091.35109.1e

2003

 * Danielson, C. K., Youngstrom, E. A., Findling, R. L., & Calabrese, J. R. (2003). Discriminative validity of the General Behavior Inventory using youth report. Journal of Abnormal Child Psychology, 31, 29-39. doi: 10.1023/A:1021717231272

2002

 * Findling, R. L., Youngstrom, E. A., DelPorto, D., Papish-David, R., Townsend, L., *Danielson, C. K., & Calabrese, J. R. (2002). Clinical decision-making using the General Behavior Inventory. Bipolar Disorders, 4, 34-42. doi: 10.1034/j.1399-5618.2002.40102.

2001

 * Youngstrom, E.A., Findling, R.L., *Danielson, C.K., & Calabrese, J.R. (2001). Discriminative validity of parent report of hypomanic and depressive symptoms on the General Behavior Inventory. Psychological Assessment, 13, 267-276. doi: 10.1037//1040-3590.13.2.267