Evidence-based assessment/Step 4: Revise probabilities based on intake assessments

=Prediction: Revise Probabilities Based on Initial Assessments=

Steps to put into practice
The idea of updating a probability based on new information is an old one.

A probability nomogram is a visual way of updating probabilities. It turns Bayes' Theorem into a connect-the-dots exercise.

Psychometric Considerations
With regard to the concern that the DLRs are influenced by the reliability, convergent, and discriminant validity, and validity of the collateral informants, this is all true. It also is already “baked in” the DLR. The DLR is an alternate form of effect size; in fact, it is possible to convert r or Cohen’s d into an area under the curve (which is a plot of paired coordinates of sensitivity and specificity, meaning that the DLR is tied at a deep level to the effect size -- Swets, Dawes & Monahan talked about the slope of the line tangent to the AUC at a given threshold as a way of expressing this relationship). Why bother with a different effect size? Because DLRs make it possible to translate the assessment result into an updated probability for the individual client. Cohen’s d or Pearson’s r do not by themselves, and they would require even more arcane formula to get there (see Wiggins. 1973, Chapter 4, for example). The strength of the EBM approach to DLRs is that it establishes a common framework for translating all assessment results into a common metric that updates the probability for an individual client. If the reliability of the measure attenuates the observed effect size, then the DLR would be smaller (and the amount that it would change the posterior probability would shrink). If the convergent validity is higher, then the effect size would be larger (and the DLR would be more informative again). More subtly, if the discriminant validity is lower, then the scores would often be elevated for reasons besides the presence of the intended target. This would degrade the specificity, again reducing the AUC (or the Cohen’s d for the separation between the distribution of scores for the target condition versus the distribution for the comparison condition). Again, the net result would be a lower effect size, and necessarily a more modest DLR. At a deep level, the psychometric concerns are already summarized in the DLR -- with the same caveat that the DLR is a property of test scores or assessment results in the context of a particular sample and for a particular purpose… the same caveat that applies to all assessment per current standards (***).

Using Diagnostic Likelihood Ratios
Straus, S. E., Glasziou, P., Richardson, W. S., & Haynes, R. B. (2011). Evidence-based medicine: How to practice and teach EBM (4th ed.). New York, NY: Churchill Livingstone. This is an excellent introduction to Evidence Based Medicine. The chapters about diagnosis and harm both describe and illustrate the use of diagnostic likelihood ratios with probability nomograms.