Meta-analysis

is a systematic technique for reviewing, analysing, and summarising quantitative research studies on specific topics or questions.

This page provides information and resources about how to conduct a meta-analysis. The target audience includes post-graduate students conducting a meta-analysis or beginning researchers in meta-analysis. These pages could also be used by students involved in research methods coursework.

Introduction

 * 1) A meta-analysis is quantitative technique for conducting a "study of studies".
 * 2) Use of meta-analysis has flourished, particularly in the social, health, and medical sciences, since it was developed in the 1970s
 * 3) Meta-analysis was initially developed in response to controversy over traditional, subjective literature review methods (specifically, at the time, those used in to review the psychotherapy outcome studies).

Lecture slides

 * 1) Practical meta-analysis (Lecture slides; Wilson, 1999)
 * 2) How to do meta-analysis (Lecture slides; Basu, 2005)
 * 3) Meta-analysis: combining information(Lecture Slides; LeBauer, 2010) — includes outline of three approaches to inferences:  moment matching, maximum likelihoo, and Bayesian.

How to do a meta-analysis

 * 1) Meta-analysis involves analysing the summary data from many studies. It can be performed by hand, using a spreadsheet and formulae, using scripts, syntax or macros with generic statistics software packages, or by using dedicated meta-analysis software packages.
 * 2) Before starting, identify a clear question(s), e.g., "What are the outcomes of psychotherapy?"
 * 3) Questions can also involve the effect of independent variables, e.g., "Are the outcomes of psychotherapy similar for males and females?"
 * 4) Read other related meta-analyses to get a feel for the kinds of questions asked.
 * 5) Make sure that any independent variables (IVs) and dependent variables (DVs) are very clearly defined.
 * 6) Because of the importance of establishing a well-defined question and variables, developing a peer-reviewed proposal for a meta-analytic study is strongly recommended.
 * 7) It can be helpful to identify several similar or related meta-analytic studies as models for your meta-analytic study. Consider the strengths and weaknesses of their methodologies.
 * 8) Establish clear criteria for selection of studies, e.g., does it need to be published in a peer-reviewed journal, or will you also accept theses and non-peer reviewed papers (e.g., conference papers)?
 * 9) Conduct an exhaustive and systematic literature search, recording your steps along the way (important for the Method - must allow replication)
 * 10) Create a "coding sheet" - this is the list of fields (variables) you want to extract from each study, and how each of the variables are to be coded - get this peer-reviewed, otherwise you will limit the potential/quality of your analyses
 * 11) Enter the data - one study per row, but note that there may be multiple outcomes and/or groups of interest for each study, in which case each of these will receive their own row in the database, with a column to code which type of outcome was measured.
 * 12) Analyse the data using spreadsheet formulae, or by writing syntax commands for a generic statistics package, or by using a dedicated meta-analysis software package (with in-built meta-analysis tools).

Effect sizes

 * 1) Central to understanding meta-analysis is an understanding of effect sizes.
 * 2) The chief value of effect sizes in the context of meta-analysis is that they provide a way to standardise effects across studies using different measures, allowing for common analysis.
 * 3) There are many possible effect sizes, but essentially there are two commonly reported types in meta-analysis:
 * 4) Correlational: e.g., r (product-moment correlation)
 * 5) Mean differences: e.g., Cohen's d, Hedge's g, etc.

Limitations

 * 1) An important limitation of meta-analysis is that its results can only be as good as the original data is valid.
 * 2) Meta-analysis can only analyse the role of independent variables in explaining variance in dependent variables if sufficient data is provided in the original studies.
 * 3) "Apples and oranges" effect - i.e., there is a risk/tendency in meta-analysis to average/mash together disparate effects.
 * 4) Can lack in qualitative insight (e.g., as may be more likely to be contributed by an expert conducting a traditional literature review).

Example meta-analytic studies

 * 1) Hattie, J., Biggs, J., Purdie, N. (1996). Effects of learning skills interventions on student learning: A meta-analysis. Review of Educational Research, 66, 99-136.
 * 2) Hattie, J., Marsh, H. W., Neill, J. T., & Richards, G. E. (1997). Adventure education and Outward Bound: Out-of-class experiences that make a lasting difference. Review of Educational Research, 67, 43-87.
 * 3) Purdie, N., Hattie, J., Carroll, A. (2002). A review of the research on interventions for attention deficit hyperactivity disorder: What works best? Review of Educational Research, 77, 61-99.

Comparison table
Some dedicated meta-analysis software includes:

Alternative (software
Non-dedicated, generic statistics software which can be used for conducting meta-analysis include:
 * R
 * Stata
 * SPSS
 * Excel
 * DStat
 * NCSS

Other comparisons/lists

 * 1) Meta-analysis
 * 2) http://www.um.es/facpsi/metaanalysis/software.php
 * 3) http://www.lehanathabane.com/personal/metalinks.htm
 * 4) http://www.med.umich.edu/csp/Course%20materials/Fall%202005/Rogers_Meta%20Analysis%20software%20packages.pdf

Tasks

 * To do