Evaluation Theories/Week 2: Overview of Evaluation Theory and the Evaluation Theory Tree

Agenda

 * 1) Theory in  Evaluation
 * 2) Evaluator Roles Activity
 * 3) Intro to Eval Theory & Theorist
 * 4) Closing Activity & Pick Theorists for Panels

Theory in evaluation:

 * Social Science Theory
 * Program Theory
 * Relationships Among those

Unpack the bigger concept of evaluation theory. Talk abotu what’s under the umbrella of Eval Theory

Evaluation

 * 1. Purpose
 * 2. Roles
 * 3. Approaches

Introduce you to the theory tree: On your reading list.

Questions you’re dying to ask? (J: If I have a question, I’ll email it. . . )

Novice evaluators think everything in their life is siloe’d and compartmentalized. (J: uh. . . no. . .) -
 * 1. Social Science theory;
 * Social Science Theory attempts to provide generalizable and verifiable knowledge about the principles that shape social behavior. (J: AKA: An attempt.) (Donaldson & Lipsey, 2006)
 * 2. Program Theory;
 * 3. Evaluation Theory

The social aspect of any behavior: social psychology theories; of how humans interact; what behavior (causes?) -

Are we just referring to social psychology? (Talking about social science as a broad spectrum)

What do theories do, how do they help us?
 * 1. Guide Research
 * 2. A priori reason for doing what you’re doing rather than exploring blindly
 * 3. Make predictions (that you can then try to falsify (aka show don’t work)
 * 4. Classify
 * 5.

Examples from social sciences disciplines:
 * Self-Determination Theory …. (She lists it
 * Hope Theory; Agency + Pathways = Goal Attainment
 * Goal Disruption Theory:
 * Cognitive Theories:
 * Organizational Theories of Management Behavior
 * Plaget’s Stages of Cognitive Development:

When we are thinking about social science theory, we need to think more broadly than just “theories” - beacuse sometiems we have a lot of ressearch to support a particular phenomena, but don’t have a well-developed theory to explain that.

There is a lot of social science research around multiple topics that we need to constantly think of as we evaluate programs.

Now, how is a social science theory different than a program theory?
Program theory (J; what they mean by this is basically a program process map: it's not "theory about how to run programs") is specific to the program, whereas social science theory (J: is the body of knowledge in journals and institutions and papers and publications in the social sciences...)

What is Program Theory? -
 * A) “The theory of how the program works.” (J: They don't distinguish between different senses of the word "theory")
 * B) “A statement of the assumptions about why the intervention should affect the intended outcomes.
 * The theory includes hypothesized links between (a) the program requirements and activities, and (b) the expected outcomes; it is depicted in the logic model.” - http://www.evaluationtoolkit.org/glossary (J: To what extent do they have the resources to utilize; how those two combine impact the benefits…)
 * “A map or model of how a specific program is supposed to work.”
 * "Captures main factors that link a program with its presumed outcomes.”
 * “May be entirely implicit” - They might have never articulated before how they think a program is supposed to work.
 * “Could be rooted in social science theory.”

(J: This sounds like a model, not a theory. . .)

(J: theory |ˈTHēərē, ˈTHi(ə)rē| noun (pl. theories) a supposition or a system of ideas intended to explain something, esp. one based on general principles independent of the thing to be explained: Darwin's theory of evolution. • a set of principles on which the practice of an activity is based: a theory of education | music theory. • an idea used to account for a situation or justify a course of action: my theory would be that the place has been seriously mismanaged. • Mathematics a collection of propositions to illustrate the principles of a subject. (New Oxford American Dictionary)

Program Theory:
Org Plan
 * Program, facilities, Personnel, activities; Program-target service transactions; Target interactions with delivery system; Service Utilization Plan
 * Impact Theory; Proximal Outcomes; Distal Outcomes (Rossi et al, 2004

What I want you to do right now is think deeply about the relationship between social science theory and research and a program theory.

I want you in your groups to identify specific examples of social science theory research that can inform this model of the description that is provided to you:

(J: I don’t believe in social science theory in the absence of seeing a logic model for how it is expected to work, and an evaluation of the strength of that logic model!)

Identify a couple of examples where you can fit in Social Science theory;

Then i’ll give you 10 minutes to work in your groups of ≈4. (/whatever you want)

Where did you start?

 * 1. Developmental: You can’t have the same cookie-cutter program for each developmental program; you have to gear it to what stage they’re at. Mentor: You won’t believe how many times I have to tell people that!
 * 2. Started with target population needs. What would be a PSYCH-LIT Search term that you would use to “helping at risk kids”
 * 3. Optimal outcome: Concept self esteem - looked at self-perception theory: Attribution theory for yourself (as opposed to peers around you) -
 * 4. How to measure the outcome; see if the program is actually working

16. J: We started with Methods that help. . .Stated goals;

Do we have any research to indicate about community service learning? - Research on Advocacy, Altruism; -
 * 1. Youth Leadership; ___ and Transformational Leadership theories; Experiential Learning theories.
 * 2. Academic Support: we know a whole bunch that you’re clients don’t know.
 * 3.

Q: We want to find theories; ____ donation theory; how that applies to this program; fact that they offer lots of these actitivies fosters autonomy of getting to select what intrinsitly motivates you; and the competence piece ___; hopefully you’r emore intrinsicly motivated to go; and we knwo that people don’t receive benefits unless the program is received in sufficient dosage. M: Yeah; I use ___ quite a bit; and I want to clue in antecedents;

Staff to implement; It’s like; ‘okay; great our outcomes are ____” (A bunch of goals) - most clients out there don’t have the expertise to think abotu social science theory of research (J: It’s a walled garden: it’s ridiculous: This information should be be publically accessible: Those decades of research are behind ridiculous paywalls, in very bad knowledge management systems that require inordinate amounts of time to learn to use effectively: fundamentally inefficient given current IT capabilities – needing the creation of )

Example: S1: Peer educator / mentor aspect of it, not just having the staff; bringing members of the community in so that younger ones could look up to them; - so that they grow into a peer mentor role as well. Retention rate: Students may be successful in how they’re measuring their outcomes; but there might be other outcomes that they’re not addressing: technically successful in achieving outcomes but members would only stay for an average of 2 years then drop out of the program.

When you think about personnel selection; content expertise needed for employees to implement services effectively; how that might relate to quality of implementation of services; organizational culture - theories related to that; so what I hope this does for you is makes you think that social science theory relates to absolutely every little word on a program theory’s model / framework. - And that as you are beginning to think about this evaluation theory space, think about how we can use and actively / intentionally integrate the social science theory into that landscape:

S2: how does learning about the social science theory affect our ability as evaluators to be mindful of confirmation bias or our own tendencies to say, “There’s so much research on this; you should adapt your program to these other models” without seeing;

A: A lot of embedded questions in that one quesion:
 * 1. To what extent are the measures validated in our literature applicable to the programs receiving funds to operate? - In-line or very different? – We have a lot of data on european american middle class white kids; measurement perspective; think critically about eval design stage; “are we going to take a measure off the shelf and give it to this local context culture? - No! maybe we need to do pilot testing or other things.”
 * 2. These programs have not been empirically validated: they’ve got money; are passionate about it; want to make a difference; what they have, if there isn’t any basis in the literature for connecting the activities to the outcomes, then that’s a conversation to have: “We can invest the resources in the eval, but from the outsider perspective it doesn’t look like these are aligned very well: how does this make sense in your context or culture?” - Cultural competencies: Knowing what you don’t know.

This is the question for Eval theory: There are a lot of different roles and approaches that you can take, and soem don’t agree with the other: my approach is to work with the client where they’re at, and be as supporting, comforting, approachable, neutral as possible, – which I can’t– but if you have a different approach; we’ll talk about this; the Eval theory: you saw everybody has a different approach: Similarities; differences; write that question down.

'''Q: What about places where it’s a limited amount of money;. .. you then look for the research; because coloring can be useful for people of all ages. . . .- this would be the best way to utilize what you have. . . '''

Part of the other role of the evaluator, which is the educator; lets move on ; you just did the activity; good job.

Defining the landscape: Evaluation Theory:
Purpose : Approach : Roles.

We talked about how vast the field is; but there is still diversity within the evaluation field; in what they think the purpose is, in the role; and in the approaches that they take.

If we are to think about evaluation purpose:

“To determine the worth or merit of something”

J: New Oxford American Dictionary: evaluation |iˌvalyo͞oˈāSHən| noun "the making of a judgment about the amount, number, or value of something; assessment: the evaluation of each method | an initial evaluation of the program."

10 min (J: What people are coming up with are not definitions. . .)

When we think about commonalities across definitions; this intro snapshot to the purpose of evaluation; There are a lot of different purposes:

That systematic assessment: we’re not just throwing together some professors and seeing what happens; there’s a plan for us to put this together appropriately.

Program process vs. program outcome. When we think about the extent to which these programs were supposed to be producing outcomes;

What is Explicit Vs. Implicit Standards? - Certain outcomes come from specific groups; Grades for inner-city minorities. . ..

(J: Minority: it’s specific minorities; not all minorities)

Growth; rate of change commensurate with what the rate of change should be depending on the final endpoint; across all of the different definitions there’s still some idea of, “In comparison to what?” -

Or is there some sense of comparison for understanding and interpreting these different program effects? - So; we’re done with Purpose.

(PROGRAM) EVALUATOR ROLES:
2014-01-29 14:17:48; Short Break & ROLE SURVEY

(SKOLITS, G., MORROW, J., & BURR, E. (2009)

(J: Saw this as different competencies; not necessarily different roles.)

Evaluation Theory Tree; How these three pieces fit together to form the different pieces of the puzzle; approaches; we can see how some align more with some roles than with others. And to see where you fit within this schema; so you can decide what might work for you and what you can incorporate into your future practice.

Go back and forth talking about the classification system as a whole and its structure.

For every decision; something is lost and something is gained. Classification systems: Strengths; is there something missing from all of them: What important dimension do you think is not captured;

Towards the end of the semester you have an opportunity to make something yourself.

(J: they kept talking about the person’s ideas having an impact;)

Social Science theory

 * 1. has empirical basis
 * 2. Predictive

These are kinds of “Approach”, “Model” - but we lack an empirical basis.


 * 1. Prescriptive Theory (Model?)
 * 2. Descriptive Theory; (?)

(J: Before you use a word, you should do an analysis on whether it is the best word…but perhaps this is too much: before you REPEAT a word, you should do analysis on whether it is the best word. . .- OR when it reaches a certain level of influence, … you should do analysis on whether it is the best word)

“Our field contrasts the so-called prescriptive theory of evaluation practitioners with more traditional forms of social science theory, labeled descriptive theory (Alkin & House, 1992).” - To isolate those causal relationships.

Uses of Social Science Theory:
 * 1. A map;
 * 2. Match theories to context;
 * 3. Stimulates discussion about whether it’s a good one
 * 4. Can make theorists themselves re-evaluate where they are as a theorist

To be considered a theory you have to talk about: think about and refine important issues to evolve these theories.

It also starts helping us play ourselves; when you consider the tree as a whole you consider where you stand: your epistemology, process of inquiry, and where you see yourself in the world: where you want to help in the world.

To be considered a theory to alkin, you need to have talked about the
 * 1) Methodology
 * 2) Manner by which things were judged and valued
 * 3) Focus of the evaluation effort

And he also used questions about:
 * 1) Primary purpose of the evaluation
 * 2) Role that they made in these decisions
 * 3) Role of those involved

Choosing who got placed on what branch;

In their placement on the tree. He acknowledged that this was a simplified version of the complex ideas of the world that had been built up. When Alkin placed people on his tree he thought about,
 * "Where would these evaluators make concessions?"
 * "What would they cling to most tenaciously?"

(J: This tree describes the individuals who are part of history of Eval; it doesn’t describe EVAL.)

It is a motivation; how the field was born in the 60s; need to be accountable for investment;

Types of Accountability:
“Alkin (1972a) defines three types of accountability: (1) goal accountability, (2) process accountability, and (3) outcome accountability.” (p. 23)

Social Accountability
- Biggest questions and debates within social sciences and within evaluation; - What methods should we use to achieve this accountability? (J: This is so within the paradigm of social sciences) - Different fields come together to form our opinions about that; Physical sciences; Psychology; Anthropology (J: They don’t have much from the fields of Engineering and Information Technology. . . wow….) -

What we can know and how to know it.
- What is knowledge and how do we know? - a piece of this paradigm of an “ontology” - how you view nature of reality; within that is how you go about generating knowledge within. -

(J: Why are we discussion this in class; explaining the concept of this tree; in class, instead of writing papers on it or sending emails? What are we supposed to do in class?)

A lot of epistemologies are not made explicit in people’s writings; actually very few of them.

It’s something to keep an eye out when you’re doing independent research; – Accountability is the contextual, political, societal drive. (That’s just me- how I think about it)

POST-POSITIVISM: There is objective reality; we can approximate it but not quite get it; what’s flawed is the individual; you are looking at the things which produce the most agreed-upon information.

CONSTRUCTIVISTS: Knowledge and knower are linked; biases are always there. Thus cause and effect are almost possible to find (J: How impossible) - Post-positives attempt to control bias; constructivists tend to

PRAGMATISTS: Value itself is important in inquiry-making process.

(J: EACH OF THESE I THOUGHT, “I AM THIS!” UNTIL I READ THE NEXT ONE:) - Not it is objectivity; )

(J: This works in Arabic. . . right to left:)) - All of these are absolutely necessary!

(J: Cook; Prolific; lots of options;

Epistemology
CRONBACH - Role of Evaluation in political things; he’s methods-centric but leans in the use direction for his attention towards that. His book towards reform of program evaluation is my favorite book. He’s very broad in his writings; you can expect to be exposed to ; UTOSCronbach’s (1982) UTOS (population, treatment, observation, and setting) (p. 121)

SCRIVEN Focuses on a broad range of evaluands; often talks about Products, Personnel, this more broad schema of evaluands, whereas most of the people on this tree talk about social programs. Bad is bad quote (J: – When he goes into methods, his approach is really opaque; seems to be based on “the professional” - more on the person than the process?)

CAMPBELL

EISNER;

METHODS
HUEY CHEN Theory-Driven Evaluation- intersection between social science theory and evaluation theory. Program design side; idea of building program theory is very applicable for people who want to be on the intervention side of research.

CAROL WEISS Entire career in federal government; policy change. Strong advocate of experimental methods to provide strong ammunition to be a voice at policy-making table. Quickly realized that evidence is one tiny voice at the decision-making table; so wrote a lot about sue. “Enlightenment use’ and other ideas.

VALUING
SCRIVEN

. ..

HOUSE (J: This is about the USE of evaluation; its ridiculous to think about this being in the definition of “Evaluation” - they’re responding to specific things within)

JENNIFER GREENE Story: Got to pick her up from the airport; Stories about Cronbach; Stakeholder participation;

LINCOLN & GUBA (I think) Constructivist; sequential interviewing; Hermetic circle; building consensus through shared meaning & dialogue (J: another USE and Purpose thing)

J: Put this into BPMN!!!! (I thought of trying to do scriven but – People should be metadata: The ideas (and correlation between the ideas) should be the main object.

David Fetterman - Very friendly; a big self-promoter, which might have been part of it; founder of empowerment evaluation; sees it as something evaluators teach clients to do for themselves for the purpose of emancipation and ____.

A lot of people think that what he does is not “evaluation” because it’s not done by “Evaluation professionals” - (J: need to define evaluation and have accepted tenses of it. . .)