Evaluation Essentials - Beth Osborne Daponte

. .

Résumé :

Preface ix

Acknowledgments x

One: Introduction 1

Learning Objectives 1

Introduction 2

Size and Importance of Nonprofit Sector 3

The Evaluation Framework 5

Structure of Book 9

Summary 9

Key Terms 10

Discussion Questions 10

Two: Describing the Initiative 11

Learning Objectives 11

Introduction 12

Reasons to Rigorously Describe the Program 13

The Logistics of Describing the Program 14

Common Mistakes Made Describing Initiative and Programs 16

The Program Is Alive and So Is Its Description 18

Theory of Change 19

The Program Logic Model 25

Analyzing the Program Logic Model 34

Challenges of Multisite Programs 35

Program Implementation Model 36

Examples of Program Descriptions 37

Summary 67

Key Terms 68

Discussion Questions 68

Three: Laying the Evaluation Ground Work 69

Learning Objectives 69

Evaluation Approaches 70

Framing Evaluation Questions 72

Insincere Reasons for Evaluation 75

High Stakes Evaluation 75

Who Will do the Evaluation? 76

External Evaluators 77

Internal Evaluators 79

Independence of the Evaluator 79

External Versus Internal Evaluators 80

Confidentiality and Ownership of Evaluations 81

Endorsement and Action on Recommendations 82

Evaluation Policy 83

The Evaluation Report 84

Summary 86

Key Terms 86

Discussion Questions 86

Four: Causation 87

Learning Objectives 87

Introduction 88

Necessary and Sufficient 88

Setting Cutoff Points and Causal Interpretation 91

Intervening Variables 95

Types of Causal Effects 97

Lagged Effects 98

Permanency of Effects 98

Functional Form of Impact 99

Spectacular Causes and Effects 100

Summary 101

Key Terms 102

Discussion Questions 102

Five: the Prisms of Validity 103

Learning Objectives 103

Introduction 104

Statistical Conclusion Validity 105

Types of Erroneous Conclusions: Type 1 and Type 2 Errors 106

Threats to Statistical Conclusion Validity 107

Internal Validity 111

Threats to Internal Validity 111

Summary 118

Construct Validity 119

Threats to Construct Validity 120

Summary 122

External Validity 122

Considerations for Determining the External Validity of Studies 122

Summary 123

Conclusions 124

Key Terms 124

Discussion Questions 124

Six: Attributing Outcomes to the Program: Quasi- Experimental Design 126

Learning Objectives 126

Introduction 127

Quasi- Experimental Notation 127

Frequently used Designs that do not Show Causation 128

One- Group Posttest- Only 128

Posttest- Only with Nonequivalent Groups 129

Participants' Pretest- Posttest 130

Designs that Generally Permit Causal Inferences 131

Untreated Control Group Design with Pretest and Posttest 131

Delayed Treatment Control Group 137

Different Samples from the Same Populations Design 140

Nonequivalent Observations Drawn from One Group 141

Equivalent Groups Using Switched Measures 142

Cohort Designs 144

Time Series Designs 146

Summary 148

Key Terms 149

Discussion Questions 149

Seven: Collecting Data 150

Learning Objectives 150

Introduction 151

Informal and Loosely Structured Interviews 152

Focus Groups 154

Surveys 160

Survey Instrument Design 161

Target Populations and Sampling 169<...

Biographie:

Preface ix

Acknowledgments x

One: Introduction 1

Learning Objectives 1

Introduction 2

Size and Importance of Nonprofit Sector 3

The Evaluation Framework 5

Structure of Book 9

Summary 9

Key Terms 10

Discussion Questions 10

Two: Describing the Initiative 11

Learning Objectives 11

Introduction 12

Reasons to Rigorously Describe the Program 13

The Logistics of Describing the Program 14

Common Mistakes Made Describing Initiative and Programs 16

The Program Is Alive and So Is Its Description 18

Theory of Change 19

The Program Logic Model 25

Analyzing the Program Logic Model 34

Challenges of Multisite Programs 35

Program Implementation Model 36

Examples of Program Descriptions 37

Summary 67

Key Terms 68

Discussion Questions 68

Three: Laying the Evaluation Ground Work 69

Learning Objectives 69

Evaluation Approaches 70

Framing Evaluation Questions 72

Insincere Reasons for Evaluation 75

High Stakes Evaluation 75

Who Will do the Evaluation? 76

External Evaluators 77

Internal Evaluators 79

Independence of the Evaluator 79

External Versus Internal Evaluators 80

Confidentiality and Ownership of Evaluations 81

Endorsement and Action on Recommendations 82

Evaluation Policy 83

The Evaluation Report 84

Summary 86

Key Terms 86

Discussion Questions 86

Four: Causation 87

Learning Objectives 87

Introduction 88

Necessary and Sufficient 88

Setting Cutoff Points and Causal Interpretation 91

Intervening Variables 95

Types of Causal Effects 97

Lagged Effects 98

Permanency of Effects 98

Functional Form of Impact 99

Spectacular Causes and Effects 100

Summary 101

Key Terms 102

Discussion Questions 102

Five: the Prisms of Validity 103

Learning Objectives 103

Introduction 104

Statistical Conclusion Validity 105

Types of Erroneous Conclusions: Type 1 and Type 2 Errors 106

Threats to Statistical Conclusion Validity 107

Internal Validity 111

Threats to Internal Validity 111

Summary 118

Construct Validity 119

Threats to Construct Validity 120

Summary 122

External Validity 122

Considerations for Determining the External Validity of Studies 122

Summary 123

Conclusions 124

Key Terms 124

Discussion Questions 124

Six: Attributing Outcomes to the Program: Quasi-Experimental Design 126

Learning Objectives 126

Introduction 127

Quasi-Experimental Notation 127

Frequently used Designs that do not Show Causation 128

One-Group Posttest-Only 128

Posttest-Only with Nonequivalent Groups 129

Participants' Pretest-Posttest 130

Designs that Generally Permit Causal Inferences 131

Untreated Control Group Design with Pretest and Posttest 131

Delayed Treatment Control Group 137

Different Samples from the Same Populations Design 140

Nonequivalent Observations Drawn from One Group 141

Equivalent Groups Using Switched Measures 142

Cohort Designs 144

Time Series Designs 146

Summary 148

Key Terms 149

Discussion Questions 149

Seven: Collecting Data 150

Learning Objectives 150

Introduction 151

Informal and Loosely Structured Interviews 152

Focus Groups 154

Surveys 160

Survey Instrument Design 161

Target Populations and Sampling 169