image

CONTENTS

Figures, Tables, and Exhibits

Acknowledgments

The Authors

Introduction: The Promise and Risks of using Program Theory

Part 1 : Key Ideas in Program Theory

Chapter 1 : The Essence of Program Theory

Evaluation Without Program Theory

Evaluation with Program Theory

Summary

Exercises

Chapter 2 : Variations of Program Theory over Time

A Short History of Program Theory

Terminology in Program Theory

Key Ideas in Program Theory

Summary

Exercises

Chapter 3 : Common Myths and Traps

Some Common Myths

Traps to Avoid When Developing and Using Program Theory

Summary

Exercises

Part 2 : Assessing Your Circumstances

Chapter 4 : Scoping Intended Uses

Why Intended Use Matters

Using Program Theory

Summary

Exercises

Chapter 5 : The Nature of the Situation and the Intervention

Simple, Complicated, and Complex

Focus

Governance

Consistency

Necessariness

Sufficiency

Change Trajectory

Summary

Exercise

Part 3 : Developing and Representing Program Theory

Chapter 6 : Processes to Identify or Develop a Program Theory

Process Options for Developing Program Theory

Decision 1: Who Should Be Involved in Developing a Program Theory, and How?

Decision 2: What Is an Appropriate Mix of Approaches for Developing or Eliciting the Program Theory?

Decision 3: How Might Workshops and Interviews Be Used in Developing Program Theory?

Decision 4: As Challenges Arise, How Should They Be Addressed?

Decision 5: How Much Time and Resources Should Be Invested in Developing or Identifying a Program Theory?

Decision 6: When Is It Time to Revisit a Program Theory?

Summary

Exercise

Chapter 7 : Developing a Theory of Change

Features of a Theory of Change

Situation Analysis: Understanding the Problem, Its Causes, and Its Consequences

Focusing and Scoping

Outcomes Chain

Summary

Exercises

Chapter 8 : Developing a Theory of Action

Preparing the Theory of Action

Success Criteria for a Theory of Change

Assumptions About Factors That Affect Successful Achievement of Outcomes

How to Identify Factors That Are Likely to Affect Outcomes

Identifying What the Program Does

Pulling the Theory of Change and the Theory of Action Together in a Matrix

Summary

Exercise

Chapter 9 : Representing Program Theory

Options for Representation

Representing Complicated Program Theory

Representing Complex Program Theory

What Makes a Good Representation of Program Theory

Should Logic Models Include SMART Measures?

Summary

Exercises

Chapter 10 : Critiquing Program Theory

Criteria for Assessing Internal Validity

Criteria for External Validation

Engaging Stakeholders in the Review

Responding to the Results of a Review of a Program Theory

Summary

Exercise

Part 4 : Resources for Developing Program Theory

Chapter 11 : Some Research-Based Theories of Change

Theory of Reasoned Action and Theory of Planned Behavior

Stages of Change Theory

Empowerment Theory

Diffusion Theory

Socioecological Theory

Network Theory

Selecting and Using Theories of Change

Summary

Exercises

Chapter 12 : Some Common Program Archetypes

Some Important Program Archetypes

Advisory, Information, and Education Program Archetype

Carrots and Sticks Program Archetype

Case Management Programs Program Archetype

Community Capacity-Building Program Archetype

Product or Direct Service Delivery Program Archetype

Deciding which Program Archetype Applies to a Program

Summary

Exercise

Chapter 13 : Logic Models Resources

Pipeline Logic Models

Variations of Outcomes Chain Logic Models

Technology for Representing Program Theory

Summary

Exercise

PART 5 : Using Program Theory for Monitoring and Evaluation

Chapter 14 : Developing a Monitoring and Evaluation Plan

Using Program Theory for Performance Monitoring

Making Choices About What to Measure Within the Program Theory

Including Comparisons as Part of the Performance Information System

Using Program Theory to Plan an Evaluation

Considerations When Using Program Theory to Design Evaluations of Complicated and Complex Programs

Summary

Exercises

Chapter 15 : Causal Inference

The Need to Be Scientific and Pragmatic

A Framework for Causal Analysis Using Program Theory

Congruence

Counterfactual Comparisons

Critical Review

Summary

Exercises

Chapter 16 : Synthesis and Reporting

Synthesis and Reporting for a Single Evaluation

Synthesis and Reporting Across Evaluations

Summary

Exercises

New Frontiers for Program Theory

References

Index

image

FIGURES, TABLES, AND EXHIBITS

Figures
1.1An Evaluation of An Apple a Day Without Program Theory
1.2Simple Pipeline and Outcomes Chain Logic Models
1.3A Logic Model Showing a Simple Program Theory for An Apple a Day Based on Improved Vitamin Intake
1.4Logic Models Showing Different Possible Causal Mechanisms Involved in Eating an Apple a Day
5.1A Complicated Program Theory for a Multilevel Intervention
5.2Emergent Focus of the Intervention: Intermediate Intended Outcomes
5.3Emergent Focus of the Intervention: Long-Term Intended Outcomes
5.4Emergent Activities to Achieve Stable Intended Outcomes
5.5An Intervention Dependent on the Contribution of a Subsequent Intervention
5.6Outputs from One Intervention Forming the Inputs to a Subsequent Intervention
5.7A Complicated Program Theory for an Intervention That Works Only in Conjunction with Other Interventions
5.8Complex Situation: Where Small Differences in Context Lead to Unpredictably Different Impacts
6.1Amalgamating Different Stakeholder Theories
6.2Positive and Negative Program Theories
6.3Linear Theories with Branching Structures
7.1Outcomes Chain for the Mature Workers Program
7.2Conversion of a Flow Diagram of Activities to a Chain of Outcomes
8.1Utilization of Evaluations: Outcomes Chain and Definitions of Successful Outcomes
9.1A Pipeline Logic Model of a Computer Skills Project
9.2An Outcomes Chain Logic Model of the Computer Project
9.3Complicated Logic Model Showing Multiple Organizations
9.4Logic Model for a Program Involving Multiple Partner Organizations
9.5Stacked Logic Model
9.6Logic Model Based on Outcomes Mapping
9.7Complicated Logic Model Showing Multiple Causal Strands
9.8Logic Model Showing Two Different Sets of Intended Outcomes
9.9The Seven D Approach to Developing and Using an Emergent Program Theory
9.10The U Process Theory of Change
9.11A Broad Theory of Change with Emergent Theory of Action: Dealing with Striga
9.12A Theory of Change with Emergent Theory of Action: Evaluation Capacity Building
9.13A Theory of Change with Cyclic Learning at Each Stage
9.14A Theory of Change with Emergent Theory of Action: Mental Health
9.15A Systems Dynamics Model of a Tobacco Control Initiative
9.16A Poor Logic Model in Terms of Focusing on the Main Elements and Explaining How It Works
9.17A Poor Logic Model in Terms of a Dead End
9.18Improved Logic Model in Terms of Linking a Dead End to an Ultimate Impact
9.19Poor Logic Model in Terms of Meaningful Arrows
9.20Improved Logic Model in Terms of Meaningful Arrows
9.21More Clearly Drawn Logic Model
9.22Poor Logic Model in Terms of Representation of Direction of Intended Change
9.23Poor Logic Model in Terms of Representation of Sequence and Consequence
9.24Improved Logic Model in Terms of Sequence and Consequence
9.25Logic Model Emphasizing Four Causal Strands, Each with Two Stages
9.26Poor Logic Model in Terms of Too Many Indiscriminate Feedback Lines
9.27Poor Logic Model in Terms of Visual Elements That Do Not Add Meaning
11.1Schematic Representation of the Theory of Planned Behavior
11.2Temporal Dimension as the Basis for the Stages of Change
11.3S Curve of Diffusion Theory
11.4Socioecological Model for Child Development
11.5Evolving Networks
12.1Generic Outcomes Chain for Advisory, Information, and Education Program Archetype and Applications of the Generic Outcomes Chain
12.2Generic Outcomes Chain for the Archetype of Motivational Programs That Use Incentives and Applications of That Chain
12.3Generic Outcomes Chain for the Archetype of Motivational Programs That Use Deterrence and an Application of That Chain
12.4Generic Outcomes Chain for Case Management Archetypal Program
12.5Generic Outcomes Chain for Community Capacity-Building Archetype Program
12.6Generic Outcomes Chain for Service or Product Archetype Programs and Applications of the Archetype
13.1Charities Evaluation Planning Triangle Logic Model of a Community Health Center
13.2United Way Logic Model of a Community Health Center
13.3W. K. Kellogg Foundation Logic Model of a Community Health Center
13.4Bennett’s Hierarchy Logic Model of a Community Health Center
13.5University of Wisconsin Logic Model of a Community Health Center
13.6Example of a People-Centered Logic Model
13.7Example of ActKnowledge/Aspen Institute Theory of Change Logic Model with Assumptions and Interventions
13.8Picture Pipeline Logic Model in Word
13.9Picture Pipeline Logic Model in Excel
13.10Picture Pipeline Logic Model in Visio
13.11Outcomes Chain Logic Model in DoView
13.12InnoNET Logic Model Builder Example
14.1Simplified Outcomes Chain Showing Levels for Which the Crisis Call Center for Problem Gamblers Was Held Accountable
14.2The Logic of Planning an Evaluation
14.3Overarching Program Theory
14.4Nonlinear Program Theory
15.1Results Chain Logic Model for An Apple a Day
16.1Reporting Findings Using DoView
16.2Reporting Findings Using Shockwave
Tables
1.1Using Program Theory to Interpret Evaluation Findings
2.1Distinctions Sometimes Made Between Program Theory and Program Logic
2.2Definitions Used in This Book
3.1Some Common Myths About Program Theory
3.2Seven Traps to Avoid When Developing and Using Program Theory
4.1Implications of Different Aspects of Use for Developing and Using Program Theory
5.1Simple, Complicated, and Complex Problems
5.2Distinguishing Simple, Complicated, and Complex Aspects of Interventions
5.3A Framework for Addressing Simple, Complicated, and Complex Interventions
6.1Considerations for Choice of Approach, Methods, and Sources of Information
7.1Components of a Program Theory
7.2Features of a Theory of Change for Addressing Common Traps
8.1Components of a Program Theory
8.2Theory of Action Features That Can Avoid Common Traps
8.3Factors That Affect the Success of Different Types of Services
8.4An Alternative Approach for Focusing on Outcomes, Their Desired Attributes, and Factors That Affect Success
8.5Application of the Program Theory Matrix to One Outcome from the Outcomes Chain for the Mature Workers Program
9.1A Realist Matrix Logic Model of the Computer Project
9.2Options for Formatting Logic Models
9.3A Realist Summary of Foster and Hope’s Study of the Effects of PEP on Crime in the Hull Experimental Estate
9.4Options for Representing Interventions with Important Complex Aspects
11.1Stages of Change Theories Useful Strategies for Different Processes of Change
13.1Variations on the Layout of Components in Pipeline Logic Models
13.2Variations on the Number of Components in Pipeline Logic Models
13.3Variations on the Labels Used for Components in Pipeline Logic Models
13.4Variations on the Definitions of Labels Used in Pipeline Logic Models
13.5Additional Components Sometimes Included in Pipeline Logic Models
13.6Variations of Pipeline Logic Models
13.7Components of a Logframe
13.8Variations of Outcomes Chain Logic Models
13.9Technology for Drawing Logic Models
13.10Tabular Pipeline Logic Model in Word for Meals on Wheels
14.1Some Types and Purposes of Evaluation at Different Stages of a Program and How Program Theory Can Be Situated
14.2Performance Information Matrix for One Outcome in the Outcomes Chain
14.3Examples of Evaluation Questions, Performance Information, and Methods of Data Collection for the Outcome: Job Retention for Mature Workers
14.4Overview of Characteristics of Simple, Complicated, and Complex Programs
14.5Some Implications of Various Characteristics for Monitoring and Evaluation
14.6Complicated and Complex Aspects of the Stronger Families and Communities Strategy Evaluation
14.7Levels of the Evaluation That Provided Opportunities to Address Emergent Issues
15.1Methods and Techniques for the Three Components of Causal Analysis
15.2Interpreting Findings in Terms of Their Congruence with Program Theory
15.3Pattern of Results Suggesting Implementation Failure
15.4Pattern of Results Suggesting Engagement or Adherence Failure
15.5Pattern of Results Suggesting Theory Failure: Intermediate Outcome
15.6Pattern of Results Suggesting Theory Failure: Final Outcome
15.7Pattern of Results Suggesting Theory Failure: Alternative Causal Path
Exhibits
6.1Questions for Drawing Out Program Theories
6.2Some Challenges When Developing a Program Theory
7.1Arguments for Including in the Scope of the Program Some Outcomes That Are Not Its Focus
7.2Steps for Developing an Outcomes Chain
7.3Some Key Points About Outcomes Chains
8.1Useful Features for Inclusion in the Theory of Action
9.1Guidelines for Developing Effective Logic Models
11.1Examples of Theories of Change
11.2Examples of Blending Various Aspects of Different Theories
14.1Choosing Which Parts of the Program Theory to Measure Routinely
14.2Some Suggestions for Choosing Which Outcomes in the Outcomes Chain to Measure
16.1Using an Outcomes Chain as a Structure for an Oral Report: An Evaluation of Road Safety Presentations by Police to High School Students

ACKNOWLEDGMENTS

MANY PEOPLE HAVE helped us along the journey to this book.

Our work builds on the contributions of the pioneers and innovators in program theory evaluation and those whose work underpins key concepts in program theory. The work of Dan Stufflebeam, Carol Weiss, Joseph Wholey, Claude Bennett, Len Bickman, Carol Fitz-Gibbon, and Lynn Morris laid the foundations for program theory. Benjamin Bloom, Thomas Hastings, and George Madaus’s taxonomy of educational objectives, when linked with Bennett’s Hierarchy, gave rise to the notions of outcome chains, and Tom Hastings’s important article, “Curriculum Evaluation: The Why of the Outcomes,” expanded thinking beyond a focus on outcomes. Bryan Lenne of the New South Wales Program Evaluation Unit contributed significantly to the development of the matrix approach to program theory and the classification of program archetypes. More recently, we have learned much from the work of Michael Patton, Rick Davies, and Boru Douthwaite exploring the use of network theory and systems approaches; the development of realist evaluation and realist synthesis by Ray Pawson and Nick Tilley; and outcome mapping by Barry Kibel, Terry Smutlyo, Fred Carden, and Sarah Earl in terms of using program theory for complicated and complex situations and interventions.

Over more than twenty-five years, we have had the good fortune to work with colleagues and clients who have stretched our thinking about program theory. Sue particularly acknowledges those whose work and insights have contributed to the development of the matrix approach: Carolyn Wells for development of the concept of attributes of success as part of program theory matrices; Steve Baxter for his contribution to thinking about managing external risk factors that affect outcomes and the implications of that for developing a program theory; and Larraine Larri for her assistance with portraying and explaining program theory matrices. Sue also acknowledges the contributions of those who helped identify program theories for particular program archetypes: Bryan Lenne for his contribution to the development of the generic theory for advisory programs drawing on the work of Claude Bennett; Ross Homel for his work on regulatory programs and contribution to the development of the sticks archetypal program theory; Harry Hatry’s work on performance incentive programs that contributed to the development of the carrots archetype; Alison Matthews for her case management approach that drew attention to the need to have models that could accommodate different outcomes for different people using different paths, types of service, and activities; Greg Masters for his work on service delivery programs; and Barry Smith for his work on community capacity-building programs. She acknowledges as well the valuable work of Ellen Taylor-Powell, Larry Jones, and Ellen Henert of the University of Wisconsin Extension Center who, through their online course, Enhancing Program Performance with Logic Models, drew her attention to the potential usefulness of research-based theories of change and prompted her in-depth exploration of those theories.

Patricia’s thinking about program theory owes a particular debt to her mentor, Carol Weiss, and colleagues Tim Hacsi, Tracey Huebner, and Anthony Petrosino at the Harvard Project on Schooling and Children, where she completed a postdoctoral fellowship supported by the Spencer Foundation. She is also grateful for the insights about program theory gained from discussions and evaluation projects with Brad Astbury, Fred Carden, Margaret Cargo, Jane Davidson, Rick Davies, Julie Elliott, Gerald Elsworth, Delwyn Goodrick, Irene Guijt, Ernest House, Bron McDonald, Michael Patton, Ray Pawson, Kaye Stevens, Gill Westhorp, Bob Williams, and Jerome Winston. Thanks also to Susie Elliott, Caitlin Nash, and Russell Stanbrough for their help with editing and manuscript preparation.

The book has benefited considerably from advice and encouragement from Andy Pasternack and Seth Schwartz at Jossey-Bass and from the constructive feedback on earlier drafts of the book from Fred Carden, Jane Davidson, Ernest House, Steve Montague, and especially Michael Patton. We bear responsibility for all remaining deficiencies.

Finally, we thank our families for their support and encouragement throughout the writing of this book and the staff and other stakeholders of the many programs from which we drew the experience on which this book is based.

To Peter, my partner and soulmate, for his encouraging, good-humored, and practical support throughout the writing of this book.

—Sue

To Paul, my dear man, for all he does and has done to make this work possible.

—Patricia

THE AUTHORS

SUE C. FUNNELL is a director of Performance Improvement, a company she established in 1992. She has more than thirty-five years of experience in program design, evaluation, and performance measurement. Since the 1980s, she has been one of the key contributors to the development, dissemination, and use of program theory in Australia. She has supported local, state, national, international, and global government and nongovernment organizations in developed and developing countries, and she has successfully used program theory for evaluation, monitoring, planning, and organizational learning. Her program theory work has been with the Joint United Nations Programme on HIV/AIDS, United Nations Center for Human Settlements, World Bank, Australian National Audit Office, Australian Department of Finance, and government agencies and departments in Australia and New Zealand. She has addressed communication technology; education, training, and leadership development; employment; energy and water; environmental protection; evaluation policy; family, youth, and community services, including alcohol and other drugs, disability, housing, mental health, problem gambling, volunteering, and welfare assistance; industry development; legal systems; natural resources management; occupational health and safety; primary industries; and roads. Funnell, a past president of the Australasian Evaluation Society (AES) and now an AES Fellow, has been awarded the AES Evaluation Training and Services Award for outstanding contributions to the profession of evaluation.

PATRICIA J. ROGERS is Professor of Public Sector Evaluation at the Royal Melbourne Institute of Technology, Melbourne, Australia. She has worked in public sector evaluation and research for more than twenty-five years, with government and nongovernment organizations (international, national, state, and local) across a wide range of program areas, including agriculture, community development, criminal justice, early childhood education, health promotion, Indigenous housing, international development, and legal aid. She has worked on projects with the United Nations Development Programme, World Bank Institute, Network of Networks on Impact Evaluation, U.S. Department of Energy, the Office of the Presidency (South Africa), the Public Service Commission (South Africa), and the Development Bank of Southern Africa. She has presented keynote addresses at conferences of the Australasian, Aotearoa/New Zealand, European, United Kingdom, South African, and Swedish evaluation societies and associations and is on the editorial boards of the journals Evaluation and New Directions for Evaluation. She has been awarded the American Evaluation Association’s Myrdal Award for Evaluation Practice, the Australasian Evaluation Society’s Evaluation Training and Services Award for outstanding contributions to the profession of evaluation, the AES Caulley-Tulloch Prize for Pioneering Literature in Evaluation, and (with Sue Funnell) the AES Best Evaluation Study Award.

INTRODUCTION: THE PROMISE AND RISKS OF USING PROGRAM THEORY

THE 1920S ENTREPRENEUR Carl Weeks once wrote, “If you can dream it, you can build it.” This is the key idea that underpins program theory. Having a vision of where we are going and some clarity about how we plan to get there can help us work together to achieve our goals, and learn from both success and failure.

WHAT PROGRAM THEORY IS

A program theory is an explicit theory or model of how an intervention, such as a project, a program, a strategy, an initiative, or a policy, contributes to a chain of intermediate results and finally to the intended or observed outcomes. A program theory ideally has two components: a theory of change and a theory of action. The theory of change is about the central processes or drivers by which change comes about for individuals, groups, or communities—for example, psychological processes, social processes, physical processes, and economic processes. The theory of change could derive from a formal, research-based theory or an unstated, tacit understanding about how things work. For example, the theory of change underpinning some health promotion programs is that changes in perceived social norms lead to behavior changes. The theory of action explains how programs or other interventions are constructed to activate these theories of change. For example, health promotion programs might use peer mentors, advertisements with survey results, or some other strategy to change perceptions of social norms.

Program theory, under all its various labels, including “theories of change,” “logic modeling,” and “intervention logic,” has grown in popularity over the past twenty years or so. Many government and nongovernment organizations across the world now encourage or require its use for planning, monitoring, and evaluating.

When done well, program theory can produce many benefits. It can develop agreement among diverse stakeholders about what they are trying to do and how, or identify where there are legitimately different perspectives. It can help to improve plans by highlighting gaps and opportunities for collaboration with partners. It can help to set realistic objectives. It can support the development of meaningful performance indicators to track progress and report achievements. It can be used to identify where and why unsuccessful programs are failing or what makes successful programs work, and how they might be reproduced or adapted elsewhere. It can provide a framework to bring together information from many sites, many projects, or many evaluations so that it is possible to learn from the past to improve the future.

Program theory, however, is not always done well. And when it is done badly, it misrepresents what an intervention does and what it can achieve. It can lead to monitoring systems and evaluations that produce an incomplete or distorted picture of what is happening and mistaken judgments about what is effective or efficient. It can demotivate staff and deflect attention from what is important to only what can be easily measured. It can silence important voices or fail to touch those who can act on it. It can take up time without adding value.

The promise of good program theory and the risk of bad program theory have motivated us to write this book. Over more than twenty years, we have worked with small and large organizations in countries all over the world; with municipal, state, and federal government agencies, and nongovernment organizations; on tiny local projects, multimillion-dollar national programs, and whole-of-government strategies; with service deliverers, policymakers, and funders; and in many sectors, including health, education, agriculture, justice, infrastructure, natural resources, community services, community development, and emergency management. Over this time, we have seen diverse approaches to program theory.

What we have learned from this experience, and from the expanding library of empirical research on program theory, is that program theory should be developed, represented, and used not in a formulaic way, but thoughtfully and strategically, in ways that suit the particular situation. We call this purposeful program theory.

PURPOSEFUL PROGRAM THEORY

Greek legend tells of the fearsome hotelier Procrustes who would adjust his guests to match the length of his bed, stretching the short and trimming off the legs of the tall. Guides to program theory that are too prescriptive risk creating such a Procrustean bed. When the same approach to program theory is used for all types of interventions and all types of purposes, the risk is that the interventions will be distorted to fit into a preconceived format. Important aspects may be chopped off and ignored, and other aspects may be stretched to fit into preconceived boxes of a factory model, with inputs, processes, outcomes, and impacts.

Purposeful program theory requires thoughtful assessment of circumstances, asking in particular, “Who is going to use the program theory, and for what purposes?” and, “What is the nature of the intervention and the situation in which it is implemented?” It requires a wide repertoire, not a one-size-fits-all approach to program theory.

Purposeful program theory also requires attention to the limitations of any one program theory, which must necessarily be a simplification of reality, and a willingness to revise it as needed to address emerging issues. As the American evaluator Daniel Stufflebeam (2001) has pointed out, evaluators who continue to use an unsuitable program theory are similarly at risk of creating a Procrustean bed for the evaluation.

OVERVIEW OF THE BOOK

The book is designed to help you assess your particular circumstances and develop, represent, and use program theory in appropriate ways. It has options at every stage and examples to help you decide which options to use and how to adapt them to your circumstances. Throughout the book, we draw on examples from our own work and the work of others. (“Our work” refers to projects we have done together and individually.) Each chapter includes exercises to try out new ideas and techniques.

If you are new to program theory, it will be most useful to read the chapters in sequence. If you have some experience or are coming back to the book during an evaluation, you can select the particular chapter you need.

Key Ideas in Program Theory

Part One sets out the key ideas of program theory and how it has developed over time. We explain in Chapter One the essential features of program theory, using the broad policy objective of eating an apple a day to keep the doctor away as an example of how program theory can be used in different ways to learn from success, failure, and mixed results. Chapter Two describes how program theory has developed over time and sorts out the confusion about the different terms that have been used. And Chapter Three introduces seven widespread myths about program theory and seven common traps to avoid.

Assessing Your Circumstances

A key message of this book is the need to approach program theory in a way that suits your circumstances. Therefore, Part Two examines how to analyze the intended uses of program theory and the nature of the situation and intervention.

We explain in Chapter Four why it is important to be clear about who is going to use program theory and for what purposes. A program theory that is useful for developing internal monitoring systems for incremental correction, for example, could be inappropriate for developing performance measures for external accountability. A theory to guide the design of an impact evaluation might not be sufficient to guide a process evaluation that aims to document an unfolding innovation. Being clear about the intended uses of program theory, reviewing this as circumstances change, and considering this when making decisions is an essential part of purposeful program theory.

Chapter Five discusses how to identify simple, complicated, and complex aspects of the program or policy and the situation in which it is being implemented. Program theory can be used for interventions that are simple; that is, they have a single implementing agency and a well-understood causal process that works pretty much the same everywhere. But most interventions have important complicated or complex aspects that program theory needs to address in order not to misrepresent how it works. The implications of complicated and complex aspects of interventions for developing, representing, and using program theory are addressed throughout the book.

Developing and Representing Program Theory

The chapters in Part Three focus on ways of developing and representing program theory.

Chapter Six discusses how to combine three approaches to developing a program theory. A deductive approach focuses on stated policies and procedures and previous research. An inductive approach builds from observing the intervention in action, reviewing previous observations of it, or observing similar interventions. A mental model approach works with stakeholders to articulate their tacit understandings of how the intervention works.

Chapter Seven sets out three steps to develop a program theory. Step 1 is undertaking a situation analysis to identify problems and opportunities and understand the causes and consequences of problems. Step 2 is to decide the program scope: agreeing which aspects of the problem—its causes and consequences—the program will focus on directly and primarily and which will be beyond the direct focus. The more complex the program is, the more fluid the boundaries should be. Step 3 is to articulate an outcomes chain that shows the assumed or hypothesized cause and effect or contingency relationships between immediate and intermediate outcomes and ultimate outcomes or impacts (both short and long term). In this chapter, we address each of these tasks by applying them to an employment program for mature workers, and we provide examples of how these can be done in different ways to suit any situation.

In Chapter Eight, we introduce a structured approach to developing the second part of the program theory, the theory of action, which spells out how the intervention is intended to activate the theory of change. For example, if a program aims to change health behaviors through increasing knowledge of their consequences, will this knowledge be achieved through a public advertising campaign, personal consultations from health professionals, viral marketing from peers, or some other activities? We introduce the program theory matrix: a structured approach that explores systematically the outcomes chain developed in the theory of change. For each of the outcomes in the outcomes chain, the matrix identifies the nature and quantity of program activities that are intended to achieve this and other factors that will affect whether and how well the outcome is achieved. It also defines what success will look like for the outcome. We continue with the example of an employment program for mature workers, introduced in Chapter Seven, to demonstrate the various components of a theory of action.

We look at different types of logic models in Chapter Nine that can be used to represent program theory. Pipeline models show an intervention as a linear series of boxes labeled something like “inputs, processes, outcomes, and impacts.” Outcome chains, which show a series of results leading to the final impacts of interest, have the advantage of being able to represent more complicated and complex interventions where the activities occur throughout the causal chain and are not all present at the beginning of the process. Realist matrices focus on showing how interventions work differently for different groups or in different situations. We discuss what makes a good logic model, do some logic model makeovers, and review some technology for producing these models.

Chapter Ten discusses how to assess the quality of the program theory in terms of its internal coherence and its validity with respect to external considerations. A program theory can be poorly expressed, incompletely expressed, or just plain wrong. It is important to review it systematically during development and periodically throughout its use.

Resources for Developing Program Theory

The chapters in Part Four provide resources to help with the processes of developing and representing program theory. It can be helpful to draw on previous research and planning when developing the outcomes chain.

Chapter Eleven provides information about a number of theories of how change occurs for individuals, organizations, and communities. The theory of reasoned action (Fishbein and Ajzen, 1975), the theory of planned behavior (Ajzen, 1988), and the stages of change theory (Prochaska and DiClemente, 1983) are theories about changing behaviors of individuals. Empowerment theory (Perkins and Zimmerman, 1995) may relate to individuals, groups, or communities. Diffusion theory (E. Rogers, 1995) is largely about changing community behaviors (and behaviors of individuals en masse). Socioecological theory (Bronfenbrenner, 1979) is about mechanisms for change for individuals, families, groups, and communities and the interplay among all of those actors. Network theory (Granovetter, 1973) is about how the relationships, networks, and connections among entities, and not just the characteristics of the entities themselves, affect outcomes. The entities could be individuals, organizations, special issues groups, or even whole countries. There are many other research-based theories of change, and the chapter lists some other potentially relevant theories that could be used as the basis for an intervention’s specific theory of change.

Chapter Twelve outlines some common program archetypes that can be selected, adapted, and combined for particular situations. These include advisory, information, and education programs that seek to change individual behavior by informing decisions; “sticks and carrots,” which work through incentives and sanctions; case management; community capacity development; and direct service delivery.

Chapter Thirteen provides examples of variations on pipeline and outcomes chain logic models.

Using Program Theory for Monitoring and Evaluation

The final part of this book describes how to use program theory specifically for monitoring and evaluation.

Chapter Fourteen explains how to use program theory to identify what aspects of the intervention, the context, and results should be measured and how to use key evaluation questions to focus an evaluation in terms of data collection, analysis, and reporting. Program theory can help to structure a coherent narrative report and a focused analysis, whether reporting the results of a single evaluation or bringing together data from many studies. We provide some suggestions on ways to do this for small and large evaluations.

Even when there is credible evidence that outcomes have occurred, can we be confident that an intervention has caused them or at least contributed to them together with other factors? In recent years there has been a vigorous debate about the suitability of different methods and designs to address the issue of causal analysis. In Chapter Fifteen, we set out a three-part framework for causal analysis when using program theory that can bring to bear the full range of research designs and methods for causal analysis. The starting point is looking for congruence of results with those predicted by program theory. The second part is finding relevant comparisons that indicate the difference that the intervention has made. These can include creating a control group or a comparison group or making other relevant comparisons. The third part is checking out alternative explanations for the results and exceptions to the patterns.

Chapter Sixteen describes ways to bring together information across the different levels of a program theory, or across several interventions that use the same program theory, and how to report this coherently and effectively.

TAKING A STRATEGIC AND ADAPTIVE APPROACH

Program theory can be developed, represented, and used in many ways. Throughout this book, we invite you to take a purposeful approach to program theory, matching it to your situation, checking how it is going, and adapting it as needed to ensure that it contributes to improved interventions and the outcomes you seek.

PART 1

Key Ideas in Program Theory