Published by John Wiley & Sons, Inc., Hoboken, New Jersey.
Published simultaneously in Canada.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400, fax 978-646-8600, or on the Web at www.copyright.com. Requests to the publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, 201-748-6011, fax 201-748-6008, or online at www.wiley.com/go/permissions.

Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages. Readers should be aware that Internet Web sites offered as citations and/or sources for further information may have changed or disappeared between the time this was written and when it is read.

This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is sold with the understanding that the publisher is not engaged in rendering professional services. If legal, accounting, medical, psychological or any other expert assistance is required, the services of a competent professional should be sought.

For general information on our other products and services, please contact our Customer Care Department within the U.S. at 800-956-7739, outside the U.S. at 317-572-3986, or fax 317-572-4002.

Wiley publishes in a variety of print and electronic formats and by print-on-demand. Some material included with standard print versions of this book may not be included in e-books or in print-on-demand. If this book refers to media such as a CD or DVD that is not included in the version you purchased, you may download this material at http://booksupport.wiley.com. For more information about Wiley products, visit www.wiley.com.

Library of Congress Cataloging-in-Publication Data:

Names: Harris, Muriel J., 1955- author.
Title: Evaluating public and community health programs / Muriel J. Harris.
Description: 2nd edition. | Hoboken, New Jersey : Jossey-Bass & Pfeiffer Imprints, Wiley, [2017] | Includes bibliographical references and index.
Identifiers: LCCN 2016025319 (print) | LCCN 2016026276 (ebook) | ISBN 9781119151050 (pbk.) | ISBN 9781119151074 (epdf) | ISBN 9781119151081 (epub)
Subjects: | MESH: Community Health Services-standards | Program Evaluation-methods | Data Collection-methods | Evaluation Studies as Topic | Community-Based Participatory Research
Classification: LCC RA440.4 (print) | LCC RA440.4 (ebook) | NLM WA 546.1 | DDC 362.1072-dc23
LC record available at https://lccn.loc.gov/2016025319

Cover Design: Wiley
Cover Photo: © Mats Anda/Getty Images, Inc.

Cover
Title Page
Copyright Page
Dedication
Preface
Acknowledgments
CHAPTER 1: AN INTRODUCTION TO PUBLIC AND COMMUNITY HEALTH EVALUATION
1. Overview of Evaluation
2. Levels of Evaluation
3. Preassessment Evaluations
4. The Participatory Approach to Evaluation
5. The Participatory Model for Evaluation
6. The Precursors to Program Evaluation
7. Cultural Considerations in Evaluation
8. Summary
9. Discussion Questions and Activities
10. Key Terms
CHAPTER 2: THE COMMUNITY ASSESSMENT: AN OVERVIEW
1. Theoretical Considerations
2. The Ecological Model
3. Data Collection
4. Data Sources
5. Reviewing the Scientific Literature
6. The Report
7. Stakeholders’ Participation in Community Assessments
8. Summary
9. Discussion Questions and Activities
10. Key Terms
CHAPTER 3: DEVELOPING INITIATIVES: AN OVERVIEW
1. The Organization’s Mission
2. Planning the Initiative
3. Incorporate Theory
4. Goals and Objectives
5. The Initiative’s Activities
6. Use Existing Evidence‐Based Programs
7. The Program’s Theory of Change
8. The Logic Model Depicting the Theory of Change
9. Criteria for Successful Initiatives
10. Stakeholders’ Participation in Planning and Developing Initiatives
11. Summary
12. Discussion Questions and Activities
13. Key Terms
CHAPTER 4: PLANNING FOR EVALUATION: PURPOSE AND PROCESSES
1. The Timing of the Evaluation
2. The Purpose of Evaluation
3. The Contract for Evaluation
4. The Evaluation Team
5. Evaluation Standards
6. Managing the Evaluation Process
7. Factors That Influence the Evaluation Process
8. Planning for Ethical Program Evaluation
9. Involving Stakeholders
10. Creating and Maintaining Effective Partnerships
11. Summary
12. Discussion Questions and Activities
13. Key Terms
CHAPTER 5: DESIGNING THE EVALUATION: PART 1: DESCRIBING THE PROGRAM OR POLICY
1. The Context of the Initiative
2. The Social, Political, and Economic Environment
3. The Organizational Structure and Resources
4. The Initiative and Its Relationship to the Organization
5. The Stage of Development of the Initiative
6. Data Access and Availability
7. The Program Initiative
8. The Policy Initiatives
9. Summary
10. Discussion Questions and Activities
11. Key Terms
CHAPTER 6: DESIGNING THE EVALUATION: PART 2A: PROCESS EVALUATION
1. Purposes of Process Evaluation
2. Key Issues in Process Evaluation
3. Selecting Questions for Process Evaluation
4. Resources for Evaluation
5. Measuring Resources, Processes, and Outputs
6. Tools for Process Evaluation
7. Ethical and Cultural Considerations
8. Summary
9. Discussion Questions and Activities
10. Key Terms
CHAPTER 7: DESIGNING THE EVALUATION: PART 2B: OUTCOME EVALUATION
1. The Relationship Between Process and Outcome Evaluation
2. Sorting and Selecting Evaluation Questions
3. Summary
4. Discussion Questions and Activities
5. Key Terms
CHAPTER 8: COLLECTING THE DATA: QUANTITATIVE
1. Quantitative Data
2. Factors Influencing Data Collection
3. Using Surveys
4. Designing Survey Instruments
5. Pilot Testing
6. Triangulation
7. Institutional Review and Ethics Boards
8. The Data Collection Team
9. Managing and Storing Data
10. Stakeholder Involvement
11. Summary
12. Discussion Questions and Activities
13. Key Terms
CHAPTER 9: COLLECTING THE DATA: QUALITATIVE
1. Qualitative Data
2. Ensuring Validity and Reliability
3. Interview‐Format Approaches
4. Document and Record Review
5. Observational Approaches
6. Case Reviews
7. Digital Approaches
8. Geographic Information Systems
9. Training Data Collectors
10. Managing and Storing Qualitative Data
11. Stakeholder Involvement
12. The Data Collection Team
13. Summary
14. Discussion Questions and Activities
15. Key Terms
CHAPTER 10: ANALYZING AND INTERPRETING QUANTITATIVE AND QUALITATIVE DATA: QUANTITATIVE (PART 1)
1. Analyzing and Reporting Quantitative Data
2. Reaching Conclusions
3. Stakeholder Involvement
4. Summary
5. Discussion Questions and Activities
6. QUALITATIVE (PART 2)
7. Analyzing Qualitative Data
8. Interpreting the Data and Reaching Conclusions
9. The Role of Stakeholders
10. Summary
11. Discussion Questions and Activities
12. Key Terms
CHAPTER 11: REPORTING EVALUATION FINDINGS
1. The Content of the Report
2. The Audience for the Report
3. The Timing of the Report
4. The Format of the Report
5. Summary
6. Discussion Questions and Activities
7. Key Terms
CHAPTER 12: CASE STUDY: THE COMMUNITY ASSESSMENT
1. Background
2. Establish a Team
3. Determine the Availability of Data
4. Decide on the Data‐Collection Approaches and Methods
5. Training
6. Resource Procurement
7. Analysis and Interpretation of the Data
8. Summary of Findings
9. The Intervention
10. Design the Evaluation
11. Collect the Data
12. Analyze and Interpret the Data
13. Report the Results
14. Discussion Questions and Activities
CHAPTER 13: CASE STUDY: PROCESS EVALUATION
1. Background
2. Theoretical Framework
3. Community Assessment Findings
4. The Evaluation Plan
5. Answering the Evaluation Question
6. Reporting the Results
7. Discussion Questions and Activities
APPENDIX A: MODEL AGREEMENT BETWEEN EVALUATION CONSULTANT AND TEAM MEMBERS
APPENDIX B: MODEL PREAMBLE FOR ADULT INDIVIDUAL INTERVIEWS
APPENDIX C: MODEL DEMOGRAPHIC SHEET
APPENDIX D: MODEL FIELD NOTES REPORT
APPENDIX E: MODEL INTERVIEW REFUSAL REPORT
APPENDIX F: DATA COLLECTION TRAINING MANUAL TEMPLATE
1. Conducting the Interview
APPENDIX G: GUIDELINES FOR COMPLETING AN EVALUATION REPORT
1. Methodology
REFERENCES
INDEX
End User License Agreement

List of Tables

CHAPTER 2: THE COMMUNITY ASSESSMENT: AN OVERVIEW
1. Table 2.1 Commonly Used Theories and Models in Public Health
2. Table 2.2 Risk Factor, Protective Factors, and Level of Influence
3. Table 2.3 The Community Assessment Team
4. Table 2.4 Example of a Data‐Collection Plan
CHAPTER 3: DEVELOPING INITIATIVES: AN OVERVIEW
1. Table 3.1 Incorporating Theory in Program Planning using the Health Belief Model
2. Table 3.2 Using Multiple Theories Within an Ecological Framework
3. Table 3.3 Goal Analysis
4. Table 3.4 Example of the Application of the SMART Attributes to an Objective
5. Table 3.5 Writing Time Lines into Objectives
CHAPTER 4: PLANNING FOR EVALUATION: PURPOSE AND PROCESSES
1. Table 4.1 A Basic Action Plan
2. Table 4.2 An Action Plan for Team Recruitment and Development
3. Table 4.3 Sample Budget
CHAPTER 6: DESIGNING THE EVALUATION: PART 2A: PROCESS EVALUATION
1. Table 6.1 Activity Objective: Expected Output and Actual Output
2. Table 6.2 Components in Formulating Policy
3. Table 6.3 Workplan for Identifying Questions for Process Evaluation
4. Table 6.4 Logic Model Components: Questions, Indicators, and Data Sources
5. Table 6.5 Assessing the Context for Policy Change Using a Table Format in a Survey
6. Table 6.6 Log Frame
7. Table 6.7 Alignment of Evaluation Research Questions (EQ) With Specific Project Objectives (SPO)
8. Table 6.8 Process Evaluation Work Plan With Funding From July Year 1 to March Year 3
CHAPTER 7: DESIGNING THE EVALUATION: PART 2B: OUTCOME EVALUATION
1. Table 7.1 Tools for Outcome Evaluation (What you use to answer the questions)
2. Table 7.2 Assessing Effectiveness of Policy Change from the Perspective of Stakeholders
CHAPTER 8: COLLECTING THE DATA: QUANTITATIVE
1. Table 8.1 Questions and Response Formats
CHAPTER 9: COLLECTING THE DATA: QUALITATIVE
1. Table 9.1 Template for Document Review
CHAPTER 10: ANALYZING AND INTERPRETING QUANTITATIVE AND QUALITATIVE DATA: QUANTITATIVE (PART 1)
1. Table 10.1 Single‐Variable Table Format for Presenting Quantitative Data
2. Table 10.2 Multiple‐Variable Table Format for Presenting Quantitative Data
CHAPTER 11: REPORTING EVALUATION FINDINGS
1. Table 11.1 List of Acronyms
CHAPTER 12: CASE STUDY: THE COMMUNITY ASSESSMENT
1. Table 12.1 Potential Contributions of Team Members
2. Table 12.2 Program Goals, Expected Outcomes, and Intervention Approaches
3. Table 12.3 Outcome Objectives and Initiative Activities for Healthy‐Weight Goal
4. Table 12.4 Outcome Objectives and Initiative Activities for Affordable‐Produce Goal
5. Table 12.5 The Logic‐Model Components and Healthy Soon Project Activities
6. Table 12.6 Evaluation Question
7. Table 12.7 Indicators and Data Sources
8. Table 12.8 Addressing Threats to Internal Validity
9. Table 12.9 Sample Site‐Visit Report
10. Table 12.10 Sample Attendance Sheet
11. Table 12.11 Site‐Visit Report at 2 Weeks
12. Table 12.12 Intervention Group and Control Group at Baseline on Knowledge About Physical Activity and Nutrition
13. Table 12.13 Intervention Group and Control Group at Baseline on Attitude Toward Diabetes
CHAPTER 13: CASE STUDY: PROCESS EVALUATION
1. Table 13.1 Summary of Community Assessment Results
2. Table 13.2 Interventions of the Leighster County Youth Initiative and Equivalent Levels of Influence
3. Table 13.3 Overarching and Subquestions in the Evaluation
4. Table 13.4 Evaluation Questions, Indicators, Data Type, Data Collection Approach, and the Time Line for Data Collection
5. Table 13.5 Baseline Level of Services and Participation in the Cinapsih Wellness Center Activities
6. Table 13.6a Personnel Salaries and Annual Cost
7. Table 13.6b Volunteers and Effort
8. Table 13.7 Survey Questions From the Section on Recruitment in Assessing Levels of Participation in the Cinapsih Wellness Center Programs and Activities

List of Illustrations

CHAPTER 1: AN INTRODUCTION TO PUBLIC AND COMMUNITY HEALTH EVALUATION
1. Figure 1.1 Components for Preassessment of Program's Readiness for Evaluation
2. Figure 1.2 Framework for Program Evaluation in Public Health
3. Figure 1.3 The Participatory Model for Evaluation
4. Figure 1.4 Evaluation in Context
5. Figure 1.5 Valuable Take‐Aways
CHAPTER 2: THE COMMUNITY ASSESSMENT: AN OVERVIEW
1. Figure 2.1 The Community Assessment as a Component of the Participatory Model for Evaluation
2. Figure 2.2 The Community Assessment, the Initiative, and the Evaluation
3. Figure 2.3 The Ecological‐Model Framework
4. Figure 2.4 Core Components of the Community Assessment Process
5. Figure 2.5 Major Factors Influencing Data Collection in the Community Assessment Process
6. Figure 2.6 Major Components of a Community Assessment
7. Figure 2.7 Summarizing Steps 1–10
8. Figure 2.8 Framework for Summarizing a Literature Review
9. Figure 2.9 Valuable Take‐Aways
CHAPTER 3: DEVELOPING INITIATIVES: AN OVERVIEW
1. Figure 3.1 Outcomes of a Community Assessment Process
2. Figure 3.2 The Hierarchical Relationship of Goals and Objectives
3. Figure 3.3 Verbs for Writing Objectives
4. Figure 3.4 Components of Evidence‐Based Programs
5. Figure 3.5 Classification System for Evidence‐Based HIV/AIDS Interventions
6. Figure 3.6 Basic Framework for a Logic Model for a Health‐Related Program
7. Figure 3.7 Basic Framework for a Logic Model for a Policy Related Initiative
8. Figure 3.8 Valuable Take‐Aways
CHAPTER 4: PLANNING FOR EVALUATION: PURPOSE AND PROCESSES
1. Figure 4.1 Poster from the Revised Media Campaign
2. Figure 4.2 Evaluation Standards
3. Figure 4.3 Nurturing Strong Evaluation Partnerships
4. Figure 4.4 Table of Contents for an Inception Report
5. Figure 4.5 Valuable Take‐Aways
CHAPTER 5: DESIGNING THE EVALUATION: PART 1: DESCRIBING THE PROGRAM OR POLICY
1. Figure 5.1 The Participatory Model for Evaluation: Design the Evaluation
2. Figure 5.2 Components for Reviewing the Program or Policy Initiative
3. Figure 5.3 Components for Understanding the Context of the Initiative
4. Figure 5.4 Organizational Chart
5. Figure 5.5 Tips for Drawing a Logic Model
6. Figure 5.6 Basic Framework for Completing a Logic Model
7. Figure 5.7 The Ecological‐Model Framework Showing P and p
8. Figure 5.8 Policy Logic Model
9. Figure 5.9 Valuable Take‐Aways
CHAPTER 6: DESIGNING THE EVALUATION: PART 2A: PROCESS EVALUATION
1. Figure 6.1 The Participatory Model for Evaluation: Design the Evaluation
2. Figure 6.2 Primary Purposes of Process Evaluation
3. Figure 6.3 Relationship Between Outcome and Activity Objectives and Outputs
4. Figure 6.4 Logic Model Framework for Process Evaluation
5. Figure 6.5 Two‐by‐Two Table
6. Figure 6.6 Changes in an Indicator Across Four Observations of Three Samples
7. Figure 6.7 Logic Model for the Wesleyan Church Public Health Ministry
8. Figure 6.8 Valuable Take‐Aways
CHAPTER 7: DESIGNING THE EVALUATION: PART 2B: OUTCOME EVALUATION
1. Figure 7.1 Logic‐Model Framework for Identifying Evaluation Questions
2. Figure 7.2 Logic Model–Inspired Questions
3. Figure 7.3 Two‐by‐Two Table
4. Figure 7.4 Changes in Conditions from the First to the Fourth Quarter
5. Figure 7.5 Pre (O₁) and Post (O₂) Test Scores
6. Figure 7.6 Valuable Take‐Aways
CHAPTER 8: COLLECTING THE DATA: QUANTITATIVE
1. Figure 8.1 The Participatory Framework for Evaluation: Collect the Data
2. Figure 8.2 Valuable Take‐Aways
CHAPTER 9: COLLECTING THE DATA: QUALITATIVE
1. Figure 9.1 Uses of Focus Groups
2. Figure 9.2 Requirements for a Focus Group Discussion
3. Figure 9.3 Mini Focus Group Interview
4. Figure 9.4 Research Process for the Interview Instrument
5. Figure 9.5 Assessing Incivilities in the Park
6. Figure 9.6 Distribution of Grocery and Corner Stores
7. Figure 9.7 Valuable Take‐Aways
CHAPTER 10: ANALYZING AND INTERPRETING QUANTITATIVE AND QUALITATIVE DATA: QUANTITATIVE (PART 1)
1. Figure 10.1 The Participatory Framework for Evaluation: Analyze and Interpret the Data
2. Figure 10.2 Bar‐Chart Format for Presenting Quantitative Data
3. Figure 10.3 Pie‐Chart Format for Presenting Quantitative Data
4. Figure 10.4 Screen Shot of Qualitative‐Data Coding from NVivo® QSR
5. Figure 10.5 Valuable Take‐Aways
CHAPTER 11: REPORTING EVALUATION FINDINGS
1. Figure 11.1 The Participatory Model for Evaluation—Report the Findings
2. Figure 11.2 Criteria for an Overall Assessment of the Initiative
3. Figure 11.3 Valuable Take‐Aways
CHAPTER 12: CASE STUDY: THE COMMUNITY ASSESSMENT
1. Figure 12.1 Theory of Change
2. Figure 12.2 Two‐by‐Two Table
CHAPTER 13: CASE STUDY: PROCESS EVALUATION
1. Figure 13.1 Model of the Ecological Model interacting With the Health‐belief Model and the Social Cognitive Theory
2. Figure 13.2 Leighster County Intervention Logic Model
3. Figure 13.3 Levels of Collaboration

PREFACE

You may not know what the term evaluation means, and, like me all those years ago and many of my students now, you are probably still a little wary of the term and wondering where this is all leading. No matter where you are in your understanding of program and policy evaluation, my hope is that whether you are a practitioner, a student, or both, you will find this book helpful on your journey and on your path to understanding. Just as I did many years ago, you probably evaluate what you do all the time without giving it a name. Evaluation is often an unconscious activity that is carried out before choosing among one or many options, both informally and formally. Informal evaluations range from selecting a restaurant for dinner to selecting a course of dishes off the menu. All the decisions you make along the way have implications for the success or failure of the outing. At the end of the evening, you go over the steps you took and decide whether the trip was worth it. If it wasn't, you may decide never to go to that restaurant again. So it is with program evaluation. We assess the resources and activities that went into a program, and then we determine whether the program or policy achieved what was intended, was worth it to those who experienced it and to those who funded it.

Evaluation activities occur in a range of work‐related settings including community‐based organizations, coalitions and partnerships, government‐funded entities, the pharmaceutical industry, and the media. Program evaluations assess how an event or activity was conducted, how well it was conducted, and whether it achieved its goal. Evaluation determines the merit of a program or policy, and it forms the basis for evidence‐based decision‐making.

Evaluation is the cornerstone of program improvement and must be carefully planned and executed to be effective. It helps make the task of assessing the appropriateness of a public health intervention or the success of a program or policy explicit by using appropriate research methods. In evaluation, a plan is developed to assess the achievement of program objectives. The plan states the standards against which the intervention will be assessed, the scope of the evaluation, and appropriate tools and approaches for data collection and analysis.

There are many opportunities to conduct an evaluation during the life of an intervention, and the approaches to conducting the evaluation in each case will differ. The methods and tools for an evaluation that is conducted during the first few months of a program are different from those used when the program or participation in the program ends and the effectiveness of the program or policy is being assessed. In addition, during the life of the program, evaluation tools and approaches can be used to record program and policy participation and progress.

This book presents a model for evaluation and describes the approaches and methods for evaluating community health program and policy interventions. It is aimed at public health and community health students as well as practitioners who are new to program and policy evaluation. This book makes no assumptions of prior knowledge about evaluation. The approach to evaluation that is presented allows for the development of simple or complex evaluation plans while focusing on practical approaches. It encourages a critical thinking and reflective approach with the full involvement of multiple stakeholders throughout the evaluation process. This book provides learners with a systematic, step‐by‐step approach to program evaluation.

The book is organized into 13 chapters. It discusses the community assessment and the development of the public health initiative as the precursors to the four‐step participatory model for evaluation with stakeholders at the center of each component. It frames program evaluation in the context of community‐based participatory research. This edition also includes a chapter on process evaluation. Two case studies help the reader experience virtual evaluations, and mini‐case studies and opportunities to “Think About It” allow the reader to reflect on the material and improve critical thinking skills. Valuable Takeaways provide simple reminders of important concepts covered in the chapter. An appendix provides some additional resources for evaluation.

EVALUATING PUBLIC AND COMMUNITY HEALTH PROGRAMS

ACKNOWLEDGMENTS

PREFACE