This second book in the M&E Series begins with a brief introduction to the ROI process model and the Twelve Guiding Principles. It goes on to explain, in detail, one of the most critical activities in the evaluation process: data collection. Data are collected during every step of the ROI Methodology, and how and when these data are collected are crucial to successful program evaluation. This book provides detailed information about data collection methods and how and when to collect data at each evaluation level.
This guide details how to develop or use each of the following data collection tools:
In addition, this book explains how to select the appropriate data collection method for any situation and any level of evaluation.
This book begins by introducing and describing each data collection method and ends by discussing the methods that are best for each evaluation level. Chapter One details how to design effective questionnaires and surveys in order to maximize response rates. It also provides tips for presenting questionnaires to participants in ways that will encourage them to provide valuable data. Chapter Two discusses the major types of tests, including how to develop and administer them. When and how to use simulations to collect data are also explained.
Chapter Three explores how to conduct interviews and focus groups for optimal results, including question development and interview techniques. Guidelines for effective observation are also provided. Chapter Four explains how to use business performance monitoring, action planning, and performance contracts to collect data.
Chapter Five focuses on measuring reaction and planned action and the importance of gathering participant feedback. Chapter Six discusses the reasons for measuring learning data and then describes how to use the data. Measurement and administrative issues involved in data collection at this level are also explored.
Chapter Seven explains the importance of collecting application and implementation data. The challenges and issues of collecting application data are explored, and appropriate data collection methods are also discussed. Chapter Eight explores the critical reasons for collecting impact data. Effective impact measures, appropriate data collection measures, and linking the measures to business needs are all discussed in this chapter.
Finally, Chapter Nine illustrates how to select the appropriate data collection method for each program and each level of evaluation.
Editors
Patricia Pulliam Phillips, Ph.D.
Jack J. Phillips, Ph.D.
The ROI Six Pack provides detailed information on developing ROI evaluations, implementing the ROI Methodology, and showing the value of a variety of functions and processes. With detailed examples, tools, templates, shortcuts, and checklists, this series will be a valuable reference for individuals interested in using the ROI Methodology to show the impact of their projects, programs, and processes.
Although financial ROI has been measured for over one hundred years to quantify the value of plants, equipment, and companies, the concept has only recently been applied to evaluate the impact of learning and development, human resources, technology, quality, marketing, and other support functions. In the learning and development field alone, the use of ROI has become routine in many organizations. In the past decade, hundreds of organizations have embraced the ROI process to show the impact of many different projects and programs.
Along the way, professionals and practitioners need help. They need tools, templates, and tips, along with explanations, examples, and details, to make this process work. Without this help, using the ROI Methodology to show the value of projects and programs is difficult. In short, practitioners need shortcuts and proven techniques to minimize the resources required to use this process. Practitioners’ needs have created the need for this series. This series will provide the detail necessary to make the ROI Methodology successful within an organization. For easy reference and use, the books are logically arranged to align with the steps of the ROI Methodology.
The principal audience for these books is individuals who plan to use the ROI Methodology to show the value of their projects and programs. Such individuals are specialists or managers charged with proving the value of their particular project or program. They need detailed information, know-how, and confidence.
A second audience is those who have used the ROI Methodology for some time but want a quick reference with tips and techniques to make ROI implementation more successful within their organization. This series, which explains the evaluation process in detail, will be a valuable reference set for these individuals, regardless of other ROI publications owned.
A third audience is consultants and re-searchers who want to know how to address specific evaluation issues. Three important challenges face individuals as they measure ROI and conduct ROI evaluations: (1) collecting post-program data, (2) isolating the effects of the program, and (3) converting data to monetary values. A book is devoted to each of these critical issues, allowing researchers and consultants to easily find details on each issue.
A fourth audience is those who are curious about the ROI Methodology and its use. The first book in this series focuses specifically on ROI, its use, and how to determine whether it is appropriate for an organization. When interest is piqued, the remaining books provide more detail.
The six books are presented in a logical sequence, mirroring the ROI process model. Book one, ROI Fundamentals: Why and When to Measure ROI, presents the basic ROI Methodology and makes the business case for measuring ROI as it explores the benefits and barriers to implementation. It also examines the type of organization best suited for the ROI Methodology and the best time to implement it. Planning for an ROI evaluation is also explored in this book.
Book two, Data Collection: Planning For and Collecting All Types of Data, details data collection by examining the different techniques, processes, and issues involved in this process, with an emphasis on collecting post-program data. It examines the different data collection methods: questionnaires, interviews, focus groups, observation, action plans, performance contracts, and monitoring records.
Book three, Isolation of Results: Defining the Impact of the Program, focuses on the most valuable part of the ROI Methodology and the essential step for ensuring credibility. Recognizing that factors other than the program being measured can influence results, this book shows a variety of ways in which the effects of a program can be isolated from other influences. Techniques include comparison analysis using a control group, trend line analysis and forecasting methods, and expert input from a variety of sources.
Book four, Data Conversion: Calculating the Monetary Benefits, covers perhaps the second toughest challenge of ROI evaluation: placing monetary value on program benefits. To calculate the ROI, data must be converted to money, and Data Conversion shows how this conversion has been accomplished in a variety of organizations. The good news is that standard values are available for many items. When they are not, the book shows different techniques for converting them, ranging from calculating the value from records to seeking experts and searching databases. When data cannot be converted to money credibly and with minimum resources, they are considered intangible. This book explores the range of intangible benefits and the necessary techniques for collecting, analyzing, and recording them.
Book five, Costs and ROI: Evaluating at the Ultimate Level, focuses on costs and ROI. This book shows that all costs must be captured in order to create a fully loaded cost profile. All the costs must be included in order to be conservative and to give the analysis additional credibility. Next, the actual ROI calculation is presented, showing the various assumptions and issues that must be addressed when calculating the ROI. Three different calculations are presented: the benefit-cost ratio, the ROI percentage, and the payback period. The book concludes with several cautions and concerns about the use of ROI and its meaning.
Book six, Communication and Implementation: Sustaining the Practice, explores two important issues. The first issue is reporting the results of an evaluation. This is the final part of the ROI Methodology and is necessary to ensure that audiences have the information they need so that improvement processes can be implemented. A range of techniques is available, including face-to-face meetings, brief reports, one-page summaries, routine communications, massaudience techniques, and electronic media. All are available for reporting evaluation results. The final part of the book focuses on how to sustain the ROI evaluation process: how to use it, keep it going, and make it work in the long term to add value to the organization and, often, to show the value of all the programs and projects within a function or department.
In this series the terms program and project are used to describe many processes that can be evaluated using the ROI Methodology. This is an important issue because readers may vary widely in their perspectives. Individuals involved in technology applications may use the terms system and technology rather than program or project. In public policy, in contrast, the word program is prominent. For a professional meetings and events planner, the word program may not be pertinent, but in human resources, program is often used. Finding one term for all these situations would be difficult. Consequently, the terms program and project are used interchangeably. Table I.1 lists these and other terms that may be used in other contexts.
Table I.1 Terms and Applications
| Term | Example |
| Program | Leadership development skills enhancement for senior executives |
| Project | A reengineering scheme for a plastics division |
| System | A fully interconnected network for all branches of a bank |
| Initiative | A faith-based effort to reduce recidivism |
| Policy | A new preschool plan for disadvantaged citizens |
| Procedure | A new scheduling arrangement for truck drivers |
| Event | A golf outing for customers |
| Meeting | A U.S. Coast Guard conference on innovations |
| Process | Quality sampling |
| People | Staff additions in the customer care center |
| Tool | A new means of selecting hotel staff |
Each book in the series takes a straightforward approach to make it understandable, practical, and useful. Checklists are provided, charts are included, templates are presented, and examples are explored. All are intended to show how the ROI Methodology works. The focus of these books is implementing the process and making it successful within an organization. The methodology is based on the work of hundreds of individuals who have made the ROI Methodology a successful evaluation process within their organizations.
Pfeiffer serves the professional development and hands-on resource needs of training and human resource practitioners and gives them products to do their jobs better. We deliver proven ideas and solutions from experts in HR development and HR management, and we offer effective and customizable tools to improve workplace performance. From novice to seasoned professional, Pfeiffer is the source you can trust to make yourself and your organization more successful.
Essential Knowledge Pfeiffer produces insightful, practical, and comprehensive materials on topics that matter the most to training and HR professionals. Our Essential Knowledge resources translate the expertise of seasoned professionals into practical, how-to guidance on critical workplace issues and problems. These resources are supported by case studies, worksheets, and job aids and are frequently supplemented with CD-ROMs, Web sites, and other means of making the content easier to read, understand, and use.
Essential Tools Pfeiffer’s Essential Tools resources save time and expense by offering proven, ready-to-use materials—including exercises, activities, games, instruments, and assessments—for use during a training or team-learning event. These resources are frequently offered in looseleaf or CD-ROM format to facilitate copying and customization of the material.
Pfeiffer also recognizes the remarkable power of new technologies in expanding the reach and effectiveness of training. While e-hype has often created whizbang solutions in search of a problem, we are dedicated to bringing convenience and enhancements to proven training solutions. All our e-tools comply with rigorous functionality standards. The most appropriate technology wrapped around essential content yields the perfect solution for today’s on-the-go trainers and human resource professionals.
Essential resources for training and HR professionals
Copyright © 2008 by John Wiley & Sons, Inc. All rights reserved.
Published by Pfeiffer
An Imprint of Wiley
989 Market Street, San Francisco, CA 94103-1741
www.pfeiffer.com
No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400, fax 978-646-8600, or on the web at www.copyright.com. Requests to the publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, 201-748-6011, fax 201-748-6008, or online at http://www.wiley.com/go/permissions.
Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.
Readers should be aware that Internet Web sites offered as citations and/or sources for further information may have changed or disappeared between the time this book was written and when it is read.
For additional copies/bulk purchases of this book in the U.S. please contact 800-274-4434.
Pfeiffer books and products are available through most bookstores. To contact Pfeiffer directly call our Customer Care Department within the U.S. at 800-274-4434, outside the U.S. at 317-572-3985, fax 317-572-4002, or visit www.pfeiffer.com.
Pfeiffer also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books.
Library of Congress Cataloging-in-Publication DataPhillips, Patricia Pulliam.
Data collection: planning for and collecting all types of data / Patricia Pulliam Phillips, Ph.D., and Cathy A. Stawarski, Ph.D.
p. cm.—(The measurement and evaluation series)
Includes bibliographical references and index.
ISBN: 978-0-7879-8718-3 (pbk.)
1. Project management—Evaluation. 2. Rate of return. 3. Employees—Rating of. 4. Organizational learning—Evaluation. 5. Social sciences—Research—Methodology. I. Stawarski, Cathy A. II. Title.
HD69.P75P496 2008
001.4’33—dc22
2007035758
Production Editor: Michael Kay
Editor: Matthew Davis
Editorial Assistant: Julie Rodriguez
Manufacturing Supervisor: Becky Morgan
No project, regardless of its size or scope, is completed without the help and support of others. My sincere thanks go to the staff at Pfeiffer. Their support for this project has been relentless. Matt Davis has been the greatest! It is our pleasure and privilege to work with such a professional and creative group of people.
Thanks also go to my husband, Jack. His unwavering support of my work is always evident. His idea for the series was to provide readers with a practical understanding of the various components of a comprehensive measurement and evaluation process. Thank you, Jack, for another fun opportunity!
Many thanks go to the staff who helped make this series a reality. Lori Ditoro did an excellent job of meeting a very tight deadline and delivering a quality manuscript.
Much admiration and thanks go to Patti. She is an astute observer of the ROI Methodology, having observed and learned from hundreds of presentations, consulting assignments, and engagements. In addition, she is an excellent researcher and student of the process, studying how it is developed and how it works. She has become an ROI expert in her own right. Thanks, Patti, for your many contributions. You are a great partner, friend, and spouse.
The ROI Methodology is a step-by-step tool for evaluating any program, project, or initiative in any organization. Figure P.1 illustrates the ROI process model, which makes a potentially complicated process simple by breaking it into sequential steps. The ROI process model provides a systematic, step-by-step approach to ROI evaluations that helps keep the process manageable, allowing users to address one issue at a time. The model also emphasizes that the ROI Methodology is a logical, systematic process that flows from one step to another and provides a way for evaluators to collect and analyze six types of data.
Applying the model consistently from one program to another is essential for successful evaluation. To aid consistent application of the model, the ROI Methodology is based on twelve Guiding Principles. These principles are necessary for a credible, conservative approach to evaluation through the different levels.
Figure P.1. The ROI Process Model