Cover Page

About This Book

Why This Book Is Important

Costs and ROI: Evaluating at the Ultimate Level, the fifth book in the M&E series, discusses the ultimate level of evaluation: calculating ROI. Executives want to know whether specific programs and projects add value to their organization; calculating the ROI gives them the accountability they seek.

Part of calculating ROI is tabulating the fully loaded program costs. Fully loaded costs include all direct and indirect costs. Using fully loaded costs is critical in order to maintain a conservative approach.

What This Book Achieves

This book shows how to tabulate the fully loaded costs of a program and explains what costs should be included in this total. It also presents the different formulas that can be used to calculate ROI. Finally, measures other than ROI that can show the value of programs and projects are defined and explained.

How This Book Is Organized

This book begins with a brief introduction to the ROI process model and the Twelve Guiding Principles. Chapter One discusses why tabulating costs and calculating the ROI are important. It also examines why the ROI should be forecast. Chapter Two discusses how to efficiently track costs and explores the issues involved in cost tracking and cost categories. Cost reporting, cost accumulation, and cost estimation are also discussed.

Chapter Three details the calculation of ROI. The chapter includes examples as well as different ways to calculate the return and other measures. Chapter Four reviews the many concerns, issues, and myths surrounding the ROI Methodology.

In Chapter Five, the many benefits of developing an ROI forecast are discussed. Forecasts can be developed prior to a program’s implementation by using a pilot program, reaction data, learning data, or skills and competencies. Finally, the guidelines for forecasting are examined.

The Measurement and Evaluation Series

Editors

Patricia Pulliam Phillips, Ph.D.

Jack J. Phillips, Ph.D.

Introduction to the Measurement and Evaluation Series

The ROI Six Pack provides detailed information on developing ROI evaluations, implementing the ROI Methodology, and showing the value of a variety of functions and processes. With detailed examples, tools, templates, shortcuts, and checklists, this series will be a valuable reference for individuals interested in using the ROI Methodology to show the impact of their projects, programs, and processes.

The Need

Although financial ROI has been measured for over one hundred years to quantify the value of plants, equipment, and companies, the concept has only recently been applied to evaluate the impact of learning and development, human resources, technology, quality, marketing, and other support functions. In the learning and development field alone, the use of ROI has become routine in many organizations. In the past decade, hundreds of organizations have embraced the ROI process to show the impact of many different projects and programs.

Along the way, professionals and practitioners need help. They need tools, templates, and tips, along with explanations, examples, and details, to make this process work. Without this help, using the ROI Methodology to show the value of projects and programs is difficult. In short, practitioners need shortcuts and proven techniques to minimize the resources required to use this process. Practitioners’ needs have created the need for this series. This series will provide the detail necessary to make the ROI Methodology successful within an organization. For easy reference and use, the books are logically arranged to align with the steps of the ROI Methodology.

Audience

The principal audience for these books is individuals who plan to use the ROI Methodology to show the value of their projects and programs. Such individuals are specialists or managers charged with proving the value of their particular project or program. They need detailed information, know-how, and confidence.

A second audience is those who have used the ROI Methodology for some time but want a quick reference with tips and techniques to make ROI implementation more successful within their organization. This series, which explains the evaluation process in detail, will be a valuable reference set for these individuals, regardless of other ROI publications owned.

A third audience is consultants and researchers who want to know how to address specific evaluation issues. Three important challenges face individuals as they measure ROI and conduct ROI evaluations: (1) collecting post-program data, (2) isolating the effects of the program, and (3) converting data to monetary values. A book is devoted to each of these critical issues, allowing researchers and consultants to easily find details on each issue.

A fourth audience is those who are curious about the ROI Methodology and its use. The first book in this series focuses specifically on ROI, its use, and how to determine whether it is appropriate for an organization. When interest is piqued, the remaining books provide more detail.

Flow of the Books

The six books are presented in a logical sequence, mirroring the ROI process model. Book one, ROI Fundamentals: Why and When to Measure ROI, presents the basic ROI Methodology and makes the business case for measuring ROI as it explores the benefits and barriers to implementation. It also examines the type of organization best suited for the ROI Methodology and the best time to implement it. Planning for an ROI evaluation is also explored in this book.

Book two, Data Collection: Planning For and Collecting All Types of Data, details data collection by examining the different techniques, methods, and issues involved in this process, with an emphasis on collecting post-program data. It examines the different data collection methods: questionnaires, interviews, focus groups, observation, action plans, performance contracts, and monitoring records.

Book three, Isolation of Results: Defining the Impact of the Program, focuses on the most valuable part of the ROI Methodology and the essential step for ensuring credibility. Recognizing that factors other than the program being measured can influence results, this book shows a variety of ways in which the effects of a program can be isolated from other influences. Techniques include comparison analysis using a control group, trend line analysis and forecasting methods, and expert input from a variety of sources.

Book four, Data Conversion: Calculating the Monetary Benefits, covers perhaps the second toughest challenge of ROI evaluation: placing monetary value on program benefits. To calculate the ROI, data must be converted to money, and Data Conversion shows how this conversion has been accomplished in a variety of organizations. The good news is that standard values are available for many items. When they are not, the book shows different techniques for converting them, ranging from calculating the value from records to seeking experts and searching databases. When data cannot be converted to money credibly and with minimum resources, they are considered intangible. This book explores the range of intangible benefits and the necessary techniques for collecting, analyzing, and recording them.

Book five, Costs and ROI: Evaluating at the Ultimate Level, focuses on costs and ROI. This book shows that all costs must be captured in order to create a fully loaded cost profile. All the costs must be included in order to be conservative and to give the analysis additional credibility. Next, the actual ROI calculation is presented, showing the various assumptions and issues that must be addressed when calculating the ROI. Three different calculations are presented: the benefit-cost ratio, the ROI percentage, and the payback period. The book concludes with several cautions and concerns about the use of ROI and its meaning.

Book six, Communication and Implementation: Sustaining the Practice, explores two important issues. The first issue is reporting the results of an evaluation. This is the final part of the ROI Methodology and is necessary to ensure that audiences have the information they need so that improvement processes can be implemented. A range of techniques is available, including face-to-face meetings, brief reports, one-page summaries, routine communications, mass-audience techniques, and electronic media. All are available for reporting evaluation results. The final part of the book focuses on how to sustain the ROI evaluation process: how to use it, keep it going, and make it work in the long term to add value to the organization and, often, to show the value of all the programs and projects within a function or department.

Terminology: Programs, Projects, Solutions

In this series the terms program and project are used to describe many processes that can be evaluated using the ROI Methodology. This is an important issue because readers may vary widely in their perspectives. Individuals involved in technology applications may use the terms system and technology rather than program or project. In public policy, in contrast, the word program is prominent. For a professional meetings and events planner, the word program may not be pertinent, but in human resources, program is often used. Finding one term for all these situations would be difficult. Consequently, the terms program and project are used interchangeably. Table I.1 lists these and other terms that may be used in other contexts.

Table I.1. Terms and Applications

Term Example
Program Leadership development skills enhancement for senior executives
Project A reengineering scheme for a plastics division
System A fully interconnected network for all branches of a bank
Initiative A faith-based effort to reduce recidivism
Policy A new preschool plan for disadvantaged citizens
Procedure A new scheduling arrangement for truck drivers
Event A golf outing for customers
Meeting A U.S. Coast Guard conference on innovations
Process Quality sampling
People Staff additions in the customer care center
Tool A new means of selecting hotel staff

Features

Each book in the series takes a straightforward approach to make it understandable, practical, and useful. Checklists are provided, charts are included, templates are presented, and examples are explored. All are intended to show how the ROI Methodology works. The focus of these books is implementing the process and making it successful within an organization. The methodology is based on the work of hundreds of individuals who have made the ROI Methodology a successful evaluation process within their organizations.

About Pfeiffer

Pfeiffer serves the professional development and hands-on resource needs of training and human resource practitioners and gives them products to do their jobs better. We deliver proven ideas and solutions from experts in HR development and HR management, and we offer effective and customizable tools to improve workplace performance. From novice to seasoned professional, Pfeiffer is the source you can trust to make yourself and your organization more successful.

ffirs_Inline_10_5.gif Essential Knowledge Pfeiffer produces insightful, practical, and comprehensive materials on topics that matter the most to training and HR professionals. Our Essential Knowledge resources translate the expertise of seasoned professionals into practical, how-to guidance on critical workplace issues and problems. These resources are supported by case studies, worksheets, and job aids and are frequently supplemented with CD-ROMs, websites, and other means of making the content easier to read, understand, and use.

ffirs_Inline_10_5.gif Essential Tools Pfeiffer’s Essential Tools resources save time and expense by offering proven, ready-to-use materials–including exercises, activities, games, instruments, and assessments–for use during a training or team-learning event. These resources are frequently offered in looseleaf or CD-ROM format to facilitate copying and customization of the material.

Pfeiffer also recognizes the remarkable power of new technologies in expanding the reach and effectiveness of training. While e-hype has often created whizbang solutions in search of a problem, we are dedicated to bringing convenience and enhancements to proven training solutions. All our e-tools comply with rigorous functionality standards. The most appropriate technology wrapped around essential content yields the perfect solution for today’s on-the-go trainers and human resource professionals.

ffirs_image_10_6.jpg Essential resources for training and HR professionals

Costs and ROI

Evaluating at the Ultimate Level

Jack J. Phillips, Ph.D.

Lizette Zúñiga, M.A.

wiley Logo

Acknowledgments from the Editors

From Patti

No project, regardless of its size or scope, is completed without the help and support of others. My sincere thanks go to the staff at Pfeiffer. Their support for this project has been relentless. Matt Davis has been the greatest! It is our pleasure and privilege to work with such a professional and creative group of people.

Thanks also go to my husband, Jack. His unwavering support of my work is always evident. His idea for the series was to provide readers with a practical understanding of the various components of a comprehensive measurement and evaluation process. Thank you, Jack, for another fun opportunity!

From Jack

Many thanks go to the staff who helped make this series a reality. Lori Ditoro did an excellent job of meeting a very tight deadline and delivering a quality manuscript.

Much admiration and thanks go to Patti. She is an astute observer of the ROI Methodology, having observed and learned from hundreds of presentations, consulting assignments, and engagements. In addition, she is an excellent researcher and student of the process, studying how it is developed and how it works. She has become an ROI expert in her own right. Thanks, Patti, for your many contributions. You are a great partner, friend, and spouse.

Principles of the ROI Methodology

The ROI Methodology is a step-by-step tool for evaluating any program, project, or initiative in any organization. Figure P.1 illustrates the ROI process model, which makes a potentially complicated process simple by breaking it into sequential steps. The ROI process model provides a systematic, step-by-step approach to ROI evaluations that helps keep the process manageable, allowing users to address one issue at a time. The model also emphasizes that the ROI Methodology is a logical, systematic process that flows from one step to another and provides a way for evaluators to collect and analyze six types of data.

Applying the model consistently from one program to another is essential for successful evaluation. To aid consistent application of the model, the ROI Methodology is based on twelve Guiding Principles. These principles are necessary for a credible, conservative approach to evaluation through the different levels.

  1. When conducting a higher-level evaluation, collect data at lower levels.
  2. When planning a higher-level evaluation, the previous level of evaluation is not required to be comprehensive.
  3. When collecting and analyzing data, use only the most credible sources.
    image

    Figure P.1. The ROI Process Model

  4. When analyzing data, select the most conservative alternative for calculations.
  5. Use at least one method to isolate the effects of a project.
  6. If no improvement data are available for a population or from a specific source, assume that little or no improvement has occurred.
  7. Adjust estimates of improvement for potential errors of estimation.
  8. Avoid use of extreme data items and unsupported claims when calculating ROI.
  9. Use only the first year of annual benefits in ROI analysis of short-term solutions.
  10. Fully load all costs of a solution, project, or program when analyzing ROI.
  11. Intangible measures are defined as measures that are purposely not converted to monetary values.
  12. Communicate the results of the ROI Methodology to all key stakeholders.