Cover Page

c01f003

Software Project Estimation

The Fundamentals for Providing High Quality Information to Decision Makers

Alain Abran

Wiley Logo

Foreword

Software project estimation is a challenge in most software organizations – and it is also a challenge to their customers, who endure software projects significantly over budget, with significant delays, less functionality than promised, and with unknown levels of quality.

Is industry better today at estimating software projects than it was 40 years ago, and do today's estimation models perform better?

What has not changed much in software estimation over that period of time?

Books and tools on software estimation have been around for four decades now, and a number of solutions (estimation tools, models and techniques) have been proposed to address the challenge of software estimation.

What do managers typically know about the quality of their estimation process, or about the performance of estimation tools available in the marketplace? Usually not much! Still, management takes a lot of decisions based on the numbers that these estimation tools yield.

In estimation:

When an organization has collected its own data and developed its own set of capabilities for analyzing their data and documenting the quality of their estimation models, then it has developed:

When an organization has not measured its own productivity on past projects, it is mostly in the dark about:

In this context, which is typical of many software organizations, using estimation models originating in environments with different productivity performance ratios does not provide real value. This is all the more true when little is known about:

Those who might feel good about such models, with all their fancy features and cost drivers, deceive themselves about the numbers coming from these ‘black-boxes’.

This book teaches the way to develop estimation information (that is, numbers + context) to be used by managers in making budgeting decisions in contexts of uncertainty.

This book is not about:

This book is about some of the best engineering practices in software project estimation, including:

…and no, there is no engineering, even in software estimation, without a sound statistical foundation!

Summary

This book is not geared to readers looking for quick one-shot solutions. It is geared to those interested in building a long term and sustainable competitive advantage in software estimation by learning about the best practices and what is needed to implement them (including the necessary effort for data collection and data analysis using simple and sound statistical methods). prior to exploring much more complex statistical approaches, including for instance machine learning techniques or fuzzy logic.

Overview

In this book we share years of experience in the design of credible software estimation processes as decision support tools for managers.

We introduce the basic statistics and economics concepts needed to understand the fundamentals of the design and evaluation of software estimation models, and improvements to them.

Because quantitative data and quantitative models constitute a fundamental concept in engineering, science, and management, this book will be useful to software organizations of all sizes, and managers will find in it effective strategies for improving the quantitative aspects of software project estimation, along with numerous examples.

The book is intended for IT practitioners, managers, and auditors involved in software project estimation, and for students registered in software project management courses.

Structure and organization

The book is organized into three parts and thirteen chapters.

Part 1 Part 2 Part 3
Understanding the Estimation Process Estimation Process: What Must be Verified? Building Estimation Models: Data Collection & Analysis
Chapters 1 to 3 Chapters 4 to 7 Chapters 8 to 13

Part 1 presents various views of software estimation that both estimators and managers must be aware of when designing and using software estimation models for decision making. It explains the structure of an estimation process, including the productivity models embedded in the estimation process, and clarifies the distinct roles and responsibilities that estimators and managers have. Finally, it introduces a number of economics concepts that must be taken into consideration, such as economies/diseconomies of scale and fixed/variable costs.

Part 2 introduces the concepts and techniques necessary for understanding that the quality of the outcomes of an estimation process depends on the quality of its inputs, of the underlying productivity models it uses, and an understanding of the limitations of the factors added as adjustments for estimation purposes.

Part 3 presents a number of issues related to building estimation models, including data collection and the use of international standards for ease of comparison across groups, organizations, and countries. In addition, it looks at how to build models which have more than one independent variable, using quality data as input and based on a number of economics concepts.

  1. Part 1: Understanding the Estimation Process
    1. Chapter 1: The Estimation Process: Phases & Roles
    2. The estimation process and its various phases is introduced, along with the distinct roles and responsibilities of software estimators and their managers.
    3. Chapter 2: Engineering & Economics Concepts for Understanding Software Process Performance
    4. Some key economics concepts are presented which are useful for understanding, and then modeling, the performance of the development process underlying productivity models. In particular, the concepts of economies/diseconomies of scale and of fixed/variable costs in production models are explained, as well as some characteristics of typical and atypical datasets in software engineering, and the presence of explicit and implicit variables in the productivity model.
    5. Chapter 3: Project Scenarios, Budgeting & Contingency Planning
    6. The impact of the selection of a single budget value among a range of estimates is discussed, including the identification of scenarios and their corresponding probabilities, and the identification and management of contingencies at the project portfolio level.
  2. Part 2: Estimation Process: What Must be Verified?
    1. Chapter 4: What Must be Verified in an Estimation Process: An Overview
    2. The various parts of an estimation process that must be understood and verified when building and using productivity models are identified. We look at models from an engineering perspective, not from a ‘craft’ perspective.
    3. Chapter 5: Verification of the Dataset Used to Build the Models
    4. The criteria required to analyze the quality of the direct inputs to mathematical models are presented, that is, the independent variables used to predict the dependent variable for estimation purposes.
    5. Chapter 6: Verification of the Models
    6. The criteria required to analyze the quality of mathematical models are presented, along with the outcomes of these models. Then, by way of illustration, these quality criteria are used to evaluate the performance of the models and tools proposed to the industry.
    7. Chapter 7: Verification of the Adjustment Phase
    8. Uncertainties and errors are inherent in measurements and in models of relationships across factors. Some sources of uncertainty and error are presented, and we show how they can accumulate when additional factors are introduced into the estimation process.
  3. Part 3: Building Estimation Models: Data Collection & Analysis
    1. Chapter 8: Data Collection & Industry Standards: The ISBSG Repository
    2. Models for industry usage should be based on well defined and standardized definitions of the parameters included in the estimation process. Some standards for software projects data collection that have been defined by the International Software Benchmarking Standards Group – ISBSG – are introduced. Standardized definitions are, of course, critical for internal and external benchmarking, as well as for building productivity and estimation models.
    3. Chapter 9: Building & Evaluating Single Variable Models
    4. The way to build a model, one independent variable at a time, is illustrated, starting with the variable recognized as the most significant, that is, the size of the software to be delivered. The way to build models using the ISBSG repository of projects is also shown, including data preparation and the identification of relevant samples to handle additional descriptive variables, such as the development environment.
    5. Chapter 10: Building Models with Categorical Variables
    6. A case study on building models with industry data on project size as the key factor is presented, along with a few additional categorical variables, and the way to analyze and understand the quality of such models.
    7. Chapter 11: Contribution of Productivity Extremes in Estimation
    8. An analysis is presented on how projects with the best and worst productivity are identified, and we show how to take advantage of the lessons learned in analyzing their performance and use this information for estimation purposes.
    9. Chapter 12: Multiple Models from a Single Dataset
    10. An analysis is presented on how to identify multiple models from a single dataset by exploring concepts of economies and diseconomies of scale, as well as process performance capabilities and the impact of constraints on productivity.
    11. Chapter 13: Re-Estimation: A Recovery Effort Model
    12. Throughout a software project, a number of factors influence productivity, functions are added and/or modified, risks materialize, and so on. As a result, projects often have to be re-estimated across life cycle phases. An approach to building re-estimation models is presented.

Additional material to complement the information contained in this book can be found at http://profs.etsmtl.ca/aabran/English/Autres/index.html

The table below provides a reading guide for software managers.

Chapters to read Why? What to do with the information
Part 1
Chapter 1: full reading
Estimation is a multi phases process and estimators and managers have distinct and complementary responsibilities. Verify that your estimation process covers all the phases described in this chapter & corresponding responsibilities well understood.
Chapter 2: full reading Economics concepts are useful for estimation purposes: they help explain fundamental issues in the software project cost structure, such as fixed/variable costs and economies/diseconomies of scale in software development. Ask your software engineers:
1- What are my fixed-variable efforts in software projects?
2- Do we have economies or diseconomies of scale in our software development process?
Chapter 3: full reading Estimators should provide scenarios and plausible ranges of estimates. From these, business executives should allocate a project budget price, as well as contingency funds at the level of a portfolio of projects. The manager should be the one to select the project budget (from a range of estimates) and set aside contingency funds to manage the inherent estimation risks.
Part 2
Chapters 4 to 7: quick reading
Estimation models should produce ‘credible numbers’: the quality of the estimation models must be verified and documented – if not, estimation is reduced to ‘garbage in, garbage out’. Ask your estimators to document the quality controls implemented in your estimation process.
Ask for an audit of your estimation process.
Part 3
Chapters: 8 to 13: quick reading
Standardized definitions for the data collected allow performance comparison within your organization and across the industry.
Engineering techniques should be used for data analysis and building estimation models.
Verify that your organization is using the best industry standards for data collection.
Ask your estimators to implement the best practices recommended in these chapters.
When project budgets go off-track, regular estimation models no longer work: re-estimation models are needed. Ask your estimator to develop a re-estimation model.

The table below provides a reading guide for IT practitioners, IT auditors and undergraduate or graduate students interested in:

Chapters to read Why? What to do with the information
Part 1
Chapters 1 to 3: full reading
Estimation models must be based on a good understanding of the performance of an organization: fixed/variable costs, as well as economies/diseconomies of scale in software development. When preparing project estimates, use your organization's historical data on fixed/variable costs as the foundation for your estimation process.
Clarify the boundaries of the responsibilities between estimator and manager.
Part 2
Chapters 4 to 7: full reading
Estimation models should produce ‘information’, not only numbers. These 4 chapters illustrate what must be verified and which criteria must be used to document the quality of the outcomes of the productivity models used or to be implemented in your organization.
This chapter also illustrates that adding more factors does not automatically increase certainty.
For decision making, you must provide information: i.e. numbers and context.
This includes documenting the quality of the inputs to your productivity models, as well as the probable range of estimates.
Part 3
Chapters: 8 to 13: full reading
To design a credible estimation process, you need:
  1. standards for data collection,
At estimation time, use the proposed techniques for building sound estimation models based on relevant data sets.
  1. identification of statistical outliers,
  2. select relevant samples for data analysis,
  3. built single and multiple variable models,
  4. take into account non quantitative variables.
  5. And at re-estimation time, a number of additional constraints have to be taken into account.
At re-estimation time, include the relevant additional productivity factors.

Acknowledgments

A number of colleagues in industry and at universities across the world, as well as former graduate students, have helped clarify many of the concepts presented in this book over the years, in particular:

Chapter Contributors
2 – Engineering and Economics Concepts for understanding Software Process Performance
  • Juan Cuadrado-Gallego, University of Alcala (Spain)
3 – Project Scenarios, Budgeting and Contingency Planning
  • Eduardo Miranda, Carnegie Mellon University (USA)
7 – Verification of the Adjustment Phase
  • Luca Santillo, Agile Metrics (Italy)
8 – Data collection and industry standards: the ISBSG repository
  • David Déry (Canada)
  • Laila Cheikhi, ENSIAS (Morocco)
9 – Building and evaluating single variable models
  • Pierre Bourque, ETS – U. Québec (Canada)
  • Iphigénie Ndiaye (Canada)
10 – Building models with categorical variables
  • Ilionar Silva and Laura Primera (Canada)
11 – Contribution of productivity extremes in estimation
  • Dominic Paré (Canada)
12 – Multiple models from a single dataset
  • Jean-Marc Desharnais, ETS – U. Québec (Canada)
  • Mohammad Zarour, Prince Sultan University
  • Onur Demırörs, Middle East Technical University (Turkey)
13 – Re-estimation: a recovery effort model
  • Eduardo Miranda, Carnegie Mellon University (USA)

Special thanks to:

Above all, this book is dedicated to:

About the Author

Dr. Alain Abran is a professor of software engineering at the École de Technologie Supérieure (ETS) – Université du Québec, Montréal, Canada.

Dr. Abran has more than 20 years of industry experience in information systems development and software engineering, and 20 years of university teaching. He holds a PhD in electrical and computer engineering (1994) from École Polytechnique de Montréal (Canada) and Master's degrees in Management Sciences (1974) and Electrical Engineering (1975) from the University of Ottawa (Canada).

He is the chairman of the Common Software Measurement International Consortium (COSMIC) – www.cosmicon.com. He published Software Metrics and Software Metrology in 2010, Management of Software Maintenance1 in 2008, both at Wiley & IEEE CS Press, and co-edited the 2004 version of the Guide to the Software Engineering Body of Knowledge (www.swebok.org).

His research interests include software productivity and estimation models, software quality, software measurement, functional size measurement methods, software risk management, and software maintenance management.

Most of his publications can be downloaded from: http://www.researchgate.net

Dr. Abran can be contacted at: alain.abran@etsmtl.ca

Part I

Understanding the Estimation Process

Estimation is not at all about coming up with a magic number to which everyone must commit at the peril of their professional career (which leads to staff members spending lots of overtime attempting to meet unrealistic deadlines.)

Part 1 of this book consists of three chapters, in which some of the key concepts of an estimation process are introduced.

Chapter 1 introduces the estimation process, including:

Chapter 2 explains the relationship between the software development life process and the classic model of a process. A number of economics concepts are introduced and illustrated in the context of software projects, such as:

Chapter 3 discusses the impact of the selection of a single budget value from a range of estimates, including the identification of scenarios and their corresponding probabilities, and the identification and management of contingencies at the project portfolio level.