Cover

Table of Contents

Cover

Table of Contents

Half title page

Series page

Title page

Copyright page

FOREWORD

PREFACE

ACKNOWLEDGMENTS

ABOUT THE AUTHOR

Part 1: KEY CONCEPTS FOR THE DESIGN OF SOFTWARE MEASURES

1 INTRODUCTION

1.1. INTRODUCTION

1.2. SOFTWARE MEASUREMENT: IS IT MATURE OR NOT?

1.3. SOFTWARE MEASUREMENT AS A NEW TECHNOLOGY

1.4. THE DESIGNS OF SOFTWARE MEASURES MUST BE VERIFIED

ADVANCED READINGS: HOW MUCH MEASUREMENT SUPPORT IS RECOGNIZED WITHIN THE SOFTWARE ENGINEERING BODY OF KNOWLEDGE (SWEBOK)?

2 FROM MEASUREMENT METHODS TO QUANTITATIVE MODELS: A MEASUREMENT CONTEXT MODEL

2.1. INTRODUCTION: NUMBERS, MEASURES, AND QUANTITATIVE MODELS

2.2. MEASUREMENT CONTEXT MODEL

2.3. HOW TO DESIGN A MEASUREMENT METHOD

2.4. APPLICATION OF A MEASUREMENT METHOD

2.5. EXPLOITATION OF MEASUREMENT RESULTS IN QUANTITATIVE MODELS

2.6. VALIDATION OR VERIFICATION?

ADVANCED READINGS:

3 METROLOGY AND QUALITY CRITERIA IN SOFTWARE MEASUREMENT

3.1. INTRODUCTION TO METROLOGY

3.2. A MODEL FOR MEASURING DEVICES

3.3. QUALITY CRITERIA FOR THE DESIGN OF A MEASUREMENT METHOD

3.4. QUALITY CRITERIA FOR THE APPLICATION OF A MEASUREMENT METHOD

3.5. QUALITY CRITERIA FOR MEASUREMENT RESULTS

3.6. SUMMARY

ADVANCED READINGS: MEASURING CHAIN AND MEASURING SYSTEM

4 QUANTIFICATION AND MEASUREMENT ARE NOT THE SAME!

4.1. INTRODUCTION: NUMBERS ARE NOT ALL CREATED EQUALS

4.2. ISO 15939 MEASUREMENT INFORMATION MODEL

4.3. SCOPE OF THE ISO 15939 MEASUREMENT INFORMATION MODEL

4.4. THE METROLOGY PERSPECTIVE IN THE ISO 15939 MEASUREMENT INFORMATION MODEL

4.5. THE QUANTIFICATION OF RELATIONSHIPS IN ISO 15939

4.6. A PRODUCTIVITY MODEL: AN ISO 15939 MEASUREMENT INFORMATION MODEL

4.7. EXAMPLES: A METROLOGY DESIGN AND A QUANTIFICATION MODEL OF RELATIONSHIPS

4.8. SUMMARY

5 THE DESIGN OF SOFTWARE MEASUREMENT METHODS

5.1. INTRODUCTION

5.2. LINKING SOFTWARE MEASUREMENT AND METROLOGY

5.3. DEFINING THE MEASUREMENT PRINCIPLE

5.4. DETERMINING THE MEASUREMENT METHOD

5.5. PRODUCTS OF THE MEASUREMENT DESIGN PHASE

5.6. POST DESIGN ACTIVITY: DETERMINING A MEASUREMENT PROCEDURE

5.7. SUMMARY

ADVANCED READINGS 1: VERIFICATION CRITERIA FOR SOFTWARE MEASUREMENT METHODS

ADVANCED READINGS 2: RELATIONAL STRUCTURES IN ASSIGNING A NUMERICAL VALUE

Part 2: SOME POPULAR SOFTWARE MEASURES: HOW GOOD ARE THEY?

6 CYCLOMATIC COMPLEXITY NUMBER: ANALYSIS OF ITS DESIGN

6.1. INTRODUCTION

6.2. THE CYCLOMATIC NUMBER IN GRAPH THEORY

6.3. THE CYCLOMATIC NUMBER FOR SOFTWARE

6.4. OTHER DESIGN ISSUES: THE ENTITY AND ATTRITUBE MEASURED

ADVANCED READINGS: LACK OF CONSENSUS ON COMPLEXITY IN SOFTWARE

7 HALSTEAD’S METRICS: ANALYSIS OF THEIR DESIGNS

7.1. INTRODUCTION

7.2. HALSTEAD’S METRICS: DEFINITIONS

7.3. ANALYSIS OF THE DESIGN OF HALSTEAD’S METRICS

7.4. DISCUSSION ON THE FINDINGS

ADVANCED READINGS: OTHER HALSTEAD’S METRICS

8 FUNCTION POINTS: ANALYSIS OF THEIR DESIGN

8.1. INTRODUCTION

8.2. THE ORIGIN OF SOFTWARE FUNCTIONAL SIZE MEASUREMENT

8.3. THE DESIGN OF FUNCTION POINTS—FP

8.4. WEAKNESSES OF THE FUNCTION POINTS MEASUREMENT DESIGN

8.5. OTHER WEAKNESSES

8.6. WHAT IS A FUNCTION POINT?

8.7. OTHER RESEARCH FINDINGS

ADVANCED READINGS: EVOLUTION OF FUNCTIONAL SIZE MEASUREMENT

9 USE CASE POINTS: ANALYSIS OF THEIR DESIGN

9.1. INTRODUCTION

9.2. USE CASE POINTS DESCRIPTION

9.3. ANALYSIS OF THE UCP DESIGN

9.4. THE MODELING OF RELATIONSHIPS OF UCP WITH PROJECT EFFORT

9.5. SUMMARY

10 ISO 91261: ANALYSIS OF QUALITY MODELS AND MEASURES

10.1. INTRODUCTION TO ISO 9126

10.2. ANALYSIS MODELS OF ISO 9126: THE (QUANTITATIVE) MODELS

10.3. THE METROLOGY-RELATED PART OF ISO 9126: BASE AND DERIVED MEASURES

10.4. ANALYSIS OF DERIVED MEASURES

10.5. THE MISSING LINKS: FROM METROLOGY TO QUANTITATIVE ANALYSIS

10.6. OUTSTANDING MEASUREMENT DESIGN ISSUES: IMPROVEMENT STRATEGY

ADVANCED READINGS 1: ANALYSIS OF THE DESIGN OF THE PRODUCTIVITY MEASURE IN ISO 9126

ADVANCED READINGS 2: ATTRIBUTES AND RELATED BASE MEASURES WITHIN ISO 9126

Part 3: THE DESIGN OF COSMIC – ISO 19761

11 COSMIC: DESIGN OF AN INITIAL PROTOTYPE

11.1. INTRODUCTION

11.2. IMPROVEMENT PROJECT

11.3. LITERATURE REVIEW

11.4. DESIGN OF THE INITIAL PROTOTYPE

11.5. FIELD TESTS OF THE PROTOTYPE

11.6. FIRST-YEAR FEEDBACK FROM INDUSTRY

ADVANCED READINGS: KEY DIFFERENCES BETWEEN FP AND THE PROPOSED EXTENSION

12 COSMIC: SCALING UP AND INDUSTRIALIZATION

12.1. INTRODUCTION

12.2. SCALING UP OBJECTIVES

12.3. DESIGN DECISIONS

12.4. INDEPENDENT INDUSTRIAL FIELD TRIALS

12.5. OUTCOME: THE DESIGN OF THE COSMIC MEASUREMENT METHOD

12.6. STRENGTHS OF THE COSMIC DESIGN

12.7. SCALING UP—METROLOGY INFRASTRUCTURE

12.8. COMPETITIVE ADVANTAGES

Part 4: OTHER ISSUES IN THE DESIGN OF SOFTWARE MEASURES

13 CONVERTIBILITY ACROSS MEASUREMENT METHODS

13.1. INTRODUCTION

13.2. OVERVIEW OF PREVIOUS CONVERTIBILITY STUDIES

13.3. A CONVERTIBILITY STUDY OF AN INDUSTRY DATASET

13.4. FP TO COSMIC CONVERTIBILITY: ISSUES AND DISCUSSION

14 DESIGN OF STANDARD ETALONS: THE NEXT FRONTIER IN SOFTWARE MEASUREMENT

14.1. INTRODUCTION—MEASUREMENT STANDARD ETALON

14.2. CALIBRATION AND TESTING: REFERENCE MATERIAL AND UNCERTAINTY

14.3. RELATED WORK IN SOFTWARE MEASUREMENT

14.4. A (DRAFT) METHODOLOGY TO DESIGN AN FSM STANDARD ETALON

14.5. DISCUSSION

ADVANCED READINGS: THE DEVELOPMENT OF AN (INITIAL) DRAFT OF A MEASUREMENT STANDARD ETALON FOR COSMIC

Appendix A  LIST OF ACRONYMS

Appendix B  GLOSSARY OF TERMS IN SOFTWARE MEASUREMENT

B.1 SOME MEASUREMENT TERMS IN SOFTWARE ENGINEERING

B.2 SOME MEASUREMENT TERMS IN METROLOGY—VIM 2007

Appendix C  REFERENCES

Index

SOFTWARE METRICS AND SOFTWARE METROLOGY

Press Operating Committee

Chair

Linda Shafer

former Director, Software Quality Institute

The University of Texas at Austin

Editor-in-Chief

Alan Clements

Professor

University of Teesside

Board Members

Mark J. Christensen, Independent Consultant

James W. Cortada, IBM Institute for Business Value

Richard E. (Dick) Fairley, Founder and Principal Associate, Software Engineering Management Associates (SEMA)

Phillip Laplante, Professor of Software Engineering, Penn State University

Evan Butterfield, Director of Products and Services

Kate Guillemette, Product Development Editor, CS Press

IEEE Computer Society Publications

The world-renowned IEEE Computer Society publishes, promotes, and distributes a wide variety of authoritative computer science and engineering texts. These books are available from most retail outlets. Visit the CS Store at http: computer.org/store for a list of products.

IEEE Computer Society Wiley Partnership

The IEEE Computer Society and Wiley partnership allows the CS Press authored book program to produce a number of exciting new titles in areas of computer science, computing and networking with a special focus on software engineering. IEEE Computer Society members continue to receive a 15% discount on these titles when purchased through Wiley or at wiley.com/ieeecs

To submit questions about the program or send proposals please e-mail kguillemette@computer.org or write to Books, IEEE Computer Society, 10662 Los Vaqueros Circle, Los Alamitos, CA 90720-1314. Telephone +1-714-816-2169.

Additional information regarding the Computer Society authored book program can also be accessed from our web site at http: computer.org/cspress.

Title page

FOREWORD

Software organizations must respond to increasingly demanding customers in a globally competitive market and must implement best industry practices. With services and products available from vendors the world over, customers are insisting that their software systems be of high quality and with support services that challenge those of the competition while costing as little as possible.

To satisfy these demands, software organizations must have the ability to develop and maintain software to meet the customer’s needs, and it must have access to software that support the company’s business processes.

  • How do you know and how do you objectively demonstrate to your customers that your software organization is performing at the top of the industry?
  • Can you leverage this knowledge to develop estimation skills as a competitive advantage?

Benchmarking and estimation is based on measurements. There is a tremendous need for software measures to support software performance measurement, benchmarking and software project estimation, even more so when software is contracted out to third party suppliers.

There is currently available a large number of software measures and quantitative models proposed to the practitioners’ community for estimating software projects and measuring the quality of the software delivered. For instance, there are hundreds of measures proposed for software quality, software complexity, objects oriented as well as an impressive number of estimation models

But …

  • How many software organizations today have in place software measurement programs and use these measures and models as a basis for decision-making?

There must be then something at work that impairs the use of quantitative data for decision making in software-base organizations.

  • What is it?

Within the software measurement community that has produced this large inventory of measures and quantitative models, there is a presumption that the lack of use of software measures in industry is caused by the practitioners’ and managers’ resistance to change.

This book is based on a different analysis and understanding of this lack of use of software measures by industry: this chasm comes from a lack of credibility in the practitioners communities, and this lack of credibility comes from the immaturity and unreliability of the measures themselves proposed to date to the industry.

Up until recently, software ‘metrics’ have been most often proposed as the quantitative tools of choice in software engineering, and the analysis of these had been most often discussed from the perspective referred to as ‘measurement theory’.

However, in other disciplines, it is the domain of knowledge referred to as ‘metrology’ that is the foundation for the development and use of measurement instruments and measurement processes.

In this book, we use as a foundation the sets of measurement concepts documented in the ISO VIM (International Vocabulary of Basic and General Terms in Metrology) to document and compare the state of maturity of measures in software with respect to classic domains of science and engineering.

  • This helps in particular to document practical aspects with respect to the current design of software measures and to identify the strengths and weaknesses of their own design as measures.

What was still missing is the know-how about how to correctly design software measures, and how to recognize if a software measure is well designed, and worth using as a basis for decision-making. This book focuses precisely on these two issues.

It is up to you:

  • to acquire such know-how about the design of a software measure and
  • to run with it for the benefit of your organization.

PREFACE

A book on the design of software measures must be suited to software engineers, both practitioners and researchers.

This book presents a perspective on software measurement that, on the one hand, is new in software engineering and, on the other hand, is fairly classical in most domains of sciences, engineering, and even in all areas of business.

Here, we share years of experience in the design of software measures for their successful use as decision making tools by software managers.

Because measurement is a fundamental engineering concept, software organizations of all sizes can use this book, and managers will find in it effective strategies for improving software management, along with numerous illustrative examples.

Applying the best practices in software measurement will ensure that software engineers and managers are equipped to respond to the most demanding customers, feel supported by senior executives and are proud to be part of the software team.

In addition, this book introduces many of the theoretical concepts and references needed by professionals, managers and students to help them understand the fundamentals of the identification and evaluation of software development and maintenance processes, and of improvements to them.

This book is intended for those developing, maintaining and managing software as well as for those in software process improvements.

STRUCTURE AND ORGANIZATION OF THIS BOOK

This book is organized into four (4) parts and fourteen (14) chapters.

Part 1: Key Concepts for the Design of Software Measures

A number of the software measures proposed to the industry have deficiencies severe enough to make some of them useless to practitioners. Part 1 presents in chapters one through five the key concepts in measurement that are necessary to recognize whether the design of a software measure is sufficiently strong to be meaningful in practice. Part 1 introduces, as well, the measurement terminology that is common in most fields of science and engineering; that is, of the metrology and related ISO standards on software measurement.

Chapter 1: Introduction.

This chapter presents the current level of maturity of software measurement within the software engineering discipline.

Chapter 2: From Measurement Methods to Quantitative Models: A Measurement Context Model.

This chapter presents a model to understand the key concepts of software measurement as well as the measurement terminology that is consistent with measurement in all disciplines. This chapter also discusses the process necessary to design a software measurement method.

Chapter 3: Metrology and Quality Criteria in Software Measurement.

This chapter presents the set of classical concepts in metrology, and presents various definitions and quality criteria in classical measurement.

Chapter 4: Quantification and Measurement are not the Same.

This chapter presents some of the differences between quantification and measurement, and establishes a parallel with the ISO 15939 Measurement Information Model.

Chapter 5: The Design of Software Measurement Methods.

This chapter presents the key concepts and steps required to design and evaluate software measurement methods, including defining the measurement principle in software measurement up to post-design activities.

Part 2: Some Popular Software Measures: How Good Are They?

Some software measures are currently popular in the industry, often because they are easy to collect or because they appear to take into account a large number of the practitioners concerns. However, in software measurement, being popular and widely quoted is not synonym to being good. Part 2 uses in chapters six through ten the criteria from Part 1 to illustrate some of the major weaknesses in the design of a few of the software measures that are either widely used or widely quoted in the software industry.

Chapter 6: Cyclomatic Complexity Number: Analysis of its Design

Chapter 7: Hasltead’s Metrics: Analysis of their Designs

Chapter 8: Function Points: Analysis of their Design.

Chapter 9: Use Case Points: Analysis of their Design.

Chapter 10: ISO 9126: Analysis of its Quality Models and Measures.

Part 3: The Design of COSMIC—ISO 10761

Part 3 illustrates in chapters eleven and twelve how the lessons learned from the analysis of the key concepts for the design of a software measure have been put into practice to design a software measurement method conformant to the ISO criteria for a measurement method of the functional size of the software, that is the COSMIC—ISO 19761. Part 3 focuses on the design process rather than on the details of this specific measurement method.

Chapter 11: COSMIC: Design of an Initial Prototype.

This chapter illustrates how this software measure of the functional size of software for real-time and embedded software was designed in response to an industry need. It describes in particular the process used to design the initial prototype of COSMIC, its field trials and its initial deployment.

Chapter 12: COSMIC—Scaling up and Industralization.

This chapter illustrates the additional effort to scale up COSMIC to increase its international acceptance and to bring it to be adopted as an international standard: ISO 19761. The key concepts of the COSMIC measurement method are also presented in this chapter.

Part 4: Other Issues in the Design of Software Measures

Part 4 illustrates in chapters thirteen and fourteen some additional issues that are traditional in measurement in day-to-day life, but that have not yet been seriously addressed in software measurement. Two specific examples are presented: convertibility across measurement design and measurement standard etalons.

Chapter 13: Convertibility across Measurement Methods

While numerous software measures are proposed for the same attributes, there is a scarcity of convertibility studies across alternative ways of measuring. This chapter presents a convertibility analysis across two functional size measurement methods: IFPUG Function Points and COSMIC Function Points.

Chapter 14: Design of Standard Etalons: The Next Frontier in Software Measurement

While measurement in science relies on well established standard etalons (such as for the meter and kilograms) to ensure the correctness and consistency of measurement results across contexts and countries, not a single standard etalon has yet been established for measuring software. This chapter looks at this next frontier in software measurement and reports on an initial attempt to design a first draft of a standard etalon for a referenced set of software requirements.

This book also contains three appendices:

Appendix A: List of Acronyms

Appendix B: Glossary of Terms in Software Measurement.

Appendix C: References

Additional material to complement the information contained in this book can be found at http://profs.logti.etsmtl.ca/aabran

If you are a software manager, you should:

f02txii24w3

If you are a software engineering practitioner or a software quality analyst using or planning to use existing software measures you should:

f02txiii24wm

If you are in software process improvement or a researcher planning to analyze existing software measures or to design new software measures or if you are taking an undergraduate or graduate course on software measurement you should:

f02txiv24x3

This book is not about:

  • A compendium of all software measures:
    • The purpose of this book is not to present an exhaustive list of measures of any type, or of a specific type (for instance on OO metrics).
    • There exists already on the market a number of books presenting inventories of alternative measures, as well as hundreds of research papers on emerging designs, which at this stage would still be fairly immature.
  • A compendium of software estimation models:
    • This book does not list or discuss any of the estimation models for software.
    • For instance, COCOMO [Boehm 1981, 2000] is an ‘estimation model’ which attempts to predict the relationships across a large number of factors. COCOMO is not about measurement but a lot more about experimentation (as in science) to build prediction models. COCOMO, for instance, should be used and evaluated as an estimation model. This will be discussed in another book looking into the design and evaluation of estimation models.
  • A compendium of analyses of all software measures:
    • This book presents from chapters six through ten analyses that have already been carried out in research and published at a number of international conferences.
    • A large number of software metrics, such as the ones in (or derived from) Chidamber & Kemerer metrics suite [Chidamber 1993], has not yet been analyzed from a metrology perspective. The analysis from a metrology perspective of these other measures still has to be done.

COSMIC Function Points

The COSMIC Function Points have been adopted in 2003 as an international standard—ISO 19761—for measuring the functional size of software. Having been designed to meet metrology criteria, COSMIC Function Points are at times used in this book to illustrate a number of measurement concepts. For more details on the design of COSMIC Function Points, see Section 5 of Chapter 12.

ACKNOWLEDGMENTS

A number of collaborators, including colleagues in industry and university as well as PhD students, have helped me over the years improve my understanding of many of the concepts presented in this book, in particular:

ChapterCo-Contributor
2: From Measurement Methods to Quantitative Models: A Measurement Context ModelDr. Jean-Philippe Jacquet (France)
3: Metrology and Quality Criteria in Software MeasurementDr. Asma Sellami—University of Sfax (Tunisia)
4: Quantification and Measurement are not the Same.Dr. Jean-Marc Desharnais—Ecole de technologie supérieure (Canada) & Bogaziçi University (Turkey)
5: The Design of a Software Measurement MethodDr. Naji Habra—Facultés Universitaires Notre-Dame de la Paix—FUNDP, Namur (Belgium)
6: Cyclomatic Complexity Number: Analysis of its DesignDr. Naji Habra—FUNDP (Belgium)
7: Halstead’s Metrics: Analysis of their DesignsDr. Rafa Al-Qutaish—Alain University of Science and Technology, Abu Dhabi Campus, United Arab Emirates
8: Function Points: Analysis of their Design 
9: Use Case Points: Analysis of their designJoost Ouwerkerk—Expedia (Canada)
10: ISO 9126: Analysis of its Quality Models and MeasuresDr. Rafa Al-Qutaish—Alain University of Science and Technology, Abu Dhabi Campus, United Arab Emirates
11: COSMIC—Design of an Initial PrototypeD. St-Pierre, Dr. Desharnais, Dr. P. Bourque and M. Maya (École de technologie supérieure—University of Québec—Canada)
12: COSMIC—Scaling up and IndustralizationC. Symons, M. O’Neil, P. Fagg, and a number of the COSMIC members of the measurement practices committee
13: Convertibility Across Measurement MethodsDr Desharnais—Ecole de technologie supérieure (Canada) & Bogaziçi University Turkey
14: Design of Standards Etalons: The Next Frontier in Software MeasurementDr. Adel Khelifi—University Al Hosn (United Arab Emirates)

Above all, this book is dedicated to all those who provided me with feedback and insights on software measures over the years and who are contributing, each in his or her own way, to the improvement of software measures as a foundation for sound, quantitatively-based decision making.

ABOUT THE AUTHOR

Dr. Alain Abran is a professor and the director of the research group in Software Engineering Management at the École de Technologie Supérieure (ETS)—Université du Québec, Montréal, Canada (www.gelog.etsmtl.ca)

He is a co-editor of the Guide to the Software Engineering Body of Knowledge (www.swebok.org). He is actively involved with software engineering standards with ISO/IEC JTC1 SC7—Software and System Engineering—and has been its international secretary in 2001–2003. He is chairman of the Common Software Measurement International Consortium (COSMIC).

Dr. Abran has more than 20 years of industry experience in information systems development and software engineering and 15 years of university teaching. He holds a PhD in electrical and computer engineering (1994) from the École Polytechnique de Montréal (Canada) and Master’s degrees in management sciences (1974) and electrical engineering (1975) from the University of Ottawa (Canada).

His research interests include software productivity and estimation models, software engineering foundations, software quality, software measurement, functional size measurement methods, software risk management and software maintenance management.

Most of his publications can be downloaded from: http://profs.logti.etsmtl.ca/aabran/Publications/index.html

Part 1: KEY CONCEPTS FOR THE DESIGN OF SOFTWARE MEASURES

1

INTRODUCTION

This chapter covers:

  • Software measurement: is it mature or not?
  • Software measurement as a new technology
  • The designs of software measures must be verified
  • Advanced Readings: Measurement within the Software Engineering Body of Knowledge

1.1. INTRODUCTION

In the field of software engineering, the term “metrics” is used in reference to multiple concepts; for example, the quantity to be measured (measurand1), the measurement procedure, the measurement results or models of relationships across multiple measures, or measurement of the objects themselves. In the software engineering literature, the term was, up until recently, applied to:

  • measurement of a concept: e.g. cyclomatic complexity [McCabe 1976],
  • quality models: e.g. ISO 9126—software product quality, and
  • estimation models: e.g. Halstead’s effort equation [Halstead 1977], COCOMO I and II [Boehm, 1981, 2000], Use Case Points, etc.

This has led to many curious problems, among them a proliferation of publications on metrics for concepts of interest, but with a very low rate of acceptance and use by either researchers or practitioners, as well as a lack of consensus on how to validate so many proposals.

The inventory of software metrics is at the present time so diversified and includes so many individual proposals that it is not seen as economically feasible for either the industry or the research community to investigate each of the hundreds of alternatives proposed to date (for instance, to measure software quality or software complexity).

This chapter illustrates the immaturity of both the software measures themselves and the necessity to verify the designs of these measures.

1.2. SOFTWARE MEASUREMENT: IS IT MATURE OR NOT?

The IEEE Computer Society defines software engineering as:

“(1) The application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software; that is, the application of engineering to software.

(2) The study of approaches as in (1)” [IEEE 610.12]

From the IEEE definition of software engineering—see box; it is obvious that measurement is fundamental to software engineering as an engineering discipline.

But what is the status of measurement within software engineering, and how mature is the field of knowledge on software measurement?

In software engineering, the software metrics approach has, up to fairly recently, been the dominant approach to measurement in this new engineering discipline.

Over recent decades, hundreds of so-called software metrics have been proposed by researchers and practitioners alike, in both theoretical and empirical studies, for measuring software products and software processes [Boehm 2000, Chidamber 1993, Karner 1993, Halstead 1997, etc.]:

  • Most of these metrics were designed based either on intuition on the part of researchers or on an empirical basis, or both, and they have most often been characterized by the ease with which some entities of the development process can be counted.
  • In their analysis of some of them, researchers have most often used the concepts of measurement theory as the foundation for their analytical investigation. However, while relevant, measurement theory deals with only a subset of the classical set of measurement concepts, and software metrics researchers, by focusing solely on measurement theory, have investigated mainly the representation conditions, the mathematical properties of the manipulation of numbers, and the proper conditions for such manipulations [Fenton 1997, Zuse 1997].
  • In the scientific fields, including engineering, as well as in others, like business administration and a significant number of the social sciences, measurement is one of a number of analytical tools. Measurement in those sciences is based on a large body of knowledge built up over centuries, even millennia, which is commonly referred to as “metrology”.

In the literature on software metrics, there is almost no reference to the classical concepts of metrology in investigations into the quality of the metrics proposed to the software engineering community.

Only recently have some of the metrology-related concepts been introduced in the software engineering community, including the selection of the ISO vocabulary on metrology [VIM 2007] as the basis for measurement terminology for all future ISO standards on software measurement.

One of the peculiarities of software engineering relative to other engineering and scientific disciplines is its lack of general use of quantitative data for decision making. Symptoms of this are:

  • a very limited number of accepted software measures available to practitioners and recognized as mature enough to be recognized as international standards, and
  • a very small number of rigorous experimental studies (which constitute general practice in the engineering and medical fields, for example).

In mature disciplines, there is:

  • a large international consensus on measures, in addition to established measurement methods and an etalon for each, and
  • a significant number of measuring instruments to be used in different contexts and under various constraints, many of them certified and calibrated against internationally recognized, and traceable, etalons.

In mature disciplines, measures are also recognized as a necessary cost of doing business and of carrying on engineering activities, as well as a must for improving decision making.

In the software domain, we have none of the above, with the exception of the recent adoption of ISO standards for the measurement of the functional size of software, and some works-in-progress for the measurement of software quality.

ISO Standards for Measuring the Functional Size of Software

The ISO 14143-1 on the mandatory set of characteristics of software functional size (i.e. a meta-standard),

Five (5) ISO recognized specific measurement methods to implement the quantification of these functional size characteristics: ISO 19761- COSMIC, ISO 20926-IFPUG, ISO 20968-MKII, ISO 24570-NESMA and ISO 29881-FISMA.

Note: The software functional size measurement process is not yet mature enough for there to be a single universal way of measuring software functional size.

ISO Standards for the Measurement of Software Quality

The set of models of software product quality in ISO 9126-1 constitutes an international standard.

The three catalogs of more than 250 “metrics” in Parts 2 to 4 of ISO 9126 are still only ISO technical reports: much work remains to be done to bring them up to ISO standard status.

What does this mean for software measurement?

  • Many, if not most, of the software measures proposed to the industry have not been seriously analyzed, nor are they sufficiently mature.
  • In contrast to other fields of science and engineering, these software measures lack the credibility to be used as a basis for decision making.
  • Verification criteria for software measures should be comprehensive, carefully defined, and agreed upon.
  • Designers of software measures should document how well their proposed measures meet these verification criteria.

Impact of Lack of Credibility of Software Measures

It is not until it can be demonstrated unambiguously that a proposed measure achieves a high level of measurement quality that it can be expected to reach a level of credibility in the practitioner and manager communities, and then be used in practice on a large scale.

Impact of the absence of software measure credibility: when objective and quantitative data are required for decision making in software engineering, software engineering researchers and practitioners must often design and develop their own individual software measures and measurement methods, whereas these already exist in other fields of knowledge.

1.3. SOFTWARE MEASUREMENT AS A NEW TECHNOLOGY

Technology is defined as the set of methods and materials used to achieve industrial or commercial objectives.

This definition does not limit technology to materials alone: it also includes processes and the knowledge related to them, referred to as “know-how.”

From that perspective, software measurement is a technology.

While some technologies are quite mature and widely used by practitioners in industry, others are emerging and in need of significant improvement if they are to penetrate deeply into an industrial domain.

  • Mature technologies: they typically have been fine-tuned over many years and they have been adapted with a number of features and tools to fit various contexts and to facilitate their use by non experts.
  • Innovative and immature technologies: they require significantly more expertise for using them in their ‘immature’ status.
    • Innovative (and immature) technologies are used initially by innovators who try them, test them and invest in improving them to facilitate their use within their technical context. Innovators work at bypassing initial design weaknesses to facilitate their use and adoption by people with less expertise.

Software measurement is definitively a new technology, and, as such, it shares many of the characteristics of new technologies and as well as the constraints that must be tackled to facilitate its adoption by industry at large and by individual practitioners.

What does it take for a new technology to be adopted?

On the part of an organization:

  • The new, initially unknown technology must promise enough benefits to overcome the pain of changing from a known one.
  • The organization must have (or gain) the technological know-how to make it work.
  • The organization must be clever enough, and enthusiastic enough, to harvest its benefits, which takes time.

On the part of an industry:

  • The new technology must become integrated into the industry’s technological environment.
  • It must also become integrated into the business context (which includes its legal and regulatory aspects).
  • It must have been proven to work well in a large variety of application contexts (that is, the technology must be mature, or maturing rapidly).

What does it take for an industry to promote a new technology?

  • The industry must recognize that there is a direction that has been proven to work in similar contexts.
  • It must recognize that current practices are not good enough.
  • It must also recognize that the players will not, on their own, submit to the pain of change (unless the environmental-regulatory context forces such a change).
  • It must want to speed up the transition to the new technology.

What does it take for software measurement to be adopted as a new technology?

On the part of a software organization:

  • Software measurement must promise enough benefits to overcome the pain of changing to an initially unknown technology.
  • The organization must have the technological know-how in software measurement to make it work.
  • The organization must be clever enough, and enthusiastic enough, to harvest the benefits, which takes time.

On the part of the software industry:

  • Software measurement must become integrated into the technological environment of the software industry.
  • It must become integrated into the business context (which includes its legal and regulatory aspects).
  • Software measurement must already have been proven to work well in a large variety of contexts (that is, it must be mature as a technology, or maturing rapidly).

What does it take for an industry to promote software measurement as a new technology?

  • Software measurement must have been proven to work in similar contexts.
  • Current software measurement practices must be good enough.
  • The industry must recognize that the players will not, by themselves, submit to the pain of change (unless the environmental-regulatory context forces such a change).
  • It should want to speed up the transition to quantitative support for decision making.

The software industry has yet to resolve many of these issues, and much work remains to bring software measurement to a high enough level of quality and maturity to meet market expectations.

1.4. THE DESIGNS OF SOFTWARE MEASURES MUST BE VERIFIED

Software measurement must play an increasingly important role in software engineering if it is to truly become an engineering discipline.

  • Over the past twenty years, a significant number of software metrics have been proposed for better control and greater understanding of software development practices and products.
  • Unfortunately, very few of these metrics have been looked at closely from a measurement method perspective, and it is currently difficult, because of a lack of agreed-upon frameworks of verification and validation procedures, to analyze their quality.

What constitutes a valid metrics, or even a valid measurement method? How can a measurement method be validated?

Various authors have attempted to address these questions in recent years, and from different points of view (mathematical, practical, etc.)2.

The analytical perspective proposed in this book is complementary to previous works referred to above and discusses this issue from a measurement method point of view. Furthermore, to avoid the confusion generated by inconsistent terminology in previous studies on validation, the more precise set of concepts of verification criteria is preferred here over validation criteria.

In Part 1 of this book (Chapters 2 to 5) we will refer to specific criteria for verification, rather then to generic concepts related to validation in general.

Examples of Verification Questions that Need to be Investigated

Does the measurement method really measure the concept intended to be measured?

Has the measurement method been internally verified, i.e. in the sense that it can be shown that it gives a proper numerical characterization of the attribute to be measured?

Is the measurement method usable? A measurement method which is as perfect as possible from a mathematical viewpoint would not be of any interest if it could not be applied (if it were far too time-consuming, for example).

ADVANCED READINGS: HOW MUCH MEASUREMENT SUPPORT IS RECOGNIZED WITHIN THE SOFTWARE ENGINEERING BODY OF KNOWLEDGE (SWEBOK)?

Audience

This Advanced Reading section provides to readers an overview of how software measurement fits within the discipline of software engineering, and how much support software measurement provides to software from an engineering perspective.

Software engineering has only recently reached the status of a legitimate engineering discipline and a recognized profession.

  • Up to 2003, there was no generally accepted body of knowledge in this new field.
  • To address the issue, the IEEE-Computer Society initiated various task forces in the 1990s, including the SWEBOK (Software Engineering Body of Knowledge) project [Abran, Moore et al. 2005d], to develop an international consensus on a compendium of the body of knowledge that has been developing and evolving over the past four decades and a guide to that compendium.

Importantly, the SWEBOK is not static, and the Guide to it must, of necessity, develop and evolve as software engineering matures.

The SWEBOK was established with the following five objectives:

1. Promote a consistent view of software engineering worldwide;

2. Clarify the place and set the boundary of software engineering with respect to other disciplines, such as computer science, project management, computer engineering, and mathematics;

3. Characterize the contents of the software engineering discipline;

4. Provide a topical access to this body of knowledge;

5. Provide a foundation for curriculum development, and for individual certification and licensing material.

The material that is recognized as being within the realm of software engineering is organized into the 10 Knowledge Areas (KAs) listed in Table 1.1.

TABLE 1.1. The 10 SWEBOK Knowledge Areas

1. Software requirements
2. Software design
3. Software construction
4. Software testing
5. Software maintenance
6. Software configuration management
7. Software engineering management
8. Software engineering process
9. Software engineering tools and methods
10. Software quality

The topic of measurement was the subject of one of the editorial criteria for the initial write-up of the first draft of the SWEBOK Guide, that is, that the measurement “theme” be common to all KAs.

Measurement in engineering is an integral part of all engineering KAs, and therefore had to be incorporated into the proposed breakdown of topics in each KA for software engineering.

Since a criterion for the inclusion of knowledge in the SWEBOK Guide was that it be generally accepted, it is important to ask what did, in fact, gain approval on a consensual basis with respect to measurement, and what can be learned from this consensus, or the lack of it.

The SWEBOK Definition of “Generally Accepted”

Applies to most of the projects, most of the time, and widespread consensus validates its value and effectiveness

This definition is borrowed from the Project Management Institute (PMI).

Another tool used for the development of the SWEBOK, from an engineering viewpoint, was the Vincenti [1990] classification of engineering knowledge, including, of course, “quantitative data” as a category.

On the left-hand side of Table 1.2, the six categories of engineering knowledge are listed, while the next three columns present related sub-concepts identified in specific engineering disciplines for classification purposes.

TABLE 1.2. Classification of Engineering Knowledge —Vincenti

c01t01220cb

At the beginning of the SWEBOK project, it was suggested to the specialists in charge of a particular software engineering KA that they use this classification for their initial draft.

This was, of course, a challenging assignment, because:

  • the discipline of software engineering was not mature enough at the time, and
  • the classification could not be directly implemented in most of the KAs of the software engineering taxonomy.

The Vincenti classification for engineering knowledge types is very useful for understanding the depth of coverage of some engineering topics (including measurement) within each KA.

For instance, it can be observed that, while the term “measurement” appears by design throughout all the SWEBOK KAs (that is, it was the subject of one of the editorial criteria), neither the KA editors nor the group of reviewers has pointed to key references providing generally accepted quantitative data for any of the topics identified in each KA.

In the 2004 version of the SWEBOK Guide, there is no reference to highly credible and documented quantitative data and relevant repositories of quantitative references.

This means, for instance, that, while there are a large number of chapters and books dedicated to estimation in the software engineering literature, the raw datasets available for study often lack engineering rigor in the data collection procedures on the one hand, and, on the other, the estimation models proposed usually have both poor explanatory power and significant limitations in generalization power.

Quantitative Data in Engineering

In engineering, quantitative data does not mean raw data, but rather descriptive or prescriptive data usually derived from controlled experiments using widely recognized measurement concepts, verified measurement instruments, documented measurement protocols, and extensive testing and replication procedures to ensure both the verification of data inputs and an in-depth understanding of the phenomena under study in order to identify both the range of operations and their limitations.

Data in Software Engineering Estimation Models

In both versions of the COCOMO models [Boehm 1981, 2000], many parameters are described by linguistic values, and their influence is determined by expert opinion rather than on the basis of information from descriptive and prescriptive engineering repositories.

Significant effort still needs to be invested by model builders in terms of incorporating engineering strength into these models.

EXERCISES

1. In every organization, there is a well-staffed and well-funded accounting department. An accounting department is basically a measurement program. Why is such measurement considered as essential to any organization, from the very small to the largest?

2. In software organizations, most consider measurement programs too expensive and too time-consuming. Why?

3. Business managers and engineers thrive on measurements and quantitative data. What about software managers?

4. In all areas of an organization, staff resources willingly collect all kinds of measures for management purposes. Why are software staff so reluctant to collect software measures?

5. In various business areas, many measures are collected manually and used for taking major business decisions. Why are manually collected measures regarded with such suspicion in software organizations?

6. In engineering organizations, significant efforts are devoted to measurement? What benefits are expected?

7. In engineering organizations, significant budgets are dedicated to acquiring or developing measuring instruments. By comparison, how much is spent in software organizations on acquiring or developing automated measurement systems?

8. In engineering and in the basic sciences, a great deal of research funding goes towards conducting experiments to collect “quantifiable data”. How much of the research budget in software engineering goes into collecting such data?

9. Select three of the most often quoted “software metrics”. For which of these does your organization have the technological knowledge to make them work.

10. Select two software measures collected in your organization. Identify business decisions which have been based on results from data collected with these two measures.

11. What is the ultimate recognition for a measure in most scientific fields?

12. Which criteria would you use to verify that the design of a software measure is sound?

13. Can you have engineering without measurement? Can you have software engineering without measurement?

14. Give examples of international consensus on some software measures.

15. How much quantitative data in software engineering are now recognized as generally accepted? Provide supporting evidence for your answer.

16. List five software measures and explain why you would consider them either mature or immature.

TERM ASSIGNMENTS

1. If you consider software measurement to be a new technology, what are the typical hurdles that must be overcome to implement software measures in an organization and in the software industry?

2. Of the software measures you are currently using in your organization, for which of them do you currently have documented evidence that they are based on a sound foundation?

3. Measurement and quantitative data are fundamental in engineering. Measurement is also important in the Guide to the Software Engineering Body of Knowledge—SWEBOK www.swebok.org. Identify recent sets of quantitative data which you would recommend for inclusion in the next release of the SWEBOK.

Notes

1 A measurand is defined as a particular quantity subject to measurement; the specification of a measurand may require statements about quantities such as time, temperature, and pressure [VIM 2007].

2 See Chapter 2, Section 6.