Copyright © 2016 by John Wiley & Sons, Inc. All rights reserved
Published by John Wiley & Sons, Inc., Hoboken, New Jersey\hb
Published simultaneously in Canada
All names of products and companies used in this book may be trademarks of their corresponding owners. In particular, the following terms and abbreviations are registered trademarks: BPMN™, Hewlett-Packard®, HP®, ISTQB®, IQBBA®, iSQI®, MATLAB Simulink®, Microsoft® Windows®, Model Driven Architecture®, MOF®, Object Management Group®, OMG®, RUP®, Sparx® Systems Enterprise Architect®, Stateflow®, UML®, Unified Modeling Language™, XMI®.
No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 750-4470, or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley.com/go/permission.
Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.
For general information on our other products and services or for technical support, please contact our Customer Care Department within the United States at (800) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002.
Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic formats. For more information about Wiley products, visit our web site at www.wiley.com.
Library of Congress Cataloging-in-Publication Data
Names: Kramer, Anne, (Software engineer) author. | Legeard, Bruno, author.
Title: Model-based testing essentials : guide to the ISTQB certified
model-based tester foundation level / Anne Kramer, Bruno Legeard.
Description: Hoboken, New Jersey : John Wiley & Sons Inc., [2016] | Includes
bibliographical references and index.
Identifiers: LCCN 2015035751 | ISBN 9781119130017 (cloth)
Subjects: LCSH: Computer software--Testing--Examinations--Study guides. |
Model-based reasoning--Examinations--Study guides. | International
Software Testing Qualifications Board--Examinations--Study guides. |
Electronic data processing personnel--Certification.
Classification: LCC QA76.76.T48 K73 2016 | DDC 005.3028/7--dc23 LC record available at http://lccn.loc.gov/2015035751
To our families
When I was asked by my dear friend Bruno Legeard to provide a foreword to the “Model-Based Testing Essentials” book that he was authoring together with Anne Kramer, I was very pleased because this was a sort of flashback that reminded me of when I was younger, so much younger than today, and was preparing my PhD dissertation, which was based on automatic test generation for telecommunication systems based on SADT and SDL models further transformed into Petri Nets.
This is just a small witness of the fact that the concept of model-based testing (MBT) has been in existence for many years; nowadays methods, techniques, and tools have matured to a point at which MBT is ready for widespread adoption in many domains, starting from the ones that have historically taken advantage of such approach (e.g., aerospace, railway and telecommunications) to others, such as automotive, in which systematic structured testing techniques have been more and more applied in the last years and MBT has quickly become a de facto standard.
In my capacity of ISTQB® President, I am very happy to welcome this book that will greatly help professionals in taking the exam for ISTQB® Model-Based Tester certification; as a matter of fact, ISTQB® has approved for general market availability in October 2015 the Model-Based Tester syllabus, which is available for free download from our website (www.istqb.org) and that is the basis around which this book has been prepared, constituting an excellent study guide to the ISTQB® Certified Model-Based Tester.
ISTQB® stand for International Software Testing Qualifications Board, a not-for-profit association, which has gained the status of world's leading organization for Software Testing Certification, by running over 560,000 exams as of June 2015.
Our mission is to continually improve and advance the software testing profession by defining and maintaining a Body of Knowledge that allows testers to be certified based on best practices, connecting the international software testing community, and encouraging research.
In this context, the ISTQB® portfolio of certifications has recently been expanded in order to complement the historically “Core” certifications (Foundation, Advanced Test Manager, Advanced Test Analyst, Advanced Technical Test Analyst, Expert) with a set of certifications dedicated to Agile Testing and a brand new cluster of “Specialist” modules, whose goal is to cover specific topics/domains in a vertical way, using a drill-down/deep-dive approach.
The “Model-Based Tester” syllabus and certification is the first of several Specialist modules to be brought to market; it complements the Core Foundation as a Specialist module and it provides a practical and easy entry to the MBT approach.
The ISTQB® Glossary of software testing terms defines Model-Based Testing as “Testing based on or involving models”; by studying this book, testers will learn how to use models to drive test analysis and design, as well as how to take advantage of the models for other testing activities such as test implementation and reporting. In so doing, you will be able to improve the effectiveness and efficiency of the test process.
So, my thanks to the ISTQB® Working Group members who developed the ISTQB® MBT certification and to Anne and Bruno for providing a thorough and pragmatic study guide to it, which I am sure will be appreciated by the software testing professionals worldwide.
Gualtiero Bazzana
ISTQB® President
Although I have long contended that all testing is model based, the nature of tools and models defines substantially different approaches to software testing. After some 20 years of research and practice in automated model-based testing over thousands of projects around the world, a broad, substantial, and proven body of knowledge has been established. I have been fortunate to play a role in this evolution as a developer and user of several model-based testing tools and as the author of three dozen test design patterns (each a model.)
During these two decades, software and computing technology has become indispensable for modern life and is increasingly shaping it. Although it has become easier to produce software, it has not become any easier to be sure that our creations are always well-behaved. Model-based testing has shown that it can better meet the quality challenges of present-day systems than the wishful thinking that is often used as a testing strategy.
We know how to select, deploy, develop, and integrate model-based testing tools. We know how to create test models for all kinds of application domains and technology stacks. We know how to adapt development, testing, and maintenance processes to make best use of model-based testing, in both agile and traditional life cycles. Perhaps most important, we have learned when model-based testing is practical and when it is not.
These have been pioneering years with attendant successes and failures. Some once promising approaches and explorations have fallen away. The curiosity and energy of open-source developers and researchers have shown how formalized relationships may be realized as useful tools. Technology entrepreneurs have created a small but ever-better stream of product offerings, each testing market acceptance and rejection. Large and small organizations have tried model-based testing. Many have found it useful enough to take the difficult path of institutionalizing it.
These pioneers had to learn by trial and error, reading available publications, and sometimes plunging into the depths of research reports. They and their organizations have had to accept the risk that such experimentation brings.
I believe that in the next 5–10 years, model-based testing can become as routine and widely used as testing with open source tools such as Selenium or Junit. Credible and practical education in model-based testing will be critical for this transition.
Until now, a practitioner, manager, or agilist wondering what model-based testing is really all about had to work hard to get an answer. That meant attending an annual conference such as UCAAT and/or taking a risk to experiment with a pilot project. That pioneer would have to work out how model-based testing fit with their understanding of software development and testing, and finally how it could be socialized in their organization.
With the publication of Model-Based Testing Essentials, readers have a credible, practical, and complete introduction to and guidance for using model-based testing. This book distills hard-won knowledge in a systematic presentation of topics needed to use and deploy model-based testing. I believe it will not only be a useful guide to prepare for the ISTQB® exam, but a valuable handbook for new and experienced practitioners alike.
Robert V. Binder
Carnegie-Mellon University Software Engineering Institute, Pittsburgh
November 2015
Testing our products gives us confidence that they perform as intended. If we do not find any bugs, then everything should be fine… or not? What, if we do not really have confidence in the test itself? Are we sure we test the parts that are really important? Is our test complete and, if we think so, are we able to prove this? What about the cost of testing? Is it possible to spare time or money in the testing phases?
Questions like this brought the testing community to think about using models for testing purposes, and to develop a new testing approach called model-based testing (MBT). The initial idea of MBT dates back to 1970s, but its industrial application on a larger scale started only during the last decade. Today, the community of MBT adepts is growing constantly and the topic attracts more and more attention. Therefore, the International Software Testing Qualifications Board (ISTQB) decided to issue a new certification scheme, the “ISTQB Certified Tester Foundation Level – Model-Based Tester Certification.”
The ISTQB standard glossary of terms used in software testing [1] defines MBT as “testing based on or involving models.” By model, we mean a somehow formal or structured representation of the system under test, its environment, or the test itself, which directly supports test analysis and design activities, but also test planning and control, implementation and execution, and reporting activities.
Why MBT is worth reading a book or attending a training course? In 2014, we performed a user survey among MBT practitioners and asked them about their expectations regarding MBT (see Figure 1). 1
Figure 1 Expectations of MBT practitioners
(from 2014 MBT User Survey).
The top four expectations clearly relate to the questions we asked about testing practices in the beginning of this section:
Better tests
We expect that MBT will help us to increase the effectiveness of our test process, for example by discovering more bugs during the various test cycles.
Cheaper tests
We are interested in automated test generation to reduce the costs of test creation and maintenance and, thus, to increase the overall efficiency of our test process.
Managing complexity
Models are abstract representations of some existing or planned reality. They proved to be very helpful in system architecture and design. Using them in the test process should help us to keep the overview and to focus on what matters for our test objectives.
Start test design earlier
One of the major promises of MBT is early requirements validation. We can start the modeling activities long time before the system under test exists in reality. Thus, MBT should help us in implementing the ISTQB good practice of early testing.
This book will show you how MBT can fulfill these expectations based on the current state of the practice in this area as taught in the ISTQB Model-Based Tester – Foundation Level certification syllabus. Furthermore, we focus on essential aspects of model-based testing in practice. Beyond the syllabus, we provide additional information on MBT and share with you our personal experience with the technique.
There are many different approaches to model-based testing. Some adepts place their expectations in complete automation, both of test case generation and test execution. Others use the models mainly to describe the test idea and to generate test cases for manual execution. All those approaches are valid and have their advantages, but without a structured introduction to the topic, newcomers are rapidly lost. This is where the ISTQB certification scheme helps.
The ISTQB MBT certification extends the ISTQB Certified Tester Foundation Level and provides additional skills to professional testers. The qualification scheme addresses people having reached the first level of achievement in their software testing career, including testers, test analysts, test automation engineers, test managers, as well as system and software developers and architects. It may also be beneficial for anyone who wants a deeper understanding of software testing and of the use of models for test generation, such as project managers, quality managers, product managers, business analysts, and business process experts. In general, an ISTQB Certified Model-Based Tester has acquired the necessary skills to contribute successfully to MBT projects in a given context.
Figure 2 provides a summary of the ISTQB MBT syllabus content. The syllabus is divided into five chapters, covering the introduction of MBT (Chapter 1), the main phases of the MBT process (Chapters 2–4), and the evaluation and deployment of an MBT approach (Chapter 5).
Figure 2 ISTQB MBT syllabus – content overview.
The syllabus covers 36 learning objectives at different cognitive levels of knowledge (so-called K-levels):
In addition, the syllabus includes 10 MBT-specific terms, which are equally part of the examination.
Throughout this book, we provide you with the required knowledge to pass successfully the ISTQB MBT certification. Various exercises invite you to practice modeling and test generation activities. If you are not interested in passing the exam, you may skip them, but if you are, beware of the fact that they are essential to reach cognitive level K3.
This book consists of 12 chapters plus 3 appendices. We tried to stay as close as possible to the structure of the ISTQB MBT syllabus. However, for pedagogic reasons, there are some exceptions from this rule. To make the mapping easier for you, each chapter of this book starts with a reference to the corresponding syllabus section(s).
In Chapter 1, we introduce the topic of model-based testing and tell you what you can realistically expect from this test approach.
In Chapter 2, you find the definition of the MBT-specific terms that are part of the ISTQB MBT examination. Moreover, we define additional terms used in this book to make sure we all understand them in a similar way. Section 2.3 is very important. We present the two simple graphical modeling languages proposed by the ISTQB MBT syllabus. Unless stated otherwise, all example models in this book respect the rules of one of those two sample languages.
Chapter 3, then, goes to the root of the matter. We explain how a test process employing MBT looks like, including its activities, actors, and roles, as well as its input and output, and its integration into the global software development lifecycle.
Modeling being a core activity in MBT, three chapters of this book deal with this topic. Chapter 4 points out some fundamental aspects you should consider before starting to write an MBT model.
Chapter 5 presents a variety of different modeling languages. It shall enable you to select the most appropriate language for your particular application focus. In addition, we provide some general information on MBT modeling tools.
In Chapter 6, we present common good modeling practices and draw your attention to typical mistakes and pitfalls. Unlike the other sections of Chapter 6, Section 6.8 is not based on the syllabus. Instead, it contains a longer list of recommendations regarding modeling practices from other sources including our personal experience.
In Chapter 7, we have a short glimpse on classic test design techniques as taught in the ISTQB Certified Tester Foundation Level course. We see how MBT relates to those techniques, which it does not replace, but supports.
Chapter 8 is entirely dedicated to test case selection. We present a taxonomy of selection criteria, examples for their practical application, pros and cons, and report some recommendations. The last section of this chapter briefly deals with automated test generation in general.
Next to test generation comes test implementation and execution, which is the topic of Chapter 9. We explain the specifics of MBT test implementation, the dependencies on modeling and test generation activities, and the additional steps required to automate the generated test cases.
Chapter 10 is the last chapter having a direct correspondence in the ISTQB MBT syllabus. It deals with the introduction of MBT in a company. It is not the classic “Do a pilot first” stuff. Instead, we focus on very specific aspects of MBT. How do the objectives of your organization influence the choice of an MBT approach? Where do the cost factors originate and where may we expect financial benefits? We present metrics to measure success and good practices to apply. In a section on tool support, we consider MBT tools supporting test generation activities and their integration with other tools.
In Chapter 11, we present three radically different case studies illustrating the daily work with MBT.
The last chapter summarizes the book with an overall positive conclusion. MBT is worth trying! Finally, in the appendices you will find the solutions of the exercises, a short quiz to test your understanding, and some additional information.
We are very pleased to thank the colleagues from whom we received valuable help and contributions for this book. We are grateful to the reviewers of the initial drafts of the book for their helpful comments and suggestions: Regina Weickmann, François Guerin, Vincent Guillard, Véronique Hamel, and Norbert Kastner. As part of the ISTQB Model-Based Tester Working Group, we had many debates and influential interactions during all the periods of developing the ISTQB syllabus with the colleagues in the working group. We want to thank all the other members of the author team: Stephan Christmann, Lars Frantzen, Armin Metzger, Thomas Müller, Ina Schieferdecker, Stephan Weißleder, and also Natasa Micuda, who managed the exam material and Stephan Schulz, who managed the review process of the syllabus.
Special thanks to Arnaud Bouzy and Elizabeta Fourneret for providing material for the use cases, respectively, in Sections 11.1 and 11.3 and to Abbas Ahmad for the TTCN-3 example.
Finally, we acknowledge the strong support we had from our families during the writing of this book. Anne particularly thanks her husband, Hansgerd, and her daughter, Alice, for their enormous patience, the unceasing encouragement, and their loving support, which made this book possible. Bruno thanks his wife, Oumhanie, and children Anissa, Loïc, Fabien, and Maëva, for their constant support, love, and understanding.
Anne Kramer – Erlangen
Bruno Legeard – Besançon
July 2015