This edition first published 2018
© 2018 John Wiley & Sons Ltd
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by law. Advice on how to obtain permission to reuse material from this title is available at http://www.wiley.com/go/permissions.
The right of Andrea Ahlemeyer‐Stubbe and Shirley Coleman to be identified as the authors of this work has been asserted in accordance with law.
Registered Offices
John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, USA
John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, UK
Editorial Office
9600 Garsington Road, Oxford, OX4 2DQ, UK
For details of our global editorial offices, customer services, and more information about Wiley products visit us at www.wiley.com.
Wiley also publishes its books in a variety of electronic formats and by print‐on‐demand. Some content that appears in standard print versions of this book may not be available in other formats.
Limit of Liability/Disclaimer of Warranty
While the publisher and authors have used their best efforts in preparing this work, they make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives, written sales materials or promotional statements for this work. The fact that an organization, website, or product is referred to in this work as a citation and/or potential source of further information does not mean that the publisher and authors endorse the information or services the organization, website, or product may provide or recommendations it may make. This work is sold with the understanding that the publisher is not engaged in rendering professional services. The advice and strategies contained herein may not be suitable for your situation. You should consult with a specialist where appropriate. Further, readers should be aware that websites listed in this work may have changed or disappeared between when this work was written and when it is read. Neither the publisher nor authors shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.
Library of Congress Cataloging‐in‐Publication data applied for
ISBN: 9781119125136
Cover Design: Wiley
Cover Images: (Business people) © JohnnyGreig/Gettyimages; (Currencies) © Inok/iStockphoto
This book is dedicated to
Agnes, Albert, Christopher, Dirk, Rosie and Rowan
With thanks
Andrea Ahlemeyer‐Stubbe is Director of Strategical Analytics at servicepro‐Agentur für Dialogmarketing und Verkaufsförderung GmbH, Munich, Germany (servicepro).
Upon receiving her Master’s degree in statistics from the University of Dortmund, Andrea formed a consulting firm, offering customised professional services to her clients. She now leads servicepro’s analytics team, working on international projects for well‐known brands in Europe, United States and China, drawing on the wealth of experience gained from her 20 years in the industry, specifically in the areas of data mining, data warehousing, database marketing, CRM, big data and social CRM. She is a frequent lecturer at several universities, as well as an invited speaker at professional conferences. She writes for special interest magazines as well as marketing and management publications. She was President of ENBIS (European Network for Business and Industrial Statistics) from 2007–2009.
Dr Shirley Coleman is Principal Statistician and Technical Director at the Industrial Statistics Research Unit, School of Mathematics and Statistics, Newcastle University and a visiting scholar at the Faculty of Economics, Ljubljana University, Slovenia. She works on data analytics in small and medium enterprises and the energy sector and contributed a highly ranked impact case study to Newcastle University’s Research Excellence Framework. She publishes in trade and academic journals and is co‐editor of several books. She is an elected member of the International Statistics Institute and a Chartered Statistician of the Royal Statistical Society. She is a well‐known international invited speaker and conference chair. She is an ambassador for communication and dissemination of statistics to the wider community. She was President of ENBIS (European Network for Business and Industrial Statistics) from 2004–2005.
The authors have previously collaborated on A Practical Guide to Data Mining for Business and Industry (Wiley, May 2014).
Figure 1.1 | Where does big data come from? |
Figure 1.2 | Big data empowers business |
Figure 1.3 | Roadmap to success |
Figure 1.4 | Wish list for generating money out of data |
Figure 1.5 | Monetising data |
Figure 2.1 | Deming’s ‘Plan, Do, Check, Act’ quality improvement cycle |
Figure 2.2 | Six Sigma quality improvement cycle |
Figure 2.3 | Example of data maturity model |
Figure 2.4 | Data improvement cycle |
Figure 3.1 | Big data definition |
Figure 3.2 | Internet of things timeline |
Figure 3.3 | Example data structure |
Figure 3.4 | NoSQL management systems |
Figure 3.5 | Big data structure and latency |
Figure 4.1 | Supervised learning |
Figure 4.2 | Unsupervised learning |
Figure 4.3 | The CRISP‐DM process |
Figure 4.4 | The SEMMA process |
Figure 4.5 | General representation of the data mining process |
Figure 4.6 | Time periods for data mining process |
Figure 4.7 | Stratified sampling |
Figure 4.8 | Lift chart for model comparison |
Figure 4.9 | Lift chart at small scale |
Figure 4.10 | An example of model control |
Figure 5.1 | Raw data from a customer transaction |
Figure 5.2 | Bar chart of relative frequencies |
Figure 5.3 | Example of cumulative view |
Figure 5.4 | Example of a Pareto chart |
Figure 5.5 | Example of a pie chart |
Figure 5.6 | Scatterplot of company age and auditing behaviour with LOWESS line |
Figure 5.7 | Scatterplot of design options |
Figure 5.8 | Ternary diagram showing proportions |
Figure 5.9 | Radar plot of fitness panel data |
Figure 5.10 | Example of a word cloud |
Figure 5.11 | Example of a mind map |
Figure 5.12 | Location heat map |
Figure 5.13 | Density map for minivans |
Figure 5.14 | SPC chart of shipping journeys |
Figure 5.15 | Decision tree analysis for older workers |
Figure 5.16 | Gains chart |
Figure 5.17 | Lift chart |
Figure 5.18 | ROC curve development during predictive modelling |
Figure 6.1 | Example of logistic regression |
Figure 6.2 | Corrected logistic regression |
Figure 6.3 | Decision tree |
Figure 6.4 | Artificial neural network |
Figure 6.5 | Bayesian network analysis of survey data |
Figure 6.6 | Bayesian network used to explore what‐if scenarios |
Figure 6.7 | Plot of non‐linear separation on a hyperplane |
Figure 6.8 | Dendrogram from hierarchical cluster analysis |
Figure 6.9 | Parallel plot from K‐means cluster analysis |
Figure 6.10 | Kohonen network with two‐dimensional arrangement of the output neurons |
Figure 6.11 | SOM output |
Figure 6.12 | T‐SNE output |
Figure 6.13 | Correspondence analysis output |
Figure 6.14 | Association rules |
Figure 6.15 | Association analysis of products |
Figure 6.16 | Comparison of customer base and population |
Figure 6.17 | Relationship between energy usage and deprivation |
Figure 6.18 | Map showing prices |
Figure 7.1 | Strategic opportunities |
Figure 7.2 | How data can boost top‐ and bottom‐line results |
Figure 7.3 | Typical data request |
Figure 7.4 | Observed data and usage |
Figure 7.5 | Maslow’s hierarchy of needs |
Figure 7.6 | Data sources to empower consumer business |
Figure 7.7 | Ready information on market opportunities |
Figure 7.8 | Word cloud from keyword occurrences |
Figure 7.9 | Using different data sources for analytics |
Figure 7.10 | Daily sleep patterns |
Figure 7.11 | Predictive analytics in insurance |
Figure 8.1 | Pathways to monetising data |
Figure 8.2 | Segmentation features of walk‐in customers |
Figure 8.3 | Business opportunities |
Figure 9.1 | Paths to monetisation |
Figure 9.2 | Pareto diagram of customer compliments |
Figure 9.3 | Graphical dashboard |
Figure 9.4 | Decrypting the DNA of the best existing customers |
Figure 9.5 | Aspects of digital maturity |
Figure 9.6 | Closed loop of B2B customer profiling – continuous learning |
Figure 9.7 | Automated B2B lead generation system |
Figure 9.8 | New methods, new insights, smart business |
Figure 9.9 | Misleading scatterplots |
Figure 9.10 | Scatterplot with multiple features |
Figure 9.11 | Histogram of suspicious‐quality recordings |
Figure 10.1 | The evolution of data analytics |
Figure 10.2 | Cumulative distribution of risk scores |
Figure 10.3 | Data sources in the shipping industry |
Figure 10.4 | Optimum speed recommendation |
Figure 10.5 | Pruned decision tree |
Figure 10.6 | Detail from decision tree |
Figure 10.7 | Customised communication |
Figure 10.8 | Individualised communication |
Figure 10.9 | Complexity of data mining steps |
Figure 10.10 | Data in the customer journey |
Figure 10.11 | Intelligent profiles and segments in B2C |
Figure 10.12 | Personalised journey |
Figure 10.13 | The reach of social media |
Figure 10.14 | The power of social media |
Figure 10.15 | Using peer group behaviour |
Figure 10.16 | National statistics oil prices |
Figure 10.17 | Example of reports portal |
Figure 10.18 | Making a business out of boring statistics |
Figure 10.19 | Right place, right time |
Figure 10.20 | Social media information summarised |
Figure 10.21 | Visualisation of user engagement |
Figure 10.22 | Concept of newsletter tracking |
Figure 10.23 | Example report on testing different versions |
Figure 10.24 | Customer profile details |
Figure 10.25 | Company profile details |
Figure 10.26 | Example of glass facades in buildings |
Figure 10.27 | Half normal plot of a screening experiment |
Figure 10.28 | Predicted vs calculated resistance factor with validation |
Figure 10.29 | Residual plot of prices |
Figure 10.30 | Visualisation of groups of products |
Figure 10.31 | Open data available to enrich company data |
Figure 10.32 | Diffusion map showing clusters of shares |
Figure 10.33 | Sampling approach for benchmarking in China |
Figure 10.34 | Three‐step approach to survey analytics |
Figure 10.35 | Skateboard offer |
Figure 10.36 | Customer journey |
Figure 10.37 | Example of customer segments |
Figure 10.38 | Virtual changing room |
Figure 10.39 | Virtual supermarket at bus stop |
Figure 10.40 | Input from miscellaneous IoT sensors |
Figure 10.41 | Appealing sleep sensor display |
Figure 10.42 | Sensors connected by mobile phone |
Figure 10.43 | The connected car |
Figure 10.44 | The new connected eco‐system |
Figure 10.45 | Industry 4.0 |
Figure 10.46 | Industry 4.0 in action |
Table 2.1 | Typical internal and external data in information systems |
Table 2.2 | Extract of sales data |
Table 2.3 | Company sales data analytics |
Table 2.4 | Internal sales data enriched with external data |
Table 2.5 | Scales of measurement examples |
Table 2.6 | Checklist for data readiness |
Table 4.1 | Confusion matrix for comparing models |
Table 5.1 | Partially tamed data |
Table 5.2 | Outcomes of a hypothesis test |
Table 5.3 | Typical significance borders |
Table 5.4 | Examples of statistical tests |
Table 5.5 | Example of a contingency table |
Table 5.6 | Target proportions |
Table 5.7 | Confusion matrix |
Table 5.8 | Gains chart |
Table 5.9 | Non‐cumulative lift and gains table |
Table 6.1 | Example of a contingency table |
Table 6.2 | Analysis table for goodness of fit |
Table 8.1 | Business models for types of exchange |
Table 8.2 | Business models for B2C selling |
Table 8.3 | Business models for service providers |
Table 9.1 | Business model canvas of the comparisons between data brokers and insight innovators |
Table 10.1 | Summary of case studies |
Table 10.2 | Risk scores in a simple case |
Table 10.3 | Distribution of risk scores in different seasons |
Table 10.4 | Allowable stress for soft impact |
Table 10.5 | Parameters used to describe a four‐sided glass panel |
Table 10.6 | Data dimensions and stakeholders |
When we finished writing our Practical Guide to Data Mining for Business and Industry, we realised that there were still things to say. The growth of interest in data has been enormous and there are now even more opportunities than during the earlier years when there was a steady awakening to the importance of data for business and industry.
Data analytics appears on billboards in mainstream locations such as airports, and even mathematics is being coupled with adverts for cars in a positive way. Everyone is aware that they have data and has seen the graphs and predictions that analysis produces.
The book describes how any business can be uplifted by monetising data. We show how data is generated by sensors, smart homes, apps, website visits, social network usage, digital communication, purchase behaviour, credit card usage, connected car devices and self‐quantification. Enriched by integrating with official statistics, analysis of these datasets brings real business advantage.
The book invites the reader to think about their data resources and be creative in how they use them. The book is not organised as a technical text but includes many examples of innovative applications of statistical thinking and analytical approaches. It does not propose original statistical or machine learning methods but focuses on applications of data‐driven approaches. It is general in scope and can thus serve as an introductory text. It has a management focus and the reader can judge for themselves where they can use the ideas. The structure of the book aims to be logical and cover the whole loop of using data for business decisions. The idea of exploring and giving advice on how to convert data into money is really appealing.
Even after several years of excitement about big data, there are few practical case studies available. For this reason, we include 21 in the final chapter to give realistic suggestions for what to do. The other chapters of the book give necessary background and motivational content.
It is timely to publish this book now, as big data and data analytics have captured the imagination of business and public alike. Data can be seen as the most powerful resource of the future; we believe it has more influence on the wealth of companies and people than any other resource. The authors have long been proponents of data analysis for business advantage and so it is with delight that we can collate our experience and rationale and share it with other people.
The ideas in this book have arisen from many hours of fascinating consulting work. We have felt honoured to be allowed to immerse ourselves in the company culture and explore their data, and been able to present solutions that in many cases have brought great financial benefits.
We are grateful to all the business people we have worked with. Writing takes considerable time and our families and friends have been very accommodating. We thank them all very much.
Data awareness has swept across economic, political, occupational, social and personal life. Making sense of the fabulous opportunities afforded by such an abundance of data is the challenge of every business and each individual. The journey starts with understanding what data is, where it comes from, what insight it can give and how to extract it. These activities are sometimes referred to as descriptive analytics and predictive analytics. In descriptive analytics data is explored by looking at summary statistics and graphics, and the results are highly accessible and informative. Predictive analytics takes the analysis further and involves statistical approaches that utilise the full richness of the data and lead to predictive models to aid decision making.
This introductory chapter discusses the rise in data, changes in attitude to data and the advantages of getting to grips with accessing, analysing and utilising data. Definitions of concepts such as open data and big data are followed by guidance for reading the rest of the book.
There is much more data available and accessible than ever before.
Increasingly data is discussed in the popular press and, rather than shying away from figures, statistics and mathematics, advertisers are using these words more and more often. People are becoming more comfortable with data. This is clear from the increase in the use of self‐measurement and mapping facilities on personal devices such as mobile phones and tablets; people have a thirst for measuring everything in their daily life and like to try and control things to keep their life in good shape. Many people choose vehicles that are fitted with advanced digital measurement devices that manage engine performance and record fuel usage and location. All this is in addition to the increased automation of production lines and machinery, which have resulted in copious measurements being a familiar concept. A major contributor to the rise in importance of data is the impact of cheap data storage. For example, an external hard drive with terabytes of memory can be bought for the price of a visit to the hairdresser.
The common phrase to describe this changed world is ‘big data’ (Figure 1.1). A book on monetising data is inevitably about big data. We will interpret the term big data as data that is of a volume, variability and velocity that means common methods of appraisal are not appropriate. We need analytical methods to see the valuable patterns in it.
Since the early 2000s there has been a drive to make data more available, giving rise to the open data movement. This promotes sharing of data gathered with the benefit of public funding and includes most official statistics, academic research output and some market, product and service evaluation data. The opening up of data has led to a steep increase in requests for access to even more data; the result is a burgeoning interest in action learning and enthusiasm to understand the potential waiting to be uncovered from the data. The profession of data scientist has evolved and now encapsulates the skills and knowledge to handle and generate insights from this information.
Figure 1.2 shows how big data combined with analytics might empower different areas of any business. The aim of this book is to encourage people to use their big data to work out exciting business opportunities, make major changes and optimise the way things are run.
One of the key motivations for this book on monetising data is the sheer amount of under‐utilised data around. Hardly less important is the under‐achievement in terms of business benefit derived among those who do use their data. This suggests a two‐dimensional representation of the state of organisations, with one axis representing the usage of business data and the other axis representing the business benefit derived from it. Needless to say, the star performers are at the top right‐hand side of the resulting diagram in Figure 1.3. Being in the top and right‐hand corner is better than being at the top or at the right‐hand side of the axes because the two factors reinforce each other in a synergistic manner, giving greater benefits than either alone.
The marketplace is highly heterogeneous, with companies and institutions (all referred to as ‘organisations’ henceforth) differentiated in many ways, including:
The baseline against which organisations can benchmark themselves in Figure 1.3 is different for different types of organisation.
Familiar players using big data include retail, finance, automotive manufacturers, health providers and process industries. In addition, the following are some of the less familiar organisations likely to be in possession of big data:
All these organisations can take advantage of their data but they start from different points with different resources and capabilities; with good ideas they may have the opportunity to become winners in their own areas. Experience suggests that organisations have a secret wish list for generating money out of their data. Figure 1.4 shows the ranking we observed from our clients. However, this is just a snapshot and does not include business enrichment and transformation, which are also possible.
Figure 1.5 shows a very generalised process for monetising data. Data comes into the process and is first used for business monitoring, leading to business insights; these might generate business optimisation and might lead to monetisation and potential business transformation.
Despite differences in scale, the matrix in Figure 1.3 can help any organisation to map their current situation and plan their next steps to uplift their business.
Data is the fundamental commodity, consisting of a representation of facts. However, when the data are summarised and illustrated they can lead to meaningful information, and assessing the meaningful information in context can lead to knowledge and wisdom.
Monetising data is more than just selling data and information. It includes everything where data is used in exchange for business advantage and supports business success. Large companies are often data rich and some have realised the advantage this gives them. Others consider themselves data rich but information poor because they have lots of data but it is not in a form that they can easily interpret or use to gain business insights. Statistical enthusiasm is a rare commodity but those businesses that pay attention to their data can find the answers to many of their policy and productivity questions. For example, scrutiny of data on sales easily yields information about seasonal trends: sales per customer might show shortfalls in maximising selling opportunities; total income might show overall success in attracting buyers, and so on.
Case studies and real data from our consulting practices are used throughout the book to illustrate the ideas, methods and techniques that are involved. As will be seen, most data can be monetised to bring benefit to the organisation. However, a lot of effort has to be expended to get the data into a suitable format for analysis. Data readiness can be assessed using tools that we will discuss. As analytics progresses, guidelines for data improvement become meaningful and we introduce the concept of the data improvement cycle to help organisations in continuous improvement and moving forward with their data analytics.
This book is aimed at managers in progressive organisations: managers who are keen to develop their own careers and who have the opportunity to suggest new ideas and innovative approaches for their organisation and influence how they are taken forward. The material requires background knowledge of dealing with numbers and spreadsheets and basic business principles. More specialised techniques, such as the use of decision tree analysis and predictive models, are fully explained. The main issue is the strength of desire to join the data revolution and hopefully after reading this book you will be an excited convert.
The rest of the book is planned as follows. Chapters 2 and 3 address data collection and preparation issues, including the use of mapping and meteorological data as well as official statistics. Chapter 4 looks at general issues around data mining: as a concept and a mechanism for gathering insights from data. Chapters 5 and 6 address technical methods; Chapter 5 looks at descriptive analytics, starting with statistical methods for summarising data and graphical presentations, and Chapter 6 moves on to statistical testing, modelling, segmentation, network analysis and predictive analytics.
Chapters 7 and 8 introduce the different strategies, motivations, modes and concepts for monetising data and examine barriers and enablers for organisations seeking to realise the full potential of their data, their valuable asset. Monetisation can be viewed strategically and operationally. Strategically we can look at new business directions, step changes in thinking, disruptive innovation and new income streams. Operationally we can consider optimising current business models, and making better use of customer targeting and segmentation. In Chapter 7 we focus on strategic issues, whilst operational improvements of the existing business will be explored in Chapter 8. In Chapter 9 we will consider the practicalities of implementation, such as issues of ethics, privacy and security; loss of cultural and technical learning due to staff turnover and the other dampers that have to be overcome before we can achieve strategic steps forward and improvement of the current situation.
The mutual importance of theory and practice has long been recognised. As Chebyshev, a founding father of much statistical theory, said back in the 19th century, ‘Progress in any discipline is most successful when theory and practice develop hand in hand’. Not only does practice benefit from theory but theory benefits from practice. So in Chapter 10 we describe a set of case studies in which monetisation has brought big gains and uplifted the business. Thus we will aim to end the book on a high note and provide inspiration to move forward.
If you locate yourself within the grid in Figure 1.3 you can see which parts of the book are most relevant for you. Those readers at the bottom left are probably at the beginning of their exploration of monetisation and could well jump to the case studies in Chapter 10 for motivation and then return to Chapter 2. Those at the bottom right have already gained substantial business advantages but could benefit from learning new statistical and data‐mining techniques to make deeper use of their data, as described in the more technical Chapters 3–6. Those at the top left already have experience of analysing data but need to realise a better business advantage and could go straight to Chapters 7–9. Those at the top right can read the whole book for revision purposes and further insights!
Note that we avoid naming specific companies. Instead we refer to them in a generic way and the reader is welcome to find example companies by searching online.