Table of Contents

Cover

Preface

Introduction: 50 Years of Data Analysis: From Exploratory Data Analysis to Predictive Modeling and Machine Learning

I.1. The revolt against mathematical statistics
I.2. EDA and unsupervised methods for dimension reduction
I.3. Predictive modeling
I.4. Conclusion
I.5. References

PART 1: Clustering and Regression

1 Cluster Validation by Measurement of Clustering Characteristics Relevant to the User
1. 1.1. Introduction
2. 1.2. General notation
3. 1.3. Aspects of cluster validity
4. 1.4. Aggregation of indexes
5. 1.5. Random clusterings for calibrating indexes
6. 1.6. Examples
7. 1.7. Conclusion
8. 1.8. Acknowledgment
9. 1.9. References
2 Histogram-Based Clustering of Sensor Network Data
1. 2.1. Introduction
2. 2.2. Time series data stream clustering
3. 2.3. Results on real data
4. 2.4. Conclusions
5. 2.5. References
3 The Flexible Beta Regression Model
1. 3.1. Introduction
2. 3.2. The FB distribution
3. 3.3. The FB regression model
4. 3.4. Bayesian inference
5. 3.5. Illustrative application
6. 3.6. Conclusion
7. 3.7. References
4 S-weighted Instrumental Variables
1. 4.1. Summarizing the previous relevant results
2. 4.2. The notations, framework, conditions and main tool
3. 4.3. S-weighted estimator and its consistency
4. 4.4. S-weighted instrumental variables and their consistency
5. 4.5. Patterns of results of simulations
6. 4.6. Acknowledgment
7. 4.7. References

PART 2: Models and Modeling

5 Grouping Property and Decomposition of Explained Variance in Linear Regression
1. 5.1. Introduction
2. 5.2. CAR scores
3. 5.3. Variance decomposition methods and SVD
4. 5.4. Grouping property of variance decomposition methods
5. 5.5. Conclusions
6. 5.6. References
6 On GARCH Models with Temporary Structural Changes
1. 6.1. Introduction
2. 6.2. The model
3. 6.3. Identification
4. 6.4. Simulation
5. 6.5. Application
6. 6.6. Concluding remarks
7. 6.7. References
7 A Note on the Linear Approximation of TAR Models
1. 7.1. Introduction
2. 7.2. Linear representations and linear approximations of nonlinear models
3. 7.3. Linear approximation of the TAR model
4. 7.4. References
8 An Approximation of Social Well-Being Evaluation Using Structural Equation Modeling
1. 8.1. Introduction
2. 8.2. Wellness
3. 8.3. Social welfare
4. 8.4. Methodology
5. 8.5. Results
6. 8.6. Discussion
7. 8.7. Conclusions
8. 8.8. References
9 An SEM Approach to Modeling Housing Values
1. 9.1. Introduction
2. 9.2. Data
3. 9.3. Analysis
4. 9.4. Conclusions
5. 9.5. References
10 Evaluation of Stopping Criteria for Ranks in Solving Linear Systems
1. 10.1. Introduction
2. 10.2. Methods
3. 10.3. Formulation of linear systems
4. 10.4. Stopping criteria
5. 10.5. Numerical experimentation of stopping criteria
6. 10.6. Conclusions
7. 10.7. Acknowledgments
8. 10.8. References
11 Estimation of a Two-Variable Second-Degree Polynomial via Sampling
1. 11.1. Introduction
2. 11.2. Proposed method
3. 11.3. Experimental approaches
4. 11.4. Conclusions
5. 11.5. References

PART 3: Estimators, Forecasting and Data Mining

12 Displaying Empirical Distributions of Conditional Quantile Estimates: An Application of Symbolic Data Analysis to the Cost Allocation Problem in Agriculture
1. 12.1. Conceptual framework and methodological aspects of cost allocation
2. 12.2. The empirical model of specific production cost estimates
3. 12.3. The conditional quantile estimation
4. 12.4. Symbolic analyses of the empirical distributions of specific costs
5. 12.5. The visualization and the analysis of econometric results
6. 12.6. Conclusion
7. 12.7. Acknowledgments
8. 12.8. References
13 Frost Prediction in Apple Orchards Based upon Time Series Models
1. 13.1. Introduction
2. 13.2. Weather database
3. 13.3. ARIMA forecast model
4. 13.4. Model building
5. 13.5. Evaluation
6. 13.6. ARIMA model selection
7. 13.7. Conclusions
8. 13.8. Acknowledgments
9. 13.9. References
14 Efficiency Evaluation of Multiple-Choice Questions and Exams
1. 14.1. Introduction
2. 14.2. Exam efficiency evaluation
3. 14.3. Real-life experiments and results
4. 14.4. Conclusions
5. 14.5. References
15 Methods of Modeling and Estimation in Mortality
1. 15.1. Introduction
2. 15.2. The appearance of life tables
3. 15.3. On the law of mortality
4. 15.4. Mortality and health
5. 15.5. An advanced health state function form
6. 15.6. Epilogue
7. 15.7. References
16 An Application of Data Mining Methods to the Analysis of Bank Customer Profitability and Buying Behavior
1. 16.1. Introduction
2. 16.2. Data set
3. 16.3. Short-term forecasting of customer profitability
4. 16.4. Churn prediction
5. 16.5. Next-product-to-buy
6. 16.6. Conclusions and future research
7. 16.7. References

List of Authors

Index

End User License Agreement

List of Tables

Introduction

Table I.1. Various cases of the maximum association principle
Table I.2. An example of interval data

Chapter 1

Table 1.1. Normalized index values for four clusterings on artificial data
Table 1.2. Calibrated index values (using random clusterings with same K) for fo...
Table 1.3. Calibrated index values (using all random clusterings) for four clust...
Table 1.4. Calibrated index values (using random clusterings with same K) for ei...
Table 1.5. Calibrated index values (using all random clusterings) for eight clus...

Chapter 2

Table 2.1. Rand index for 0.001 ≤ u ≤ 0.01

Chapter 3

Table 3.1. Posterior means of the parameters under the three models
Table 3.2. Model comparison criteria for the FBR, the BRR and the beta regressio...

Chapter 4

Table 4.1. The contamination by leverage points on the level of 1%, n = 100. The...
Table 4.2. The contamination by leverage points on the level of 5%,n = 100. The ...
Table 4.3. Contamination by outliers: For randomly selected observations, we put...

Chapter 5

Table 5.1. Allocation of explained variance
Table 5.2. Ranking predictors by decreasing importance: random forest, lmg-Shapl...

Chapter 6

Table 6.1. Result of simulation (T = 300, L1 = 50)
Table 6.2. Result of simulation (T = 400, Tm = 200, L1 = 30, L2 = 60)
Table 6.3. Slected L1, L2 and K
Table 6.4. AIC of models
Table 6.5. The estimated parameters

Chapter 8

Table 8.1. Components obtained by CATPCA

Chapter 9

Table 9.1. Definition of variables
Table 9.2. Goodness of fit summary: model 1a
Table 9.3. Goodness of fit summary: model 1b
Table 9.4. Goodness of fit summary: model 2a
Table 9.5. Goodness of fit summary: model 2b
Table 9.6. Model 3: goodness of fit summary

Chapter 10

Table 10.1. Error tolerance and stopping criteria by iterative methods
Table 10.2. Quartiles, number of iterations and solutions using stopping criter...

Chapter 11

Table 11.1. Pollutant concentration
Table 11.2. Intensity of radiation

Chapter 12

Table 12.1. The specific cost estimates (€) for 1,000 € of wheat gross product, ...

Chapter 13

Table 13.1. Variables that were recorded by the weather stations
Table 13.2. Evaluation results of ARIMA and linear regression models

Chapter 14

Table 14.1. Question categories based on good and bad efficiencies
Table 14.2. Special question categories based on good and bad efficiencies

Chapter 16

Table 16.1. Performance of the linear model and benchmark model
Table 16.2. Sign and significance of some of the most significant variables of t...
Table 16.3. Performance of linear models on the test set for different forecasti...
Table 16.4. Performance of the best linear model (logit) and the best nonlinear ...
Table 16.5. Performance of logit models in predicting the next product bought by...

List of Illustrations

Chapter 1

Figure 1.1. Artificial data set. Left side: clustering by 3-means. Right side: c...

Chapter 2

Figure 2.1. Processing path. For a color version of this figure, see www.iste.co...
Figure 2.2. Graphical representation of the number of microclusters generated at...

Chapter 3

Figure 3.1. Some examples of FB distributions. Left panel: p = 0.5 and {λ1 = 0.5...
Figure 3.2. From the left: fitted regression curves for the mean of the response...

Chapter 4

Figure 4.1. The examples of possible shapes of weight function. For a color vers...

Chapter 5

Figure 5.1. Squared CAR scores. Diabetes data. Predictor bp’ added correlated to...

Chapter 6

Figure 6.1. An example of the trend
Figure 6.2. An example of Intervention on the trend
Figure 6.3. A scatter plot of Pt and Dt
Figure 6.4. The histogram of estimated value of γ(m)
Figure 6.5. TOPIX (Tokyo Price Index)
Figure 6.6. Returns of TOPIX
Figure 6.7. Estimated trend (all)
Figure 6.8. Estimated trend (t=1,2,…,250)
Figure 6.9. Estimated volatality (all)
Figure 6.10. Estimated volatality: (t=1,2,…,250)

Chapter 7

Figure 7.1. ACF and PACF of the artificial data generated from model [7.11]
Figure 7.2. ACF and PACF of the residuals obtained after fitting an AR(1) model ...
Figure 7.3. ACF and PACF of the artificial data generated from model [7.12]
Figure 7.4. ACF and PACF of the artificial data generated from the linear approx...
Figure 7.5. On the left, ACF of the residuals of the ARMA approximation of the d...
Figure 7.6. ACF (left panel) and PACF (right panel) of the residuals of the ARMA...
Figure 7.7. (a) ACF of the artificial data (Yt) generated from model [7.13]; (b)...

Chapter 8

Figure 8.1. The structural equation model based on the constructed components

Chapter 9

Figure 9.1. Measurement model 1a
Figure 9.2. Estimated model 1b
Figure 9.3. Estimated model 2a
Figure 9.4. Estimated model 2b
Figure 9.5. Estimated model 3

Chapter 10

Figure 10.1. Convergence of five criteria using power series/Jacobi iterative me...
Figure 10.2. Index of first 100 top ranks and error (criterion I) against iterat...
Figure 10.3. Convergence of top 100 and top 300 ranks using Kendall’s τ correlat...

Chapter 11

Figure 11.1. Restrictions on the use of the proposed method
Figure 11.2. Point-source pollutant. For a color version of this figure, see www...
Figure 11.3. Point-source of radiation. For a color version of this figure, see ...

Chapter 12

Figure 12.1. The first factorial plan (F1×F2) of the V-PCA, with the conditional...
Figure 12.2. The first factorial plan (F1×F2) of the V-PCA, with the empirical d...
Figure 12.3. S-PCA of empirical distributions of specific costs, with the projec...
Figure 12.4. S-PCA of empirical distributions of specific costs, with the projec...
Figure 12.5. The interval divisive hierarchical clustering of empirical distribu...

Chapter 13

Figure 13.1. Non-stationary time series
Figure 13.2. ACF before differencing
Figure 13.3. PACF before differencing
Figure 13.4. Differenced time series
Figure 13.5. ACF after differencing
Figure 13.6. PACF after differencing
Figure 13.7. ARIMA(4,1,4) forecast from 22:00 until sunrise. The observed temper...
Figure 13.8. ARIMA(4,1,4) forecast shown with the temperature of the previous th...
Figure 13.9. Count of p
Figure 13.10. Count of q
Figure 13.11. Conditional plot of p versus q at given d

Chapter 14

Figure 14.1. Results for Exam1: success rates and efficiencies for 10 multiple-c...
Figure 14.2. Results for Exam2: success rates and efficiencies for 10 multiple-c...
Figure 14.3. Comparison of the non-weighted grades (plotted in ascending order) ...
Figure 14.4. Histograms of the non-weighted and adjusted weighted grades for Exa...

Chapter 15

Figure 15.1.The probabilities of death [log10(qx)] and the Heligman–Pollard form...
Figure 15.2. The health state function and some health indicators. For a color v...
Figure 15.3. The mortality diagram used in the µx-based method. For a color vers...
Figure 15.4.Final health state function and the supporting curve of the simple m...
Figure 15.5. (A) Population health state, (B) deterioration function, (C) stocha...

Chapter 16

Figure 16.1. Relation between customer wealth and profitability for the bank in ...
Figure 16.2. Relation between customer wealth and profitability for the bank in ...
Figure 16.3. Relation between the profitability of the clients in January and F...
Figure 16.4. Relation between the profitability of the clients in June and July
Figure 16.5. Histogram with monthly values of customer profitability

First published 2019 in Great Britain and the United States by ISTE Ltd and John Wiley & Sons, Inc.

Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced, stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers, or in the case of reprographic reproduction in accordance with the terms and licenses issued by the CLA. Enquiries concerning reproduction outside these terms should be sent to the publishers at the undermentioned address:

ISTE Ltd

27-37 St George’s Road

London SW19 4EU

www.iste.co.uk

John Wiley & Sons, Inc.

111 River Street

Hoboken, NJ 07030

USA

www.wiley.com

The rights of Christos H. Skiadas and James R. Bozeman to be identified as the authors of this work have been asserted by them in accordance with the Copyright, Designs and Patents Act 1988.

Library of Congress Control Number: 2018965155

British Library Cataloguing-in-Publication Data

A CIP record for this book is available from the British Library

ISBN 978-1-78630-382-0

Preface

Thanks to significant work by the authors and contributors, we have developed these two volumes on new and classical approaches to Data Analysis and Applications.

The data analysis field has been continuously growing over recent decades following the wide applications of computing and data collection along with new developments in analytical tools. Therefore, the need for publications is evident. New publications appear as printed or e-books covering the need for information from all fields of science and engineering, thanks to the wide applicability of data analysis and statistics packages.

These two volumes thus present collected material in over 30 chapters, divided into seven parts, in a form that will provide the reader with theoretical and applied information on data analysis methods, models and techniques along with appropriate applications.

In addition to the chapters, we include in both volumes an excellent introductory and review paper entitled “50 Years of Data Analysis: From Exploratory Data Analysis to Predictive Modeling and Machine Learning” by Gilbert Saporta, a leading expert in the field. The paper was based on his speech given at the celebration of his 70th birthday at the ASMDA2017 International Conference in London (held in De Morgan House of the London Mathematical Society).

Volume 1 contains the following three parts:

Part 1, Clustering and Regression, includes four chapters: “Cluster Validation by Measurement of Clustering Characteristics Relevant to the User” by Christian Hennig; “Histogram-Based Clustering of Sensor Network Data” by Antonio Balzanella and Rosanna Verde; “The Flexible Beta Regression Model” by Sonia Migliorati, Agnese M. Di Brisco and Andrea Ongaro; and “S-Weighted Instrumental Variables” by Jan Amos Visek.

Part 2, Models and Modeling, includes seven chapters: “Grouping Property and Decomposition of Explained Variance in Linear Regression” by Henri Wallard; “On GARCH Models with Temporary Structural Changes” by Norio Watanabe and Okihara Fumiaki; “A Note on the Linear Approximation of TAR Models” by Francesco Giordano, Marcella Niglio and Cosimo Damiano Vitale; “An Approximation to Social Well-Being Evaluation Using Structural Equation Modeling” by Leonel Santos-Barrios, Monica Ruiz-Torres, William Gómez-Demetrio, Ernesto Sánchez-Vera, Ana Lorga da Silva and Francisco Martínez-Castañeda; “An SEM Approach to Modeling Housing Values” by Jim Freeman and Xin Zhao; “Evaluation of Stopping Criteria for Ranks in Solving Linear Systems” by Benard Abola, Pitos Biganda, Christopher Engström and Sergei Silvestrov; and “Estimation of a Two-Variable Second-Degree Polynomial via Sampling” by Papatsouma Ioanna, Farmakis Nikolaos and Ketzaki Eleni.

Part 3, Estimators, Forecasting and Data Mining, includes five chapters: “Displaying Empirical Distributions of Conditional Quantile Estimates: An Application of Symbolic Data Analysis to the Cost Allocation Problem in Agriculture” by Dominique Desbois; “Frost Prediction in Apple Orchards Based upon Time Series Models” by Monika A. Tomkowicz and Armin O. Schmitt; “Efficiency Evaluation of Multiple-Choice Questions and Exams” by Evgeny Gershikov and Samuel Kosolapov; “Methods of Modeling and Estimation in Mortality” by Christos H. Skiadas and Konstantinos Zafeiris; and “An Application of Data Mining Methods to the Analysis of Bank Customer Profitability and Buying Behavior” by Pedro Godinho, Joana Dias and Pedro Torres.

Volume 2 continues with a further four parts as follows:

Part 1, Applications, includes six chapters: “Context-specific Independence in Innovation Studies” by Federica Nicolussi and Manuela Cazzaro; “Analysis of the Determinants and Outputs of Innovation in the Nordic Countries” by Catia Rosario, Antonio Augusto Costa and Ana Lorga da Silva; “Bibliometric Variables Determining the Quality of a Dentistry Journal” by Pilar Valderrama, Manuel Escabias, Evaristo Jiménez-Contreras, Mariano J. Valderrama and Pilar Baca; “Analysis of Dependence among Growth Rates of GDP of V4 Countries Using 4-Dimensional Vine Copulas” by Jozef Komornik, Magda Komornikova and Tomas Bacigal; “Monitoring the Compliance of Countries on Emissions Mitigation Using Dissimilarity Indices” by Eleni Ketzaki, Stavros Rallakis, Nikolaos Farmakis and Eftichios Sartzetakis; and “Maximum Entropy and Distributions of Five-Star Ratings” by Yiannis Dimotikalis.

Part 2, The Impact of the Economic and Financial Crisis in Europe, contains one chapter about credit: “Access to Credit for SMEs after the 2008 Financial Crisis: The Northern Italian Perspective” by Cinzia Colapinto and Mariangela Zenga. This is followed by two chapters on the labor market: “Gender-Based Differences in the Impact of the Economic Crisis on Labor Market Flows in Southern Europe”, and “Measuring Labor Market Transition Probabilities in Europe with Evidence from the EU-SILC, both by Maria Symeonaki, Maria Karamessini and Glykeria Stamatopoulou.

Part 3, Student Assessment and Employment in Europe, has an article concerning university students who are about to graduate and hence are close to employment that is related to Part 4: “Almost Graduated, Close to Employment? Taking into Account the Characteristics of Companies Recruiting at a University Job Placement Office” by Franca Crippa, Mariangela Zenga and Paolo Mariani, followed by a paper on how students are assessed: “How Variation of Scores of the Programme for International Student Assessment Can be Explained through Analysis of Information” by Valérie Girardin, Justine Lequesne and Olivier Thévenon.

Part 4, Visualization, examines this topic in computing: “A Topological Discriminant Analysis” by Rafik Abdesselam, followed by “Using Graph Partitioning to Calculate PageRank in a Changing Network” by Christopher Engström and Sergei Silvestrov, and in politics: “Visualizing the Political Spectrum of Germany by Contiguously Ordering the Party Policy Profiles by Andranik Tangian.

We deeply thank the authors and contributors to this book. We pass on our sincere appreciation to the referees for their hard work and dedication in providing an improved book form. Finally, we express our thanks to the secretariat and, of course, the publishers.

December 2018

Christos H. SKIADAS, Athens, Greece

James R. BOZEMAN, Bormla, Malta

Data Analysis and Applications 1

Clustering and Regression, Modeling-estimating, Forecasting and Data Mining

Preface