All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by law. Advice on how to obtain permission to reuse material from this title is available at http://www.wiley.com/go/permissions.

The right of David Nunes, Jorge Sá Silva and Fernando Boavida to be identified as the authors of this work has been asserted in accordance with law.

John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, UK

For details of our global editorial offices, customer services, and more information about Wiley products visit us at www.wiley.com.

Wiley also publishes its books in a variety of electronic formats and by print-on-demand. Some content that appears in standard print versions of this book may not be available in other formats.

While the publisher and authors have used their best efforts in preparing this work, they make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives, written sales materials or promotional statements for this work. The fact that an organization, website, or product is referred to in this work as a citation and/or potential source of further information does not mean that the publisher and authors endorse the information or services the organization, website, or product may provide or recommendations it may make. This work is sold with the understanding that the publisher is not engaged in rendering professional services. The advice and strategies contained herein may not be suitable for your situation. You should consult with a specialist where appropriate. Further, readers should be aware that websites listed in this work may have changed or disappeared between when this work was written and when it is read. Neither the publisher nor authors shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.

Names: Nunes, David, 1987- author. | Silva, Jorge Sá, author. | Boavida, Fernando, 1959- author.

Title: A practical introduction to human-in-the-loop cyber-physical systems / David Nunes, Jorge Sá Silva, Fernando Boavida.

Description: First edition. | Hoboken, NJ : John Wiley & Sons, 2018. | Includes bibliographical references and index. |

Identifiers: LCCN 2017025006 (print) | LCCN 2017042126 (ebook) | ISBN 9781119377801 (pdf) | ISBN 9781119377788 (epub) | ISBN 9781119377771 (cloth)

Subjects: LCSH: Cooperating objects (Computer systems) | Human-computer interaction.

Classification: LCC TJ213 (ebook) | LCC TJ213 .N86 2017 (print) | DDC 621.39-dc23

List of Figures

Figure 2.1 In [1], books and other common objects were augmented with RFID tags and associated with virtual documents by PDAs.
Figure 2.2 Shaman [2] acted as a representative for the connected LiteServers, offering Java and HTML interfaces.
Figure 2.3 Device web presence in Cooltown [3]. Source: Adapted from Kindberg et al. 2002.
Figure 2.4 JXTA [4] peers created virtual ad hoc networks which served to abstract the real ones.
Figure 2.5 Works such as [5] and [6] used proxies to offer embedded devices' capabilities through RESTful web services.
Figure 2.6 The SenseWeb [7] architecture.
Figure 2.7 WikiCity [8] interfaced between virtual data and the physical world through a semantically defined format for data exchange.
Figure 2.8 Nokia 6101 vs iPhone 6s/LG Nexus 5X.
Figure 2.9 HiTL technologies evolution timeline.
Figure 3.1 Basic processes of human-in-the-loop control.
Figure 3.2 Taxonomy of human control.
Figure 3.3 Taxonomy of human roles.
Figure 4.1 SenQ's query system stack shown side-by-side with the topology and components of AlarmNet, a prototypical implementation for assisted-living [9]. Source: Adapted from Wood 2008.
Figure 4.2 The architecture of CenceME [10], one of MetroSense's implementations.
Figure 4.3 The three key components of BCI using smartphones [11]. Source: Adapted from Lathia et al. 2013.
Figure 4.4 SociableSense architecture [12]. Source: Adapted from Rachuri 2011.
Figure 4.5 Control architecture for energy saving with HiTL [13]. Source: Adapted from Liang 2013.
Figure 4.6 Architecture of an HiTL HVAC system [14]. Source: Adapted from Agarwal 2011.
Figure 4.7 Diagram showing the main components of CAALYX's roaming monitoring system [15]. Source: Adapted from Boulos et al. 2007.
Figure 4.8 A semi-autonomous wheelchair receives brain signals from the user and executes the associated tasks of path planning, obstacle avoidance, and localization [16]. Source: Adapted from Schirner 2013.
Figure 4.9 A mockup of a map interface similar to the Highlight application.
Figure 4.10 Overview of the system proposed in [17]. Source: Adapted from W.-H. Rho and S.-B. Cho 2014.
Figure 5.1 HappyWalk HiTL control.
Figure 5.2 HappyWalk's architecture.
Figure 5.3 Android's activity lifecycle.
Figure 5.4 HappyWalk's Android class structure.
Figure 5.5 An overview of HappyWalk Android app's main classes.
Figure 5.6 An overview of HappyWalkServer's main classes.
Figure 5.7 A typical artificial neural network architecture.
Figure 5.8 Sound signal in the time domain (left side) analyzed through a Fourier transformation to show its frequency domain (right side).
Figure 5.9 HappyWalk's Emotional Feedback.
Figure 5.10 HappyWalk's neural network design.
Figure 6.1 Installing Java SE Development Kit 7u79.
Figure 6.2 Installing Android Studio and Android SDK.
Figure 6.3 Canceling the setup wizard.
Figure 6.4 Opening the Android SDK manager.
Figure 6.5 Installing Android API 21.
Figure 6.6 Opening the standalone SDK manager.
Figure 6.7 Installing Android SDK Build-tools 21.1.2.
Figure 6.8 Installing Git #1. (a) Adding Git to the PATH, on Windows (b) Choose Checkout Windows-style.
Figure 6.9 Installing Git #2. (a) We recommend using MinTTY (b) Uncheck Enable file system caching.
Figure 6.10 Importing HappyWalk from Git.
Figure 6.11 Cloning the HappyWalk project.
Figure 6.12 Opening the HappyWalk project.
Figure 6.13 Choosing HappyWalk's project folder.
Figure 6.14 Do not upgrade Android Gradle or its plugin.
Figure 6.15 Running HappyWalk.
Figure 6.16 HappyWalk's first launch.
Figure 6.17 Obtaining the Android debug key.
Figure 6.18 Creating a project to obtain a Google Maps Android API key.
Figure 6.19 Creating the Google Maps Android API key.
Figure 6.20 Obtaining the Google Maps Android API key.
Figure 6.21 Changing into the project's view.
Figure 6.22 Opening app/debug/res/values/google_maps_api.xml.
Figure 6.23 Choosing PostgreSQL superuser's password.
Figure 6.24 No need to launch Stack Builder.
Figure 6.25 Clone from a URI.
Figure 6.26 Introduce the URI corresponding to HappyWalk's server.
Figure 6.27 Select the master branch.
Figure 6.28 Selecting the local storage directory.
Figure 6.29 Select the option Import existing Eclipse projects.
Figure 6.30 Tick the checkbox of the HappyWalkServer project.
Figure 6.31 Creating a Foursquare® app.
Figure 6.32 Foursquare®'s Client ID and Client Secret.
Figure 6.33 Navigating into the server's GlobalVariables.
Figure 6.34 Log in to the PostgreSQL 9.3 server.
Figure 6.35 Create a new database.
Figure 6.36 Name the new database as happywalk.
Figure 6.37 Select the correct SQL script.
Figure 6.38 Populating the database.
Figure 6.39 Create a new server.
Figure 6.40 Define a new Tomcat 7 installation.
Figure 6.41 Installing Tomcat 7 from Eclipse.
Figure 6.42 Adding HappyWalk to Tomcat 7.
Figure 6.43 Running the HappyWalk server.
Figure 6.44 Select the newly created Tomcat 7.
Figure 6.45 The HappyWalk server is up and running.
Figure 6.46 The ipconfig command.
Figure 6.47 HappyWalk's map screen.
Figure 7.1 Creating a new class.
Figure 7.2 AS cannot resolve symbol issue.
Figure 7.3 Importing the appropriate class.
Figure 7.4 Creating a new package.
Figure 7.5 Creating the sensor processors.
Figure 7.6 Signal processing overview.
Figure 7.7 Current state of our HiTLCPS at the end of Chapter 7.
Figure 8.1 An example of a sigmoid activation function.
Figure 8.2 Creating a new basic activity.
Figure 8.3 Name the activity as EmotionFeedback.
Figure 8.4 The files that compose the EmotionFeedback activity.
Figure 8.5 Our goal for the EmotionSpace view.
Figure 8.6 Creating the EmotionSpace class.
Figure 8.7 Create EmotionSpace constructor matching super.
Figure 8.8 Choose View(context:Context, attrs:AttributeSet).
Figure 8.9 Changing from the layout Design view to Text view.
Figure 8.10 Creating a new Values resource file.
Figure 8.11 Naming the Values resource file.
Figure 8.12 The coordinates of the EmotionSpace view.
Figure 8.13 The emotion feedback notification.
Figure 8.14 Creating TaskSendEmotion.
Figure 8.15 Current state of our HiTLCPS at the end of Chapter 8.
Figure 9.1 HappyWalk's database conceptual schema.
Figure 9.2 Creating a new class in Eclipse.
Figure 9.3 Naming RequestSetEmotion.
Figure 9.4 Generating the Constructors, toString(), and the Getters and Setters.
Figure 9.5 Generating a Constructor using fields.
Figure 9.6 Generating a Constructor from Superclass.
Figure 9.7 Generating the Getters and Setters.
Figure 9.8 Overriding the default toString() method.
Figure 9.9 The location of the HappyWalkServer's web.xml.
Figure 9.10 The emotion alert dialog.
Figure 9.11 The emotion heatmaps
Figure 9.12 Final state of our HiTLCPS at the end of Chapter 9.
Figure 10.1 The HiTL resilience paradigm.
Figure 11.1 Lessons learned towards human-in-the-loop control.

Foreword

Our world keeps being an increasingly technological one. As first put forward by the renowned computer scientist Mark Weiser, we continue to see that, as devices get smaller in size, more mobile, powerful, and efficient, they begin to “disappear”. Technology is now so intrinsic to our everyday lives that it has become an inherent part of our existence. This is the premise behind concepts such as the Internet of things and cyber-physical systems, in which distributed technology is used to monitor and control the environment. However, our current technological advancement still falls short of Weiser's ideas. Each time we have to hurdle through unintuitive configuration menus, errors, and software incompatibilities we become stressed by our computers and appliances. Weiser argued that the ultimate form of computers was an extension of our subconscious. To him, the ideal computer would be capable of truly understanding people's unconscious actions and desires. Instead of humans adapting to technology and learning how to use it, it would be technology that would adapt to the disposition and uniqueness of each human being.

In fact, systems that consider the human context are becoming increasingly more important, and there are strong indications that most future technologies will most likely be much more human-aware. This book focuses on the realm of human-in-the-loop cyber-physical systems (HiTLCPSs), that is cyber-physical systems that take human response into consideration. HiTLCPSs infer the user's, intents, psychological states, emotions, and actions through sensors, using this information to determine the system's actions. This involves using a large variety of sensors and mobile devices to monitor and evaluate human nature. Therefore, this technology has strong ties with wireless sensor networks, robotics, machine learning, and the Internet of things.

This book is useful to BSc and MSc students, as well as to PhD students, researchers, and professors addressing the areas of ubiquitous computing, Internet of things, cyber-physical systems, and human–computer interaction. It can also be useful to professional developers that intend to introduce HiTL concepts into their mobile apps and/or Internet of things/cyber-physical system applications.

Throughout its pages, the book will guide the reader through a journey into this novel and exciting area of research and technological development. As such, it is intended to be used as a primer on HiTLCPSs, providing some insights into the research being done on this topic, current challenges, and requirements. One of the book's objectives is to introduce the reader to the practical usage of HiTL paradigms within software development. Therefore, we included a comprehensive hands-on tutorial where the major theoretical concepts behind HiTLCPSs are applied to a sample mobile application and explained from a practical perspective. This tutorial requires some knowledge of Android and the Java programming language, as well as some notions about databases and RESTful web services. It is accompanied by a base source code repository and several code snippets which the reader can extensively modify.¹ It is not our intention to provide in-depth knowledge about the programming languages, and/or the machine learning techniques, necessary to create complex HiTL systems. Instead, the tutorial aims at illustrating and consolidating some of the book's theoretical ideas.

Finally, we would like to thank you, the reader, for your interest. We would also like to ask you to contact us and tell us about your experience with our book. Your feedback is a very valuable resource towards improving the book. Send your email to dsnunes@dei.uc.pt, sasilva@dei.uc.pt or boavida@uc.pt.

Preface

The Internet has changed our whole life and it will have further impact on how we live and how we work. Most of the cyber-physical systems (CPSs) make use of the Internet and even define parts of it. Let me cite Wikipedia in this preface, even though it is not very scientific so to do. Understanding the CPS as “a mechanism controlled or monitored by computer-based algorithms, tightly integrated with the internet and its users” means that users, humans, are essential for any CPS. The National Institute of Standards and Technology of the US Department of Commerce (NIST) goes even further, stating that “these systems will provide the foundation of our critical infrastructure, form the basis of emerging and future smart services, and improve our quality of life in many areas”. Looking at the examples mentioned in Wikipedia, “smart grid, autonomous automobile systems, medical monitoring, process control systems, robotics systems, and automatic pilot avionics”, human are always involved.

Humans are not only involved; humans are the essential part of CPSs; CPSs have to serve us! With the basic idea, to incorporate humans as being in the system, we encounter human-in-the-loop (HiTL). It comprises a model, an adequate representation of the human behavior in order to treat it as an integral part of the whole system. Just as one example, let me cite Carsten Binning et.al. at his preface of the Proceedings of the first Workshop on Human-In-the-Loop Data Analytics HILDA of June 26th, 2016, in San Francisco, California: “A major bottleneck in data analytics today is to efficiently leverage the human capabilities to formulate questions and understand answers of data analytics systems … Recent technology trends (such as touchscreens, motion detection, and voice recognition) are widening the possibilities for users to interact with data, and data-driven industries are shifting to personalized processing to better target their services to users' needs”.

Hence it seems somewhat natural to look at both topics together in a kind of textbook and survey. In my six years as editor-in-chief of the journal ACM Transactions on Multimedia Computing, Communications, and Applications (ACM TOMM), I have, unfortunately, not come across a comprehensive high-quality survey paper of CPS HiTL; it has been even more serious: nobody even tried to cover with a survey this essential area on multimedia computing, communications, and its applications. No one did so far!

At the present time, writing this preface, I was only able to read parts of this book; I am looking forward to reading it all together–the whole book.

The authors of this book, David Nunes, Jorge Sá Silva, and Fernando Boavida from the University of Coimbra provide an in-depth view to HiTLCPS evolution, theory, technologies, and applications. Moreover, they illustrate how to apply HiTLCPS concepts to a sample smartphone application, through a hands-on approach that guides the reader from the development environment to the final product, including data acquisition, state inference, and actuation. With (1) their profound technical knowledge of many areas in computing and communications, as well as with (2) their expertise and experience as authors of other textbooks, the authors are certainly key for this book being a long-term successful scientific book in this area. Congratulations!

Dr. Ralf Steinmetz

Fellow of the IEEE and Fellow of the ACM

Director, Multimedia Communications Laboratory, Technische Universität Darmstadt

Chairman of the Board, Hessian Telemedia Technology CompetenceCenter, Germany

Darmstadt, March 2017

Acknowledgments

A book such as this would not have been possible without the help and support of many people and institutions.

First of all, we would like to thank our base institutions—the Department of Informatics Engineering, and the Center for Informatics and Systems, both from the University of Coimbra—in the scope of which we carry out our teaching and research activities, for the provided facilities and research environment. With their effort and contributions, enthusiasm, discussions, and suggestions during several years of joint research activities and human-in-the-loop social interaction, our students and our colleagues were instrumental in making this book a reality.

We also thank IMDEA Networks Institute, in Madrid, for the support provided during Fernando Boavida's sabbatical in 2015/2016, and especially to its leading computer scientist, Arturo Azcorra, for his support; to Antonio Fernández Anta, Miguel Péon, Jeanet Birkkjaer; and Rosa Gómez for their encouragement; and to all its researchers and staff in general.

Some of the research that formed the basis for this book was carried out in the scope of financed research projects and initiatives and, thus, it is also right to thank the entities that made the referred research possible, namely the Portuguese Foundation for Science and Technology (FCT), FCT's POPH/FSE program, and the SOCIALITE Project (PTDC/EEI-SCR/2072/2014), supported by COMPETE 2020, Portugal 2020, Operational Program for Competitiveness and Internationalization (POCI), and the European Union's ERDF (European Regional Development Fund).

We would also like to thank David Hutchison, from Lancaster University, for believing in us and putting us in contact with the excellent editorial team at John Wiley & Sons.

Finally, we would like to thank our families, for their unconditional love and support.

List of Abbreviations

AI	Artificial Intelligence
ANN	Artificial Neural Network
API	Application Programming Interface
AS	Android Studio
AV	Autonomous Vehicle
BCC	Body-Coupled Communication
BCI	Behavior Change Interventions
CHIL	Computers in the Human Interaction Loop
CoAP	Constrained Application Protocol
cOre	Constrained RESTful environments
CPS(s)	Cyber-Physical System(s)
CPU	Central Processing Unit
DAO	Data Access Object
ECG	Electrocardiography
EEG	Electroencephalography
ESM	Experience Sampling Method
FCT	Fast Cosine Transform
FFT	Fast Fourier Transformation
GPRS	General Packet Radio Service
GPS	Global Positioning System
GSM	Global System for Mobile Communications
HiTL	Human-in-the-Loop
HiTLCPS(s)	Human-in-the-Loop Cyber-Physical System(s)
HTML	HyperText Markup Language
HTTP	Hypertext Transfer Protocol
HVAC	Heating, Ventilation, and Cooling
ID	Identification
IFR	International Federation of Robotics
IoA	Internet of All
IoT	Internet of Things
IP	Internet Protocol
IDE	Integrated Development Environment
IEEE	Institute of Electrical and Electronics Engineers
IETF	Internet Engineering Task Force
ISM band	Industrial, Scientific, and Medical radio bands
Java EE	Java Enterprise Edition
Java SE	Java Standard Edition
JDK	Java Development Kit
JSON	JavaScript Object Notation
LTE	Long-Term Evolution
M2M	Machine-to-Machine
MPTCP	MultiPath Transmission Control Protocol
NAT	Network Address Translation
NSF	National Science Foundation
OSI	Open Systems Interconnection
OS	Operating System
P2P	Peer-to-Peer
POI(s)	Point(s) of Interest
RAM	Random-Access Memory
REST	Representational state transfer
RF	Radio Frequency
RFID	Radio-Frequency Identification
RSSI	Received Signal Strength Indication
SCTP	Stream Control Transmission Protocol
SDK	Software Development Kit
sMAP	Simple Monitoring and Action Profile
SMS	Short Message Service
SOAP	Simple Object Access Protocol
SQL	Structured Query Language
TCP	Transmission Control Protocol
UDP	User Datagram Protocol
URI	Uniform Resource Identifier
URL	Uniform Resource Locator
UUID	Universally Unique Identifier
VoIP	Voice Over Internet Protocol
WSDL	Web Service Description Language
WSN(s)	Wireless Sensor Network(s)
XML	Extensible Markup Language

A Practical Introduction to Human-in-the-Loop Cyber-Physical Systems