All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by law. Advice on how to obtain permission to reuse material from this title is available at http://www.wiley.com/go/permissions.
The right of David Nunes, Jorge Sá Silva and Fernando Boavida to be identified as the authors of this work has been asserted in accordance with law.
Registered Office(s)
John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, USA
John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, UK
Editorial Office
The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, UK
For details of our global editorial offices, customer services, and more information about Wiley products visit us at www.wiley.com.
Wiley also publishes its books in a variety of electronic formats and by print-on-demand. Some content that appears in standard print versions of this book may not be available in other formats.
Limit of Liability/Disclaimer of Warranty
While the publisher and authors have used their best efforts in preparing this work, they make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives, written sales materials or promotional statements for this work. The fact that an organization, website, or product is referred to in this work as a citation and/or potential source of further information does not mean that the publisher and authors endorse the information or services the organization, website, or product may provide or recommendations it may make. This work is sold with the understanding that the publisher is not engaged in rendering professional services. The advice and strategies contained herein may not be suitable for your situation. You should consult with a specialist where appropriate. Further, readers should be aware that websites listed in this work may have changed or disappeared between when this work was written and when it is read. Neither the publisher nor authors shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.
Library of Congress Cataloging-in-Publication Data
To my parents, Jorge and Eulália, and to my brother, Telmo.
David Nunes
To Fátima, Catarina, Pedro, Jojó, and my parents
Jorge Sá Silva
To Maria Joào and our three daughters-Susana, Inês, and Catarina
Fernando Boavida
List of Figures
Figure 2.1 In [1], books and other common objects were augmented with RFID tags and associated with virtual documents by PDAs.
Figure 2.2 Shaman [2] acted as a representative for the connected LiteServers, offering Java and HTML interfaces.
Figure 2.3 Device web presence in Cooltown [3]. Source: Adapted from Kindberg et al. 2002.
Figure 2.4 JXTA [4] peers created virtual ad hoc networks which served to abstract the real ones.
Figure 2.5 Works such as [5] and [6] used proxies to offer embedded devices' capabilities through RESTful web services.
Figure 2.6 The SenseWeb [7] architecture.
Figure 2.7 WikiCity [8] interfaced between virtual data and the physical world through a semantically defined format for data exchange.
Figure 2.8 Nokia 6101 vs iPhone 6s/LG Nexus 5X.
Figure 2.9 HiTL technologies evolution timeline.
Figure 3.1 Basic processes of human-in-the-loop control.
Figure 3.2 Taxonomy of human control.
Figure 3.3 Taxonomy of human roles.
Figure 4.1 SenQ's query system stack shown side-by-side with the topology and components of AlarmNet, a prototypical implementation for assisted-living [9]. Source: Adapted from Wood 2008.
Figure 4.2 The architecture of CenceME [10], one of MetroSense's implementations.
Figure 4.3 The three key components of BCI using smartphones [11]. Source: Adapted from Lathia et al. 2013.
Figure 4.4 SociableSense architecture [12]. Source: Adapted from Rachuri 2011.
Figure 4.5 Control architecture for energy saving with HiTL [13]. Source: Adapted from Liang 2013.
Figure 4.6 Architecture of an HiTL HVAC system [14]. Source: Adapted from Agarwal 2011.
Figure 4.7 Diagram showing the main components of CAALYX's roaming monitoring system [15]. Source: Adapted from Boulos et al. 2007.
Figure 4.8 A semi-autonomous wheelchair receives brain signals from the user and executes the associated tasks of path planning, obstacle avoidance, and localization [16]. Source: Adapted from Schirner 2013.
Figure 4.9 A mockup of a map interface similar to the Highlight application.
Figure 4.10 Overview of the system proposed in [17]. Source: Adapted from W.-H. Rho and S.-B. Cho 2014.
Figure 5.1 HappyWalk HiTL control.
Figure 5.2 HappyWalk's architecture.
Figure 5.3 Android's activity lifecycle.
Figure 5.4 HappyWalk's Android class structure.
Figure 5.5 An overview of HappyWalk Android app's main classes.
Figure 5.6 An overview of HappyWalkServer's main classes.
Figure 5.7 A typical artificial neural network architecture.
Figure 5.8 Sound signal in the time domain (left side) analyzed through a Fourier transformation to show its frequency domain (right side).
Figure 5.9 HappyWalk's Emotional Feedback.
Figure 5.10 HappyWalk's neural network design.
Figure 6.1 Installing Java SE Development Kit 7u79.
Figure 6.2 Installing Android Studio and Android SDK.
Table 6.1 Summary of the steps necessary to install AS 2.1.3
Table 6.2 Summary of the steps necessary to set up HappyWalk's Android project
Table 6.3 Summary of the steps necessary to deploy HappyWalk's server
Table 6.4 Summary of the steps necessary to test the base HappyWalk system
Table 10.1 Summary of the identified HiTL requirements and challenges
Foreword
Our world keeps being an increasingly technological one. As first put forward by the renowned computer scientist Mark Weiser, we continue to see that, as devices get smaller in size, more mobile, powerful, and efficient, they begin to “disappear”. Technology is now so intrinsic to our everyday lives that it has become an inherent part of our existence. This is the premise behind concepts such as the Internet of things and cyber-physical systems, in which distributed technology is used to monitor and control the environment. However, our current technological advancement still falls short of Weiser's ideas. Each time we have to hurdle through unintuitive configuration menus, errors, and software incompatibilities we become stressed by our computers and appliances. Weiser argued that the ultimate form of computers was an extension of our subconscious. To him, the ideal computer would be capable of truly understanding people's unconscious actions and desires. Instead of humans adapting to technology and learning how to use it, it would be technology that would adapt to the disposition and uniqueness of each human being.
In fact, systems that consider the human context are becoming increasingly more important, and there are strong indications that most future technologies will most likely be much more human-aware. This book focuses on the realm of human-in-the-loop cyber-physical systems (HiTLCPSs), that is cyber-physical systems that take human response into consideration. HiTLCPSs infer the user's, intents, psychological states, emotions, and actions through sensors, using this information to determine the system's actions. This involves using a large variety of sensors and mobile devices to monitor and evaluate human nature. Therefore, this technology has strong ties with wireless sensor networks, robotics, machine learning, and the Internet of things.
This book is useful to BSc and MSc students, as well as to PhD students, researchers, and professors addressing the areas of ubiquitous computing, Internet of things, cyber-physical systems, and human–computer interaction. It can also be useful to professional developers that intend to introduce HiTL concepts into their mobile apps and/or Internet of things/cyber-physical system applications.
Throughout its pages, the book will guide the reader through a journey into this novel and exciting area of research and technological development. As such, it is intended to be used as a primer on HiTLCPSs, providing some insights into the research being done on this topic, current challenges, and requirements. One of the book's objectives is to introduce the reader to the practical usage of HiTL paradigms within software development. Therefore, we included a comprehensive hands-on tutorial where the major theoretical concepts behind HiTLCPSs are applied to a sample mobile application and explained from a practical perspective. This tutorial requires some knowledge of Android and the Java programming language, as well as some notions about databases and RESTful web services. It is accompanied by a base source code repository and several code snippets which the reader can extensively modify.1 It is not our intention to provide in-depth knowledge about the programming languages, and/or the machine learning techniques, necessary to create complex HiTL systems. Instead, the tutorial aims at illustrating and consolidating some of the book's theoretical ideas.
Finally, we would like to thank you, the reader, for your interest. We would also like to ask you to contact us and tell us about your experience with our book. Your feedback is a very valuable resource towards improving the book. Send your email to dsnunes@dei.uc.pt, sasilva@dei.uc.pt or boavida@uc.pt.
Preface
The Internet has changed our whole life and it will have further impact on how we live and how we work. Most of the cyber-physical systems (CPSs) make use of the Internet and even define parts of it. Let me cite Wikipedia in this preface, even though it is not very scientific so to do. Understanding the CPS as “a mechanism controlled or monitored by computer-based algorithms, tightly integrated with the internet and its users” means that users, humans, are essential for any CPS. The National Institute of Standards and Technology of the US Department of Commerce (NIST) goes even further, stating that “these systems will provide the foundation of our critical infrastructure, form the basis of emerging and future smart services, and improve our quality of life in many areas”. Looking at the examples mentioned in Wikipedia, “smart grid, autonomous automobile systems, medical monitoring, process control systems, robotics systems, and automatic pilot avionics”, human are always involved.
Humans are not only involved; humans are the essential part of CPSs; CPSs have to serve us! With the basic idea, to incorporate humans as being in the system, we encounter human-in-the-loop (HiTL). It comprises a model, an adequate representation of the human behavior in order to treat it as an integral part of the whole system. Just as one example, let me cite Carsten Binning et.al. at his preface of the Proceedings of the first Workshop on Human-In-the-Loop Data Analytics HILDA of June 26th, 2016, in San Francisco, California: “A major bottleneck in data analytics today is to efficiently leverage the human capabilities to formulate questions and understand answers of data analytics systems … Recent technology trends (such as touchscreens, motion detection, and voice recognition) are widening the possibilities for users to interact with data, and data-driven industries are shifting to personalized processing to better target their services to users' needs”.
Hence it seems somewhat natural to look at both topics together in a kind of textbook and survey. In my six years as editor-in-chief of the journal ACM Transactions on Multimedia Computing, Communications, and Applications (ACM TOMM), I have, unfortunately, not come across a comprehensive high-quality survey paper of CPS HiTL; it has been even more serious: nobody even tried to cover with a survey this essential area on multimedia computing, communications, and its applications. No one did so far!
At the present time, writing this preface, I was only able to read parts of this book; I am looking forward to reading it all together–the whole book.
The authors of this book, David Nunes, Jorge Sá Silva, and Fernando Boavida from the University of Coimbra provide an in-depth view to HiTLCPS evolution, theory, technologies, and applications. Moreover, they illustrate how to apply HiTLCPS concepts to a sample smartphone application, through a hands-on approach that guides the reader from the development environment to the final product, including data acquisition, state inference, and actuation. With (1) their profound technical knowledge of many areas in computing and communications, as well as with (2) their expertise and experience as authors of other textbooks, the authors are certainly key for this book being a long-term successful scientific book in this area. Congratulations!
Dr. Ralf Steinmetz
Fellow of the IEEE and Fellow of the ACM
Director, Multimedia Communications Laboratory, Technische Universität Darmstadt
Chairman of the Board, Hessian Telemedia Technology CompetenceCenter, Germany
Darmstadt, March 2017
Acknowledgments
A book such as this would not have been possible without the help and support of many people and institutions.
First of all, we would like to thank our base institutions—the Department of Informatics Engineering, and the Center for Informatics and Systems, both from the University of Coimbra—in the scope of which we carry out our teaching and research activities, for the provided facilities and research environment. With their effort and contributions, enthusiasm, discussions, and suggestions during several years of joint research activities and human-in-the-loop social interaction, our students and our colleagues were instrumental in making this book a reality.
We also thank IMDEA Networks Institute, in Madrid, for the support provided during Fernando Boavida's sabbatical in 2015/2016, and especially to its leading computer scientist, Arturo Azcorra, for his support; to Antonio Fernández Anta, Miguel Péon, Jeanet Birkkjaer; and Rosa Gómez for their encouragement; and to all its researchers and staff in general.
Some of the research that formed the basis for this book was carried out in the scope of financed research projects and initiatives and, thus, it is also right to thank the entities that made the referred research possible, namely the Portuguese Foundation for Science and Technology (FCT), FCT's POPH/FSE program, and the SOCIALITE Project (PTDC/EEI-SCR/2072/2014), supported by COMPETE 2020, Portugal 2020, Operational Program for Competitiveness and Internationalization (POCI), and the European Union's ERDF (European Regional Development Fund).
We would also like to thank David Hutchison, from Lancaster University, for believing in us and putting us in contact with the excellent editorial team at John Wiley & Sons.
Finally, we would like to thank our families, for their unconditional love and support.
List of Abbreviations
AI
Artificial Intelligence
ANN
Artificial Neural Network
API
Application Programming Interface
AS
Android Studio
AV
Autonomous Vehicle
BCC
Body-Coupled Communication
BCI
Behavior Change Interventions
CHIL
Computers in the Human Interaction Loop
CoAP
Constrained Application Protocol
cOre
Constrained RESTful environments
CPS(s)
Cyber-Physical System(s)
CPU
Central Processing Unit
DAO
Data Access Object
ECG
Electrocardiography
EEG
Electroencephalography
ESM
Experience Sampling Method
FCT
Fast Cosine Transform
FFT
Fast Fourier Transformation
GPRS
General Packet Radio Service
GPS
Global Positioning System
GSM
Global System for Mobile Communications
HiTL
Human-in-the-Loop
HiTLCPS(s)
Human-in-the-Loop Cyber-Physical System(s)
HTML
HyperText Markup Language
HTTP
Hypertext Transfer Protocol
HVAC
Heating, Ventilation, and Cooling
ID
Identification
IFR
International Federation of Robotics
IoA
Internet of All
IoT
Internet of Things
IP
Internet Protocol
IDE
Integrated Development Environment
IEEE
Institute of Electrical and Electronics Engineers
IETF
Internet Engineering Task Force
ISM band
Industrial, Scientific, and Medical radio bands
Java EE
Java Enterprise Edition
Java SE
Java Standard Edition
JDK
Java Development Kit
JSON
JavaScript Object Notation
LTE
Long-Term Evolution
M2M
Machine-to-Machine
MPTCP
MultiPath Transmission Control Protocol
NAT
Network Address Translation
NSF
National Science Foundation
OSI
Open Systems Interconnection
OS
Operating System
P2P
Peer-to-Peer
POI(s)
Point(s) of Interest
RAM
Random-Access Memory
REST
Representational state transfer
RF
Radio Frequency
RFID
Radio-Frequency Identification
RSSI
Received Signal Strength Indication
SCTP
Stream Control Transmission Protocol
SDK
Software Development Kit
sMAP
Simple Monitoring and Action Profile
SMS
Short Message Service
SOAP
Simple Object Access Protocol
SQL
Structured Query Language
TCP
Transmission Control Protocol
UDP
User Datagram Protocol
URI
Uniform Resource Identifier
URL
Uniform Resource Locator
UUID
Universally Unique Identifier
VoIP
Voice Over Internet Protocol
WSDL
Web Service Description Language
WSN(s)
Wireless Sensor Network(s)
XML
Extensible Markup Language
About the Companion Website
Don't forget to visit the companion website for this book:
www.wiley.com/go/nunesloop
There you will find valuable material designed to enhance your learning, including: