Cover Page

A Practical Introduction to Human-in-the-Loop Cyber-Physical Systems


David Nunes

University of Coimbra


Jorge Sá Silva

University of Coimbra


Fernando Boavida

University of Coimbra




Wiley Logo

To my parents, Jorge and
Eulália, and to my brother,


David Nunes


To Fátima, Catarina,
Pedro, Jojó, and my parents


Jorge Sá Silva


To Maria Joào and our
three daughters-Susana,
Inês, and Catarina


Fernando Boavida

List of Figures

  1. Figure 2.1 In [1], books and other common objects were augmented with RFID tags and associated with virtual documents by PDAs.
  2. Figure 2.2 Shaman [2] acted as a representative for the connected LiteServers, offering Java and HTML interfaces.
  3. Figure 2.3 Device web presence in Cooltown [3]. Source: Adapted from Kindberg et al. 2002.
  4. Figure 2.4 JXTA [4] peers created virtual ad hoc networks which served to abstract the real ones.
  5. Figure 2.5 Works such as [5] and [6] used proxies to offer embedded devices' capabilities through RESTful web services.
  6. Figure 2.6 The SenseWeb [7] architecture.
  7. Figure 2.7 WikiCity [8] interfaced between virtual data and the physical world through a semantically defined format for data exchange.
  8. Figure 2.8 Nokia 6101 vs iPhone 6s/LG Nexus 5X.
  9. Figure 2.9 HiTL technologies evolution timeline.
  10. Figure 3.1 Basic processes of human-in-the-loop control.
  11. Figure 3.2 Taxonomy of human control.
  12. Figure 3.3 Taxonomy of human roles.
  13. Figure 4.1 SenQ's query system stack shown side-by-side with the topology and components of AlarmNet, a prototypical implementation for assisted-living [9]. Source: Adapted from Wood 2008.
  14. Figure 4.2 The architecture of CenceME [10], one of MetroSense's implementations.
  15. Figure 4.3 The three key components of BCI using smartphones [11]. Source: Adapted from Lathia et al. 2013.
  16. Figure 4.4 SociableSense architecture [12]. Source: Adapted from Rachuri 2011.
  17. Figure 4.5 Control architecture for energy saving with HiTL [13]. Source: Adapted from Liang 2013.
  18. Figure 4.6 Architecture of an HiTL HVAC system [14]. Source: Adapted from Agarwal 2011.
  19. Figure 4.7 Diagram showing the main components of CAALYX's roaming monitoring system [15]. Source: Adapted from Boulos et al. 2007.
  20. Figure 4.8 A semi-autonomous wheelchair receives brain signals from the user and executes the associated tasks of path planning, obstacle avoidance, and localization [16]. Source: Adapted from Schirner 2013.
  21. Figure 4.9 A mockup of a map interface similar to the Highlight application.
  22. Figure 4.10 Overview of the system proposed in [17]. Source: Adapted from W.-H. Rho and S.-B. Cho 2014.
  23. Figure 5.1 HappyWalk HiTL control.
  24. Figure 5.2 HappyWalk's architecture.
  25. Figure 5.3 Android's activity lifecycle.
  26. Figure 5.4 HappyWalk's Android class structure.
  27. Figure 5.5 An overview of HappyWalk Android app's main classes.
  28. Figure 5.6 An overview of HappyWalkServer's main classes.
  29. Figure 5.7 A typical artificial neural network architecture.
  30. Figure 5.8 Sound signal in the time domain (left side) analyzed through a Fourier transformation to show its frequency domain (right side).
  31. Figure 5.9 HappyWalk's Emotional Feedback.
  32. Figure 5.10 HappyWalk's neural network design.
  33. Figure 6.1 Installing Java SE Development Kit 7u79.
  34. Figure 6.2 Installing Android Studio and Android SDK.
  35. Figure 6.3 Canceling the setup wizard.
  36. Figure 6.4 Opening the Android SDK manager.
  37. Figure 6.5 Installing Android API 21.
  38. Figure 6.6 Opening the standalone SDK manager.
  39. Figure 6.7 Installing Android SDK Build-tools 21.1.2.
  40. Figure 6.8 Installing Git #1. (a) Adding Git to the PATH, on Windows (b) Choose Checkout Windows-style.
  41. Figure 6.9 Installing Git #2. (a) We recommend using MinTTY (b) Uncheck Enable file system caching.
  42. Figure 6.10 Importing HappyWalk from Git.
  43. Figure 6.11 Cloning the HappyWalk project.
  44. Figure 6.12 Opening the HappyWalk project.
  45. Figure 6.13 Choosing HappyWalk's project folder.
  46. Figure 6.14 Do not upgrade Android Gradle or its plugin.
  47. Figure 6.15 Running HappyWalk.
  48. Figure 6.16 HappyWalk's first launch.
  49. Figure 6.17 Obtaining the Android debug key.
  50. Figure 6.18 Creating a project to obtain a Google Maps Android API key.
  51. Figure 6.19 Creating the Google Maps Android API key.
  52. Figure 6.20 Obtaining the Google Maps Android API key.
  53. Figure 6.21 Changing into the project's view.
  54. Figure 6.22 Opening app/debug/res/values/google_maps_api.xml.
  55. Figure 6.23 Choosing PostgreSQL superuser's password.
  56. Figure 6.24 No need to launch Stack Builder.
  57. Figure 6.25 Clone from a URI.
  58. Figure 6.26 Introduce the URI corresponding to HappyWalk's server.
  59. Figure 6.27 Select the master branch.
  60. Figure 6.28 Selecting the local storage directory.
  61. Figure 6.29 Select the option Import existing Eclipse projects.
  62. Figure 6.30 Tick the checkbox of the HappyWalkServer project.
  63. Figure 6.31 Creating a Foursquare® app.
  64. Figure 6.32 Foursquare®'s Client ID and Client Secret.
  65. Figure 6.33 Navigating into the server's GlobalVariables.
  66. Figure 6.34 Log in to the PostgreSQL 9.3 server.
  67. Figure 6.35 Create a new database.
  68. Figure 6.36 Name the new database as happywalk.
  69. Figure 6.37 Select the correct SQL script.
  70. Figure 6.38 Populating the database.
  71. Figure 6.39 Create a new server.
  72. Figure 6.40 Define a new Tomcat 7 installation.
  73. Figure 6.41 Installing Tomcat 7 from Eclipse.
  74. Figure 6.42 Adding HappyWalk to Tomcat 7.
  75. Figure 6.43 Running the HappyWalk server.
  76. Figure 6.44 Select the newly created Tomcat 7.
  77. Figure 6.45 The HappyWalk server is up and running.
  78. Figure 6.46 The ipconfig command.
  79. Figure 6.47 HappyWalk's map screen.
  80. Figure 7.1 Creating a new class.
  81. Figure 7.2 AS cannot resolve symbol issue.
  82. Figure 7.3 Importing the appropriate class.
  83. Figure 7.4 Creating a new package.
  84. Figure 7.5 Creating the sensor processors.
  85. Figure 7.6 Signal processing overview.
  86. Figure 7.7 Current state of our HiTLCPS at the end of Chapter 7.
  87. Figure 8.1 An example of a sigmoid activation function.
  88. Figure 8.2 Creating a new basic activity.
  89. Figure 8.3 Name the activity as EmotionFeedback.
  90. Figure 8.4 The files that compose the EmotionFeedback activity.
  91. Figure 8.5 Our goal for the EmotionSpace view.
  92. Figure 8.6 Creating the EmotionSpace class.
  93. Figure 8.7 Create EmotionSpace constructor matching super.
  94. Figure 8.8 Choose View(context:Context, attrs:AttributeSet).
  95. Figure 8.9 Changing from the layout Design view to Text view.
  96. Figure 8.10 Creating a new Values resource file.
  97. Figure 8.11 Naming the Values resource file.
  98. Figure 8.12 The coordinates of the EmotionSpace view.
  99. Figure 8.13 The emotion feedback notification.
  100. Figure 8.14 Creating TaskSendEmotion.
  101. Figure 8.15 Current state of our HiTLCPS at the end of Chapter 8.
  102. Figure 9.1 HappyWalk's database conceptual schema.
  103. Figure 9.2 Creating a new class in Eclipse.
  104. Figure 9.3 Naming RequestSetEmotion.
  105. Figure 9.4 Generating the Constructors, toString(), and the Getters and Setters.
  106. Figure 9.5 Generating a Constructor using fields.
  107. Figure 9.6 Generating a Constructor from Superclass.
  108. Figure 9.7 Generating the Getters and Setters.
  109. Figure 9.8 Overriding the default toString() method.
  110. Figure 9.9 The location of the HappyWalkServer's web.xml.
  111. Figure 9.10 The emotion alert dialog.
  112. Figure 9.11 The emotion heatmaps
  113. Figure 9.12 Final state of our HiTLCPS at the end of Chapter 9.
  114. Figure 10.1 The HiTL resilience paradigm.
  115. Figure 11.1 Lessons learned towards human-in-the-loop control.

List of Tables

  1. Table 4.1 Summary of some of the technologies/solutions that support HiTLCPS
  2. Table 4.2 Summary of experimental HiTLCPS projects
  3. Table 5.1 Machine learning approaches for sensing context in smartphones [18]. Source: Adapted from Guinness 2013
  4. Table 5.2 Testing training performance (150 emotions)
  5. Table 5.3 Testing neural network accuracy (41 emotions)
  6. Table 6.1 Summary of the steps necessary to install AS 2.1.3
  7. Table 6.2 Summary of the steps necessary to set up HappyWalk's Android project
  8. Table 6.3 Summary of the steps necessary to deploy HappyWalk's server
  9. Table 6.4 Summary of the steps necessary to test the base HappyWalk system
  10. Table 10.1 Summary of the identified HiTL requirements and challenges


Our world keeps being an increasingly technological one. As first put forward by the renowned computer scientist Mark Weiser, we continue to see that, as devices get smaller in size, more mobile, powerful, and efficient, they begin to “disappear”. Technology is now so intrinsic to our everyday lives that it has become an inherent part of our existence. This is the premise behind concepts such as the Internet of things and cyber-physical systems, in which distributed technology is used to monitor and control the environment. However, our current technological advancement still falls short of Weiser's ideas. Each time we have to hurdle through unintuitive configuration menus, errors, and software incompatibilities we become stressed by our computers and appliances. Weiser argued that the ultimate form of computers was an extension of our subconscious. To him, the ideal computer would be capable of truly understanding people's unconscious actions and desires. Instead of humans adapting to technology and learning how to use it, it would be technology that would adapt to the disposition and uniqueness of each human being.

In fact, systems that consider the human context are becoming increasingly more important, and there are strong indications that most future technologies will most likely be much more human-aware. This book focuses on the realm of human-in-the-loop cyber-physical systems (HiTLCPSs), that is cyber-physical systems that take human response into consideration. HiTLCPSs infer the user's, intents, psychological states, emotions, and actions through sensors, using this information to determine the system's actions. This involves using a large variety of sensors and mobile devices to monitor and evaluate human nature. Therefore, this technology has strong ties with wireless sensor networks, robotics, machine learning, and the Internet of things.

This book is useful to BSc and MSc students, as well as to PhD students, researchers, and professors addressing the areas of ubiquitous computing, Internet of things, cyber-physical systems, and human–computer interaction. It can also be useful to professional developers that intend to introduce HiTL concepts into their mobile apps and/or Internet of things/cyber-physical system applications.

Throughout its pages, the book will guide the reader through a journey into this novel and exciting area of research and technological development. As such, it is intended to be used as a primer on HiTLCPSs, providing some insights into the research being done on this topic, current challenges, and requirements. One of the book's objectives is to introduce the reader to the practical usage of HiTL paradigms within software development. Therefore, we included a comprehensive hands-on tutorial where the major theoretical concepts behind HiTLCPSs are applied to a sample mobile application and explained from a practical perspective. This tutorial requires some knowledge of Android and the Java programming language, as well as some notions about databases and RESTful web services. It is accompanied by a base source code repository and several code snippets which the reader can extensively modify.1 It is not our intention to provide in-depth knowledge about the programming languages, and/or the machine learning techniques, necessary to create complex HiTL systems. Instead, the tutorial aims at illustrating and consolidating some of the book's theoretical ideas.

Finally, we would like to thank you, the reader, for your interest. We would also like to ask you to contact us and tell us about your experience with our book. Your feedback is a very valuable resource towards improving the book. Send your email to, or


The Internet has changed our whole life and it will have further impact on how we live and how we work. Most of the cyber-physical systems (CPSs) make use of the Internet and even define parts of it. Let me cite Wikipedia in this preface, even though it is not very scientific so to do. Understanding the CPS as “a mechanism controlled or monitored by computer-based algorithms, tightly integrated with the internet and its users” means that users, humans, are essential for any CPS. The National Institute of Standards and Technology of the US Department of Commerce (NIST) goes even further, stating that “these systems will provide the foundation of our critical infrastructure, form the basis of emerging and future smart services, and improve our quality of life in many areas”. Looking at the examples mentioned in Wikipedia, “smart grid, autonomous automobile systems, medical monitoring, process control systems, robotics systems, and automatic pilot avionics”, human are always involved.

Humans are not only involved; humans are the essential part of CPSs; CPSs have to serve us! With the basic idea, to incorporate humans as being in the system, we encounter human-in-the-loop (HiTL). It comprises a model, an adequate representation of the human behavior in order to treat it as an integral part of the whole system. Just as one example, let me cite Carsten Binning at his preface of the Proceedings of the first Workshop on Human-In-the-Loop Data Analytics HILDA of June 26th, 2016, in San Francisco, California: “A major bottleneck in data analytics today is to efficiently leverage the human capabilities to formulate questions and understand answers of data analytics systems … Recent technology trends (such as touchscreens, motion detection, and voice recognition) are widening the possibilities for users to interact with data, and data-driven industries are shifting to personalized processing to better target their services to users' needs”.

Hence it seems somewhat natural to look at both topics together in a kind of textbook and survey. In my six years as editor-in-chief of the journal ACM Transactions on Multimedia Computing, Communications, and Applications (ACM TOMM), I have, unfortunately, not come across a comprehensive high-quality survey paper of CPS HiTL; it has been even more serious: nobody even tried to cover with a survey this essential area on multimedia computing, communications, and its applications. No one did so far!

At the present time, writing this preface, I was only able to read parts of this book; I am looking forward to reading it all together–the whole book.

The authors of this book, David Nunes, Jorge Sá Silva, and Fernando Boavida from the University of Coimbra provide an in-depth view to HiTLCPS evolution, theory, technologies, and applications. Moreover, they illustrate how to apply HiTLCPS concepts to a sample smartphone application, through a hands-on approach that guides the reader from the development environment to the final product, including data acquisition, state inference, and actuation. With (1) their profound technical knowledge of many areas in computing and communications, as well as with (2) their expertise and experience as authors of other textbooks, the authors are certainly key for this book being a long-term successful scientific book in this area. Congratulations!

Dr. Ralf Steinmetz

Fellow of the IEEE and Fellow of the ACM

Director, Multimedia Communications Laboratory, Technische Universität Darmstadt

Chairman of the Board, Hessian Telemedia Technology CompetenceCenter, Germany

Darmstadt, March 2017


A book such as this would not have been possible without the help and support of many people and institutions.

First of all, we would like to thank our base institutions—the Department of Informatics Engineering, and the Center for Informatics and Systems, both from the University of Coimbra—in the scope of which we carry out our teaching and research activities, for the provided facilities and research environment. With their effort and contributions, enthusiasm, discussions, and suggestions during several years of joint research activities and human-in-the-loop social interaction, our students and our colleagues were instrumental in making this book a reality.

We also thank IMDEA Networks Institute, in Madrid, for the support provided during Fernando Boavida's sabbatical in 2015/2016, and especially to its leading computer scientist, Arturo Azcorra, for his support; to Antonio Fernández Anta, Miguel Péon, Jeanet Birkkjaer; and Rosa Gómez for their encouragement; and to all its researchers and staff in general.

Some of the research that formed the basis for this book was carried out in the scope of financed research projects and initiatives and, thus, it is also right to thank the entities that made the referred research possible, namely the Portuguese Foundation for Science and Technology (FCT), FCT's POPH/FSE program, and the SOCIALITE Project (PTDC/EEI-SCR/2072/2014), supported by COMPETE 2020, Portugal 2020, Operational Program for Competitiveness and Internationalization (POCI), and the European Union's ERDF (European Regional Development Fund).

We would also like to thank David Hutchison, from Lancaster University, for believing in us and putting us in contact with the excellent editorial team at John Wiley & Sons.

Finally, we would like to thank our families, for their unconditional love and support.

List of Abbreviations

AI Artificial Intelligence
ANN Artificial Neural Network
API Application Programming Interface
AS Android Studio
AV Autonomous Vehicle
BCC Body-Coupled Communication
BCI Behavior Change Interventions
CHIL Computers in the Human Interaction Loop
CoAP Constrained Application Protocol
cOre Constrained RESTful environments
CPS(s) Cyber-Physical System(s)
CPU Central Processing Unit
DAO Data Access Object
ECG Electrocardiography
EEG Electroencephalography
ESM Experience Sampling Method
FCT Fast Cosine Transform
FFT Fast Fourier Transformation
GPRS General Packet Radio Service
GPS Global Positioning System
GSM Global System for Mobile Communications
HiTL Human-in-the-Loop
HiTLCPS(s) Human-in-the-Loop Cyber-Physical System(s)
HTML HyperText Markup Language
HTTP Hypertext Transfer Protocol
HVAC Heating, Ventilation, and Cooling
ID Identification
IFR International Federation of Robotics
IoA Internet of All
IoT Internet of Things
IP Internet Protocol
IDE Integrated Development Environment
IEEE Institute of Electrical and Electronics Engineers
IETF Internet Engineering Task Force
ISM band Industrial, Scientific, and Medical radio bands
Java EE Java Enterprise Edition
Java SE Java Standard Edition
JDK Java Development Kit
JSON JavaScript Object Notation
LTE Long-Term Evolution
M2M Machine-to-Machine
MPTCP MultiPath Transmission Control Protocol
NAT Network Address Translation
NSF National Science Foundation
OSI Open Systems Interconnection
OS Operating System
P2P Peer-to-Peer
POI(s) Point(s) of Interest
RAM Random-Access Memory
REST Representational state transfer
RF Radio Frequency
RFID Radio-Frequency Identification
RSSI Received Signal Strength Indication
SCTP Stream Control Transmission Protocol
SDK Software Development Kit
sMAP Simple Monitoring and Action Profile
SMS Short Message Service
SOAP Simple Object Access Protocol
SQL Structured Query Language
TCP Transmission Control Protocol
UDP User Datagram Protocol
URI Uniform Resource Identifier
URL Uniform Resource Locator
UUID Universally Unique Identifier
VoIP Voice Over Internet Protocol
WSDL Web Service Description Language
WSN(s) Wireless Sensor Network(s)
XML Extensible Markup Language

About the Companion Website

Don't forget to visit the companion website for this book:


There you will find valuable material designed to enhance your learning, including:

Scan this QR code to visit the companion website.