Copyright © 2015 by John Wiley & Sons, Inc. All rights reserved
Published by John Wiley & Sons, Inc., Hoboken, New Jersey
Published simultaneously in Canada
No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 750-4470, or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley.com/go/permissions.
Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.
For general information on our other products and services or for technical support, please contact our Customer Care Department within the United States at (800) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002.
Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic formats. For more information about Wiley products, visit our web site at www.wiley.com.
Library of Congress Cataloging-in-Publication Data:
Cook, Diane J., 1963-
Activity learning : discovering, recognizing, and predicting human behavior from sensor data / Diane J. Cook, Narayanan C. Krishnan.
pages cm. – (Wiley series on parallel and distributed computing)
Includes bibliographical references and index.
ISBN 978-1-118-89376-0 (hardback)
1. Active learning–Data processing. 2. Detectors–Data processing. 3. Multisensor data fusion. I. Title.
LB1027.23.C665 2015
371.3–dc23
2014039501
Activity learning from sensor data is a research topic that is making its way into many fields including machine learning, pervasive computing, psychology, and sociology. The ability to discover and model activities based on collected sensor data has matured. These methods are now able to handle increasingly complex with increasingly complex situations including unscripted activities, interrupted or interwoven activities, real-time learning, and learning from multiple users or residents. At the same time, there have been advances in the application of the theoretical techniques to challenge real-world problems including health monitoring and home automation.
The goal of this book is to define the notion of an activity model learned from sensor data and to present the key algorithms that form the core of the field. While many current efforts have contributed to the ability to automatically recognize known activities, there are other aspects of activity learning that we want to include in this treatment. These include discovering activity patterns from unlabeled data and predicting when specific activities will occur in the future, as well as mapping sensor event sequences to predefined activity labels. We also want to discuss the challenges that are faced when these theoretical techniques are applied to real-world problems and suggest methods for addressing the challenges.
This book is designed for an interdisciplinary audience who would like to use or design activity learning techniques. As a result, most of the ideas are presented from the ground up, with little assumptions about the background of the reader. Ideally, the book will provide some helpful background and guidance to researchers, undergraduate or graduate students, or practitioners who want to incorporate the ideas into their own work.
The best way to understand activity learning techniques is to look at activity sensor data, play with existing tools, and writing your own code. To help with this process, we provide code for many of the techniques described in this book. A variety of activity sensor datasets are available for your use as well. All of the related book materials can be found online at http://eecs.wsu.edu/∼cook/albook.
We would like to thank the many individuals who helped us with this book. Jacqueline Southwick took the activity photographs that we included and the team of Kyle Elsalhi and Anthony Simmons, who generated the Kinect-based motion history image. Numerous individuals contributed to the data collections we made available with this book and who provided feedback on drafts of the book. These include Larry Holder, Maureen Schmitter-Edgecombe, Aaron Crandall, Brian Thomas, Yasmin Sahaf, Kyle Feuz, Prafulla Dawadi, Ehsan Nazerfard, Bryan Minor, Adriana Seelye, Carolyn Parsey, Jennifer Walker, Alyssa Weakley, Sue Nelson, Thomas Cowger, Selina Akter, and Prasanth Lade. Our editors at Wiley provided guidance throughout the entire process and for that we are grateful. Last but not least, we want to thank our colleagues, friends, and family who were unfailingly supportive of this effort. Diane would like to thank her family, Larry, Abby, and Ryan, who kept her going with ideas, encouragement, and lots of humor. She dedicates this book to them. Narayanan dedicates this book to his family, Krishnan, Geetha, and Karthik for their constant encouragement and support.
Diane J. Cook and
Narayanan C. Krishnan
Figure 1.1 The role of activity learning in the design of an intelligent agent.
Figure 2.1 Examples of individual group actions and activities that are found in common everyday life settings.
Figure 2.2 Relationship between state (S), action, and activity.
Figure 3.1 PIR infrared sensor packaging (left) and internal architecture (right).
Figure 3.2 The two components of a magnetic contact switch encased in plastic (left) and a magnetic contact switch acting as a door open/shut sensor (right).
Figure 3.3 Tactile pressure sensors positioned in a chair (left) and interpreted as a pressure map (right).
Figure 3.4 RFID tag positioned beneath visible object barcode.
Figure 3.5 Increased acceleration values can indicate a start or stop of a motion.
Figure 3.6 A smart phone with a three-axis accelerometer (left) and a three-axis gyroscope (right).
Figure 3.7 Floor plan and sensor layout for the first floor of a smart home. Sensor identifiers starting with “M” are infrared motion sensors, sensors starting with “D” are magnetic door closure sensors, and sensors starting with “T” are ambient temperature sensors. The hand soap object sensor is located near the sink and the additional object sensors are in the kitchen closet. All object sensors are indicated with stars. Additional sensors included in these datasets are A001 (gas usage sensor attached to burner), A002 and A003 (hot and cold water consumption sensors, respectively), and P001 (whole-home power usage meter).
Figure 3.8 Positioning of wearable accelerometers on an individual's dominant arm (left) and hip (right).
Figure 3.9 Time plot of sensor activity during the Hand Washing activity. The x axis shows the time of day when the sensor message was generated. The y axis shows the identifier (on the left) and functional location (on the right) for each sensor generating messages.
Figure 3.10 Time plot of normalized accelerometer readings during the Hand Washing activity. The x axis shows the time of day when the sensor message was generated. The y axis shows the identifier (on the left) and placement (on the right) for each sensor. There are three lines for each sensor, corresponding to the x, y, and z axes.
Figure 3.11 Time plot of sensor activity during the Sweeping activity. The x axis shows the time of day when the sensor message was generated. The y axis shows the identifier (on the left) and functional location (on the right) for each sensor generating messages.
Figure 3.12 Time plot of normalized accelerometer readings during the Sweeping activity. The x axis shows the time of day when the sensor message was generated. The y axis shows the identifier (on the left) and placement (on the right) for each sensor. There are three lines for each sensor, corresponding to the x, y, and z axes.
Figure 3.13 Heat map of discrete event sensor usage in a smart home (left, motion sensor locations represented by circles in the image) and motion history image of a sit-to-stand transition movement (right). In both images, the pixel intensities indicate activities occurring in the corresponding region of the space. In the left picture, darker circles indicate areas in which more time was spent. In the right picture, lighter regions indicate more recent movement occurring in the region.
Figure 3.14 Plot of acceleration (Ax, Ay, Az) and rotational velocity (Rx, Ry, Rz) values for HIP accelerometer from the Sweeping activity data sample.
Figure 3.15 Frequency content in the accelerometer and gyroscope signal for the sensor placed on the hip while performing a Sweeping activity.
Figure 4.1 Plot illustrating the distribution of the start time of the Eating activity from a synthetic dataset. The three solid curves represent the distribution for breakfast, lunch, and dinner. The dashed curve is the sum of these distributions.
Figure 4.2 The GMM_EM algorithm.
Figure 4.3 Graphical representation of an HMM. The shaded nodes represent the observed variables—the sensor events or the feature vectors from the sensor stream. The white nodes represent the hidden variables, which correspond to the underlying activity labels. Note the direction of the arrow indicating that the underlying hidden states (activities) generate the observations (sensor events).
Figure 4.4 A HMM used to recognize four distinct activities.
Figure 4.5 The Viterbi algorithm to generate the most likely sequence of activity labels from a sequence of sensor observations.
Figure 4.6 A sample decision tree for classifying a data point as Bed Toilet Transition, Personal Hygiene, Take Medicine, or Other.
Figure 4.7 ID3 decision tree algorithm.
Figure 4.8 Illustration of a SVM for a linearly separable classification problem.
Figure 4.9 Illustration of the soft-margin concept for SVM classification in the nonseparable scenario.
Figure 4.10 The two-class SVM algorithm.
Figure 4.11 Graphical representation of a linear chain CRF.
Figure 4.12 The Boosting algorithm to combine multiple classifiers.
Figure 4.13 The Bagging algorithm to combine classifiers.
Figure 4.14 The filter-based feature selection algorithm.
Figure 4.15 The PCA dimensionality reduction algorithm.
Figure 4.16 Plot of the data points projected onto a three-dimensional subspace using PCA. The axes in the figure represent the three principal components.
Figure 5.1 The AR process includes the stages of raw sensor data collection, sensor preprocessing and segmentation, feature extraction and selection, classifier training, and data classification.
Figure 5.2 Illustration of alternative approaches to processing sensor data streams for AR. Sensor event types and timings are depicted by the vertical lines, where the line type indicates the type, location, and value of the sensor message. The sensor windows are obtained using explicit segmentation, sliding window extraction with a fixed window time duration, or sliding window extraction with a fixed window length (number of sensor events).
Figure 5.3 Rule-based activity segmentation algorithm.
Figure 5.4 Illustration of time dependency in a sliding window of sensor events.
Figure 5.5 Effect of 
on weights.
Figure 5.6 Illustration of sensor dependency in a sliding window of sensor events.
Figure 5.7 Intensity-coded MI between every pair of motion sensors (right) in a smart home (left).
Figure 5.8 Online change point detection algorithm.
Figure 5.9 Hybrid bottom-up/sliding-window segmentation algorithm.
Figure 5.10 Example ROC curve (left) and PRC (right).
Figure 5.11 Event-based evaluation of Hand Washing AR. The vertical solid lines indicate the performance segment boundaries. The top row shows the ground truth labeling of hand washing or sweeping and the bottom row shows the labels generated by an AR algorithm. Each segment mismatch is labeled with the type of error that is represented, either an overfill (O), underfill (U), insertion (I), deletion (D), merge (M), or fragmenting (F).
Figure 6.1 Time spent by Americans as reported by the Bureau of Labor Statistics14.
Figure 6.2 Example activity-attribute matrix. Each row represents a distinct activity and each column represents a separate high-level feature. The contents of the matrix indicate whether the feature is associated with the activity 
or not 
.
Figure 6.3 Frequency-based sequence mining algorithm.
Figure 6.4 Highest-value patterns found by frequency mining.
Figure 6.5 Example of the hierarchical discovery algorithm applied to an input sequence S. Each unique sensor event symbol is represented by a unique circle pattern in the diagram. A sequence pattern (P) is identified and used to compress the dataset into a smaller sequence 
. A new best pattern 
is found in the second iteration of the hierarchical discovery algorithm.
Figure 6.6 Hierarchical compression-based sequence mining algorithm.
Figure 6.7 Illustration of the topic model process. Each sensor event sequence is a mixture of topics (high-level features), each topic is a distribution over low-level sensor features, each sensor feature is drawn from one of the topics.
Figure 7.1 LZ78 string parsing algorithm.
Figure 7.2 The trie formed by the LZ78 parsing of the location sequence 
. The numbers in parentheses indicate the frequency of the corresponding phrase encountered within the sequence as it is parsed.
Figure 7.3 ALZ string parsing algorithm.
Figure 7.4 The trie formed by the ALZ parsing of the location sequence aaababbbbbaabccddcbaaaa.
Figure 7.5 Forecasted time elapsed values for Enter Home activity.
Figure 7.6 A regression tree to output expected number of seconds until the next occurrence of a Bed Toilet Transition activity. Each internal node queries the value of a particular variable and each node contains a multivariate linear regression function. Each of the regression functions generates a prediction value as a function of the lags and other input feature values. The features include lag variables, or values of the predicted variable at previous time units. Lag1 indicates the prediction value at time 
and Lag3 indicates the prediction value at time 
.
Figure 7.7 An order 2 AR-HMM.
Figure 7.8 Actual and predicted class values over a three day period based on a regression tree model. In this graph, the x-axis represents the time of day (from the beginning of day one to the end of day three) and the y-axis represents the number of seconds (predicted or actual) that will elapse before the next occurrence of the target activity, Bed Toilet Transition. The class variable is the number of seconds that will elapse before the next occurrence of the target activity.
Figure 8.1 The VARS video annotation tool allows a researcher to label aspects of an activity scene while watching the corresponding video.
Figure 8.2 An experimenter observes a participant performing everyday activities via web cameras and provides activity labels remotely using the RAT.
Figure 8.3 Example tweets that indicate an individual's activities.
Figure 8.4 Illustration of a social network used to train an activity classifier for user 
.
Figure 8.5 Example of SMOTE synthetic data point creation for 
nearest neighbors of point 
.
Figure 8.6 GAL generalized active learning algorithm.
Figure 8.7 Learning curves for two algorithms, A and B. Algorithm A converges to optimal performance more quickly and thus has a larger ALC value than Algorithm B.
Figure 8.8 Depiction of difference between supervised learning and transfer learning approaches to activity modeling.
Figure 8.9 The TrAdaBoost algorithm to transfer instances from the source to target domains.
Figure 8.10 Sample WikiHow pages for the activities Cook, Shower, and Jog.
Figure 8.11 The CDAR algorithm to transfer instances between activity labels.
Figure 8.12 Multi-view transfer learning.
Figure 8.13 The Co-Training algorithm.
Figure 8.14 The Co-EM algorithm.
Figure 8.15 The Teacher–Learner algorithm.
Figure 8.16 The Manifold Alignment algorithm.
Figure 8.17 The number of distinct activity labels associated with data points sampled from a two-resident smart home.
Figure 8.18 The binary relevance multi-label learning algorithm.
Figure 8.19 The principal label space transform multi-label learning algorithm.
Figure 8.20 The multi-label output code algorithm using canonical correlation analysis.
Figure 8.21 Graphical representation of CHMM model for two users, denoted 
and 
.
Figure 8.22 Factored CRF representing a sequence of activities for two users, 
and 
.
Figure 8.23 Flow network for three sensor events with one cluster (path) highlighted.
Figure 8.24 The STAR particle filter-based algorithm for tracking multiple users.
Figure 9.1 The COACH hand-washing assistance system (a) Activity forecasting-based prompts can be delivered to a computer, television (b), or mobile device (c).
Figure 9.2 A “Them and Us” game replayed inside the after-game visualization interface.
Figure 9.3 Example hierarchical model learned from MavHome sensor data.
Figure 9.4 Activity profiles based on visualization of accelerometer data.
Figure 9.5 Sample activity density maps.
Figure 9.6 Radial bar chart for the Bed Toilet Transition activity.
Figure 9.7 Visualization of activity density, activity times, and activity trends.
Figure 9.8 Visualization of an activity daily diary.
Figure 9.9 Wi-Fi cluster co-occurrence plot.
Figure 9.10 Visualization of how surveyed Americans spent their time in 2008259.
Figure 9.11 Mobility patterns grouped by cell tower region for two time periods. The bars on the right correspond to the number of calls per hour on a log scale.
Figure 9.12 Plot of relative activity level as a function of the hour of the day (left) and relative activity duration as a function of the activity class (right).
Figure 9.13 The meaningfulness of a place is indicated by friendships that are made there.
“All growth depends upon activity.”
—Calvin Coolidge
“From time to time, it is worth wandering around the fuzzy border regions of what you do, if only to remind yourself that no human activity is an island.”
—Julian Baggini
“People love chopping wood. In this activity one immediately sees results.”
—Albert Einstein
“Nothing is more terrible than activity without insight.”
—Thomas Carlyle
“We are what we repeatedly do.”
—Aristotle
Learning and understanding observed activities is at the center of many fields of study. An individual's activities affect that individual, those around him, society, and the environment. In the past, theories about behavior and activities were formed based on limited observation. Over the past decade, the maturing of sensor technologies has made it possible to automate activity learning. Sensors have become miniature, low power, low cost, and high capacity. At the same time, we have witnessed tremendous progress in the areas of wireless networks, data processing, and machine learning. These advances have shifted researchers' focus from low level data collection and transmission to high-level information collection, inference, and recognition.
Activity learning plays a key role in the design of intelligent agents. Russell and Norvig1 define an agent as an entity that perceives its environment through sensors and acts upon the environment through actuators. The techniques we describe in this book add intelligence and activity-awareness to this concept of an agent, shown in Figure 1.1. The state of individuals and their environment is perceived by sensors and stored as raw data. The activity learning agent processes this data in order to analyze it, make inferences from it, and describe it in terms of recognizable activities. Activity patterns can then be used to make a more informed decision about actions that can be taken to change the environment in a way that achieves the agent's goal.
Figure 1.1 The role of activity learning in the design of an intelligent agent.
Because activity learning technologies have become more robust, automated modeling and tracking of activities have become integral components of numerous solutions to real-world problems. For example, surveillance and security systems try to recognize and predict activities in order to address terrorist threats. Ambient-assisted living exploits activity discovery and monitoring to support independent living for individuals with cognitive and physical impairments. Activity-aware services have transformed ideas such as intelligent meeting rooms, automated homes, and personalized digital assistants from science fiction to everyday reality. Activity learning has also recently made its debut as a core component in products including game consoles and fitness smart phone apps.
As a result of the technology push and application pull, the number and diversity of projects that are designing or utilizing activity learning is exploding. Activity discovery and recognition is found in settings where data can be captured by cameras, body sensors, phones, or buildings. Entire conferences and journals are dedicated to reporting the latest advances in the field. Over the last 3 years, there have been two dozen workshops specifically devoted to activity recognition. The interest and enthusiasm for this topic is still increasing.
The goal of this text is to provide an in-depth look at computational approaches to activity learning from sensor data. Activity learning here refers to several aspects of activity processing. These include features that can be used to represent and model activities from sensor data. Also included are basic techniques for learning activity concepts and recognizing them in offline or streaming mode from sensor data. A companion to the idea of learning activity concepts from labeled data is the idea of automatically discovering activity patterns from unlabeled data, which we describe in detail. The third major component of activity learning that we include is the prediction or forecasting of activities that will occur at a time point in the future.
This book contains a pragmatic introduction to the primary algorithms and approaches to activity learning. Because the emphasis is on computational approaches, we will provide an algorithmic perspective. We detail the methods using pseudocode and provide implementations of the techniques in the source code that accompanies this book. We also illustrate the methods using examples of sensor data collected in real-world settings.
We note that the book is intended to be an algorithmic reference. It is written, where possible, to be independent of any specific sensor device or class of devices. Some of the examples used to illustrate the techniques are based on data collected from a particular type of sensor such as environment sensors, smart phone sensors, or cameras. However, the data features that are used to represent and recognize the data can be extracted and abstracted from a wide variety of low-level data collection sources. Here is a brief overview of the remaining chapters in the book.
Chapter 2 introduces the terminology that will be used throughout the text. The definitions of actions, interactions, and activities are provided and related to each other. A discussion of activity classifications and activity ontology generation is provided.
Chapter 3 provides an overview of sensors that are commonly used for observing and learning activities. In addition to describing sensor types and capabilities, the chapter includes descriptions of high-level features that can be extracted from low-level sensor data.
Chapter 4 introduces machine learning techniques that will be used throughout the remainder of the text. In addition to supervised classification techniques, the chapter also discusses approaches to dimensionality reduction that is valuable when learning concepts from high-dimensional sensor features.
Chapter 5 presents concepts and approaches for activity recognition. Additionally, methods for segmenting data into distinct activities are described together with methods to process streaming data. The chapter also includes performance metrics that can be used to evaluate and compare alternative activity recognition algorithms.
Chapter 6 describes algorithmic techniques for discovering activity patterns from sensor data. These techniques include zero-shot learning, sequence mining, clustering, and topic models. Methods for evaluating the results of activity discovery are summarized.
Chapter 7 covers activity prediction. Activity prediction includes identifying the next activity in a sequence as well as forecasting the next point in time when a particular activity will occur. As with activity recognition and discovery, metrics for evaluating the performance of activity prediction are included.
Chapter 8 discusses activity learning “in the wild.” This refers to computational techniques that can be used to address problems that are encountered when activity learning is used in complex, real-world situations. These problems include obtaining sufficient high-quality labeled activity sensor data, transferring data or learned activity models to new settings with new users, mapping sensor sequences to multiple activity labels, and learning activities in the presence of multiple individuals.
Chapter 9 provides an overview of activity learning applications that are currently being investigated. Ongoing work is described that is related to activity recognition for health monitoring and intervention, marketing and activity-aware automation, surveillance and emergency management, describing routine behaviors in terms of activities, and analyzing human dynamics for entire populations.
Chapter 10 provides a conclusion of the book. This chapter poses some grand challenge problems for activity learning as well as uses for activity learning in other fields.
Additional materials including chapter updates, data sets, implementation of algorithms described in the book, and links to other tools and ongoing activity learning research projects are available online. These resources can be found at http://eecs.wsu.edu/cook/albook.
Activity learning is an important concept because it is critical for understanding human behavior as well as designing human-centric technologies. As activity learning projects tend to focus on a specific subset of activity patterns and use a subset of available sensor types, it can be difficult to reach an agreement about the expectations of activity learning and how to compare alternative approaches. Our goal is to provide a unifying treatment of activity learning. As a beginning, we provide definitions for sensor events and activities that we will use as the basis of our discussions throughout the rest of the book.
As Figure 1.1 illustrates, an activity learner receives data from sensors that are used to perceive the state of an individual and/or an environment. We specify input data to an activity learner as a sequence of sensor events. Each sensor event, 
, takes the form 
where 
denotes a timestamp, 
denotes a sensor ID, and 
denotes the sensor message. We define an activity instance or activity occurrence as a sequence of 
sensor events, 
. An activity represents the collection of all of its instances. An activity learner may represent an activity as an abstraction of this collection of activity instances.
The terms “action” and “activity” are frequently used interchangeably in activity learning research. In actuality, there is a tremendous diversity of concepts that are classified as activities in the literature. The activity classes that are investigated differ in terms of activity complexity, the type of sensor modality that is typically used to capture the activity, and the computational techniques that are most effective for learning the activity. Figure 2.1 illustrates some of the activities that can be represented and learned using techniques described in this book. While sitting, standing, waving, and walking appear at one end of the spectrum of activities, the other end consists of longer, more complicated tasks such as cooking, marching in formation, and playing a football game.
Figure 2.1 Examples of individual group actions and activities that are found in common everyday life settings.
Throughout this book, we refer to action as a simple ambulatory behavior executed by a single person and typically lasting short durations of time. Similarly, we refer to interaction as a short, single-movement action that involves multiple individuals. By contrast, by activity we refer to complex behaviors consisting of a sequence of actions that can be performed by a single individual or several individuals interacting with each other. They are typically characterized by longer temporal durations. At the other end of the spectrum are individual states such as human (or animal) postures and poses, environmental state, and object location. While such states are indicative of human behavior, they occur at a single point in time and can be differentiated from actions and activities in this way. Note that these definitions are hierarchical. An action may consist of a sequence of states, while an activity may contain any number of actions. Activities may also be described in terms of environment states or influences that the environment has on the individual performing the activity. Figure 2.2 represents the relationship between these subsets of human behavior.
Figure 2.2 Relationship between state (S), action, and activity.
While activity sensor-based datasets are currently available for analysis, the activities that are labeled and learned vary greatly among these datasets. We provide one possible taxonomy of activity classes in Table 2.1. As can be seen in this list, there exist implicit and explicit relationships between activity classes. A model that can be used to represent the activity cooking may also be used to represent the subset cooking breakfast. On the other hand, a cooking breakfast model may not be an effective representative of all cooking tasks. Additionally, some activity classes may be considered functionally different and yet bear a striking resemblance in terms of the sensor events they generate. For example, an individual may fill a glass with water to drink or fill a kettle with water to boil. Such similarities between activities means that additional contextual information, including time of day and the previous activity, need to be considered when learning the activity class. Depending on the sensor modality, some pairs of classes may be virtually indistinguishable. For example, the movements corresponding to lying down on a rug and falling down may generate identical sensor sequences if only infrared motion sensors are used. In these cases, the activities can only be distinguished by incorporating additional sources of information such as different sensor modalities.
Table 2.1 Categories of Routine Activities
Actions
|
Activities
|
A number of authors have considered ways to categorize classes of activities for modeling, detecting, and recognizing. Candamo et al.2 and Chaquet et al.3 distinguish single person activities from multiple person activities and interactions. Borges et al.4 offer a characterization that breaks activities into gestures, actions, and interactions in a manner similar to the discussion found in this chapter. Krishnan et al.5 consider methods of learning activity taxonomies directly from sensor data. Chen et al.6 and Bae7 employ a method for activity learning that involves hand-constructing an activity taxonomy then using occurrences of the activities to refine the taxonomy. These learned activity ontologies can form a knowledge-driven approach to learning and reasoning about activities8,9.
In this book, we will refer to examples activities that appear in Figure 2.1 and Table 2.1. However, this is by no means a comprehensive list nor is it the only taxonomical organization of activity classes. Other taxonomies have been generated for use in specific domains. For example, clinicians may evaluate an individual's cognitive and physical health based on their ability to independently complete known activities. As a result, in the health literature activities classes of activities are often categorized as activities of daily living (ADLs)10 or instrumental activities of daily living (iADLs)11. Many resources are available that provide labels, categories, and examples of classes of activities. This includes work by Tapia12, the Compendium of Physical Activities13, the American time use survey14, the Facial Action Coding System15, the SCARE joint task corpus16, and the UCF sports dataset17.