A widely accepted prediction is that computing will move to the background, weaving itself into the fabric of our everyday living spaces and projecting the human user into the foreground. If this prediction is to come true, then next generation computing, which we will call human computing, should be about anticipatory user interfaces that should be human-centered, built for humans based on human models. They should transcend the traditional keyboard and mouse to include natural, human-like interactive functions including understanding and emulating certain human behaviors such as affective and social signaling. This article discusses a number of components of human behavior, how they might be integrated into computers, and how far we are from realizing the front end of human computing, that is, how far are we from enabling computers to understand human behavior.

The ability to understand and manage social signals of a person we are communicating with is the core of social intelligence. Social intelligence is a facet of human intelligence that has been argued to be indispensable and perhaps the most important for success in life. This paper argues that next-generation computing needs to include the essence of social intelligence – the ability to recognize human social signals and social behaviours like turn taking, politeness, and disagreement – in order to become more effective and more efficient. Although each one of us understands the importance of social signals in everyday life situations, and in spite of recent advances in machine analysis of relevant behavioural cues like blinks, smiles, crossed arms, laughter, and similar, design and development of automated systems for Social Signal Processing (SSP) are rather difficult. This paper surveys the past efforts in solving these problems by a computer, it summarizes the relevant findings in social psychology, and it proposes a set of recommendations for enabling the development of the next generation of socially-aware computing.

In this paper we review the major approaches to Multimodal Human Computer Interaction, giving an overview of the field from a computer vision perspective. In particular, we focus on body, gesture, gaze, and affective interaction (facial expression recognition and emotion in audio). We discuss user and task modeling, and multimodal fusion, highlighting challenges, open issues, and emerging applications for Multimodal Human Computer Interaction (MMHCI) research.

We present the design, implementation, and deployment of a wearable computing platform for measuring and analyzing human behavior in organizational settings. We propose the use of wearable electronic badges capable of automatically measuring the amount of face-to-face interaction, conversational time, physical proximity to other people, and physical activity levels in order to capture individual and collective patterns of behavior. Our goal is to be able to understand how patterns of behavior shape individuals and organizations. By using on-body sensors in large groups of people for extended periods of time in naturalistic settings, we have been able to identify, measure, and quantify social interactions, group behavior, and organizational dynamics. We deployed this wearable computing platform in a group of 22 employees working in a real organization over a period of one month. Using these automatic measurements, we were able to predict employees ’ self-assessments of job satisfaction and their own perceptions of group interaction quality by combining data collected with our platform and e-mail communication data. In particular, the total amount of communication was predictive of both of these assessments, and betweenness in the social network exhibited a high negative correlation with group interaction satisfaction. We also found that physical proximity and e-mail exchange had a negative correlation of r = −0.55 (p <0.01), which has far-reaching implications for past and future research on social networks.

Abstract — Dominance- a behavioral expression of power- is a fundamental mechanism of social interaction, expressed and perceived in conversations through spoken words and audio-visual nonverbal cues. The automatic modeling of dominance patterns from sensor data represents a relevant problem in social computing. In this paper, we present a systematic study on dominance modeling in group meetings from fully automatic nonverbal activity cues, in a multi-camera, multi-microphone setting. We investigate efficient audio and visual activity cues for the characterization of dominant behavior, analyzing single and joint modalities. Unsupervised and supervised approaches for dominance modeling are also investigated. Activity cues and models are objectively evaluated on a set of dominance-related classification tasks, derived from an analysis of the variability of human judgment of perceived dominance in group discussions. Our investigation highlights the power of relatively simple yet efficient approaches and the challenges of audio-visual integration. This constitutes the most detailed study on automatic dominance modeling in meetings to date. Index Terms — Group Meetings, dominance modeling, nonverbal communication, audio-visual activity cues

Mobile telephones, company ID badges, and similar common devices form a sensor network which can be used to map human activity, and especially human interactions. The most informative sensor data seem to be measurements of personto-person proximity, and statistics of vocalization and body movement measurements. Using this data to model individual behavior as a stochastic process allows prediction of future activity, with the greatest predictive power obtained by modeling the interactions between individual processes. Experiments show that between 40 % and 95 % of the variance in human behavior may be explained by such models.

A widely accepted prediction is that computing will move to the background, weaving itself into the fabric of our everyday living spaces and projecting the human user into the foreground. To realise this prediction, next-generation computing should develop anticipatory user interfaces that are human-centred, built for humans and based on naturally occurring multimodal human communication. These interfaces should transcend the traditional keyboard and mouse and have the capacity to understand and emulate human communicative intentions as expressed through behavioural cues, such as affective and social signals. This article discusses how far we are to the goal of human-centred computing and Human-Centred Intelligent Human-Computer Interaction (HCI²) that can understand and respond to multimodal human communication.

Studies have shown that programmers frequently seek external information during programming, from source code and documents, as well as from other programmers because much of the information remains in the heads of programmers. Programmers therefore often ask other programmers questions to seek information in a timely fashion to carry out their work. This information seeking entails several conflicting factors. From the perspective of the information-seeking programmer, not asking questions degrades productivity. Conversely, asking questions interrupts other programmers and degrades their productivity, and may be frowned upon by peers due to the perceived social inconsideration of the information seeker. From the perspective of the recipients of the question, even though helping is costly, not helping also incurs social costs due to the deviation from social norms. To balance all these factors, this paper proposes the STeP_IN (Socio-Technical Platform for In situ Networking) framework to guide the design of systems that support information seeking during different phases of programming. The framework facilitates access to the information in the heads of other programmers while minimizing the negative impacts on the overall productivity of the team.

Automatically detecting human social intentions from spoken conversation is an important task for dialogue understanding. Since the social intentions of the speaker may differ from what is perceived by the hearer, systems that analyze human conversations need to be able to extract both the perceived and the intended social meaning. We investigate this difference between intention and perception by using a spoken corpus of speed-dates in which both the speaker and the listener rated the speaker on flirtatiousness. Our flirtationdetection system uses prosodic, dialogue, and lexical features to detect a speaker’s intent to flirt with up to 71.5 % accuracy, significantly outperforming the baseline, but also outperforming the human interlocuters. Our system addresses lexical feature sparsity given the small amount of training data by using an autoencoder network to map sparse lexical feature vectors into 30 compressed features. Our analysis shows that humans are very poor perceivers of intended flirtatiousness, instead often projecting their own intended behavior onto their interlocutors. 1

We are developing a new wearable electronic badge that will enable people working in large organizations to communicate, find information, and interact in more efficient and intelligent ways. The badge will perform speech analysis and speech recognition using a microphone and state-ofthe-art micro-power electronics. It will be capable of playing audio messages and reminders through a speaker. An accelerometer will allow us to study how people move and behave throughout the day: Are they walking to a meeting? Are they talking to someone? Are they sitting in front of their computers? An infrared sensor will be used to capture face-to-face interactions and study social networks. A 2.4 GHz radio transceiver will send and receive information from base stations distributed along a specific area and a Bluetooth module will enable it to interface with cell phones, PDAs, portable computers, and other Bluetoothenabled sensors and devices.