Sage Journals: Discover world-class research

Abstract

Affective computing draws more and more attention to the human-computer interaction. Based on physiological signals acquired by body sensor network, within the affection recognition process, the problem that training samples have larger class distance and smaller intraclass distance must be considered. For the class divisibility and intraclass compactness problem, researching method of samples validity was proposed based on metric multidimensional scaling. With dissimilarity matrix, scalar product matrix was calculated. Subsequently, individual attribute reconstructing matrix could be got using principal components factor analysis to display samples difference in low dimension. By means of experiment results, training and testing samples for sentiment classifier will be selected instructionally.

1. Introduction

To realize natural human-computer interaction, more personalized services are required during the interactive process. Therefore, it is an effective way to bring in a user's affection with body sensor network to the service. The artificial psychological and affective computing are becoming new research directions in the field of artificial intelligence [1]. Generally, researches focus on simulating affective recognition, modeling, and expression mainly. The affective recognition consists of two types. One is recognizing user's affection which is user centered, and the other is recognizing affection in content, such as light effect, context, and characters in film, music, and art painting, which is content centered [2].

For the first type, researches can use multitypes of sensors to measure user's affection. Kim et al. surveyed the amount of sweat, pulse, body temperature, and blood pressure. Based on this physiological information, they determined the user's affective state after data fusion and fuzzy inference [3]. Nguyen used fewer sensors, only with pulse and body temperature, to acquire affection [4]. Arapakis proposed a method with facial expression [5]. In addition, the measuring method also includes the questionnaire or its extended form. It is an indirect way. González et al. used emotion quotient (EQ) test table [6]. Ai Thanh Ho provided a group of different colors to select for user and determined the user's affection based on his selection. The features of these affection acquiring methods are compared in Table 1.

Table 1

Comparison among the methods of affection accessing.

Method	Portability	Continuity	Sensibility	Comfort
Expression	Poor	Poor	General	General
Voice	General	Poor	General	Good
Eye movement	General	General	General	Poor
Brainwave	Poor	Good	Good	Poor
Behavior	Poor	Poor	General	General
Physiological information	Good	Good	Good	General
Questionnaire	General	Poor	General	Poor

The method based on the user's physiological information to judge the affective state is better than others in portability, signal continuity, sensibility, and comfort as shown in Table 1. Therefore, researches on physiological divisibility of affection attract the interest of researchers in the field of human-computer interactionMany psychologists got different conclusions on autonomic nervous system (ANS). On the one hand, paper [7], regarded that the “detest” affection has specific ANS proposed. But on the other hand, the opposite researching view was also studied in [8]. However, the existence of specific affection ANS has got more and more support from recent studies [7–16]. In the research of affection recognition based on physiological signals, some affective physiological data sample libraries were established as shown in Table 2. However, these libraries provide a foundation for comparison and analysis of affective physiological feature selection and sentiment classification, whether they are valid, that is, whether or not having a larger class distance and a smaller intraclass distance, as training data is worthy of study. For the affective physiological database proposed in [17], this paper analyzes the divisibility and compactness of 20 groups of samples with 8 kinds of affections.

Table 2

Library of affective physiological samples.

Researcher	Inducing method	Affection types	Physiological signals	Sample size
Picard et al. [17]	Image	Anger, hate, grief, platonic love, romantic love, joy, and reverence	EMG, pulse, skin conductance, and respiratory	20 groups of a single participant
Kim et al. [18]	Audio and video	Sadness, depression, anger, and surprise	ECG, pulse rate, skin temperature, and skin conductance	175 groups of 125 participants and 50 participants
Li and Chen [19]	Movie clips	Fear, happiness, and easily	ECG, skin temperature, skin conductance, and respiratory	55 participants
Kim and André [20]	Music	Happiness, anger, sadness, and joy	EMG, skin conductance, ECG, and respiration	90 groups of three participants
Wen et al. [16]	Movie clips	Happiness, surprise, disgust, sadness, anger, and fear	Skin conductance, heart rate, pulse, ECG, respiratory, facial electromyography, and frontal lobe EEG	300 participants

2. Affective Physiological Samples

Affective database DataSet I, which is established by Picard, recorded physiological data of 20 days. The samples include 4 types: myoelectricity E, pulse B, skin conductance S, and respiratory R. And the affective states have 8 classes. They are calmness (N), anger (A), hate (H), grief (G), platonic love (P), romantic love (L), joy (J), and reverence (R). These signals were sampled in 100 seconds. The length of the time series is $N = 2001$ . So daily physiological data has 8 × 4 × 2001 = 64032 in total. The organization of affection database DataSet I is shown in Figure 1.

Figure 1

Data organization of affection database DataSet I.

3. Analysis of Divisibility and Compactness

In the affective state discriminating process with physiological signals, for feature-based classification methods, the training samples and selected feature impact on the classifier directly. Due to the different deployment of sensors in body sensor network and the participant's daily life, training samples have more noise. So samples, which have a larger class distance and a smaller intraclass distance, should be selected to train the classifier. In addition, for different feature sets, dispersion between classes or within one class should be discussed. Therefore, after selecting a certain feature set, validity analysis for affective physiological sample is one of the key factors in training the affective state classifier. Based on the multidimensional scaling theory, this paper illustrates a general validity analysis method by using the dataset proposed in [17]. From the view of similarity or dissimilarity of affective physiological signals, we represent the samples in the low-dimensional space to reveal the underlying structure of the sample.

The literature in psychophysiology and related disciplines described features about affective state recognition based on facial muscle movement, heart rate changes, and so on [10]. Picard et al. proposed 6 static characteristics. They are mean value $μ_{i}$ , standard deviation $σ_{i}$ , mean value of the first-order differential absolute value $δ_{i}$ , its normalized value ${\tilde{δ}}_{i}$ , mean value of the second-order differential absolute value $γ_{i}$ , and its normalized value ${\tilde{γ}}_{i}$ [17]. Suppose that the ith $i \in {1,2, 3,4}$ type of physiological signal acquiring devices gets the nth $n \in {1,2, \dots, N}$ measured value $X_{n}^{i}$ . And its normalized value is ${\tilde{X}}_{n}^{i}$ . So the 6 static features can be obtained by formulas (1) and (2) as follows:

\begin{matrix} μ_{i} = \frac{1}{N} \sum_{n = 1}^{N} X_{n}^{i}, \\ σ_{i} = {[\frac{1}{N - 1} \sum_{n = 1}^{N} {(X_{n}^{i} - μ_{i})}^{2}]}^{1 / 2}, \\ δ_{i} = \frac{1}{N - 1} \sum_{n = 1}^{N - 1} | X_{n + 1}^{i} - X_{n}^{i} |, \\ {\tilde{δ}}_{i} = \frac{1}{N - 1} \sum_{n = 1}^{N - 1} | {\tilde{X}}_{n + 1}^{i} - {\tilde{X}}_{n}^{i} | = \frac{δ_{i}}{σ_{i}}, \end{matrix}

(1)

where

{\tilde{X}}_{n}^{i} = (X_{n}^{i} - μ_{i}) / σ_{i}

\begin{matrix} γ_{i} = \frac{1}{N - 2} \sum_{n = 1}^{N - 2} | X_{n + 2}^{i} - X_{n}^{i} |, \\ {\tilde{γ}}_{i} = \frac{1}{N - 2} \sum_{n = 1}^{N - 2} | {\tilde{X}}_{n + 2}^{i} - {\tilde{X}}_{n}^{i} | = \frac{γ_{i}}{σ_{i}} . \end{matrix}

(2)

Based on the features above, time series acquired by the ith type of physiological signal acquiring devices can be described with

{\vec{F}}_{i} = [\begin{bmatrix} μ_{i} & σ_{i} & δ_{i} & {\tilde{δ}}_{i} & γ_{i} & {\tilde{γ}}_{i} \end{bmatrix}]

. So the feature vector of the jth class of affective states is denoted as

{\vec{S}}_{j} = [\begin{bmatrix} {\vec{F}}_{1} & {\vec{F}}_{2} & {\vec{F}}_{3} & {\vec{F}}_{4} \end{bmatrix}] \in ℜ^{1 \times 24}

. This dataset contains 8 types of affective state data in 20 days. In order to discuss the difference in these data, that is, dispersion between classes or within one class, two affective state feature matrixes

S^{D}

and

S^{*}

are established with

{\vec{S}}_{j}

S^{D}

is composed of 8 types of affective state feature vectors on day

D \in {1,2, \dots, 20

; that is,

S^{D} = {[\begin{bmatrix} {\vec{S}}_{1} & {\vec{S}}_{2} & \dots & {\vec{S}}_{8} \end{bmatrix}]}^{T} \in ℜ^{8 \times 24}

. And

S^{*}

is composed of the same types of affective state feature vectors in 20 days; that is,

S^{*} = {[\begin{bmatrix} {\vec{S}}_{1} & {\vec{S}}_{2} & \dots & {\vec{S}}_{20} \end{bmatrix}]}^{T} \in ℜ^{20 \times 24}

. According to multidimensional scaling theory [21], build a dissimilarity matrix

Δ_{L \times L}

with affective state feature matrix and especially

L \in {8,24}

in this paper.

ϖ_{i j}

Δ

can be calculated as

\begin{array}{l} ϖ_{i j} = d_{i j} = {[{({\vec{S}}_{i} - {\vec{S}}_{j}) ({\vec{S}}_{i} - {\vec{S}}_{j})}^{'}]}^{1 / 2} = {[\sum_{k} {({\vec{S}}_{i k} - {\vec{S}}_{j k})}^{2}]}^{1 / 2} \\ \forall i, j \in {1,2, \dots, L}, k \in {1,2, \dots, 24} . \end{array}

(3)

Then, calculate the inner product matrix

Γ

. Its element

r_{i j}

\begin{matrix} r_{i j} = - 0.5 \times (ϖ_{i j}^{2} - ϖ_{i •}^{2} - ϖ_{• j}^{2} + ϖ_{• •}^{2}), \end{matrix}

(4)

where

\begin{matrix} ϖ_{i •}^{2} = \frac{1}{L} \sum_{j} ϖ_{i j}^{2}, \\ ϖ_{• j}^{2} = \frac{1}{L} \sum_{i} ϖ_{i j}^{2}, \\ ϖ_{• •}^{2} = \frac{1}{L^{2}} \sum_{i} \sum_{j} ϖ_{i j}^{2} . \end{matrix}

(5)

Designate reconstructed matrix in low-dimensional space as

{\hat{Ω}}_{L \times M} (M < 24)

, which could be got through affective state feature matrix. And correspondingly, its affective state dissimilarity matrix is recorded as D. Based on the theory of metric multidimensional scaling,

Δ

is similar to D in a sense. Then,

\begin{matrix} Γ = {\hat{Ω} \hat{Ω}}^{'} . \end{matrix}

(6)

Solving (6), affective state feature reconstructed matrix in low-dimensional space

\hat{Ω}

could be got. That is to say affective state feature could be presented in low-dimensional space.

At the same time, it is easy to obtain eigenvalues $λ_{j}$ which the jth dimension of $\hat{Ω}$ is corresponding to:

\begin{matrix} λ_{j} = \sum_{i} {\hat{ψ}}_{i j}^{2}, \end{matrix}

(7)

where

{\hat{ψ}}_{i j}

are elements of

\hat{Ω}

4. Experiment

4.1. Class Divisibility

Take 8 classes of affective state feature vector on the day D to form a matrix $S^{D} = {[\begin{bmatrix} {\vec{S}}_{1} & {\vec{S}}_{2} & \dots & {\vec{S}}_{8} \end{bmatrix}]}^{T} \in ℜ^{8 \times 24}$ . And based on the theory of metric multidimensional scaling above, after reconstructing affective state feature matrix in low-dimensional space, analyze the class distance of samples measured by the body sensor network. According to Kruskal's stress requirements, in this paper, each affection feature vector is reconstructed in two-dimensional space as shown in Figure 2.

Figure 2

Restructuring graph of affective eigenvectors on the Dth day.

In Figure 2, N, A, H, G, P, L, J, and R represent 8 classes of affective states, respectively. The axes of the left figures mean relative distance between reconstructive points. Take Figure 2(a) as an example; points L and J are closer, which demonstrates that the two types of affection, romantic love L and happiness J, only have few differences for the first day samples while affective physiological data are described with 6 static features above. Their class divisibility is too poor to train an effective classifier. Points R, P, H, and G have few differences in dimension II but some differences in dimension I. However, there are large differences for points A and N no matter in dimension I or II.

The right part of Figure 2(a) shows the linear fit scatter plot of reconstruction process. The x-axis means the samples disparity, and the y-axis reflects the fitting distance. Fitting coefficient $R^{2} = 0.999$ is good enough. Young's S-stress and Kruskal's stress reach 0.407% and 1.264%, respectively. And the determination coefficient RSQ is 0.99922, which means that the proportion of explaining by relative spatial distance is larger. The affection eigenvector reconstruction process data of the Dth day are summarized in Table 3.

Table 3

Summary of affective eigenvectors restructuring process on the Dth day.

Day	Young's S-stress	Kruskal's stress	Determination coefficient RSQ	Sample with smaller class distance (affection)
1	0.00407	0.01264	0.99922	L, J
2	0.00451	0.01007	0.99954	—
3	0.00102	0.00473	0.99993	R, G, P, J, L
4	0.00132	0.00160	0.99999	—
5	0.00347	0.01847	0.99919	N, H, P, J, L
6	0.00121	0.00319	0.99996	J, G
7	0.00038	0.00617	0.99994	N, G and H, P, J, L, R
8	0.00954	0.00997	0.99936	—
9	0.00877	0.01504	0.99912	H, P, J, R
10	0.00446	0.00973	0.99968	—
11	0.00559	0.00875	0.99968	P, R
12	0.01004	0.02010	0.99876	H, P, L, R
13	0.00196	0.00451	0.99992	—
14	0.00394	0.00689	0.99988	P, J, L
15	0.00485	0.00568	0.99988	P, L
16	0.00699	0.01512	0.99902	P, L, R
17	0.00888	0.01170	0.99940	J, R
18	0.04481	0.08798	0.96343	A, G and H, N
19	0.00187	0.00909	0.99979	—
20	0.05319	0.07584	0.97629	—

4.2. Intraclass Compactness

Affective state feature matrix $S^{*} = {[\begin{bmatrix} {\vec{S}}_{1} & {\vec{S}}_{2} & \dots & {\vec{S}}_{20} \end{bmatrix}]}^{T} \in ℜ^{20 \times 24}$ consists of the same type of affective state feature vector for 20 days. Based on metric multidimensional scaling theory stated above, reconstruct affective state feature in low-dimensional space to analyze the intraclass distance of samples. The reconstruction in two-dimensional space is as shown in Figure 3.

Figure 3

Restructuring graph of 8 types of affective eigenvectors.

In Figure 3, points Day 1–Day 20 denote the eigenvectors of 20 days for the same kind of affective state. From the left part of Figure 3, the axes also mean relative distance between reconstructive points. Points Day 1, Day 3, Day 4, and Day 5 are far away from the rest of the reconstructive points. That is to say intraclass distances of the same type of affective physiological data for the first day and the 3th–5th days are larger. Their intraclass compactness is poor. In addition, the 7th day samples also have a larger intraclass distance for anger affective state.

The right part of Figure 3 shows the linear fit scatter plot of reconstruction process. The x-axis means the samples disparity, and the y-axis reflects the fitting distance. Fitting coefficient $R^{2}$ is close to 1. Young's S-stress, Kruskal's stress, and determination coefficient RSQ for the reconstructive process of 8 types of affective state feature vector are summarized in Table 4.

Table 4

Summary of 8 types of affective eigenvectors restructuring process.

Affection	Young's S-stress	Kruskal's stress	Determination coefficient RSQ	Sample with bigger intraclass distance (day)
N	0.02821	0.04515	0.99073	1, 3, 4, 5
A	0.01213	0.05945	0.99445	1, 3, 4, 5, 7
H	0.02534	0.04995	0.99102	1, 3, 4, 5
G	0.05502	0.07292	0.98062	1, 3, 4, 5
P	0.01861	0.04275	0.99384	1, 3, 4, 5
L	0.03656	0.06468	0.98685	1, 3, 4, 5
J	0.02771	0.06226	0.98909	1, 3, 4, 5
R	0.03550	0.07736	0.98321	1, 3, 4, 5

5. Conclusions

As training samples of a classifier, the validness of affective physiological feature vectors is worthy of study. Larger classes distance and smaller intraclass distance are needed. An analysis method is proposed for body sensor network, which collects affective physiological data. Particularly, in this paper, we take the affective physiological database DataSet I as an example, and analyze divisibility and compactness of 20 groups of samples belonging to 8 types of affections.

Based on the 6 static features described by formulas (1)–(2), with the samples in DataSet I, class divisibility is poor for the 4 types of affection P, L, J, and R in one day. And for the same affective type, intraclass compactness is poor for the 1st, 3rd, 4th, and 5th days.

Footnotes

Acknowledgment

This work was supported by the Foundation for Young Scholars of Hebei Educational Committee (QN20131152, Q2012070).

References

Wang

Artificial psychology—a most accessible science research to human brain

Journal of University of Science and Technology Beijing 2000 22 5 478 481

2-s2.0-77949722743

Oliveira

Ribeiro

N. M.

Chambel

Towards enhanced video access and recommendation through emotions

Proceedings of the ACM Conference on Human Factors in Computing Systems

2010

New York, NY, USA

ACM

Kim

S. J.

Kim

J. S.

Kim

S. H.

Evolvable recommendation system in the portable device based on the emotion awareness Heidelberg

Proceedings of 9th International Conference on Knowledge-Based & Intelligent Information & Engineering Systems (KES)

2005

Berlin, Germany

Springer

251 257

Nguyen

T. L.

EmuPlayer: Music Recommendation System Based on User Emotion Using Vital-Sensor 2010

Kanagawa, Japan

Keio University

Arapakis

Moshfeghi

Joho

Ren

Hannah

Jose

J. M.

Integrating facial expressions into user profiling for the improvement of a multimodal recommender system

Proceedings of the IEEE International Conference on Multimedia and Expo (ICME '09)

July 2009

New York, NY, USA

1440 1443

2-s2.0-70350711890

10.1109/ICME.2009.5202773

González

López

De la Rosa

J. L.

Managing emotions in smart user models for recommender systems

Proceedings of the 6th International Conference on Enterprise Information Systems

2004

Atlanta, Ga, USA

AllBusiness

187 194

J. P.

Zhang

Wang

L. F.

Experiment study of autonomic nervous response patterns in five basic emotions

Chinese Journal of Behavioral Medical Science 2005 14 2 257 259

Christie

I. C.

Friedman

B. H.

Autonomic specificity of discrete emotion and dimensions of affective space: a multivariate approach

International Journal of Psychophysiology 2004 51 2 143 153

2-s2.0-0346881151

10.1016/j.ijpsycho.2003.08.002

Bradley

M. M.

Lang

P. J.

Affective reactions to acoustic stimuli

Psychophysiology 2000 37 2 204 215

2-s2.0-0034011248

10.1017/S0048577200990012

10.

Cacioppo

J. T.

Berntson

G. G.

Larsen

J. T.

Lewis

Haviland-Jones

J. M.

The psychophysiology of emotion

The Handbook of Emotion 2000

New York, NY, USA

Guilford Press

173 191

11.

Gomez

Stahel

W. A.

Danuser

Respiratory responses during affective picture viewing

Biological Psychology 2004 67 3 359 373

2-s2.0-2942717634

10.1016/j.biopsycho.2004.03.013

12.

Gomez

Zimmermann

Guttormsen-Schär

Danuser

Respiratory responses associated with affective processing of film stimuli

Biological Psychology 2005 68 3 223 235

2-s2.0-11144272810

10.1016/j.biopsycho.2004.06.003

13.

Nyklíček

Thayer

J. F.

Van Doornen

L. J. P.

Cardiorespiratory differentiation of musically-induced emotions

Journal of Psychophysiology 1997 11 4 304 321

2-s2.0-0031460889

14.

Palomba

Sarlo

Angrilli

Mini

Stegagno

Cardiac responses associated with affective processing of unpleasant film stimuli

International Journal of Psychophysiology 2000 36 1 45 57

2-s2.0-0033975671

10.1016/S0167-8760(99)00099-9

15.

Rainville

Bechara

Naqvi

Damasio

A. R.

Basic emotions are associated with distinct patterns of cardiorespiratory activity

International Journal of Psychophysiology 2006 61 1 5 18

2-s2.0-33745353942

10.1016/j.ijpsycho.2005.10.024

16.

Wen

W. H.

Qiu

Y. H.

Liu

G. Y.

Construction and cross-correlation analysis of the affective physiological response database

Scientia Sinica 2011 41 1 77 89

17.

Picard

R. W.

Vyzas

Healey

Toward machine emotional intelligence: analysis of affective physiological state

IEEE Transactions on Pattern Analysis and Machine Intelligence 2001 23 10 1175 1191

2-s2.0-0035481502

10.1109/34.954607

18.

Kim

K. H.

Bang

S. W.

Kim

S. R.

Emotion recognition system using short-term monitoring of physiological signals

Medical and Biological Engineering and Computing 2004 42 3 419 427

2-s2.0-2942731250

10.1007/BF02344719

19.

Chen

J. H.

Pan

Z. G.

Cheok

Haller

Emotion recognition using physiological signals

Lecture Notes in Computer Science 2006

Berlin, Germany

Springer

437 446

20.

Kim

André

Emotion recognition based on physiological changes in music listening

IEEE Transactions on Pattern Analysis and Machine Intelligence 2008 30 12 2067 2083

2-s2.0-56649124649

10.1109/TPAMI.2008.26

21.

Davison

M. L.

Multidimensional Scalin 1983

New York, NY, USA

Wiley

Divisibility and Compactness Analysis of Physiological Signals for Sentiment Classification in Body Sensor Network

Abstract

1. Introduction

2. Affective Physiological Samples

3. Analysis of Divisibility and Compactness

4. Experiment

4.1. Class Divisibility

4.2. Intraclass Compactness

5. Conclusions

Footnotes

Acknowledgment

References