Sage Journals: Discover world-class research

Abstract

Facial emotion recognition has been used as a representative pedestrian activity in studies examining the effect of changes in road lighting. Past studies have drawn conclusions using results averaged across performance with the six universally recognised expressions. This paper asks whether expression choice matters. A reanalysis of past data for each unique expression does not suggest a change in the conclusion that facial emotion recognition is not significantly affected by the spectral power distribution of the lighting.

1. Introduction

Making a judgement about the intentions of other people is an assumed critical visual task for pedestrians.^1–3 There is support for this assumption in studies using eye tracking. When looking at static images, observers will tend to look at the people in a scene with a frequency significantly greater than chance.⁴ In natural outdoor settings, there is a probability of over 80% that another person in the field of view will be fixated at least once^5,6 and fixations on other people can be sufficiently important to demand significant cognitive attention.⁷ The assumed reason for observing others is that a pedestrian ‘need[s] to be able to take a “good look” at the other users of streets – identification of persons or of intentions …’,² that is whether they are friendly, aggressive or indifferent⁸ and thus whether it is safe to approach them or if avoiding action is required. After dark, on roads where significant pedestrian activity is expected, road lighting should be designed to enhance the performance of such interpersonal judgements.

Facial emotion recognition is the identification of a person’s emotional state from their facial expression. Facial emotions have been linked with the approach–avoid response⁹ and thus three experiments have been carried out to investigate how this task is affected by changes in road lighting.^10–12 These studies used photographs from the FACES database of actors portraying the six universally recognised facial expressions of emotion: anger, disgust, fear, happiness, neutrality and sadness.^13,14 A photograph was presented for a brief (0.5 or 1.0 s) observation and a six-alternative forced choice of expression was sought. These studies used the faces of four actors (a young male, a young female, an old male and an old female) each portraying all six expressions giving 24 target images in total. An example of the target images is shown in Figure 1. The targets were observed under lighting of different spectral power distribution (SPD), luminance and viewing distance as simulated by manipulation of image size (Table 1). In two studies,^10,12 the photographs were presented on a non-self-luminous screen with the surrounding environment being lit by the test lighting. In the third study,¹¹ the photographs were projected onto a screen, with the projector light and surrounding field providing similar SPD and luminance.

Figure 1.

Sample of facial expressions from the FACES database.¹³ These are a younger female with expressions (from left to right) of angry, disgust, fear, happy, neutral and sadness. Website for image database: http://faces.mpdl.mpg.de/faces/.

Table 1

Experimental conditions used in past studies of facial emotion recognition.

Study	Lamp type^a	Luminance on target (cd/m²)	Simulated distance (m)	Duration of presentation (ms)	Sample size
Fotios et al.¹²	HPS, MH	0.01, 0.1, 1.0	4, 10, 15	1000	30
Yang and Fotios¹⁰	HPS, MH, CPO	0.01, 0.03, 0.1, 0.33, 1.00, 3.33	4, 15	500, 1000	20
Fotios et al.¹¹	HPS, MH	0.1, 0.33, 1.0	4, 15	500	28

Note: Fotios et al.¹¹ also varied target colour (black and white or coloured photographs).

Lamp types: HPS: High pressure sodium, (2000 K, S/P = 0.57, Ra = 25), MH: Metal halide (4200 K, S/P = 1.77, Ra = 92), CPO: Metal halide (2868 K, S/P = 1.22, Ra = 70).

The results from the three experiments are shown in Figure 2. Analyses of these data confirmed that luminance and distance were significant factors in correct recognition frequency. At low luminance (0.01 cd/m²) performance is at chance level, increasing towards a plateau of maximum performance with higher luminance. Closer (i.e. larger) targets were correctly identified with a significantly higher frequency than targets further away (i.e. smaller). In neither study, however, was SPD suggested to be a significant effect. In Figure 2, different SPDs (i.e. lamp types) are given different line types, and these graphs do not indicate any significant or consistent trends.

Figure 2.

Frequency of correct recognition of facial emotion plotted against luminance. These are the data from Fotios et al.¹² (top left), Fotios et al.¹¹ (top right) and Yang and Fotios¹⁰ (bottom).

These conclusions were drawn from consideration of expression recognition performance averaged across all six expressions. The peer reviewer of one study¹⁰ commented that some facial expressions may be easier to detect than others and therefore that the analysis of lighting effects should be repeated but for individual expressions. It may also be the case that recognition of some expressions is more important than others for a pedestrian’s reassurance, or more difficult to discriminate, and hence that particular expressions are differently affected by changes in lighting. This paper therefore presents a further analysis of the results of the facial emotion recognition experiments of Yang and Fotios¹⁰ and Fotios et al.¹¹ with emotion recognition evaluated for individual expressions.

2. Choice of facial expression

In the past studies,^10–12 all six universally recognised facial expressions were used as targets, and the data used to analyse the effect of changes in lighting was the frequency of correct response averaged across all six expressions. Some expressions may, however, be more difficult to discriminate than others because they are more ambiguous or more complex.¹⁵ Expression portrayal in the FACES database, evaluated under good lighting conditions with unlimited exposure durations, demonstrated that different expressions have different recognition rates. The happy and neutral expressions were correctly identified most frequently and the sad and disgust expressions were correctly identified least frequently (Table 2).

Table 2.

Proportion of correct identification of unique facial expressions as reported by Ebner et al.¹³ and Yang and Fotios¹⁰

Expression	Proportion of correct identification
Expression	Ebner et al.¹³	Yang and Fotios¹⁰
Happy	0.96	0.95
Neutral	0.87	0.93
Angry	0.81	0.80
Fear	0.81	0.79
Sad	0.73	0.81
Disgust	0.68	0.77

Note: For Yang and Fotios these are data determined under their best visual conditions (i.e. 3.33 cd/m², 4 m simulated distance). The expressions are listed in descending order as defined by the results of Ebner et al.

Of these six facial expressions, four might be considered negative emotions (angry, disgust, fear and sad)¹⁶ one to be a positive emotion (happy), and one to be ambivalent (neutral). This negativity is recognised by observers; Willis et al.¹⁷ found that happy faces were judged more positively than all other emotions, while neutral faces were judged more favourably than faces displaying negative emotions. Angry and disgusted faces were given the most negative ratings, significantly more so than sad and fearful faces. It might therefore be pertinent to ask whether it is appropriate in investigations of road lighting to use all six expressions, or whether it might be interesting to pick the most salient expressions for interpersonal evaluations (which is not yet known) or to balance the number of positive and negative emotions presented during trials.

Some past studies investigating the influence of facial expression have used a sub-set of the six expressions rather than all six (Table 3). These studies have examined how we look at others,¹⁸ how we evaluate others^9,16 and how we take cues from others.¹⁹ What these indicate is that there is no apparent consensus as to the right expression(s) to use.

Table 3.

Facial expressions used in past studies

Study	Aim	Target type	Emotion conveyed by facial expression
Study	Aim	Target type	Happy	Angry	Neutral	Fear	Disgust	Sadness
Gallup et al.¹⁹	Gaze following tendency	Actor	•		•	•
Kanan et al.¹⁸	Scan path for judging emotion when viewing a face	Photo	•	•	•
Mienaltowski et al.¹⁶	Facial emotion recognition	Photo		•		•	•	•
Willis et al.⁹	Approachability	Photo	•	•	•
Fotios et al.^11,12 Yang and Fotios¹⁰	Facial emotion recognition under variation in lighting	Photo	•	•	•	•	•	•

Note: Gallup et al.¹⁹ also included a ‘suspicion’ expression.

3. Analysis with individual expressions

For this paper, the data from two studies of facial emotion recognition were re-analysed.^10,11 In the original analyses, a participants’ probability of correctly identifying the target expression was determined as a proportion of the sample of 24 expressions observed (4 actors, 6 expressions) under each combination of luminance, lamp type, target colour and simulated distance. For the current analyses, conclusions regarding the effect of SPD were drawn from separate analyses of the six expressions.

Tables 4 and 5 show the expression discrimination results for each expression, and for each combination of distance, luminance and lamp type. These data are the median number of correct emotion recognition responses, summated for each test participant across the four actors portraying that emotion and the two levels of either duration¹⁰ or target colour.¹¹ The maximum possible number of correct responses was therefore eight in both studies.

Table 4.

Median frequencies of correct emotion discrimination for each expression, after Yang and Fotios¹⁰

			Median frequency of correct response according to luminance (cd/m²)
Expression	Distance (m)	Lamp type	3.33	1.0	0.33	0.1	0.03	0.01
Angry	4	HPS	6	7	6	6	4	1
Angry	4	MH	6	7	6.5	6	4	2
Angry	4	CPO	6	7	6	6	4.5	1.5
Angry	15	HPS	5	3.5	2	1	0.5	0.5
Angry	15	MH	5	4	2	0	0	1
Angry	15	CPO	5	4	2	0.5	0	0.5
Disgust	4	HPS	7	5.5	5.5	4.5	2	1
Disgust	4	MH	5.5	6	5.5	5.5	3	1
Disgust	4	CPO	7	6	5	5	2	1
Disgust	15	HPS	3.5	1	1	1	1	1
Disgust	15	MH	2.5	2	1	1	0	1
Disgust	15	CPO	4	2.5	1	1	0.5	1
Fear	4	HPS	6	6	6	5	3.5	1
Fear	4	MH	6	6	5.5	5	4	1
Fear	4	CPO	6	6	5	4	4	1
Fear	15	HPS	4	3	2.5	1.5	0	1
Fear	15	MH	4	3.5	1.5	1	0	1
Fear	15	CPO	4	3	1	1.5	1	0
Happy	4	HPS	8	8	8	7	5.5	2
Happy	4	MH	8	8	8	7	6	3
Happy	4	CPO	8	8	8	7	5	2
Happy	15	HPS	7	6	5	3	1	0
Happy	15	MH	6.5	6	5	2	0.5	1
Happy	15	CPO	6	6	5	2	1	1
Neutral	4	HPS	7	7	8	6.5	5	3
Neutral	4	MH	7.5	8	8	7	6	2.5
Neutral	4	CPO	8	8	8	7	5	3
Neutral	15	HPS	7	6	5.5	4.5	3	2
Neutral	15	MH	6	6	4.5	4	2	1
Neutral	15	CPO	6	6.5	4.5	3	3	2
Sad	4	HPS	7	6	7	5	3	1
Sad	4	MH	7.5	7	7	5	2.5	1
Sad	4	CPO	6	6	6.5	6	3	1
Sad	15	HPS	2	2	0.5	1	0	0
Sad	15	MH	1.5	1.5	1	0.5	0	1
Sad	15	CPO	2	2	1	0	0.5	0

Note: For each test participant the results are summated for both durations, and hence from the four target actors the maximum possible score was 8.

Table 5.

Median frequencies of correct emotion discrimination for each expression, after Fotios et al.¹¹

Expression	Distance (m)	Lamp type	Median frequency of correct response according to luminance (cd/m²)
Expression	Distance (m)	Lamp type	1.0	0.33	0.1
Angry	4	HPS	8	8	8
Angry	4	MH	8	8	8
Angry	15	HPS	6	6	5
Angry	15	MH	6	6	4
Disgust	4	HPS	7	6	6
Disgust	4	MH	6	6	6
Disgust	15	HPS	4	4	2
Disgust	15	MH	4	5	3
Fear	4	HPS	8	8	8
Fear	4	MH	8	8	8
Fear	15	HPS	5	5	4
Fear	15	MH	5	4	4
Happy	4	HPS	8	8	8
Happy	4	MH	8	8	8
Happy	15	HPS	7	6	6
Happy	15	MH	7	6	6
Neutral	4	HPS	8	8	8
Neutral	4	MH	8	8	8
Neutral	15	HPS	7	6	6
Neutral	15	MH	6	6	6
Sad	4	HPS	8	7	7
Sad	4	MH	7	8	7
Sad	15	HPS	5	4	2
Sad	15	MH	5	4	2

Note: For each test participant the results are summated for both target colours, and hence from the four target actors the maximum possible score was 8.

Figures 3 and 4 show these data further summated across lamp type to show separately the expression recognition rate for each expression. The maximum number of correct responses in these data are therefore 24 for Yang and Fotios¹⁰ who used three lamps and 16 for Fotios et al.¹¹ who used two types of lamp. It can be seen that trials with different expressions led to different frequencies of correct responses. The happy and neutral expressions tended to be identified more frequently than were the other expressions across all luminances, consistent with Table 2.

Figure 3.

Median frequencies for correct identification of emotion from facial expression for the six expressions at the two test distances (as identified in the legend) after Yang and Fotios¹⁰

Figure 4.

Median frequencies for correct identification of emotion from facial expression for the six expressions at the two test distances (as identified in the legend) after Fotios et al.¹¹

These data were not drawn from a normally distributed population and hence analyses of differences were done using non-parametric statistical tests for repeated measures, the Wilcoxon test (two samples) and the Friedman test (k samples). While repeated application of such tests raises the risk of falsely indicating a significant effect (a type I error), adjusting the threshold p-value to compensate raises the risk of failing to identify a significant effect (a type II error): Rothman²⁰ recommends that we do not make adjustment for multiple comparisons. Following the example of previous work,²¹ we retained the standard threshold of p < 0.05 and planned to draw conclusions by consideration of the overall pattern of results rather than by placing emphasis on any one result.

In past studies^10–12 the effect of SPD on facial emotion recognition was not suggested to be significant. The focus of the current paper is to examine whether that conclusion holds when the effect of SPD is examined using the results of each expression separately rather than using the collated results of all six expressions. Tables 6 and 7 show the results of significance testing for the two studies. In the Yang and Fotios¹⁰ data, the Friedman test suggested a significant difference between the three types of lamp used (p < 0.05) in only 5 of the 144 cases (6 expressions × 24 combinations of luminance, duration and distance). In the Fotios et al.¹¹ data, the Wilcoxon test suggested a significant difference (p < 0.05) between the two types of lamp used in only 5 of the 72 cases. There is no apparent trend in the distribution of these few significant cases. These data thus confirm the conclusion drawn following analysis across all expressions, that SPD did not have significant effect on expression discrimination.

Table 6.

p-Values determined using the Friedman test to compare facial expression discrimination performance under three lamps

Luminance (cd/m²)	Distance (m)	Duration (ms)	Expression
Luminance (cd/m²)	Distance (m)	Duration (ms)	Angry	Disgust	Fear	Happy	Neutral	Sad
3.33	4	1000	0.412	0.284	0.307	0.779	0.846	0.323
	4	500	0.973	0.458	0.294	0.097	0.041	0.751
	15	1000	0.736	0.696	0.360	0.494	0.864	0.439
	15	500	0.247	0.362	0.078	0.751	0.135	0.138
1.00	4	1000	0.476	0.754	0.684	0.108	0.382	0.862
	4	500	0.819	0.657	0.735	0.651	0.110	0.103
	15	1000	0.155	0.336	0.087	0.565	0.497	0.068
	15	500	0.393	0.717	0.465	0.771	0.206	0.581
0.33	4	1000	0.321	0.034	0.282	0.811	0.140	0.581
	4	500	1.000	0.008	0.180	0.468	0.882	0.936
	15	1000	0.983	0.270	0.544	0.565	0.832	0.979
	15	500	0.607	0.708	0.744	0.526	0.687	0.569
0.10	4	1000	0.839	0.880	0.012	0.721	0.100	0.385
	4	500	0.942	0.563	0.106	0.464	0.743	0.478
	15	1000	0.938	0.172	0.288	0.236	0.437	0.884
	15	500	0.467	0.427	0.587	0.814	0.598	0.498
0.03	4	1000	0.839	0.502	0.599	0.799	0.683	0.125
	4	500	0.313	0.926	0.449	0.627	0.528	0.798
	15	1000	0.350	0.811	0.950	0.729	0.049	0.488
	15	500	0.238	0.735	0.951	0.538	0.288	0.148
0.01	4	1000	0.301	0.670	0.310	0.010	0.982	0.898
	4	500	0.307	0.879	0.146	0.859	0.849	0.397
	15	1000	0.289	0.656	0.767	0.074	0.819	0.937
	15	500	0.157	0.305	0.439	0.627	0.863	0.015

Note: This is a post hoc analysis of the data from Yang and Fotios.¹⁰ Values in bold are those where p < 0.05.

Table 7.

p-Values determined using the Wilcoxon test to compare facial expression discrimination performance under two lamps

Luminance (cd/m²)	Distance (m)	Expression
Luminance (cd/m²)	Distance (m)	Angry	Disgust	Fear	Happy	Neutral	Sad
		Grey scale targets
1.0	4	0.317	0.458	1.000	0.414	0.705	0.317
0.33	4	0.088	0.485	1.000	0.655	0.48	0.365
0.1	4	0.205	0.034	0.642	0.059	0.782	0.499
1.0	15	0.302	0.706	0.785	0.09	0.361	0.496
0.33	15	0.194	0.861	0.414	0.195	0.018	0.831
0.1	15	0.941	0.008	0.185	0.405	0.705	0.605
		Coloured targets
1.0	4	0.394	0.302	0.705	0.157	0.705	0.251
0.33	4	0.804	0.728	0.527	0.739	0.527	0.403
0.1	4	0.449	0.811	0.266	0.157	0.417	0.225
1.0	15	0.549	0.231	0.128	1.000	0.320	0.209
0.33	15	0.456	0.024	0.806	0.416	0.398	0.415
0.1	15	0.617	0.202	0.559	0.177	0.756	0.672

Note: This is a post hoc analysis of the data from Fotios et al.¹¹ Values in bold are those where p < 0.05.

According to Table 2, a happy expression tends to receive the highest probability of correct identification and disgust the least probability, suggesting disgust to be the more difficult expression to recognise. With tasks of greater difficulty, it appears that SPD is more likely to have an effect²² and thus the disgust expression is more likely to suggest significant effects of SPD. In Tables 6 and 7, the disgust expression has the most cases where a significant effect of SPD is indicated (5/36), while the happy expression has only two such cases. This suggests that the higher difficulty involved in recognising the disgust expression did lead to higher likelihood of indicating an effect of SPD, but this is not, however, significant nor consistent trend in these data.

The effects of further independent variables were also examined within the individual expressions. Fotios et al.¹¹ also considered the target colour, using coloured and grey scale versions of the target photographs. Analysis of these data using the Wilcoxon test suggested a significant effect in only 9/72 cases. Effects of luminance and distance were also re-analysed. In these tests luminances ranged from 0.01 cd/m² to 3.33 cd/m² (Yang and Fotios¹⁰) and 0.10 cd/m² to 1.0 cd/m² (Fotios et al.¹¹). It was confirmed that there is a significant effect of luminance for targets seen at 15 m, suggested to be the critical distance,²³ with higher luminances enabling a greater probability of correct identification. This effect of luminance is less prevalent with the targets simulating a distance of 4 m. It was confirmed that there is a significant effect of distance, with a greater probability of correctly identifying expression at 4 m than at 15 m.

4. Conclusion

Previous studies suggested that facial emotion recognition is significantly affected by luminance and observation distance but not by lamp spectrum. These conclusions were confirmed in the current analysis which considered each expression in isolation rather than considering the combined results of all six expressions. This conclusion therefore does not suggest the decision to employ the full range of facial expressions rather than a specific selection in previous work^10–12 was erroneous.

Footnotes

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Engineering and Physical Sciences Research Council (EPSRC) grant numbers EP/H050817 and EP/M02900X/1.

Acknowledgement

Images used in the original trials were taken, with permission, from the FACES database developed by the Max Planck Institute for Human Development.

References

British Standards Institution BS 5489-1:2013. Code of Practice for the Design of Road Lighting Part 1: Lighting of Roads and Public Amenity Areas. London: BSI, 2012.

Caminada

Van Bommel

WJM

. New lighting criteria for residential areas. Journal of the Illuminating Engineering Society 1984; 13: 350–358.

Simons RH, Hargroves RA, Pollard NE, Simpson MD. Lighting criteria for residential roads and areas: Proceedings of the CIE, Venice, 1987: pp. 274–277.

Fletcher-Watson

Findlay

Leekam

Benson

. Rapid detection of person information in a naturalistic scene. Perception 2008; 37: 571–583.

Foulsham

Walker

Kingstone

. The where, what and when of gaze allocation in the lab and the natural environment. Vision Research 2011; 51: 1920–1931.

Fotios

Uttley

Yang

. Using eye-tracking to identify pedestrians’ critical visual tasks. Part 2. Fixation on pedestrians. Lighting Research and Technology 2015; 47: 149–160.

Fotios

Uttley

Cheal

Hara

. Using eye-tracking to identify pedestrians’ critical visual tasks. Part 1. Dual task approach. Lighting Research and Technology 2015; 47: 133–148.

British Standards Institution. BS 5489-3: 1992. Road Lighting – Part 3: Code of Practice for Lighting for Subsidiary Roads and Associated Pedestrian Areas. London: BSI, 1992.

Willis

Palermo

Burke

. Judging approachability on the face of it: the influence of face and body expressions on the perception of approachability. Emotion 2011; 11: 514–523.

10.

Yang

Fotios

. Lighting and recognition of emotion conveyed by facial expressions. Lighting Research and Technology 2015; 47: 964–975.

11.

Fotios

Castleton

Cheal

Yang

. Investigating the chromatic contribution to recognition of facial expression through lamp spectrum and target colour. Lighting Research and Technology. First published 24 November 2015. doi:10.1177/1477153515616166.

12.

Fotios

Yang

Cheal

. Effects of outdoor lighting on judgements of emotion and gaze direction. Lighting Research and Technology 2015; 47: 301–315.

13.

Ebner

Riediger

Lindenberger

. FACES – a database of facial expressions in young, middle-aged, and older women and men: development and validation. Behavior Research Methods 2010; 42: 351–362.

14.

Etcoff

Magee

. Categorical perception of facial expressions. Cognition 1992; 44: 227–240.

15.

Adolphs

. Recognizing emotion from facial expressions: Psychological and neurological mechanisms. Behavioral and Cognitive Neuroscience Reviews 2002; 1: 21–62.

16.

Mienaltowski

Johnson

Wittman

Wilson

A-T

Sturycz

Norman

. The visual discrimination of facial expressions by younger and older adults. Vision Research 2013; 81: 12–17.

17.

Willis

Palermo

Burke

. Social judgements are influenced by both facial expression and direction of eye gaze. Social Cognition 2011; 29: 415–429.

18.

Kanan

Bseiso

DNF

Ray

Hsiao

Cottrell

. Humans have idiosyncratic and task-specific scanpaths for judging faces. Vision Research 2015; 108: 67–76.

19.

Gallup

Chong

Kacelnik

Krebs

Couzin

. The influence of emotional facial expressions on gaze-following in grouped and solitary pedestrians. Scientific Reports 2014; 4: 5794–5794.

20.

Rothman

. No adjustments are needed for multiple comparisons. Epidemiology 1990; 1: 43–46.

21.

Boyce

Cuttle

. Effect of correlated colour temperature on the perception of interiors and colour discrimination. Lighting Research and Technology 1990; 22: 19–36.

22.

Lin

Fotios

. Investigating methods for measuring facial recognition under different road lighting conditions. Lighting Research and Technology 2015; 47: 221–235.

23.

Fotios

Yang

Uttley

. Observing other pedestrians: Investigating the typical distance and duration of fixation. Lighting Research and Technology 2015; 47: 548–564.

Does expression choice affect the analysis of light spectrum and facial emotion recognition?

Abstract

1. Introduction

2. Choice of facial expression

3. Analysis with individual expressions

4. Conclusion

Footnotes

Declaration of conflicting interests

Funding

Acknowledgement

References