Face colour under varying illumination - analysis and applications

Birgitta Martinkauppi

Department of Electrical and Information Engineering, University of Oulu

Abstract

The colours of objects perceived by a colour camera are dependent on the illumination conditions. For example, when the prevailing illumination condition does not correspond to the one used in the white balancing of the camera, the object colours can change their appearance due to the lack of colour constancy capabilities. Many methods for colour constancy have been suggested but so far their performance has been inadequate. Faces are common and important objects encountered in many applications. Therefore, this thesis is dedicated to studying face colours and their robust use under real world illumination conditions. The main thesis statement is "knowledge about an object"s colour, like skin colour changes under different illumination conditions, can be used to develop more robust techniques against illumination changes".

Many face databases exist, and in some cases they contain colour images and even videos. However, from the point of view of this thesis these databases have several limitations: unavailability of spectral data related to image acquisition, undefined illumination conditions of the acquisition, and if illumination change is present it often means only change in illumination direction. To overcome these limitations, two databases, a Physics-Based Face Database and a Face Video Database were created. In addition to the images, the Physics-Based Face Database consists of spectral data part including skin reflectances, channel responsivities of the camera and spectral power distribution of the illumination. The images of faces are taken under four known light sources with different white balancing illumination conditions for over 100 persons. In addition to videos, the Face Video Database has spectral reflectances of skin for selected persons and images taken with the same measurement arrangement as in the Physics-Based Face Database. The images and videos are taken with several cameras.

The databases were used to gather information about skin chromaticities and to provide test material. The skin RGB from images were converted to different colour spaces and the result showed that the normalized colour coordinate was among the most usable colour spaces for skin chromaticity modelling. None of the colour spaces could eliminate the colour shifts in chromaticity. The obtained chromaticity constraint can be implemented as an adaptive skin colour modelling part of face tracking algorithms, like histogram backprojection or mean shift. The performances of these adaptive algorithms were superior compared to those using a fixed skin colour model or model adaptation based on spatial pixel selection. Of course, there are cases when the colour cue is not enough alone and use of other cues like motion or edge data would improve the result. It was also demonstrated that the skin colour model can be used to segment faces and the segmentation results depend on the background due to the method used. Also an application for colour correction using principal component analysis and a simplified dichromatic reflection model was shown to improve colour quality of seriously clipped images. The results of tracking, segmentation and colour correction experiments using the collected data validate the thesis statement.

Table of Contents
Acknowledgements
List of symbols
List of original publications
1. Introduction
1.1. Background
1.2. The scope and contributions of the thesis
1.3. The outline of the thesis
2. An overview of colour-based face image and skin analysis
2.1. Some basic concepts in colour theory and spaces
2.2. Properties of human skin
2.3. Skin reflectances, PCA and ICA
2.4. Face databases
2.5. Studies of skin colours at different spaces
2.6. Colour based detection, localization and tracking of skin
3. Colour image acquisition by a CCD camera
3.1. Overview
3.2. Illuminants
3.2.1. Responses of the human eye and a colour camera
3.2.2. Non-idealities of real colour cameras
3.2.3. White balance or white calibration
3.3. The RGB response of a camera
3.4. Colour spaces
3.5. Evaluation of camera performance
4. Acquisition of face images by a colour camera
4.1. Overview
4.2. The Physics-based Face Database
4.3. Analysis of spectral characteristics of skin
4.4. Making overclipped facial images useful
5. Skin chromaticities seen by a colour camera
5.1. Basic principles
5.2. Skin locus from an image series
5.3. Skin locus from basis functions
5.4. Behavior of skin colour
6. Skin locus in face tracking
6.1. Face Video Database
6.2. Ratio histogram and histogram backprojection
6.3. Adaptive ratio histogram
6.4. Tracking with skin locus: settings and results
6.5. Comparison with other tracking methods
6.6. Robustness to localization errors
6.7. Mean shift with skin locus
7. Conclusions
References
A. Transforms from RGB to other colour spaces
B. Visualization of skin chromaticities at different colourspaces
C. Mean shift algorithm
D. Errata
List of Tables
1. Face databases.
2. Colour spaces for pixel labelling.
3. The values of metameric indices for NCS samples.
4. Number of metameric pairs for CIE Lab difference formulae (NCS samples).
5. Values of colour differences in the Temet TVI RGB camera space for pairs with ΔEab<= 3.0.
6. Values of colour differences ΔEab for the similar colour pairs for the 8-bit TVI camera.
7. Values of colour differences ΔEab for the similar colour pairs for the 12-bit TVI camera.
8. Mean error for 20 corrected face images taken under AD conditions.
9. Reconstruction error.
10. Reconstruction error with illumination normalized by Euclidean rule.
11. Reconstruction error with illumination normalized by Euclidean rule.
12. Overlapping percentage between different skin groups.
13. Overlapping percentage between different skin groups.
14. Overlapping between cameras and size of skin locus in the colour space.
15. Sixteen faces for Nogatech.
16. Selected frames from an outdoor Nogatech video.
17. Selected frames from an indoor Alaris video.
18. Adaptive face tracking with a skin locus (video 1)
19. Adaptive face tracking with a spatial constraint (video 1).
20. .Face tracking with a static object model (video 1).
List of Figures
1. Structure of the skin. Structure of the epidermis: (1) Keratin, (2) Horny layer, (3) Lucid layer, (4) Granular layer, (5) Spinous layer, (6) Basal layer and (7) Dermis.
2. Examples of different SPDs: (a) CIE standard daylight spectra (Hunt 1987), (b) CIE representative distributions for fluorescent lamps (Hunt 1987), and (c) calculated Planckian radiator spectra (Wyszecki & Stiles 2000). Note: SPD of F11 and of Planckian 2300 K are not shown in their full range.
3. Human and machine vision systems have different light responses: (a) relative spectral sensitivities of SONY DXC-755P 3CCD colour camera as given by the manufacturer and (b) RGB spectral sensitivities for 1964 Supplementary Observer (Wyszecki & Stiles 2000).
4. The response of a camera for intensity in a channel: (a) linear response, (b) non-linear response caused by two different linear regions (pre-knee) and (c) non-linear response by gamma.
5. Skin chromaticities from images taken under different white balancing illumination. The different colours are used to separate the chromaticities obtained from different white balancing cases. The r and g chromaticities shown in the axes are parameters of the normalized colour coordinates obtained by a conversion from skin RGB values.
6. The grey world algorithm can fail: (a) original image, and (b) image corrected by the grey world algorithm. The channel average value is set to 100 by the user.
7. Experimental setup for the Physics-based Face Database.
8. The imaging procedure.
9. An example from Physics-based Face Database: 16 images of a face.
10. Skin spectral reflectance measured at three points for a person.
11. Histograms of Δδavgi,k for all skin complexions: (a) for spectral reflectances of cheeks (left cheek = 1, right cheek = 3); (b) for spectral reflectances of a left cheek and a forehead (left cheek = 1, forehead = 2); (c) for a spectral reflectances of a right cheek and a forehead (right cheek = 3, forehead = 2); and (d) for all three spectral curves (left cheek = 1, right cheek = 3), (left cheek = 1, forehead = 2), and (right cheek = 3, forehead= 2).
12. Spectral reflectance for skin when diffusely reflected light (SCE), and diffusely and specularly reflected light (SCI) are measured.
13. Spectral reflectance curves (SCE): (a) Caucasian; (b) Asian; and (c) Negroid complexions; (d) average curves for each group.
14. RGB pixels of an AD image extracted from an unsaturated skin part (nose). AD means that the image is taken under daylight illuminant D when the white balancing of the camera was done under incandescent illuminant A.
15. Testing the colour correction: (a) the original AD image, and (b) the overclipping image in which black means pixels with one or more overclipped channel and white shows those pixels which do not have overclipped values.
16. Colour correction results for (a) RGB eigenfaces, which produce a greenish cast on some pixels, and (b) RGB eigenfaces and method of ratios. Both of these images are geometrically normalized due to the requirements of PCA. (c) is the original image in which both white balancing and the prevailing illumination type is incandescent A.
17. An image series for the Nogatech camera.
18. Extracted facial skin areas.
19. Skin locus obtained from extracted facial skin areas.
20. The three first basis functions of colour signals using a Planckian set with (a) simple scaling normalization and (b) Euclidean normalization, and an artificial set with (c) simple scaling and (d) Euclidean normalization.
21. First three coefficients of basis functions form a quadratic slope when a Planckian set with illumination normalization of scaling is used. The viewing angle is set to maximize the visibility of the slope and due to this the coefficient 2 axis is almost perpendicular. The calculated coefficients of basis functions for one light source situation (referred to as the original and marked with *) and for different mixtures of two light sources (referred as the new illuminant and marked with .) like combinations of Planckians of 2300K and 6500K; and of 2400K and 6000K etc.
22. The mark ‘*’ shows coefficients of basis functions for one light source situation. It is easy to show that the coefficients for mixture illumination can be obtained by a linear combination of the coefficients from one light source case. Here the mark ‘.’ shows calculated coefficients for different combinations of three Planckian illuminants (2300 K, 3400 K and 6500 K) and the linear relationship between the coefficients from one light source case is obvious.
23. Computed skin loci for the Sony camera: (a) with all data, (b) with the first three basis functions and (c) with the first five basis functions. The light sources were taken from an artificial group. The r and g chromaticities shown in the axes are parameters of the normalized colour coordinates obtained by a conversion from the calculated RGB values.
24. Winnov locus in NCC rgb from (a) original data and (b) data associated with values higher than 25 in each channel (maximum value 255).
25. There was no single cluster for (a) HS-chromaticities in Cartesian coordinates or (b) TS-chromaticities.
26. Skin chromaticities of HS-space plotted in polar coordinates.
27. Face localization by (a) a box and (b) a polygon.
28. Skin locus for (a) the Nogatech camera and (b) the Alaris camera. The straight or dashed lines visualize the pair of quadratic functions used to define the upper and lower bound for the skin chromaticity cluster. (b) also shows the white point and the surrounding to be excluded from the image (marked with *).
29. Behavior of the ratiohistogram. The upper image row shows how the ratiohistogram of a face image is affected by the colour appearance of skin. The skin locus is visualized with two straight lines. The reddish colour in the upper row means high values whereas the blue indicates low values.
30. Adaptive skin locus based face tracking for video sequences of the Nogatech webcamera: (a) indoor conditions, and (b) outdoor conditions (tracked with a limited skin locus). White bounding boxes indicate localized face whereas magenta boxes show the search area.
31. A few selected frames from the skin locus based adaptive face tracking for a video taken by an Alaris webcamera. White boxes visualize localization of the bounding box and magenta boxes show the search area.
32. Numerical evaluation of tracking results. The skin locus was used as a constraint on the histogram adaptation. The blue line indicates the goodness of the tracking result (‘A’) and the dashed line shows the error count value (anything over 1 means a frozen bounding box).
33. Face tracking with fixed histogram. The magenta box shows the search area while the white box and cross display the face localization.
34. Numerical evaluation of tracking results. The skin locus was used as a constraint on the histogram adaptation. The blue line indicates the goodness of tracking result (‘A’) and the dashed line shows the error count value (anything over 1 means a frozen bounding box).
35. Adaptive tracking using an elliptical constraint. The magenta box shows the search area while the white box and cross display the face localization.
36. An example of displacement for elliptical constraint. Ellipses show a displacement in four directions, up, down, right and left, while the x are the middle points and the white dashed line displays the manually selected ground truth bounding box.
37. Error D due to relative displacement eB for adaptive tracking schemes based on elliptical and skin locus constraints.
38. Skin locus for the Alaris camera.
39. A few visualizations of the goodness and error measure.
40. Localization error of the bounding box for three different tracking methods with a static colour model, an adapted model based on geometrical constraint and an adaptive model based on chromaticity constraint.
41. True positives for video 1.
42. False positives for video 1.
  Next
  Acknowledgements

Homepage of this publication

Library Units | Collections | Databases | Library News | Library Services | Electronic Collection | Links elsewhere | Alphabetical Index

© 2002 Oulu University Library