Colour is a very useful cue for object characterization, detection and localization. Because it is a low level feature, it is computationally inexpensive and therefore suitable for real-time applications. In addition, it is robust to occlusion and rotation. Unfortunately, colour has also a serious drawback, illumination dependency, and due to this it had limited use in practical applications. Many colour correction and colour constancy schemes have been presented but up to now, they have been unsuccessful for machine vision applications working in real world conditions. It is therefore important to develop techniques which can cope with these situations.
This thesis investigates the use of illumination robust colour techniques on facial skin. Faces were selected as a target because they are very important and common objects in images and video sequences. First, the data on the facial skin needs to be collected under varying illumination conditions. To do this, two databases have been created: the Physics-based Face Database and the Face Video Database. The Physics-based Face Database has images of 125 persons with pale, yellow and dark skin tones. The light sources and their SPDs are known. The images are taken with four white balancing illumination and four prevailing light sources. This makes it possible to compare not only between different canonical images but also between different unbalanced conditions. The skin chromaticities in these unbalanced images differ strongly from those of skin tones. It was also noticed that the direction of illumination colour temperature change effects the results. If the prevailing light source has a higher colour temperature, then the colours in the images shift towards blue. In the opposite case, a shift toward red is observed. In addition to images, the reflectances of skins were measured with a spectrophotometer. Because also the responses of the camera were known, this makes possible a physical modelling of image formation. The Face Video Database contains images as well as videos taken by several cameras. The images are taken under the same conditions as earlier with the Physics-based Face Database. The videos are imaged both under indoor and outdoor lighting conditions, with challenging illumination changes.
Next, the collected data, spectral reflectances and images are analysed and shown to be useful in different applications. The evaluation of skin spectral uniformity revealed that in general, it is quite uniform over the pure facial skin. In some cases, the cheeks’ and forehead’s reflectances have minor differences but this might be due to difference in the structure of the positions. The matt approximation of skin seems to be quite valid because the difference between SCI and SCE measurements was only 1-1.8 % per wavelength. This is very close to the result (5 %) given in the literature. The reflectances from facial skin seem to be very smooth, slowly varying and similar in shape for different skin tone groups.
Before studying the possibility to model skin colour change, the different colour spaces were evaluated. The choice of a proper colour space is an important part of any algorithm and it can be evaluated using criteria like discriminability of colours or effect of colour changes caused by illumination. For example, if the colours cannot be separated in RGB, they are not that even after conversion to a human colour space. On the other hand, this nonlinear conversion can make two discriminable colours in RGB more difficult to separate. In addition, the transformation matrix to a human colour space is illumination dependent and therefore it cannot be used for colour constancy. Due to these reasons, one should consider very carefully the use of human colour space in those applications which do not require evaluation of human vision point of view. If the evaluation is important, then the method presented in Paper I can be used as a selection criteria for cameras. Because in our research evaluation was not important, the skin chromaticities from different balanced and unbalanced cases were investigated only in 17 device dependent colour spaces. The intensity data was excluded because it is very sensitive to environmental changes. It was noticed that many colour spaces are suitable for skin colour modelling if the criteria were overlapping between skin chromaticity regions of two different skin tone groups, the total area occupied in the space and the uniformity of the chromaticity blob in the space. The overlap of skin chromaticities has some dependence on the camera used and between all cameras it was not very high. But if the comparison was done between cameras with a gain controller and without a gain controller, the overlap increased. From these colour spaces, NCC rgb were selected for further use because of good overall performance. The skin chromaticities in NCC rg were found to be modellable with two quadratic functions and in NCC rb with three straight lines. The purpose of the modelling was to find a limited region for possible skin chromaticities under varying illumination (chromatic constraint). It was also shown that basis functions of skin colour signals can be used for this kind of modelling and evaluation of colour appearance under mixed illumination condition.
The knowledge obtained from data is shown to be useful in three applications. The first application is skin colour correction in colour images with severe overclipping. This technique requires for the overclipped image that there is an unclipped area and at least one channel is free from clipping in the other areas. The unclipped area is used to define ratios between channels and these ratios are used to approximate the pixels’ values in overclipped channels. After removing overclipping, the image is subjected to PCA. To correct skin colours so that they have canonical appearances, the first coefficients of the obtained eigenfaces are replaced by those obtained after applying PCA to a canonical image. In the second application, faces of persons are tracked in video sequences under drastic illumination changes using a chromaticity constraint. The chromaticity constraint is used to select pixels for updating the skin colour model of the face to be tracked. It does not assume any probability for a skin colour; it just defines if a camera can perceive a skin with a chromaticity pair. This adaptive tracking is shown to be superior to fixed model based tracking and to adaptive tracking with spatial pixel selection. In the last application, the tracking and segmentation are combined: the chromatic constraint is used to filter out non-skin coloured pixels to achieve the segmented image. The segmented image is again used in face tracking, and the localization is utilized in updating the colour model for the face and approximate the possible range of chromaticities in the next frame.
These results confirm the validity of the main thesis statement: knowledge about an object’s colour, like skin colour changes under different illumination conditions, can be used to develop more robust techniques against illumination changes.