| Experimental ergonomic evaluation with user trials: EEE product development procedures | ||
|---|---|---|
| Prev | Chapter 2. Background review of evaluation with user trials | Next |
A question that often arises during the planning of an experiment or a test is whether to obtain objective performance data or subjective data, e.g. data related to preference setting. Objective performance data are usually preferred for experiments. In addition, they are required for design evaluations whenever the evaluation criteria are objective. Unfortunately, however, objective measurements are frequently more difficult – even impossible - to carry out, and the process of collecting objective data is usually more time-consuming and costly. In contrast, subjective data may be obtained easily, quickly, and inexpensively. The subjective measurement technique also provides the only direct means for the assessment of user opinion and preferences. (Cushman & Rosenberg 1991)
The sources of objective data that are frequently used in user trials can be divided into three categories (McClelland 1995):
direct objective measurements of the user,
directly recorded data resulting from users’ actions, registered by the investigator or by some remote means, such as video or automatic event recording,
data measured directly from the product on the completion of or during the trial.
Many kinds of objective data can be measured when, for instance, all the components of a balanced system are considered (Fig. 2). This system is applicable to both working and living contexts in the field. The same fact is often relevant in simulations (see 2.6). Baber and Stanton (1996) have discussed the strengths and weaknesses of observations.
The typical methods used in subjective measurement (regarding fonts in Sinclear"s example) are (Sinclair 1995):
ranking methods,
rating methods,
questionnaire methods
interviews
checklists.
However, subjective data and preference data must be interpreted with caution. Cushman and Rosenberg (1991) suggested that the following points should be considered when evaluating subjective data:
If the subjects in experiments and tests do not fit the user profile compiled during the planning phase, their opinions and preferences may not accurately reflect those of the intended users of the product. Conclusions based on data obtained from inappropriate subjects may not be valid.
Attitude measures and self-reports may be distorted by biasing factors, such as the ”halo effect”, acquiescence, and cognitive dissonance (Rubinstein & Hersh 1984 in Cushman & Rosenberg 1991).
Subjects’ preferences are affected by events in the recent past.
Cushman and Rosenberg (1991) recommended collection of both objective and subjective data during experiments and tests whenever feasible. Collecting subjective data will add little to the cost of the study, but may provide significant insights not obtainable by objective methods. In addition, subjective data may be particularly useful if objective measurements fail to detect any differences between conditions. This study also emphasised the need for both subjective and objective data to support or complement each other in the evaluation.