Experimental ergonomic evaluation with user trials: EEE product development procedures

Heli Kirvesoja

Department of Process and Environmental Engineering, University of Oulu

Abstract

The main difficulty in the usability evaluation of a product concept or a prototype is that it is very difficult to take reliably into account many different characteristics or attributes, which cannot be measured with the same unit, scale or instrument. Secondly, evaluation always involves some uncertainty. One can never be really sure that all the essential aspects that affect the final user emphasis are taken into consideration. This uncertainty can be decreased by involving diverse people in the evaluation process during R&D or, before the final decision, by elaborating the best ideas to a level at which they can be really used on a pilot scale in the field. Most often, though, the latter takes too much time and would result in many other problems. Evaluation can possibly be best enhanced by using enough involved people, i.e. various experts and especially end-users, who need or use the product in question. A key ingredient to the success of product development, in addition to ergonomic knowledge, is often thought to be active involvement of the intended product users by (1) measuring user-product interaction and (2) participation in design decisions. When a potential end-user experiments with the product, both quantitative and qualitative evaluation methods can be used. This thesis shows some methodological possibilities of evaluation, especially through simulation. It also describes in detail the practical phases of the experiments. For example, a lot of development was needed to find out how to communicate product alternatives and their concepts to (elderly) users. And most importantly, this thesis aims to give evidence of how the procedure called experimental ergonomic evaluation (EEE) should be feasibly implemented and statistically confirmed for significance and consistency.

A special focus in the experiments was placed on elderly end-users. Since the number of elderly citizens is increasing, there is a need for products to help the elderly live independently at their homes. Studying and understanding how users accomplish their tasks helps to identify their needs and to formulate implications for the design of technology to satisfy those needs. Thus, user studies conducted before beginning to design a new technology provide a proactive way of involving users in the design process. The first prototypes then enable usability studies, such as user trials. With an emphasis on usability engineering, trials can be developed into more feasible EEE procedures for industrial companies. All the developed and applied EEE procedures were based on a user-centred approach with different user trial types (N = 15). The users as subjects (N = 264) performed as real tasks as possible and, based on their perceptions during the trials, gave their preferences or scored certain variables. The subjects were also observed and measured by the researcher. The products or other technologies in the trials comprised a total of 9 cases, ranging from “low-tech” steps and chairs to “high-tech” information and communication technology (ICT) applications. The perceived preference and observed performance measures were then combined. Different methods are needed simultaneously to make the results more accurate. The present EEE procedures proved to be cost-effective, efficient and sufficiently valid at least in a research context. The EEE procedures ranged from subjective estimations, such as rating and ranking, to more complex multi-criteria methods that can be used to facilitate decision-making, such as conjoint analysis, Michael’s paired comparison and use-value analysis. Objective evaluation was also used, including measurement of products and users as well various observations. Both experts and end-users (subjects) had their own important roles in the experiment. Based on this study, EEE procedures are easy to implement in industry for routine usability testing in the course of product development. EEE with its wide coverage yields more universal and absolute usability values, not only ones based on direct benchmarking.

Table of Contents
Acknowledgements
Symbols and abbreviations
Key definitions
List of original papers
1. Introduction
1.1. Basics of user-centred product design
1.2. Research approach and study design
1.3. Scope of the study
2. Background review of evaluation with user trials
2.1. User-centred product design
2.2. A trend among the user population: ageing - gerontechnology
2.3. User trial and usability testing
2.4. Performance vs. preference
2.5. Subjective vs. objective
2.6. Simulation
2.7. Methods to help evaluation and decision-making
2.7.1. Unit measurement by an instrument (I, II, III, IV)
2.7.2. Rating (II, VII)
2.7.3. Ranking (VI)
2.7.4. Paired comparison (IV, V)
2.7.5. Conjoint analysis (III, IV)
2.7.6. Use-value analysis (V)
3. Objectives of the study
4. Materials and methods
4.1. Subjects
4.2. User trials of various technologies
4.3. Methods and procedures
4.3.1. Unit measurement by an instrument (I, II, III, IV)
4.3.2. Rating (II, VII)
4.3.3. Ranking (VI)
4.3.4. Michael’s paired comparison (IV, V)
4.3.5. Conjoint analysis (III, IV)
4.3.6. Multi-criteria evaluation procedure (V)
4.3.7. Statistical methods
5. Results
5.1. Unit measurement by an instrument (I, II, III, IV)
5.2. Rating (II, VII)
5.3. Ranking (VI)
5.4. Mitchell’s paired comparison (IV, V)
5.5. Conjoint analysis (III, IV)
5.6. Multi-criteria evaluation procedure (V)
6. Discussion
6.1. Experimental ergonomic evaluation (EEE) procedures
6.2. Evaluation of procedures
6.2.1. Consistency
6.2.2. Validity
6.3. Evaluation with user trials
6.3.1. Recommendations and legislation
6.3.2. Technology and markets
6.3.3. Approaches
6.3.4. Context
6.3.5. Simulation
6.3.6. Subjects
6.3.7. Methods
6.4. Relevance to industry
6.5. Future needs for research
7. Conclusions
References
List of Tables
1. Classification of data gathering methods, modified from Meister and Rabideau (1965 presented by Sinclear 1995).
2. The elderly study subjects, representing users.
3. The study subjects other than the elderly.
4. Targets to be evaluated in the papers and places of user trials.
5. Comparison of the scaling methods commonly used in product design research (Cushman & Rosenberg 1991) and the paper in which the method is used in this study.
6. The significance of the results was clarified with help of different statistical methods.
7. Consistency of the results.
8. Applicability of the EEE procedures generated in the study.
List of Figures
1. User-centred design: the ergonomics system consists of the product, the user and the task in interaction (Pheasant 1996).
2. A balanced model of a work system (Smith & Sainfort 1989).
3. A rationalistic design model is shown in the middle of the figure (Cohill 1991, Simon 1969). The model was supported by both ergonomic knowledge (left) and ergonomics method (right) (Väyrynen et al. 1999).
4. Interdependence of human-centred design activities (ISO 13407 1999).
5. Mind map of the main background factors of the study approach described by key terms.
6. Papers related to the methods used in this study. * Patients" experiences of video consultation at the implementation phase.
7. Main phases in a typical user trial (McClelland 1995).
8. Motivation process of consumers (Kemp & van Gelderen 1996, adapted from Engel et al. 1990).
9. An example of the different scaling methods used in the experiments.
10. An example of an EEE1 procedure with generalisation: the process of measuring the suitable work surface height in a task-surface experiment (paper II).
11. An example of an EEE2 procedure with generalisation: the process of rating fixture height in the home simulator and observations by an expert in the task-surface experiment (paper II).
12. An example of an EEE3 procedure with generalisation: the process of ranking telephone types for the elderly in the telephone experiment (paper VI).
13. An example of an EEE4 procedure with generalisation: the process of Mitchell’s paired comparison in the chair experiment (paper IV).
14. An example of an EEE5 procedure with generalisation: the process of conjoint analysis in the assessment of microwave oven workstations (paper III).
15. An example of an EEE6 procedure with generalisation: the process of multi-criteria evaluation in the chair experiment (paper V).
16. The EEE cluster of procedures in view of main features and application in this study.
17. Value profiles for the comparison of two adjustable wrenches (Väyrynen 1996).
18. Aims of ergonomics in product development (Wilson 1995).
19. The EEE application of the Deming and Taguchi cycle presented by Logothetis (1992) and a short illustration of main phases of the EEE.
  Next
  Acknowledgements

Homepage of this publication

Library Units | Collections | Databases | Library News | Library Services | Electronic Collection | Links elsewhere | Alphabetical Index

© 2000 Oulu University Library