1.2. Research approach and study design
Almost every decision made during the process of designing a system includes some evaluation (Roozenburg & Eekels 1995, Sanders & McCormick 1993). In addition to design, evaluation is also important when an organisation or an individual compares products in testing or purchasing situations. Sanders and McCormick (1993) have discussed the characteristics of ergonomics studies, which can be divided into (a) descriptive studies, (b) experimental research and (c) evaluative research. Descriptive studies seek to characterise a population in terms of certain attributes by, for example, surveying the dimensions of people’s bodies and the weights of boxes people are willing to lift. The purpose of experimental research is to test the effects of some variables on behaviour, e.g. to find out how well people can edit manuscripts with partial-line, partial-page, and full-page computer displays. The key issue in experimental research is usually whether a variable has an effect on behaviour and the direction of that effect. Evaluative research is similar to experimental research in that it aims to assess the effect of “something“. In evaluative research, the ”something” is usually a system or a product. Evaluative research is also similar to descriptive research in that it seeks to describe the performance and behaviours of the people using the system and product.
The main emphasis of this study was on means of evaluative research, but some experimental research was also involved. These methods allow a smaller number of subjects than descriptive studies. Descriptive studies were made to take anthropometric measures of the elderly subjects. However, because the number of subjects was quite small, no major conclusions concerning the whole population can be derived. That data were aimed at giving values for background and control variables.
In an experiment, the investigator sets users a series of tasks and quantifies their performance in terms of, for example, the time spent on the task and the number of errors (Jordan & Kerr 1993). Experiments need to be strictly controlled, which is why the measurements taken are fairly precise. Chapanis (1959, quoted by McClelland 1995) defines the experimental method as a series of controlled observations undertaken in an artificial situation with the aim of deliberately manipulating some variables in order to confirm one or more specific hypotheses. The laboratory context enables control of the experiment. The laboratory was used to simulate the actual use context of the products in most papers of this study (papers I, II, III, IV and V).
According to the international standard for interactive IT systems (ISO 13407 1999), the incorporation of a human-centred approach involves the following characteristics (Fig. 4):
active involvement of users and a clear understanding of user and task requirements,
appropriate allocation of functions between users and technology,
iteration of design solutions,
multi-disciplinary design.
The nature of iterative design is based on support from the management, and the process is actually a repeated cycle, in which understanding of user needs and contexts of use, user requirements, and design solutions is elaborated until the system satisfies the specified user and organisational requirements. Fig. 3 presents another possible R&D process with iterations. Figures 3 and 4 as well as common sense propose that the iterations and evaluations are normally closely related to each other.
This study concentrates on the item ”evaluate design against requirements” in Fig. 4, i.e. on the need to conduct various usability studies (Fig. 3). Product evaluation has been defined as the measurement of products to verify that they will do what they are intended to do (Meister & Rabideau 1965). The alternatives, e.g. product variants, must meet all criteria as well as possible. The evaluation and the subsequent decision-making are based on the determined “value” or “worth” of the alternatives in the light of the aims that were set (Roozenburg & Eekels 1995). The product is valuable to the extent that it meets the objectives of design specifications. Evaluation is an essential step in human-centred design and should take place at all stages of the system’s life cycle (ISO 13407 1999). Systematic and comprehensive evaluation is needed, especially during the conceptual design and embodiment design stages (Pahl & Beitz 1988) of the product design process. Evaluation can be used (ISO 13407 1999):
to provide feedback to improve design,
to assess whether the user and organisational objectives have been achieved, and
to monitor the long-term use of the product or system.
A product can and will naturally be evaluated by the designer him/herself. When inspecting the product, he or she utilises heuristic rules, detailed checklists, his/her own experiences and various research results and literature. During the product development process, however, the designers should be interested in the kind of user behaviour that their design provokes, so that they can improve their design, if necessary, based on experimental testing.
Fig. 5 shows all the key terms used as the framework in this study. The main emphasis of this study is on the methods that can be used to assist decision-making, with an emphasise on the user aspects.
The recommendations, legislation and standards give a framework and guidelines for development work. The development is also influenced by the different stakeholders who develop the technology and the markets with the need for the product. Both users and experts are essential for a successful evaluation of product variants or systems with different approaches. Simulation helps the users to evaluate the product. Evaluation of the product in its actual context or as close to the actual circumstances as possible is important for validity. What we truly need is easy, quick and cheap evaluation methods to help us attain the aims.
Essential key terms, in addition to those listed in Fig. 5, include product characteristics and iterative design. Market researchers tend to concentrate more on specifying the desirable attributes of a new product, whereas designers and engineers concentrate more on the product’s engineering characteristics (usually in terms of its physical properties) (Cross 1989). Other synonyms include parameters and features, whose values and magnitudes correspond to the levels. The terms ‘criterion’ and ‘objective’ are related to the requirement specifications. In this study, a variety of terms are used, depending on the method applied. In the same way, the final score or overall value indicates the final worth of the product or system to be evaluated. The model of design mainly used in this study is the one introduced by Pahl and Beitz (1988): clarification of the task, conceptual design, embodiment design and detail design (see paper V Fig. 1).
