| Experimental ergonomic evaluation with user trials: EEE product development procedures | ||
|---|---|---|
| Prev | Chapter 5. Results | Next |
The multi-criteria evaluation procedure is a hybrid one, because it combines the expertise of both experts and end-users. It can be used as a systematic tool for user participation during designing and testing, i.e. users are not only subjects but also evaluators, designers and decision or trade-off makers. The multiple criteria were ranked in an order of importance by the subjects, i.e. in this case by the elderly end-users. The evaluation procedure, including the quantitative importance of each criterion, could thus be determined for a product, a multi-purpose chair for aged people. Concrete illustrations and user-testable prototypes were utilised to express how the product in question looks and feels.
The weightings of criteria were done with Mitchell’s paired comparison. The score values were attained with use-value analysis. With the use-value analysis, the properties of each product alternative corresponding to every criterion can be measured as magnitudes of parameters (concrete quantitative) or as subjective verbal assessments (qualitative). In both cases, the same fixed quantitative scale (0 to 4) can be utilised, as was also done in the paper V.
The evaluation procedure was applied to the three prototype chairs by three researchers. The experiment was successful and gave reliable results. The inter-rater consistency of the scores was found to be moderate.
The following procedure for multi-criteria evaluation was developed (based mainly on the use-value analysis, partly supported by Mitchell). The results in paper V supported the applicability of the procedure. The steps of the EEE6 procedure are:
Experts collect data on the product and its users.
Experts as a team formulate a hierarchic objective tree for the product.
The final objectives are listed as criteria, their total number being 8-15.
The multiple criteria are ranked through paired comparisons by end-users, done as proposed by Mitchell (1992), supported by communicative real prototypes and sketches. Each subject"s Mitchell ranks can be seen as entries to a preference matrix (see paper IV Table 4 and paper V Table 3).
By collecting and summing up all entries, the total matrix can be statistically tested, and the statistical significance of the agreement between the users can be determined.
The Mitchell votes for each criterion can be directly calculated and transformed into exact weighting factors.
Each criterion in the specification can be operationalised as measurable properties of a product (quantitative or qualitative). Further, each property attribute can be transformed into effectiveness scores, which are presented on a five-point scale of 0-4.
An evaluation document can be collected, including a list of criteria, their weights and the procedures and forms that facilitate various score ratings.
Each property of all product variants can be scored according to the evaluation document. The overall value (utility) based on multiple criteria can be calculated and compared between product variants.
Thus, a sounder basis for decision-making concerning design is available, because the presented evaluation method is more quantitative and transparent. It helps to make systematic optimal user-preferred trade-offs in the context of modern user-centred design, utilising co-operation between users and experts.