| Experimental ergonomic evaluation with user trials: EEE product development procedures | ||
|---|---|---|
| Prev | Chapter 5. Results | Next |
This method is more complicated with its two stages: weighting of criteria and the possibility to rank the products as far as their properties assessed by each criterion are concerned. In the chair experiment (papers IV and V), Mitchell’s paired comparison method was used to derive the criterion weights for eight different criteria of the chair. Further, in the second part of Mitchell’s method (paper IV), the criterion weight was multiplied by the value assigned to each of the three chair configurations to obtain a final configuration score for each chair. After that, the chairs could be compared to each other.
Inter-rater consistency showed a strong agreement between the subjects in their ratings of criterion importance. The intra-rater consistency as the degree of agreement in criterion weights was good, in some cases even excellent. The differences in ranking were consistent.
Mitchell’s paired comparison method gives an opportunity to evaluate how people rank and rate product criteria and to compare different product variants. The experiment is based on a simple step-by-step design, in which a subject compares one pair at a time and chooses the product or item he/she prefers. If very detailed and multiple criteria were used in Mitchell’s method, the comparison of pairs would take too much time. The number of judgements required per subject increases by a factor of [n (n - 1)/2], as n, i.e. the number of entities, increases (Sinclair 1995). This may require considerable time, and the bored subject towards the end of a session is a very different person from the keen, perhaps slightly apprehensive, subject who started the session. This was not true in the chair experiment (paper IV), however, because n was only eight and the comparisons were easy to make. The experiment convinced the authors of both the advantages and the disadvantages presented by Mitchell (1992).
No more than eight features should be weighted. Moreover, the number of products to be compared in the second part must be kept to the minimum, since the number of judgement required per subject increases by a factor of [n(k-1)!], as k, i.e. the number of product variants, increases.
The EEE4 procedure can be described as follows:
Experts collect data on the product and its users.
Experts as a team formulate criteria for the product, the recommended maximum total number of criteria being 8.
End-users compare the criteria in pairs, as proposed by Mitchell (1992), supported by sketches. Paired comparison along with sketches and kind instructions are means of effective communication. Each subject’s Mitchell ranks can be seen as entries to a preference matrix (see paper IV Table 4 and paper V Table 3).
By summing up the entries, the total preference matrix can be statistically tested, and the statistical significance of the agreement between the users can then be determined.
The Mitchell votes for each criterion can be directly calculated and transformed into exact weighting factors.
End-users compare real products (max. number 4) two at a time for every criterion, as proposed by Mitchell (1992). The results can be calculated as values.
The weightings and product criterion values can be used to calculate the final score for every product.