Assessment of user-centred design processes as a basis for improvement action

An experimental study in industrial settings

Abstract

Many software-intensive products and systems on the market today reveal a poor level of usability despite the availability of knowledge about how to develop good usability, namely user-centred design. Improving the status of user-centred design has proved to be a challenge in product development. An effective step to start out on the road to such improvements is to carry out a current state analysis. Our research problem was to learn how to perform an effective current state analysis, i.e. an assessment, of user-centred design processes to provide a basis for improvement action. As the main part of our research, we carried out five experimental assessments in industrial settings during the years 2000 and 2001.

The main result of the research is a novel assessment approach. The objective of the approach is to provide a basis for improvement of the performance of user-centred design in individual product development projects. The particular features of the approach are a new user-centred design process model, a three-dimensional process performance model and implementation of the assessment in the form of a workshop with extensive participation by the project members. The user-centred design process model is method-independent, and consists of six processes that are defined through outcomes. The performance of processes is assessed from the viewpoints of quantity, quality, and integration.

Our theoretical examinations show that different assessment categories can be identified depending on the different foci and purposes of an assessment. Moreover, it may even be appropriate to have different assessment approaches for different categories. In the category of our focus and purpose, i.e. 'assessment of user-centred design processes for performance improvement', the focus of an assessment should be in the substance rather than in the management of UCD processes. An assessment approach should be regarded as an artefact and it should be subject to ongoing refinement. An assessment should be an intervention that provides both an effective training session and a meaningful experience to the participants. Clarity in both the concepts and the results seems to take precedence over having standard reference models.

In addition, we created a preliminary theory of usability capability. It identifies three dimensions of usability capability: user-centred design infrastructure, performance of user-centred design in product development projects and usability in business strategy. We also propose some new assessment artefacts for other assessment categories and contrast our research experience with established research frameworks.

Table of Contents

Acknowledgements

I. SETTING THE SCENE

1. Introduction

1.1. Research area
1.2. Motivation
1.3. Results and findings by other researchers
1.4. Research setting
1.5. The problem definition
1.6. Structuring of the research

2. Research framework

2.1. Research framework of March and Smith
2.2. Research framework of Järvinen & Järvinen
2.3. Interpreting the artefacts in the domain of UCA
2.4. Interpreting the research activities

3. Overview of the existing UCA approaches

3.1. Illustrating usability capability assessment procedure
3.2. Overview of the literature
3.3. Main features of the existing UCA approaches
3.4. Summary

4. Theorising about usability capability

4.1. Implementation of UCD in a product development project
4.2. Mechanism for ensuring routine use of UCD
4.3. Keeping infrastructure of UCD at competitive level
4.4. A proposal for dimensions of usability capability model

II. EXPERIMENTS

5. Assessment at Buscom in May 2000

5.1. Building activities
5.2. Implementation of assessment
5.3. Feedback
5.4. Conclusions
5.5. Implications for further assessments

6. Assessment at Teamware in June 2000

6.1. Implementation of the assessment
6.2. Evaluation of assessment
6.3. Conclusions
6.4. Implications for further assessments

7. Assessment at NET in October/November 2000

7.1. Building activities
7.2. Implementation of assessment
7.3. Feedback
7.4. Conclusions
7.5. Implications for further assessments

8. Assessment at NET in December 2000

8.1. Building activities
8.2. Implementation of assessment
8.3. Feedback
8.4. Conclusions
8.5. Implications for further assessments

9. Assessment at NMP in April 2001

9.1. Building activities
9.2. Implementation of assessment
9.3. Feedback
9.4. Conclusions
9.5. Implications for further assessments

10. Summarising the experiments

10.1. Evolvement of the assessment approach
10.2. Drift of focus
10.3. Changes in challenges
10.4. Contrasting the assessment experiences with the ones of the TRUMP project

III. RESEARCH RESULTS

11. The artefacts: An approach for UCD performance assessment (UPA)

11.1. Constructs
11.2. Models
11.3. Methods
11.4. Instantiations
11.5. Benefits of KESSU UPA
11.6. Maturity of KESSU UPA
11.7. Contrasting KESSU UPA with ISO 18529 and QIU/HSL

12. Theoretical examinations

12.1. Understanding assessments
12.2. Model of usability capability

13. Research framework – lessons learnt

13.1. Artefacts
13.2. Research activities
13.3. Summary

14. Discussion

14.1. Flow of research
14.2. Summarising the research results
14.3. Limitations
14.4. Implications for practitioners
14.5. Proposals for new research topics

List of Tables
1. Evaluation criteria of artefacts
2. Interpreting research outputs in the domain of UCA
3. Ehrlich and Rohn define four stages of acceptance of user-centred design.
4. Examples of usability practices at different levels of capability in Trillium
5. The IBM model consists of nine attributes that fall into three categories
6. Structure of HUSAT model
7. Capability areas and attributes of the HUSAT model
8. Examples of management practices at different levels of capability in INUSE HCS
9. There are six levels of capability in the INUSE Processes model.
10. Classification of the UCA approaches
11. Suitability of the different assessment approaches for different purposes
12. Support for practitioners
13. Feedback on usability and understandability of processes (scale 1 low … 7 high).
14. Feedback on the usefulness of the assessment (scale 1 low … 7 high).
15. Perceived understandability of different concepts (1 low – 7 high)
16. Feedback from persons whose work is related to user-centred design activities
17. Lessons learnt about the constructs
18. Lessons learnt about the model
19. Lessons learnt about the method
20. Conclusions related to the constructs
21. Conclusions related to the model
22. Conclusions related to the method
23. The analysis of the context of use
24. Evaluation criteria derived from context of use analysis
25. Qualitative feedback from results reporting session
26. Conclusions related to the constructs
27. Conclusions related to the models
28. Conclusions relating to the methods
29. Conclusions related to the constructs
30. Conclusions related to the model
31. Conclusions related to the methods
32. Summary of assessments
33. Drift of the focus of assessments
34. New constructs provided by KESSU UPA
35. Part of UCD-PP Findings table
36. Contrasting artefacts of KESSU UPA, ISO 18529 and HSL
37. Illustration of differences between KESSU and HSL process definitions: Context of Use Process
38. Evaluation criteria of artefacts

List of Figures
1. Activities of user-centred design according to ISO 13407 (ISO/IEC 1999)
2. Relationships between the different artefacts.
3. The research framework of March and Smith (March and Smith 1995).
4. Research framework of Järvinen (x = a viable research effort). (Järvinen 2000).
5. The focus of research is the development of a UCA approach. While, simultaneously, we identify the criteria for, and to increase, successful assessments.
6. Illustration of an assessment cycle.
7. Example of the results of a usability capability assessment with the INUSE Processes approach: the higher the rating, the better the capability of a process.
8. The family tree of the different UCA approaches.
9. Successful UCD requires the presence of many elements in product development projects.
10. Management Demand on Usability Leads to Effective User-centred Design.
11. Model of usability capability: Effective user-centred design requires infrastructure and commitment from business management.
12. Main dimensions of usability capability.
13. Format of describing the experiments.
14. Assessment at Buscom.
15. Example of an SIBC matrix (with arbitrary results).
16. Process model used in the assessment at Buscom.
17. Illustration of a simplified presentation of results (arbitrary).
18. Flow of the assessment at Teamware.
19. Assessment at NET.
20. The interpretation and interview cycle.
21. UCD process model developed during the interviews.
22. The process performance dimensions.
23. The style of visualising the quantitative results (arbitrary).
24. Presenting qualitative findings: two examples (arbitrary). In presentation, one set of qualitative findings was shown at a time.
25. Process model that evolved in the reporting phase of the assessment.
26. Quantitative feedback from results reporting session.
27. Implementation of the assessment of the customer documentation at NET.
28. Instantiation of the assessment at NMP.
29. KESSU improvement model: UCD improvement action may take place in different organisational contexts.
30. A past (reference) project is assessed to provide a basis for planning UCD activities in a customer project.
31. Introduction to assessment in the interview session.
32. Overview of the main steps in the development of the assessment approach.
33. KESSU UCD Processes model: the latest version.
34. KESSU UCD Processes model: processes and their outcomes.
35. KESSU process performance dimensions with scales of rating.
36. Visual representation of quantitative results of assessment in UCD-PP Profile diagram.
37. Example of UCD-PP Profile in a simple case. The quantity performance dimension is presented only.
38. An example of UCD-PP Profile in a more comprehensive case. The quantity performance dimension is presented only.
39. Phases of carrying out an assessment.
40. Assessment categories.
41. Characteristics of assessment approaches in different categories.
42. Breakdown of assessment approaches into different categories. The bigger font, the better performance.
43. Dimensions of Usability Capability.

		Next
		Acknowledgements

Homepage of this publication

Library Units | Collections | Databases | Library News | Library Services | Electronic Collection | Links elsewhere | Alphabetical Index

© 2000 Oulu University Library