The Role of Commitment in Software Process Improvement

Pekka Abrahamsson

Abstract

Software process improvement (SPI) approaches have been designed to produce changes at many levels, i.e. in the strategies, culture and working practices, of software development. Studies have shown that nearly two thirds of all SPI efforts have failed or fallen short of expectations. It is often stated in SPI-related literature and practice that “commitment” to SPI plays an important part in determining whether an SPI endeavor ultimately becomes a success or a failure. However, it often remains unclear what this concept actually means and how it affects SPI.

This thesis argues for a scientifically grounded concept of commitment and delivers a description and a definition of this concept in the context of software process improvement. The elaboration of the concept is based on a literature study, which makes the research done in behavioral psychology and organizational science applicable in the field of software process improvement. This thesis shows that current thinking relies on practical models of commitment, and the empirically validated analysis conducted within this study reveals a number of common misleading assumptions regarding the notion and development of commitment in SPI. On this basis, this thesis suggests that the commitment phenomenon is better explained through what can be called strategic, operational and personal commitment nets. This framework can be used for analyzing the unfolding and alteration of commitment towards a specific entity, in this case a software process improvement endeavor, through time and changing circumstances.

The viability and usefulness of the commitment nets framework is demonstrated through an analysis of four SPI cases in two software organizations. As a result, it is shown that even though the objective features of SPI in terms of costs and benefits may be dominating in the project initiation phase, their role tends to lose strength later on due to an inability of the SPI effort to produce quick and meaningful results, even if these are explicitly sought for. This phenomenon gives rise to a need for enhancing the role of social and psychological drivers. If this is not achieved, SPI activities are likely to cease to exist.

The empirical analysis demonstrates that the use of the commitment nets model enables a more precise analysis of the various aspects involved in the commitment phenomenon than what would have been possible with current commitment models. Commitment, as conceptualized and operationalized in this thesis, makes a significant contribution to the outcome of the SPI initiative. The empirical evidence shows that, eventually, even well-planned SPI initiatives may fail to reach the goals set for them due to changes in commitment nets.


Table of Contents
Preface
Abbreviations
List of original publications
1. Introduction
1.1. Research focus
1.2. Research problem and results
1.3. Overview of the dissertation
2. Software process improvement typology
2.1. SPI Management
2.2. Approaches to SPI
2.2.1. Evolutionary approaches
2.2.2. Norm based approaches
2.2.3. Commitment based approaches
2.3. Improvement types
2.4. Phases in SPI
2.5. SPI outcome
2.6. Summary
3. A typology of commitment
3.1. Commitment in organizational behavior studies
3.1.1. Two schools of thought
3.1.2. Conceptualization of commitment
3.1.3. Commitment drivers identified in OB literature
3.2. Commitment in information systems studies
3.2.1. Two streams of IS related commitment research
3.2.2. Commitment Drivers identified in IS literature
3.3. Summary - towards a typology of commitment
3.3.1. Common aspects of commitment
3.3.2. Definition of commitment
4. The Dynamic structure of the commitment process
4.1. Analysis of dynamic commitment models
4.1.1. Scientific commitment models
4.1.2. Practical commitment models
4.2. Assumptions in current thinking
4.2.1. Causality in commitment process
4.2.2. Controllability of the commitment process
4.2.3. The notion of a singular commitment construct
4.2.4. Commitment as an all-positive phenomenon
4.2.5. Shared development of commitment
4.3. Model of commitment nets
4.3.1. (Re)discovery of commitment nets
4.3.2. Actors in commitment nets model
4.3.3. Commitment drivers
4.3.4. Concern and action
4.3.5. Outcomes
4.3.6. Commitment process
4.4. Summary
5. Research Design
5.1. Research evolution
5.1.1. Metrics focus
5.1.2. Conceptualization
5.1.3. Empirical study
5.1.4. Abstraction
5.2. Research approach and methods
5.3. Research setting and case selection
5.4. Data collection
5.5. Data analysis
5.5.1. Narrative strategy
5.5.2. Alternate templates strategy
5.5.3. Grounded theory strategy
5.5.4. Visual Mapping Strategy
5.5.5. Synthetic strategy
5.5.6. Cross-case analysis
5.6. Summary
6. The role of commitment in four SPI cases
6.1. ntroduction to cases
6.2. Case 1: Improving usability processes
6.2.1. Longitudinal analysis of commitment nets
6.2.2. Empirical conclusions
6.3. Case 2: Improving module testing processes
6.3.1. Longitudinal analysis of commitment nets
6.3.2. Empirical conclusions
6.4. Case 3: Improve geographically distributed software development
6.4.1. Longitudinal analysis of commitment nets
6.4.2. Empirical conclusions
6.5. Case 4: Improving software process improvement processes
6.5.1. Longitudinal analysis of commitment nets
6.5.2. Empirical conclusions
7. Cross-case evaluation
7.1. Analysis of commitment assumptions
7.1.1. Causality and controllability
7.1.2. Singularity and all-positive role
7.1.3. Shared development of commitment
7.1.4. Conclusions
7.2. Commitment process
7.2.1. Operational level commitment process
7.2.2. Means used for creating concern
7.3. Commitment vs. SPI outcome
8. Evaluation of the results
8.1. Implications for theory
8.2. Implications for practice
8.2.1. Voluntary involvement
8.2.2. Embedded SPI
8.2.3. Emphasis on the environment
9. Conclusions
9.1. Summary of the results
9.2. Limitations of the study
9.3. Future research
References
A. Appendix I
List of Tables
1. Generic phases of an SPI project.
2. Success dimensions in SPI (Abrahamsson 2000b).
3. SPI concepts used in empirical analysis.
4. Summary of commitment drivers identified in IS literature.
5. Common aspects of commitment.
6. Assumptions have shaped the SPI research.
7. Actors in SPI context.
8. Drivers for SPI projects.
9. Research approaches utilized.
10. Data collection methods utilized.
11. Strategies for sensemaking used in this study.
12. Summary of the cases.
13. Commitment net drivers in the project initiation phase.
14. Commitment net drivers in the project operation phase.
15. Commitment net drivers in the project closure phase.
16. Case 1: Commitments reflecting the progress of the improvement project.
17. Case 1: Commitment process in the SPI project.
18. Change achieved in actors’ commitment nets.
19. Commitment net drivers in the initiation phase.
20. Commitment drivers in the operation phase.
21. Case 2: Commitments reflecting the progress of the improvement project.
22. Case 2: Commitment process in the SPI project.
23. Case 2 – Focal organization: Management’s commitment net.
24. Change achieved in actors’ commitment nets.
25. Commitment drivers in the initiation phase.
26. Commitment drivers in the operation phase.
27. Commitment drivers in the closure phase.
28. Case 3: Commitments reflecting the progress of improvement project.
29. Case 3: Commitment process in the SPI project.
30. Change achieved in actors’ commitment nets.
31. Commitment net drivers in the initiation phase.
32. Commitment drivers in the operation phase.
33. Commitment net drivers in the closure phase.
34. Case 4: Commitments reflecting the progress of improvement project.
35. Case 4: Commitment process in the SPI project.
36. Change achieved in actors’ commitment nets.
37. Assumptions: Current research and empirical findings mapped.
38. Commitment processes – operational level
39. Means used for transferring a concern to operational level commitment net.
40. Performance of the basic SPI elements.
41. Improvements to commitment nets model.
List of Figures
1. Software process improvement types (Seppänen et al. 2001)
2. Commitment defined
3. . The dynamic commitment model (Newman & Sabherwal 1996)
4. The theoretical model (of commitment) (Sabherwal & Elam 1995)
5. Development of commitment to change (Conner & Patterson 1982).
6. Development of management commitment to "new thinking" (Huge 1990).
7. Commitment Nets Model
8. Research evolution phases.
9. An extract from a case narrative (case two).
10. A foreword email for case narratives (case four).
11. An example of practitioners’ critique concerning a case narrative (case four).
12. A draft model of commitment nets.