| The paradigms of e-Education: An analysis of the communication structures in the research on information and communication technology integration in education in the years 2000–2001 | ||
|---|---|---|
| Prev | Chapter 3. The design of the research | Next |
In qualitative research the requirements of validity and reliability are under enthusiastic discussion. There are interpretations that these traditional measures of reliability are not applicable at all in qualitative research because of the nature of the methods and epistemological assumptions of the research, which promote the uniqueness of the research. On the other hand, there are also demands for using the same criteria for qualitative and quantitative research when evaluating the trustworthy of the research. Between these poles are many different variations for justifying the results of the research. However, the issue of trustworthiness cannot be avoided whatever the epistemological approach of the research (Gibbs 2002, 13).
In theory, trustworthiness, i.e. credibility and validity of qualitative research can be considered from two different perspectives depending on the epistemological foundation of the research and the epistemologically biased arguments of the evaluation. Ideally, both of the considerations are based on the same epistemological foundation. In many cases, the external evaluation of the trustworthiness of the research has a different epistemic basis than the analysis itself, which can be confusing.
The researcher can influence directly only the epistemological basis of the research, not the evaluator’s bias, but of course clear criteria stated by the researcher makes it more valid for the reader to evaluate the trustworthiness of the research. Therefore it is very important that the researcher him/herself will set a clear basis for the evaluation. In this chapter, the problem of the credibility and validity of this research will be explored and criteria for trustworthiness set in an epistemologically relevant context.
According to Johnson (1997), in qualitative research three types of validity can be discussed. First, descriptive validity refers to the factual accuracy of the account as reported by the qualitative researcher. Second, interpretive validity is obtained to the degree that the participants’ viewpoints, thoughts, intentions, and experiences are accurately understood and reported by the qualitative researcher. Third, theoretical validity is obtained to the degree that a theory or theoretical explanation developed from a research study fits the data and is, therefore, credible and defensible. To demonstrate these measures of validity, Johnson (1997) describes thirteen strategies used to promote research validity in qualitative research (table 3).
Most of the strategies mentioned by Johnson (1997) argue from two different presumptions. First, implicitly the strategies take the assumptions of validity from quantitative research as a starting point, which aims to avoid subjectivity in research results. Second, strategies usually presuppose that the source of information in qualitative analysis is subjective expressions of the research informants, from which the researcher then will interpret more or less objective meaning.
Table 3. Strategies used to promote qualitative research validity (Johnson 1997).
| Strategy | Description |
| Researcher as "Detective" | A metaphor characterizing the qualitative researcher as he or she searches for evidence about causes and effects. The researcher develops an understanding of the data through careful consideration of potential causes and effects and by systematically eliminating "rival" explanations or hypotheses until the final "case" is made "beyond a reasonable doubt." The" detective" can utilize any of the strategies listed here. |
| Extended fieldwork | When possible, qualitative researchers should collect data in the field over an extended period of time. |
| Low inference descriptors | The use of description phrased very close to the participants’ accounts and researchers’ field notes. Verbatim (i.e., direct quotation) is a commonly used type of low inference descriptor. |
| Triangulation | "Cross-checking" information and conclusions through the use of multiple procedures of sources. When the different procedures or sources are in agreement you have "corroboration." |
| Data triangulation | The use of multiple data sources to help understand a phenomenon. |
| Methods triangulation | The use of multiple research methods to study a phenomenon. |
| Investigator triangulation | The use of multiple investigators (i.e., multiple researchers) in collecting and interpreting the data. |
| Theory triangulation | The use of multiple theories and perspectives to help interpret and explain the data. |
| Participant feedback | The feedback and discussion of the researcher’s interpretations and conclusions with the actual participants and other members of the participant community for verification and insight. |
| Peer review | Discussion of the researcher’s interpretations and conclusions with other people. This includes discussion with a "disinterested peer" (e.g., with another researcher not directly involved). This peer should be sceptical and play the "devil’s advocate," challenging the researcher to provide solid evidence for any interpretations or conclusions. |
| Negative case sampling | Locating and examining cases that disconfirm the researcher’s expectations and tentative explanation. |
| Reflexivity | This involves self-awareness and "critical self-reflection" by the researcher on his or her potential biases and predispositions as these may affect the research process and conclusions. |
| Pattern matching | Predicting a series of results that form a "pattern" and then determining the degree to which the actual results fit the predicted pattern. |
Some of the strategies, mentioned by Johnson (1997), are related to descriptive measures of validity. This issue is a pertinent one within psychological research as psychology has been based traditionally on naturalistic epistemologies and hence concerned with establishing objective and reliable methods of investigations (Madill et.al. 1, 2000). To achieve greater descriptive validity in this thesis, the extended data collection (the number of analysed articles), reflectiveness of the researcher (analysing the content in micro-context and macro-context and reflecting on the results critically compared with the pre-understanding of the researcher) and pattern matching (making the matching patterns of the text and the matching rule visible to the reader) will be deployed.
When emphasizing the interpretive validity the aim is often to find the “original” meaning exactly as the research informant or target group originally experienced or expressed it. This understanding is related to epistemic theories where the foundation of social interaction is simply seen as an outcome of social interaction, and therefore the individual subjects of the research are the best informants, with their own stories or other subjectively expressed information collected by the researcher. It is worth mentioning that reciprocity between the researcher and the researched (Harrison et al. 2001) makes this type of the validity very complex.
The interpretative validity (Johnson 1997) can easily be questioned in qualitative research and the interpretation would easily be understood as a personal opinion, and not considered scientific research. Especially the research approaches with realistic epistemologies, which are seeking objective evidence for the generalization of the result, would not accept this kind of research without measures like methodological and investigator or data triangulation (Johnson 1997).
In this thesis the information analysed is considered as a sample of social communication. Socially communicated information can no longer be returned back to the intentions of the communicator. Any message, which is socially communicated, can be understood differently depending on the information and utterance of the message (Luhmann 1995). This means that the analysis of the social communication should not try to interpret the “original” meaning of the communication. Rather, it should concentrate on analysing and explicating the semantic structures of the communication: make the structure visible. In this analysis the strategies promoting trustworthiness and validity from the point of view that the researcher tries to seek the original meaning of the information source (in this case: the writer of the article), like investigator triangulation and participant feedback or consideration of reciprocity, are inappropriate.
The language issue in the semantic analysis of the articles is one issue related to interpretative validity. Being a non-native researcher and articles being written by non-native speakers makes the understanding process of the research communication somewhat complex. The means that can be used for ensuring the validity of the interpretations of texts are use of a native peer-reviewer and explaining the analysis for the readers. In this research both of the measures will be applied. The language and interpretations will be double checked by a native English speaking colleague and the analysis process will be written in such a way that the reader can also double check the interpretations. Of course these measures do not eliminate the personal touch of the researcher – which can be considered both negative and positive, depending the epistemic bias of the criticism.
The third form of validity, theoretical validity, is the most difficult to ensure and to evaluate. It can be seen as an internal validity of the theoretical framework, and the evaluation is, more or less, an analysis of the theoretical cohesion of the argumentation in the research. It can be seen also as an external form of validity, when the viability of the theoretical framework is questioned by using an alternative theoretical approach. This kind of external validity is easily questionable in social research, because of the incompatibility of the epistemologically different theories.
In this research, most of the basic epistemological assumptions are based on Luhmann’s theory of social systems. From the theoretical validity point of view, the criticism of this research from theoretical point of view can be addressed to this epistemological foundation. Luhmann’s social systems theory can be seen as a continuum of the dialectical tradition of philosophy (Wagner 1997). The similarity with Hegelian theories like critical theory is that Luhmann tries to find the epistemic ground from the concept of difference, that is close to the idea of polar oppositions and the concept of self-reference, that is close to the concept of reflection (Wagner 1997). The difference compared to other many other philosophers like Habermas is that Luhmann emphasizes the system instead of action (Wagner 1997). This distinction defines whether social is seen as an environment for individuals and classes of society or as a system. In this way Luhmann’s systems theory avoids taking any sides in terms of political agendas or gender in social research. In theory, that makes it possible to research even those phenomena themselves in the research communication without resorting to a biased – and therefore instrumental analysis of science.
Another strong attribute of using Luhmann’s theory of social systems is the notion of the fundamental difference between material, temporal and social systems. It can offer a contributive position for different scientific approaches rather than nullify the other approaches. From this point of view, it can also avoid the relativity of other constructive theories.
One important part of the theoretical validity of this research is the context analysis, setting the interim scope of the research field of ICT integration in education. The context analysis is used also for selecting the research articles for closer analysis of the communicative structures. The context analysis will not follow the traditional instructional technologies research framework, and this is also the intention. The problem inherited from this is that it is difficult to use any general concepts describing ICT integration in education without being theoretically biased. The neutral “ICT integration in education” may not work perfectly in encapsulating all the phenomena and aspects of the research field, but at least it makes it possible to capture most of the metaphors within one concept.
One could claim that the results would be different if some other journals had been selected. This is true but only in respect to the provisional representation of each of the issues researched. By analyzing 194 different research articles a good understanding of the communication structures, metaphors and rhetoric of the communication is obtained. That can be applied more generally to research communication of ICT integration in education – no broader than that scope. In general, journals and articles on the research of technology, education and the social, each of the structures would be emphasized differently. Also the usage of multiple theoretical views (social, learning and technology as it is explicated in the context analysis later in this research) in the synthesis of the analysis will also increase the theoretical validity by providing the macro context for the analysis.
In the science of science one problem is that it will easily see science isolated from other social systems. In this research only the logic of communication will be analyzed in the context of scientific communication structures called paradigms. Paradigms are seen as major structures in the change of the research field. One can ask, what is the role of science policy and research funding in the change of the research field of ICT and education? Platt (1996), for example has argued that the development of empirical work in sociology has been influenced by patterns of research funding. Insofar as the argument relates to research methods, such funding is seen as having played an important role in promoting quantification, instrumentalism and other ‘scientific’ aspects of method - which the writers commonly characterize as a distortion. Plat (1996) concludes that, in the field of research methods, the counterfactual possibilities of scientific discourse, politics and funding mechanisms are more plausible. If they are, it follows that the role of external funding has been important in relation to methods, but not so determinative or so independent of factors internal to the discipline as some writers have suggested, so that the difference made cannot be characterized as a distortion.
In the field of ICT, the massive research programmes and political programmes influence the research. What is the influence of these programmes on the research and the paradigms steering the research? This question has been left out of the scope of this thesis and therefore, in that respect, the external validity of the research approach can be questioned. However, this issue can also be seen as an interesting research question in the further research of the dynamics of research paradigms as communication structures of the research field.