The Reading Matrix
Vol. 2, No. 2, June 2002

Abstract

In this study the impact of the web instructional medium on second language (L2) comprehension during independent reading was investigated. A 2X3 factorial design was chosen for this study with factors being English as a second language proficiency as measured by TOEFL and a medium of instruction (Web, Traditional print and Control). The results of the study suggest that when teaching content through self-directed reading, web medium can evoke better reading comprehension than traditional print medium. Interestingly, the ESL students with a higher TOEFL score did not show a significantly higher level of reading comprehension of the text when compared with those ESL students with a lower TOEFL score. Discussion of the pedagogical implications of the results and recommendations for future research are included.

INTRODUCTION / LITERATURE REVIEW

Research has shown that presentation of information in several modes, such as text in combination with sound, pictures, animations or video has a positive effect on vocabulary acquisition and overall comprehension (Clark & Paivio, 1991; Chun & Plass, 1996b; Mayer, 1997). For example, in his integrative review of multimedia research, Mayer, 1997 showed that students who received explanations in several modes performed significantly better than those who received only verbal explanations. Other researchers discovered that the effects of learning from different formats can be hindered by such factors as task interference, attention problems, and individual differences (Chun & Plass, 1997, p. 66).

Web environments offer new possibilities to combine visual, verbal and auditory modes in multimedia presentations. The effectiveness of these capabilities available to L2 learners via multimedia has been the focus of several studies (Adamson et al., 1995; Chun & Plass, 1996b; Hong, 1997).
Adamson et al. (1995) compared the efficacy of two presentational modes (traditional versus multimedia) on material comprehension among ESL students of two different proficiency levels. The results of the study demonstrated that the use of multimedia computer-based instruction resulted in better comprehension. The study did not confirm the prediction that the multimedia subjects with the lower proficiency would perform significantly better on the post-treatment test than the multimedia subjects with the higher proficiency.

Chun & Plass (1996b) investigated the effects of multimedia on reading comprehension. They conducted three studies involving a total of 160 students of German using CyberBuch, a multimedia software application. The results of these studies showed that the use of multimedia facilitated overall reading comprehension and that vocabulary annotations consisting of both visual and verbal information were more effective than verbal information exclusively.

In his 1997 study Hong investigated the effects of multimedia on reading comprehension in a business Chinese course, concentrating his efforts on speakers at an advanced proficiency level. Participants were randomly assigned to two groups and asked to read two texts, one using a multimedia software package and one using traditional print texts. The results indicated that students read Chinese business texts with a higher comprehension rate in half the time when they utilized multimedia.

In contrast to this research, other studies on the effectiveness of computer multimedia were not able to prove that computers improve student performance (Donato & Coen, 1987; Davis et al., 1997; Xiufeng et al., 1998). Donato & Coen (1987) claimed that they were not able to prove that computer use resulted in higher achievement than traditional modes of instruction. Xiufeng et al. (1998) did not find computer integration to have a significant effect on student achievement. (p. 124). Davis & Lyman-Hager (1997) investigated the ways intermediate college-level students of French employed on-line glosses in a multimedia computer program. The results of this study showed unanimously favorable attitudes towards the software on the part of the participants, but no evidence was found that the use of this software facilitated their reading comprehension.

Previous studies also specifically explored the relation between the use of computerized dictionaries/online glossing and vocabulary acquisition and overall reading comprehension (Lyman-Hager et al., 1993; Knight, 1994; Aweiss, 1994; Chun & Plass, 1996; Lomicka, 1998, Gettys et al., 2001).

Lyman-Hager et al. (1993) examined the effects of using computerized dictionaries on vocabulary acquisition. In this project students studying French were randomly assigned to two groups, one that had instant access to computerized glossing through a software program and one that used print-out pages of glosses drawn from the same program. The results showed that those who used computerized glossing exhibited a higher retention rate on a vocabulary quiz than those who used a printout version of the same materials.

Knight, 1994 studied electronic dictionary use and its effects on vocabulary acquisition for students of different verbal abilities. She concluded that students who had access to a computerized dictionary generally learned more words than those who did not. When the participants were separated according to their verbal ability, only a lower ability group produced "proportionately similar or greater increases in vocabulary learning and in reading comprehension" (p. 295).

Aweiss (1994) explored the effect of computer-mediated reading supports on comprehension during independent reading. The most significant finding of this study was that readers with access to computer-assisted reading aids scored significantly higher on the immediate recall protocol than those with no access to any reading aids. The design of this study, however, did not make it possible for the researcher to investigate whether the use of traditional reading supports would have rendered similar results from the participants.

In another study Chun & Plass (1996a) investigated the impact of multimedia word annotations on incidental vocabulary acquisition. The results indicated a higher than expected rate of incidental vocabulary learning, which the researchers attributed to such features of CyberBuch as the availability of multimedia annotations and ease of word look-up.

Lomicka (1998) researched the impact of online multimedia annotations on the levels of comprehension and discovered that computerized reading with full glossing promoted a deeper level of text comprehension. Gettys et al. (2001) investigated the optimal on-line glossing format and arrived at the conclusion that on-line glossing facilitated higher vocabulary retention rates when basic dictionary-form annotations were used instead of sentence-level translation equivalent glosses.

In summary, although the effectiveness of computerized multimedia on L2 reading comprehension has been a focus of several studies, research so far yielded somewhat conflicting results, especially when multimedia was directly compared with traditional print medium. It is also unclear whether lower level proficiency learners benefit more from computerized reading when compared to higher proficiency learners. The ever-widening use of the Internet has triggered interest among educators in the issues related to web-based instruction (WBI) including its effectiveness. WBI research is closely related to multimedia studies as the web's nature allows multiple ways of information delivery.

The focus of this paper is the investigation of the impact of the web medium on L2 reading comprehension. The paper reports the results of an exploratory quantitative study conducted at the University of Mississippi in the fall and spring semester of 1999-2000. In this study an attempt was made to determine whether the use of web medium versus traditional print for L2 independent reading would result in different levels of text comprehension. Web features that were explored in this study include web multimedia capabilities in combination with on-line reading aids. A 2X3 factorial design was chosen for this study with factors being English proficiency as measured by TOEFL and a medium of instruction.

PROJECT DESCRIPTION

For the purpose of this study the researcher wrote an original text on the history and culture of the state of Mississippi. The text contained approximately 2500 words. A number of photographs also were specifically produced for this project. The text was written for intermediate to advanced ESL students. The researcher utilized a learner's dictionary, Longman Dictionary of American English (2000) as a reference to ensure that lexical items included in the text were appropriate for intermediate/advanced students. The researcher employed two native Mississippians who were at the time graduate students at the School of Education to serve as cultural and language informants. These two informants had access to the text at all stages of writing and made language and cultural suggestions, which added more authenticity to the text. Five Ph.D. candidates in the TESOL program at the University of Mississippi were contacted to determine which of the terms used in the resource might be difficult to comprehend for ESL students at the intermediate and advance language proficiency levels. Based on their feedback necessary glosses were provided.

The text together with the photographs and vocabulary explanations was transformed into a web resource, known as the Mississippi Web ESL Project (for a sample of the web page see Appendix I). A traditional print version of the resource was also created.

The web resource was completed with Adobe PageMill 3.0, a specialized authoring software for Macintosh computers. Multimedia capabilities of the World Wide Web, including graphics, animations and hyperlinks, were widely employed in the web resource's design. Links to vocabulary explanations of possibly new or difficult terms illustrated by visuals as well as a link to an online version of the Merriam-Webster dictionary were provided on every page of the web resource.

The Mississippi Web ESL Project's traditional print version was created using Microsoft Word 98 for Macintosh in such a way that it resembled a book or a magazine. It had eleven pages of information plus seven pages of vocabulary notes in 12 point Times font with 1.5 line spacing. The information given in the print version was identical to the information in the web resource.

The researcher had to keep in mind intrinsic format differences between web and print media making necessary adjustments to both versions of the Project. For example, in the print version traditional non-illustrated glosses were used instead of links to the online multimedia vocabulary notes, an online Merriam-Webster dictionary was substituted with a bound version of the same dictionary and web animations were replaced with static illustrations in the main body of the text.

HYPOTHESES AND ASSESSMENT

For the purposes of this study the participants were randomly assigned to the Control, Traditional print and Web groups. The following three null hypotheses were formulated:

H#1: There will no be significant difference between the posttest scores of the ESL students from the two treatment groups and the posttest scores of the ESL students from the Control group;

H#2: There will no be significant difference between the posttest scores of the ESL students from the Web group and the posttest scores of the ESL students from the Traditional print group;

H#3: There will no be significant difference between the posttest scores of the ESL students with a higher TOEFL score and the posttest scores of the ESL students with a lower TOEFL score.

The reading comprehension level was determined by comparing the results of two equivalent forms of a specifically developed reading comprehension test, henceforth referred to as a pretest and a posttest (see Appendix II & III). The idea behind this design was to create two tests comprised of different items but measuring the same variable, namely knowledge of information about the state of Mississippi.

The researcher compiled a bank of twenty-one subtopics dealing with various aspects of Mississippi culture, geography and history covered in the Mississippi Web ESL Project. For each topic two equivalent-form questions were created which were then randomly assigned to the pretest and posttest. Twenty-two questions dealt with various aspects of Mississippi culture, geography and nature, whereas twenty-two questions dealt with Mississippi history. A small pilot study* of the pretest/posttest was conducted prior to the date of the actual experimentation session. As a result of this pilot study two redundant items were removed, leaving 20 questions in each of the two equivalent forms.

The pretest and posttest's content validity was established by expert judgment**. The tests' reliability was measured by the performance of the Control Group during the experimentation session. The reliability coefficient was calculated to be .7034. The standard error of measurement was 1.6. Taking into consideration that this was a newly developed test the coefficient of reliability and the standard error of measurement seemed to show acceptable levels of reliability.

METHODOLOGY

The population for this study was defined as all newly enrolled ESL students who came to the United States one to two weeks prior to the beginning of this research project and began attending an American medium-size public university or its Intensive English Program during the fall semester of 1999. The population consisted of ninety students. The size of the sample was 31 voluntary participants, including 12 Chinese students, 5 students from the former Soviet Union, 4 students from Malaysia, 2 from Germany, and one from each of the following countries: Nepal, India, Saudi Arabia, Thailand, Japan, and Venezuela. All participants were over 18 years old and for all of them this was their first visit to the United States. None of the participants had previously lived in an English-speaking country. The gender distribution of the sample was 16 female and 14 male subjects.

The subjects' TOEFL score ranged from 400 to 647 with 597 being the median. The subjects were divided into two subgroups: those with a higher

TOEFL score (scores higher than and equal to 597) and those with a lower TOEFL score (scores below 597). Then the subjects from each subgroup were randomly assigned to the Web group, Traditional print group or Control group.

During the experimentation session, which lasted approximately two hours, all participants were gathered in the university computer laboratory to fill out a demographic survey by hand and to take a computerized pretest. Both the pretest and the posttest were administered via computer, using the multiple-choice test format of CourseInfo, a software package for online instruction. All participants were instructed on how to use this software package. The test was administered only when there was a general consensus that everyone understood how to use CourseInfo. It took approximately 15 to 20 minutes for the participants to complete the pretest.

The students then were given their assigned tasks. It was clearly explained to the treatment groups that their task was to look through the material presented to them and learn as much as possible about Mississippi culture and history through independent reading. The researcher also explained that it was their decision on how much time they would spend on reading and what reading strategies they would utilize.

Those who were placed into the Control group were taken to a nearby classroom and asked to watch a segment from "Ferris Bueller's Day Off" for approximately an hour. Afterwards, the students were encouraged to participate in a teacher-led discussion of the differences and similarities between high school education in the United States and in their native countries. The discussion lasted approximately half an hour and as indicated by several participants left them with a better understanding of American high school education. The video segment and the topics that emerged in the classroom discussion were unrelated to the topics covered in the Mississippi Web ESL Project and in this way did not have any inherent learning value pertinent to this research.
Those who were placed in the Web group were asked to remain in the lab and work with the Mississippi Web ESL Project using the web medium. They were shown how to navigate the web, use online glosses and the online version of the Merriam-Webster dictionary. They were encouraged to practice with the computer before starting to read and ask questions if they had technical problems while on task. It took approximately one hour and twenty minutes for everyone in the Web group to complete the assignment. Although technical assistance was offered to the subjects, very few questions were asked.

The Traditional print group participants were taken to a nearby classroom and presented with a book version of the Merriam-Webster dictionary and a print version of the Mississippi Web ESL Project. They completed their reading in approximately an hour and ten minutes***.

In approximately an hour and a half from the beginning of the experiment all the subjects from the Control, Web and Traditional print groups took a posttest in the computer lab. It took approximately twenty minutes for all students to complete the posttest.

RESULTS

Descriptive statistics of the sample involved in the study, including subjects' pretest/posttest means and corresponding standard deviations are given in Table 1.

Table 1
Pretest/Posttest Score Means by Mode of Presentation

For the purpose of this research project, statistical significance was accepted at the .05 level of confidence. Analysis of Covariance (ANCOVA) was employed to test Hypothesis 1 and Hypothesis 2. Factorial Analysis of Variance (ANOVA) was utilized to test Hypothesis 3.

In compliance with the assumptions underlying ANCOVA, a preliminary investigation of the data was conducted. Lack of a curvilinear relationship between the covariate and dependent variable, as well as lack of significant differences between the groups involved in the study (see Table 2) showed the appropriateness of the use of ANCOVA as a major statistical device for the evaluation of Hypothesis 1 and 2.

Table 2
One-Way Analysis of Variance (ANOVA)

H#1: There will no be significant difference between the posttest scores of the ESL students from the two treatment groups and the posttest scores of the ESL students from the Control group.

The obtained significant value of .000 shown in Table 2 achieved the .05 level of statistical significance. This led to the conclusion that there was indeed a significant difference between posttest means of at least two groups involved in the study. Further analysis of the data showed that at the .05 level of significance there was noted a statistically significant difference between the posttest scores of the subjects from the Web group as compared to the posttest scores of the subjects from the Control group. Results of the analysis of covariance (Table 2) show that when leveled on their pretest scores, the program participants who received instruction, either in web or traditional format, did score significantly higher than the program participants from the control group. Consequently, the null hypothesis was rejected.

H#2: There will no be significant difference between the posttest scores of the ESL students from the Web group and the posttest scores of the ESL students from the Traditional print group.

As shown in Table 3, the post hoc tests showed a significant difference between the posttest scores of the Web group and the Traditional print group (the significant value of .021) at .05 level significance. Thus, the null hypothesis was rejected.

Table 3
Analysis of Covariance (ANCOVA)

W vs C - Web group vs Control group
T vs C - Traditional print group vs Control group
W vs T - Web group vs Traditional print group

H#3: There will no be significant difference between the posttest scores of the ESL students with a higher TOEFL score and the posttest scores of the ESL students with a lower TOEFL score.

As shown in Table 4, no significant differences were found at the .05 level of significance among TOEFL High and TOEFL Low groups, and no significant interaction between assignment to groups and TOEFL score was found. The appropriate null hypothesis failed to be rejected.

Table 4
Factorial Analysis of Variance (ANOVA)

The statistical analysis of the data showed that the ESL students who utilized the Mississippi Web ESL Project, either in its web or traditional format, achieved significantly higher posttest scores than the ESL students who were in the Control group. The analysis also revealed that the ESL students who utilized the web exhibited a significantly higher level of reading comprehension than those who used the traditional format. And, finally, the analysis failed to prove that the ESL students with a higher TOEFL score achieved a significantly higher level of reading comprehension than those ESL students with a lower TOEFL score.

CONCLUSIONS, DISCUSSION AND RECOMMENDATIONS

In this study two modes of reading were compared in regards to their effectiveness for L2 reading comprehension: the web and the traditional print mode. It was hoped that reading on the web would produce better performance than traditional reading method. Consistent with Kozma (1991), Lyman-Hager (1993), Adamson et al. (1995), and Hong (1997), the results of this study indicate that the medium of instruction does have an impact on the level of reading comprehension, with the web mode resulting in better performance when compared to the traditional print mode. The overall significantly higher scores with the web are attributed to several features of this medium that were incorporated into the Mississippi Web ESL Project, including online glosses, electronic dictionaries, web animations and fast and convenient delivery of these multimedia reading supports through well-designed navigational strategies and orientation cues.

For ESL educators these results suggest that when teaching content through self-directed reading, web medium can evoke better reading comprehension than traditional print medium. Online glosses and dictionaries enable students to access necessary lexical items much faster than traditional print reading supports, resulting in more efficient reading. In addition, online glosses provide contextual annotations, which prevent the reader from having to choose from several meanings and thus eliminating the risk of making a mistake (Hong, 1997, p. 339). On the other hand, web animations and graphics add a visual aspect to the text, serving 'as an aid for text comprehension and functions as supplemental information that is added to the mental model of the text…" (Chun et al., 1997, p. 65).

In agreement with Adamson et al. (1995) and in contrary to Knight (1994) the results of this study indicate that students at various levels of English proficiency benefited equally from participation in this project by learning new material about the regional culture of the state of Mississippi. This finding suggests that further research, including longitudinal studies, is needed to further investigate the impact of the web medium on reading comprehension among L2 learners at various proficiency levels. On the other hand, the alternative interpretation of this finding is that perhaps when reading content-rich material on the web, the level of English proficiency measured only by four traditional skills and exemplified by a TOEFL score may not necessarily ensure better performance. Other factors, including previous exposure to the target culture, country of origin, and an array of other individual differences might interact and influence learners' performance.

And, finally, consistent with Hanson-Smith (1997), the study shows that new technologies, such as the Internet are able to support various instructional innovations, including content-based student-centered instruction. The success of the Mississippi Web ESL Project, both in its web and traditional print version, advocates the necessity for creating content-based web resources geared specifically towards ESL audiences with American regional cultures being the content and English being the language of instruction.

STUDY LIMITATIONS AND FUTURE RESEARCH RECOMMENDATIONS

In order to obtain simple and concise results in this exploratory study, the researcher consciously chose to concentrate on the impact of only two variables on L2 reading comprehension, namely reading mode and language proficiency. However, other relevant factors that were beyond the scope of this study, might influence the way L2 learners process information when they utilize different reading media (Mayer, 1994; Mayer & Sims, 1994, Chun et al., 1997, Mayer, 1997). These factors include but are not limited to participants' individual differences, such as gender, native language, country of origin, age, and previous English experience. Psychosocial factors, including learning style preference, motivation, gloss/look-up behavior, personality type should also be considered. Although the scope of this study did not make it possible to explore these issues, it is hoped that future research will incorporate the investigation of these factors into its design.

The other limitations of this study include a small sample size, a broad range of TOEFL scores and a short duration of the experiment. It is hoped that in the studies to come, these limitations will be taken into account, resulting in more generalizable outcomes. This project should be regarded, then, as a preliminary investigation, one that demonstrates a need for similar studies on a broader scale. In order to establish results with a higher level of validity and reliability, similar longitudinal studies with a larger number of participants are recommended to consider.

Other issues that need to be resolved to move this research beyond its preliminary stages are reading material content difficulty, text authenticity, language proficiency assessment, and, finally, issues of statistical analysis. To warrant result generalizability, it is recommended to consider the use of authentic texts with different levels of content difficulty in the future. It also would be advisable to employ alternative holistic assessment instruments, which in contrast to such traditional measures as TOEFL do not only focus on the four language skills, but account for what is known as communicative competence, including sociolinguistic, discourse, grammatical and strategic competence (Canale & Swain, 1980). To obtain higher credibility results, in the future it might be also useful to utilize more conservative non-parametric tests.

Finally, there is also a need for investigation of other issues related to the ESL web design, such as the use of alternative navigational techniques and pop-up menus, the use of color, animation, sound, movies and electronic mail. The rapid development of web applications makes it important for researchers to conduct studies not only comparing various media of instruction (e.g. web versus traditional) but also concentrating on the web medium itself (e.g. frame versus non-frame version of the same web page).

In conclusion, the results of this research raise an important question, namely that there might be a discernable difference between the comprehension rate of the learners using traditional print media and those using web media, with the latter holding a demonstrable if yet not generalizable advantage. With this in mind, it is hoped that new studies will follow this research in its footsteps in order to provide a conclusive answer to this critical question.

NOTES

*Pilot study - Two exchange graduate students and two American
college students were asked to fill out a pilot version of the computerized pretest and posttest. These students were presented with a pretest and then asked to watch a one-hour segment video and took the posttest. The researcher interviewed them to find out if they felt that any of the questions were unclear or misleading. The results of this small pilot study showed that the participants obtained the same score on both forms of the instrument and that two items of the instrument were redundant and had to be removed.

**Expert judgment - Content validity of the instrument employed in this study was determined by expert judgment of five Ph.D. candidates from the TESOL program at the University of Mississippi. These experts were presented both with the information disseminated through the Mississippi Web ESL Project and several drafts of the developed instrument and asked to make a judgment on whether all subtopics had been proportionately represented in the instrument. Necessary changes were made.

***Time on task - Though time on task was not an issue investigated in this study, two possible explanations may be offered to account for the fact that the participants from the Web group seemed to spend more time on the assignment. The first one is that the web version offered its users more options to explore in a more accessible way, raising student interest and motivation to read more. This explanation seems to be quite plausible taking into consideration the fact that the Web group participants did score significantly higher on the posttest. However, it is also possible that there were too many options and too many distracters in the web version, making it more difficult for the participants to concentrate, which ultimately resulted in more time on task. Future research needs to explore actual behavior patterns of L2 web users, shedding light on user preferences and factors that might account for them.

Julia Stakhnevich is currently employed as an assistant professor at Bridgewater State College, Massachusetts. She received her doctorate from the University of Mississippi, Oxford. Her research interests include computer-assisted language learning with the focus on the web and the study of language shift, including word-borrowing patterns in various Indo-European languages.

APPENDIX

References