Research Methods to be Employed
The proposed dissertation project will be true experimental research (Gay, 1996) about Multi-user domain Object Oriented (MOO), an Internet communications program that has been claimed to be a beneficial foreign language instructional procedure at the high school level (Sanchez, 1996a), as well as being a beneficial procedure for older students (Falsetti, 1995; Pinto, 1996). As reported in Chapter One of this dissertation, the specific research questions of this study will be:
- Are high school foreign language students motivated to use MOO to the extent that merits the adoption of this procedure?
- If MOO is adopted as a foreign language procedure on the high school level, can the students indicate which of the preexisting procedures should be replaced by MOO?
- Do gender, keyboard skills, or general computer anxiety influence the students' motivation to use MOO as a foreign language procedure at the high school level?
A Pretest-Posttest Control Group Design will be used in this experiment (Gay, 1996). The participants will be the total population (N=62) of the most advanced level of English as a Foreign Language (EFL) in the 11th grade of the Har V'Gai Regional School at Kibbutz Dafna, Israel. The population will be divided randomly and equally into three groups: two experimental classes and one control class. The experimental classes will experience MOO as one of many instructional procedures during the academic year while the control class will not use MOO during that year. Instead, the control group will receive a greater amount of other procedures which all EFL students, including the experimental students, routinely use in class at the Har V'Gai Regional School. It should be noted that none of the students in the 11th grade have ever experienced MOO as an in-class instructional procedure in any subject.
The Review of Literature suggests two possible areas of research to test the value of MOO as an foreign language instructional procedure: the cognitive domain and the affective domain. In the cognitive domain, the quality and quantity of interaction (including input, negotiation, and output) could be studied. However, due to technical considerations, described later in this chapter in a section relating to limitations, such an approach is beyond the scope of this dissertation. In the remaining area, the affective domain, the Review of Literature offers the constructs of anxiety and motivation in the foreign language classroom as fertile ground for empirical research concerning MOO. The data of such research will be collected on questionnaires administered to the experimental and control groups before and after the treatment.
This project will attempt to ascertain if there is any difference in general classroom anxiety levels and motivation levels between the experimental group and the control group. Also, an attempt will be made to find significant relationships between anxiety/motivation while using MOO on the one hand, and gender, keyboard skills, and general computer anxiety on the other. Moreover, given that all classroom time before this experiment will have been taken up by procedures other than MOO, the study will attempt to discover which of these preexisting procedures the students would sacrifice, totally or partially, in order to accommodate the use of MOO. (While it is not intended that student opinion be the decisive factor in curriculum decisions, knowing what motivates the students and what creates anxiety can play a part in such decisions.)
Specific Procedures to be Employed
Description of the Experimental Population
As noted, the participants in the experiment will be the entire population (N=62) of the 11th grade EFL students at the Har V'Gai Regional School (Kibbutz Dafna, Israel) at the Five Point Bagrut Level. This is the highest level of English instruction as defined by the Israeli Ministry of Education and is formally assessed during a national examination, usually taken at the end of the 12th grade.
The students studying at the Four Point, Three Point, and One Point levels will not be part of the experiment. (There is no Two Point level.) In addition, a remedial Five Point class will not participate in the experiment for a number of reasons. First, it was deemed that they would not be on the same level as the normal Five Point students. Second, according to a decision of the English staff, the class will not broken up, thus making randomization impossible. Finally, since this is a remedial class, they will not be studying exactly the same material as the regular classes. Therefore, only the regular Five Point Bagrut students will participate in this experiment.
Description of the Experimental Sites
The Har V'Gai Regional School is a coeducational rural school, ranging from seventh to twelfth grade. There are approximately 1,000 students, coming from middle and upper-middle class Jewish settlements in the Upper Galilee and Golan Heights. There is an ethnic mix between Ashkenazim (with origins in Northern and Western Europe) and Sephardim (with origins in Southern Europe, North Africa, and the Near East), but no exact statistics exist about the percentage of each group in the school. There are also about 50 new immigrant students from the former Soviet Union, whose native language is Russian. The predominant language heard in the school is Hebrew. English language studies are mandatory and Arabic (Israel's second official language) is taught as an elective. The Russian speaking immigrants also have the option of continuing their Russian language studies. The English language requirements, as opposed to the other language electives, reflect the linguistic realities of the country: English is extremely important economically and culturally in Israel, more so than any other foreign language, Arabic or Russian.
All the English teachers at the Har V'Gai Regional school are certified by the Ministry of Education and work according to the syllabus written by the Ministry. The teachers are periodically observed, in class, by visiting English Inspectors from the Ministry of Education. According to Hadasi Label (personal communication, June 15, 1998), who administers Bagrut procedures at the school, recent English Bagrut examination averages of Five Point students have been close to the national average. In 1997, the school average was 0.3 points below the national average; in 1996, 3.2 points above; in 1995, 3.5 points above; in 1994, 1.2 points above; and in 1993, 5.6 points below the national average.
The school is connected to the Internet by a frame relay. The Internet Lab, with 15 computers, is adjacent to the main reading room of the school library, which has another nine computers connected to the Internet. The school administration is keen on integrating the Internet into the daily curriculum.
The virtual experimental site will be schMOOze University, which was created in July 1994 by Julie Falsetti and Eric Schweitzer of Hunter College, City University of New York. SchMOOze can be accessed via telnet://schmooze.hunter.cuny.edu:8888 at a server at Hunter College which keeps it running 24 hours a day. The basic metaphor of schMOOze is a small college campus with dorms, a library, classrooms, a conference center, an administration building, a student union, an international culture center, an off-campus disco/bar, various malls, gardens, and other collegiate locations. The users who frequent schMOOze are mostly students and teachers of English as a second/foreign language, but there are many native English speakers, with no connection to language instruction, who visit schMOOze as well. According to the census of schMOOze, as of November 13, 1998, there were a total of 592 permanent members of the community. This does not include the numerous guests who visit without registering. (Awaji, 1998; Falsetti, 1995; schMOOze, 1994).
The primary purpose of schMOOze University is to offer a sheltered environment for students of English to meet native speakers and non-native speakers. Through conversations about real life matters, or conversations about the fantasy environment offered by the MOO site, the students will progress in their acquisition of English. If the students log in when there aren't many other students available for a chat, the MOO fantasy will offer an opportunity to read and write English in a game-like environment. This includes interacting with existing MOO objects and building new ones for other characters to examine. The secondary purpose of schMOOze University is to offer a professional and social meeting place for teachers of English as a second/foreign language from all over the world (Awaji, 1998; Falsetti, 1995).
Steps to be Taken
At the end of the 10th grade (the year before the experiment), all the students at the Five and Four Point Bagrut Levels will be given consent forms for their parents to sign. The Four Point Bagrut Level will be included for two reasons. First, some of the students may move up to the Five Point Level during the summer before the 11th grade. Second, if the students believe that the Four Point level is included in the project, perhaps the experimental nature of the MOO procedure will be less obvious, thus avoiding the Hawthorne Effect.
At the end of the 10th grade (the year before the experiment), the Five and Four Point students will be asked to suggest three personal code names. Their 10th grade EFL teachers will explain that these code names will be used for computer programs the following year, but MOO will not be mentioned specifically. Those participants who are later randomly assigned to the experimental group will receive character names based on their requests. If a certain character name is already taken, a similar name will be requested.
Due to the novelty of the topic, there are no standardized tests available for examining affective elements of using MOO in foreign language instruction. As a result, questionnaires will be constructed for this project at the end of the academic year prior to the experiment. As with all nonstandardized instruments, there will be potential problems of validity and reliability (Gay, 1996). To deal with content validity, all the Five Point teachers on the Har V'Gai English staff will examine the questionnaires. (One of these teachers is also an instructor of EFL methodology at the Oranim Teachers' College. Another is the Ministry of Education's National Counselor for CALL - Computer-Aided Language Learning.) After incorporating the comments of the teachers, the questionnaires will be translated into Hebrew and given to a graduating Five Point 11th grade English class (in the year before the experiment) with MOO experience. The responses of this class will serve as a pilot study for the questionnaires and the general experiment. After completing the pilot-study questionnaire, there will be an in-class discussion about the students' perceptions of the items on the questionnaire. Any relevant comments by the students will be incorporated into the questionnaire. At this point, a Russian translation will be made for the 15 Russian speaking immigrant students who may have trouble with the Hebrew on the questionnaire.
During the summer before the experiment, the students who will be in regular Five Point Bagrut classes in the 11th grade will be divided randomly into two experimental classes and one control class. Teachers will be assigned randomly to the three classes. An independent consultant (i.e., a teacher not from the English staff) will perform the randomization, using Microsoft Excel (version 7).
The names, e-mail addresses, and requested character names (pseudonyms) of the students in the two experimental classes will be sent to the administrators of the schMOOze University MOO site for registration. The administrators of schMOOze have agreed to assist with the proposed project by registering the students en bloc and facilitating other procedural matters that may arise during the experimental period.
During the first English class of the experimental year, a preliminary questionnaire (the pretest) will be given to the experimental and control classes. The questionnaire will present 12 commonly used EFL instructional procedures: intensive reading with exercises, extensive reading without exercises, grammar exercises, listening to frontal teacher explanations, taking quizzes and tests, listening to taped songs followed by exercises, listening to taped comprehension exercises, speaking English in a class discussion, frontal oral presentations in English, speaking English while in small groups, watching video (clips or full movies) followed by exercises, and writing exercises in class. Using modified Likert scales, the students will be asked to indicate their feelings of anxiety and motivation towards each of these procedures and towards English lessons in general. While anxiety will be treated as a single construct, motivation will be divided into the four subcategories suggested by Dornyei (1994): interest, relevance, expectancy (locus of control), and satisfaction. The questionnaire will be administered in Hebrew, with Russian translations for those students who need them. Written Hebrew and Russian explanations will help the students distinguish between the four subcategories. The students will be able to query the teachers if the written explanations are not sufficient. The questionnaire will also ask for the name of previous and current English teachers (for administrative purposes), as well as the participant's gender. Since MOO has not been used in the school with these particular students, it is assumed that MOO is unknown. Thus, no questions will be asked about MOO in the preliminary questionnaire. The English version of this questionnaire will be included in the appendix of the dissertation report.
The treatment (the use of MOO as a major EFL instructional procedure) will be administered, at certain intervals, during the academic year. The experimental students will use MOO at least ten times, each time for 45 minutes. Because there are only 15 stations in the school's Internet lab, the two experimental classes will be divided into smaller groups to ensure that each student has a separate computer. Each group will use the Internet lab the same number of times and within a few days of the other groups. Students who miss a regular session, will be able to make up that session with another group. Thus, all the students should have approximately the same amount of class time at schMOOze University, spaced out over the academic year. There will be additional noncomputerized material (reading comprehension exercises, discussions, etc.) to prepare the students to use MOO effectively, to register the students' opinions of their MOO experiences, and to integrate MOO into the curriculum (according to the recommendations of the literature). This off-line material, structured as a teacher's guide, will appear as an appendix in the final dissertation report.
During the academic year, the control class will use nonMOO procedures. The two experimental teachers and the one control teacher will loosely coordinate the content of their lessons, according to the demands of the Ministry of Education. They will not attempt to scientifically control the nonMOO instruction in any of the three classes.
In the beginning of the academic year, the experimental students will be given an explanation about how they can use MOO by themselves (at home or during free periods in school. It should be noted no other chat-like program is permitted in the school, without specific permission of a teacher.) Although no MOO homework will be given, the students will be encouraged to use the MOO if they wish. In the Internet lab and in the library, the computers will use the AvPlay MOO client, a program that facilitates using the MOO environment (Austin, 1995). The AvPlay connect option will be set to schMOOze University. In their instructions for home use, the students will be directed to schMOOze University. The students will be urged to use only their own character names while visiting schMOOze University during free time as well as during class. These arrangements and recommendations will result in a record of all MOO activity of the students while they visit schMOOze University, using their own character name. (Beyond these option settings, directions, and recommendations, it will not be possible to prevent the students from visiting other MOO sites during their private time. Nevertheless it is assumed that most voluntary use of MOO will be done in the familiar environment of schMOOze University.)
On two different days at the end of the academic year, the participants will be given questionnaires. The first will be basically the same as the preliminary questionnaire. The critical difference will be the addition of MOO as a procedure for the experimental group. The question about MOO will be placed in the middle of the list of questions. The control group will not receive the question about MOO because they will not have experienced it as a procedure. Another addition, there will be an new item about using the World Wide Web during English class. With the opening of the new Internet lab, the use of the World Wide Web will become a normal part of EFL instruction at Har V'Gai Regional School. Both the control group and experimental groups will have used the World Wide Web and will be asked about it on the posttest.
On a subsequent day, only the experimental group will receive a second post treatment questionnaire. Using modified Likert scales, the students will respond to questions about self-assessed keyboard skills, self-assessed general computer anxiety, and opinions about MOO's efficacy in improving English reading, English writing, and general English skills. Finally, two open questions will ask the students to respond freely about their MOO experiences. Once again, all the questions on both questionnaires will be in Hebrew, with Russian translations for the 15 Russian speaking immigrant students. The students will be asked to respond to the open questions in English. (By the end of the 11th grade, Five Point Bagrut students are expected to write short essays in comprehensible English. Those motivated to express optional opinions about MOO will do so.)
During the first week in June, a record will be made of the total time each registered character from the experimental group will have spent at schMOOze University. This is public information, which the MOO program presents upon request. The MOO program lists the total amount of time characters have been active and the average length of visits to the MOO site. By subtracting the amount of class time spent in the MOO site, it is possible to obtain an approximate figure for voluntary use of MOO. This assumes that students will always visit the MOO site using their registered character names, and will only visit schMOOze University. Given the imprecision and limitations of this information about voluntary use of MOO, such information will be used only as supplementary data.
After the data are collected via the questionnaires and from the MOO program, the following steps will be taken:
- Perform the statistical analysis using the SPSS statistical program.
- Describe results.
- Write the draft report.
- Obtain feedback.
- Revise report.
- Submit final report.
Formats for Presenting Results
As reported in Chapter One of this dissertation proposal, the experiment will attempt to support, or reject, the following hypotheses about using MOO in the high school foreign language classroom:
Using the data from the experimental group, a descending list of means of all the procedures will be generated for anxiety and the four motivational subconstructs. The means regarding the use of MOO in the experimental class will be compared to the higher anxiety means and to the lower motivational means. One-tailed t-tests for paired samples will be used because the two sets of data will be generated by the same population.
- The students will report less than average anxiety and more than average interest, relevance, expectancy, and satisfaction about using MOO in high school foreign language class in comparison to other instructional procedures used.
The statistical procedure for examining this hypothesis will be the comparison of the means regarding MOO and the average of the means of all the other procedures. The one-tailed t-test for paired samples will be the statistical tool used. The samples will be paired because this will be a comparison of two pieces of data generated by the same students (i.e., those in the group using MOO). The statistical analysis will be performed five times, once each for: anxiety, interest, relevance, expectancy, and satisfaction.
- Gender will not be significantly related to self-reported anxiety, interest, relevance, expectancy, and satisfaction while using MOO as a foreign language procedure at the high school level.
The statistical procedure for examining this hypothesis will be the comparison of gender with the means regarding MOO. The one-tailed t-test for independent samples will be the statistical tool. The samples will independent since the data will be generated by two different groups: the male students and the female students. The statistical analysis will be performed five times, once each for: anxiety, interest, relevance, expectancy, and satisfaction.
- Self-reported keyboard skills will correlate negatively with self-reported anxiety and positively with self-reported interest, relevance, expectancy, and satisfaction while using MOO as a foreign language procedure at the high school level.
The statistical procedure for examining this hypothesis will be the generation of correlation coefficients. The statistical analysis will be performed five times, once each for: anxiety, interest, relevance, expectancy, and satisfaction.
- Self-reported general computer anxiety will correlate positively with self-reported anxiety and negatively with self-reported interest, relevance, expectancy, and satisfaction while using MOO as a foreign language procedure at the high school level.
The statistical procedure for examining this hypothesis will be the generation of correlation coefficients. The statistical analysis will be performed five times, once each for: anxiety, interest, relevance, expectancy, and satisfaction.
- The students using MOO in class will report lower levels of anxiety and higher levels of interest, relevance, expectancy, and satisfaction about foreign language instruction (in general) than students not using MOO in class.
The statistical procedure for examining this hypothesis will be the comparison of means of the experimental group and means of the control group. The one-tailed t-test for independent samples will be the statistical tool. The samples will be independent since the data will be generated by two different groups: experimental and control. The statistical analysis will be performed five times, once each for: anxiety, interest, relevance, expectancy, and satisfaction. In additional, a one-tailed t-test for independent samples will be used to ascertain if the teaching style of the individual teachers influenced the findings in the two experimental classes. The samples will be independent since the data will be generated by the two different experimental classes.
- In terms of anxiety and motivation, students will express clear opinions about which foreign language procedures should be displaced in order to include MOO in the curriculum.
According to the theoretical material presented in the Review of Literature, the students using MOO as an instructional procedure in the foreign language classroom should report more motivation and less anxiety than students in the control class (i.e., those not using MOO). However, there are two major caveats to this prediction, making it false for some of the students. It is assumed that self-reported keyboard skills will positively correlate with the students' motivational attitudes towards MOO and negatively correlate with the students' anxieties about using MOO. It is also assumed that the positive reactions towards MOO will decrease if the students suffer anxiety about computers in general. Thus, it is predicted that motivation to use MOO and feelings of anxiety about the procedure will be strongly influenced by general considerations of educational computation, as well as specific considerations about MOO itself. Furthermore, although there are no predictions about the relationship of gender to the attitudes towards MOO, it should be noted that any relationship between gender and general educational computation will also influence the relationship between gender and student perception of MOO.
In addition, it is assumed that MOO will prove to be more popular than some of the other instructional procedures. Taking examinations and giving oral presentations will probably head this list. Teachers will be able to use this list when considering future curricula. Although the teachers ultimately decide on instructional procedures, such decisions should be made with the knowledge of student preferences. Even a necessary procedure might be used less frequently if it creates high levels of anxiety, thus decreasing the over-all efficacy of the instruction.
As stated before, the empirical research project will be done at the Har V'Gai Regional School at Kibbutz Dafna, Israel. Most of the required resources will be provided by the school. The participants will be the 11th grade high school EFL students at the Five Point Bagrut Examination level. (As noted before, the Bagrut is the Israeli national matriculation examination. In EFL there are various levels: Five Points, Four Points, Three Points, and One Point, where Five Points is the most advanced.) The students will be reorganized into three classes, on a statistically random basis. Two classes will serve as experimental groups while one class will be the control group. Experienced teachers will teach the three classes.
While the linguistic competency of the 12th grade classes would be more suitable for the study, these students are under the pressure of the Bagrut Examinations in many subjects. As a result, the school year is shortened and the regular classes are frequently interrupted. This situation would defeat any attempt at a major experiment with 12th grade students. Thus, the population for this experiment will be all the regular Five Point 11th grade students.
Based on previous EFL projects, full cooperation of the administration, the computer staff, and English staff is expected. The school's recent migration to a networked environment with a high level of connectivity to the Internet will aid the current research project in a number of ways. First, there will be enough Internet computers to make this project possible. Second, the administration will be eager to receive a reasonable return for the school's investment in the network and will support any serious use of the Internet lab. In addition, the proposed research will aid the school's attempts in obtaining "experimental school" status. Receiving this classification would bring substantial governmental funds to the school. Thus, the administration has already expressed its interest in the research project and has given top scheduling priority to the project, thus avoiding the potentially major problem of lack of access to the Internet lab. Third, the novelty of the Internet lab means that there will be minimum competition for use of the room, at least in the beginning of the year. Because technology takes time to be adopted by teachers, the current dissertation proposal will enjoy the status of being the first in the field in regard to Internet use at the school. The local computer staff will be anxious to have the Internet lab used, and has promised full cooperation in regard to the proposed project. The computer staff will download the AvPlay MOO client from the Internet and install it, with the AvPlay icon appearing on the desktop of all computers in the lab. The English staff has also promised full cooperation in the proposed experiment, despite the problems of reorganizing the classes and extra work involved. Known for their creativity and flexibility, the Har V'Gai English teachers have welcomed the experiment and will learn how to use the MOO program.
Other resources will be found at the MOO site itself. The administrators of schMOOze University have expressed interest in the proposed experiment. They have agreed to process the registration of the students en masse, and deal with other administrative procedures as fast as possible, thus facilitating the students' activities.
Finally, a local statistician, using version 8.0 of SPSS software (SPSS, 1997), will help in the statistical procedures of the empirical study.
Questions of Validity and Reliability
According to Gay (1996), the use of a true experimental design will control for most sources of internal and external invalidity. The proposed dissertation will use a Pretest-Posttest Control Group Design, whose only potential weakness, according to Gay, is the possible interaction between the pretest and the posttest. This weakness will be neutralized by three factors: the nature of the pretest, the novelty of other controlled procedures, and the duration of the treatment. Since the students will have never used MOO in any of their classes before the experimental period, MOO will not be mentioned in the pretest. As a result, the students will not become aware of the exact nature of the experiment from taking the pretest. Although the students' parents will have signed prior permission for participation in an experiment in English class (thus alerting the students to some type of experimentation), the students will experience a number of novel procedures, some of which are standard in the 11th grade, some of which are novel because the opening of the Internet lab. The Five Point Bagrut syllabus will bring a number of new procedures to all the classes. At the same time, the new Internet lab will facilitate the use of the World Wide Web for all the classes. In addition, the control class will be told that their MOO experience will come towards the end of the year, because of scheduling considerations. Thus, it may not be obvious to the students that using MOO is an experiment and that there is a significant difference between the procedures given to the experimental group and the control group. Finally, the length of the treatment period (nine months) will minimize the influence of the pretest on the posttest.
Although the use of a Pretest-Posttest Control Group Design with randomized groups should control for most sources of internal and external invalidity, the nature of this project raises the problem of potential instrumentation (Gay, 1996). The novelty of the instructional procedure to be examined and the resultant need to design a questionnaire inevitably lead to caveats regarding the reliability of the experiment. Only similar experiments in the future will be fully able to establish the reliability of the current project. For the same reasons of novelty and of the use of a nonstandard instrument, there may be queries about validity. Questions about content validity will be answered by the use of a panel of experts while constructing the questionnaire, but there seems to be no solution for problems of construct validity and concurrent validity. Although the Review of Literature notes that foreign language anxiety has been used as a construct in research for many years (MacIntyre & Gardner, 1991), Dornyei's (1994) four subconstructs for course-specific foreign language motivation have not been thoroughly validated. Nevertheless, Dornyei's theoretical work offers the best framework for the proposed project, thus construct validity will be assumed pending future validation. Questions about concurrent validity will also have to wait for future experimentation. Constructing and administering additional instruments for the sake of validation is beyond the scope of this project.
This dissertation will be a true experimental project: randomly dividing a population into a experimental group and a control group, administering a pretest, administering the treatment, and then administering posttests. The central research questions and resulting hypotheses will examine the use of MOO as a high school foreign language classroom procedure. Although the theory of second/foreign language acquisition suggests research in both the cognitive and affective domains, institutional limitations to the study eliminate a meaningful examination of cognitive elements. Thus, the research will focus on the affective elements of anxiety and motivation in the foreign language classroom. Anxiety will be used as a single construct and motivation will be divided into the following subconstructs: interest, relevance, expectancy, and satisfaction.
If the research does support the claim that MOO is a worthwhile procedure for high school foreign language classes, there should be some indication of which existing procedures should be decreased, or totally eliminated, to make room for MOO in the school schedule. In addition, if there is support for the claim that MOO is a worthwhile classroom procedure, teachers will be able to use the experimental procedures, reported in the appendix, as classroom material. Nevertheless, given the caveats stemming from the limitations and delimitations of the project, all conclusions arising from this project should be taken as conditional, demanding further validation in future experiments both in the affective and cognitive domains.