"A comparative analysis of performance and behavioral outcomes in technology-based learning"
Katia Passerini, 
Doctoral Candidate in the Information and Decisions Systems, pkatia@gwu.edu
Department of Management Science, School of Business and Public Management
The George Washington University
Introduction and Summary
A systematic understanding of the impact of different technological media on the achievement of instructional goals enables educators to deliver the subject matter more effectively.This research focuses on the impact of selected instructional delivery modes in supporting the learning of business management topics. Among the various instructional technologies that advance learning, interactive multimedia is recurrently identified as a very powerful tool. The dissertation measures the effects of multimedia technology, compared to both traditional classroom and text-based instruction. It analyzes learners’ performance within a knowledge representation framework that looks at recall and application[1] of facts, concepts, principles, or procedures, as representations of instructional outcomes. The research uses a pretest-posttest quasi-experimental design to address concerns expressed by earlier comparison analyses. The design supports the examination of selected instructional objectives achieved within a specific timeframe, and isolates the effect of the medium on students’ (graduate and undergraduate) learning and satisfaction outcomes.
Prior Research
There is an on-going controversy in the literature dealing with the question of the learning effectiveness of instructional technology. In the last three decades, every time a new instructional technology has come about, researchers have tried to assess its impact on learning. Some scholars (Kozma, 1991) claim that media influences learning. Many others (Clark, 1983, 1985, 1991, 1994; Souder, 1993; Martin & Rainey, 1993; Barry & Runyan, 1995; McClure, 1996) claim that there is no difference in learning outcomes based on the medium. Few scholars (Moore & Kearsley, 1996) argue that researchers should not be asking effectiveness questions because “for any group of students, the environment in which learning occurs and the medium of communication between the teacher and the learners are not significant as predictors of achievement” (pg. 65). More recently, others (Jones & Paolucci, 1998) are calling for further research. There is no final answer yet. In spite of the several studies (Nugent, 1982; Mayer & Anderson,1991) and meta-analyses (Hartley, 1977; Burns & Bozeman, 1981; Kulik & Bangert-Drowns, 1983, Bosco, 1986; Kulik, Kulik & Shwalb, 1986; Kulik & Kulik, 1989, 1991; Roblyer, Castine & King, 1988; Kulik, 1994; Khalili & Shashaani, 1994; Fletcher-Flinn & Gravatt, 1995;) competing to answer the question: “do computers and other related technologies make a difference in learning?” this controversy is still alive.
Theoretical Background
The most relevant learning theories supporting the model tested in this research are related to studies on interactive multimedia and its impact on the cognitive system (figure 1).
Research Goals and Methodology
There are three main goals associated with this research:
1.The elaboration of a simplified theoretical model to assess learning.
2.The application of a research methodology that overcomes limitations – such as temporal and instructor-based biases - of earlier studies.
3.The delineation of a task-driven matrix on the comparative advantages of different instructional environments.
 
To achieve these goals, the dissertation applies a quasi-experimental design, which involves one factor (learning project management topics such as scheduling tools and change control) with three treatments (learning from the in-class lecture, reading the textbook, or using the interactive multimedia CD-ROM). The participants in the study are graduates and undergraduates enrolled in business administration degree programs at a major University in the East Coast. The study is replicated, under comparable conditions, with the two groups. The purpose of the replication is to investigate whether effectiveness (in recall and application tasks) and satisfaction vary by age (graduate and undergraduate), and by subject topic characteristics (complexity). The latter include topics such as ‘scheduling tools’ (higher complexity topic, or ‘hard-topic’) and ‘change control” (lower complexity topic, or ‘soft-topic’) – where complexity definition is based on a well know cognitive taxonomy (Bloom 1953).

The main and secondary research questions focus on performance and behavioral outcomes:

Question 1: Which instructional delivery environment best supports learning: multimedia, textbooks, or in-class lectures?
§Question 1.1: Which environment is more effective at achieving the learning objectives of recall or application?
§Question 1.2: Which environment is more effective at delivering a low-complexity and high-complexity topic?
Question 2: Which learning environment is more appealing: multimedia, textbooks, or in-class lectures?
§Question 2.1: Which environment, if any, is more appealing to graduates and undergraduates?
 
Figure 2 presents the learning model tested in the dissertation, and summarizes the expected media relationships developed on the basis of an extensive review of the literature.
To test the model presented in Figure 2, several univariate and multivariate data analysis techniques are employed. In addition, exploratory factor analysis is used to identify the underlying model, the number of factors in the model, and the variables associated with each factor. The expectation is that the analysis supports at least a 2-factor model (the recall and application groupings), although it may support other combinations and explain knowledge hierarchies. 
Expected Contributions
This research contributes to research on technology effectiveness in several ways:

§Offering an actionable theoretical framework that integrates different learning models, enabling the research to draw conclusions on the comparative effects of the medium as a whole, rather than on the individual components, methods, or specific media characteristics. 

§Presenting a unique comparative situation that enables the researcher to control for content-differences biases found in earlier studies. The research design overcomes several limitations of these earlier studies (controlling for time, instructor, and content-differences), and presents a framework that addresses previous concerns.
§Differentiating the instructional outcomes. The use of a recognized performance-content matrix (Merrill, 1983) enables to define learning in terms of widely recognized dimensions. 
§Replicating this research with undergraduate and graduate students, with soft and hard topics, and contrasting direction and magnitude (effect size), this research offers further validation procedures. 
Current Status
Currently, the dissertation defense, data collection and data analysis have been completed bye the researcher. The preliminary results from the data analysis are presented in the interim conclusions.
Interim Conclusions
The findings from the data analysis indicate that the effectiveness of technology-based learning is dependent upon the nature of the presented topic. In-class instruction is more suitable for high-complexity topics, while those studying lower-complexity topics benefit from self-paced learning using interactive multimedia software. In terms of learning objectives, student recall performance is higher than application performance in a short-module of instruction. Positive attitudes toward interactive multimedia and textbooks are higher than in-class instruction when the latter uses an average speaker to deliver a soft topic. Attitudes toward in-class instruction are higher when the speaker is an above average presenter or when the topic of instruction is highly complex. In addition, differences in the relationships on comparative performance depend on the nature of the learning task: learning effectiveness of a medium is also related to whether the student is tasked with showing proficiency in the recall or application of facts, concepts, principles, or rules. In an unexpected finding, student performance is not found to be significantly correlated with their attitudes toward the instructional environment.
Open Issues
Was the research successful in solving the effectiveness controversy? The research shows that statistically significant differences exist when comparing learning in short-instructional modules. The research also shows that interesting differences in performance and attitude are found when looking at the subset of learning tasks. In fact, the latter contributions are even more important. The comparisons of learning and attitude are not as indicative and actionable as those that relate recall and application with tasks and topic complexity. The trade-offs of such a detailed, bottom-up approach are represented by the need for focusing the complex results in a few straightforward outcomes. The detailed analysis presents a high challenge: bringing together the several separate pieces and generalizing the findings in immediate applications. This is not an easy task. However, it is in this detailed approach that the research can bear most of its fruits: it shows that focusing on what is learned (and not only how much is learned) offers important insights into the way humans perform in a computer-mediated environment. It also offers several insights on what mix of applications should be used to build effective instructional environments. It remains in the hands of developers, instructional designers, and implementing stakeholders (trainers and educators) the task of skillfully building the right program combinations for supporting specific instructional outcomes and further evaluate the effectiveness of the combination.

REQUIRED INFORMATION
Current Stage in Your Program
I have successfully completed:
§Coursework (Fall 1999)
§Comprehensive exams (Summer 1999)
§Dissertation Proposal defense (March 2000)
And met the requirements set forth in the approved study plan.
What you hope to gain from participating in the Doctoral Consortium
By participating in the SIGCSE Doctoral Consortium, I hope to:
§Receive constructive feedback to improve the dissertation and further expand personal future research focus
§Interact with colleagues and mentors on activities and information sessions on the opportunities and challenges facing an academic career
§Have an opportunity to share research findings as well as lessons learned from conducting an experimental study
§Learn from experiences and findings of the research conducted by other doctoral students
§Expand research horizons by participating to presentations on research in related, as well as diverse, research areas from my own.
BIBLIOGRAPHIC REFERENCES (entire dissertation)
ACM Curriculum Committee for Human-Computer Interaction. (1992). New York: The Association of Computing Machinery Inc.
Arnone, M., and Grabowski, B. (1991). Effects of variations in learner control on children's curiosity and learning from interactive video. Proceedings of Selected Research Presentations at the Annual Convention of the AECT .
Azevedo, R., Shaw, S. G., and Bret, P. M. (1995). The effectiveness of computer-based hypermedia teaching modules for radiology students. Annual Meeting of the American Educational Research Association Proceedings .
Baggett, P. (1979). Structurally equivalent stories in movie and text and the effect of the medium on recall. Journal of Verbal Learning and Behavior, 18, 333-356.
Bagui, S. (1998). Reasons for increased learning using multimedia. Journal of Educational Multimedia and HyperMedia, 7(1), 3-18.
Bain, A., Houghton, S., Sah, F. B., and Carroll, A. (1992). An evaluation of the application of interactive video for teaching social problem solving to early adolescents. Journal of Computer-Based Instruction, 19(3), 92-99.
Bangert-Drowns, R. L., Kulik, J. A., and Kulik, C. C. (1985). Effectiveness of computer-based education in secondary schools. Journal of Computer-Based Instruction, 12(3), 59-68.
Barker, S. P. (1988). Comparison of effectiveness of interactive videodisc versus lecture-demonstration instruction. Physical Therapy, 68(5), 699-703.
Barnes, W. G. W. (1994). Constructing knowledge from an ill-structured domain:Testing a multimedia hamlet. Annual Meeting of the American Educational Research Association Proceedings .
Barrow, L. C., and Westley, B. H. (1959). Comparative teaching effectiveness of radio and television. Audio Visual Communication Review, 7, 14-23.
Barry, M., and Runyan, G. (1995). A Review of Distance-Learning Studies in the U.S. Military . The American Journal of Distance Education, 9(3), 37-47.
Bartram, D. J. (1980). Comprehending spatial information:The relative efficiency of different methods of presenting information about bus routes. Journal of Applied Psychology, 65, 103-110.
Bloom, B. S., Englehart, M. D., Furst, E. J., Hill, W. H., and Krathwohl, D. R. (1956). A taxonomy of educational objectives:Handbook I.The cognitive domain. New York: McKay.
Bork, E. (1987). An investigation of hypermedia. Reading Research and Instruction, 27(4), 5-24.
Borsook, T. (1991). Harnessing the power of interactivity for instruction. In Proceeding of the 1991 Convention of the Association for Educational Communications and Technology (pp. 103-117). Orlando, FL: Association for Educational Communication and Technology.
Bosco, J. (1986). An analysis of evaluations of interactive video. Education Technology, 25, 7-16.

Brown, J. S., Collins, A., and Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32-42.

Buchholz, W. (1991). A learning activity for at-risk ninth through twelfth grade students in creating a computer-generated children's storybook design. Unpublished doctoral dissertation, New York Institute of Technology, New York. ERIC Document Reproduction.

Burns, P. K., and Bozeman, W. C. (1981). Computer-assisted instruction and mathematics achievement:Is there a relationship? Educational Technology, 21, 32-39.

Burton, J. K., Moore, D. M., and Holmes, G. A. (1995). Hypermedia concepts and research:An overview. Computers in Human Behavior, 11(3/4), 345-369.

Campbell, D. T., and Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Boston: Houghton Mifflin Company.

Chen, A. (1993). The use of interactive multimedia systems to improve the learning of classroom practices. Education Research and Perspective, 20(2), 24-32.

Chen, P. (1996). The Effectiveness of Computerized Adult Literacy Education:A Meta-analysis of Literature from 1979 to 1994. Unpublished doctoral dissertation, Purdue University.

Clark, R. E. (1985). Evidence for confounding in computer-based instruction studies:Analyzing the meta-analyses. Educational Communications and Technology Journal, 33(4), 249-262.

Clark, R. E. (1994). Media will never influence learning. Educational Technology Research and Development, 42(2), 21-29.

Clark, R. E. (1983). Reconsidering research on learning from media. Review of Educational Research, 53(4), 445-459.

Clark, R. E. (1991). When researchers swim upstream:Reflections on an unpopular argument about learning from media. Educational Technology, 31(2), 34-40.

Clark, R. E., and Estes, F. (1998). Technology or Craft:What are we doing? Educational Technology, September-October, 5-11.

Clark, R. E., and Sugrue, B. M. (1995). Research on Instructional Media, 1978-1988. G. J. Anglin (Ed.), Instructional Technology: Past, Present, Future (2nd ed., pp. 348-364). Englewood, Colorado: Libraries Unlimited, Inc.

Cohen, M., and Riel, M. (1989). The effect of distance audiences on children's writings. American Educational Research Journal, 26(2), 143-159.

Conklin, J. (1987). Hypertext:An introduction and survey. IEEE Computing, 20(9), 17-40.

Corry, M. D. (1998). Mental Models and Hypermedia User Interface Design. Educational Technology Review, Spring/Summer(9), 20-24.

Craiger, J. P. (2000). Psychology of software design: Usability evaluation. The Industrial Organization Psychologist, 21, 134-144.

Craiger, J. P. (2000) Understanding Users [Web Page]. URL http://hci.ist.unomaha.edu [2000, September 15]. 

Cunningham, D. J., Duffy, T. M., and Knuth, R. A. (1993). The textbook of the future. In C. McKnight, A. Dillon, and J. Richardson (Eds.), Hypertext: A psychological perspective (pp. 19-50). Chichester, England: Ellis Horwood.

D'Alessandro, M. P., Galvin, J. R., Erkonen, W. E., Albanese, M. A., Michaelsen, V. E., Huntley, J. S., McBurney, R. M., and Easley, G. (1993). The instructional effectiveness of a radiology multimedia textbook (HyperLung) versus a standard lecture. Investigative Radiology, 26(1), 643-648.

Dale, E. (1946). Audiovisual methods in teaching. New York: Holt, Rinehart, and Winston.

Davis, D. (1988). Computer-supported cooperative learning systems:Interactive group, technologies and open learning. Programmed Learning and Educational Technology, 25, 205-215.

Delclos, V. R., and Hartman, A. (1993). The impact of an interactive multimedia system on the quality of learning in educational psychology:An exploratory study. Journal of Research on Computing Education, 26(1), 83-93.

Dillon, A., and Zhu, E. (1997). Designing Web-based instruction: A human-computer interaction perspective. In B. Kahn (Editor), Web-based instruction (pp. 221-224). Englewood Cliffs, NJ: Educational Technologies Publications.

Driscoll, M. P. (1994). Psychology of learning for instruction. Boston: Allyn and Bacon.

Duffy, T. M., and Knuth, R. A. (1990). Hypermedia and instruction.Where is the match? In D. H. Jonassen, and H. Mandl (Eds.), Designing hypermedia for learning (Vol. 67pp. 199-225). Berlin: Springer-Verlag.

Dusick, D. M. (1998). The learning effectiveness of educational technology: what does that really mean? Educational Technology Review, Autumn/Winter(10), 10-12.

Engelkamp, J., and Zimmer, H. D. (1994). The human memory: Amultimodal approach. Seattle/Toronto: Hogrefe and Huber Publishers.

Ericsson, K. A., and Simon, H. A. (1984). Protocol analysis: Verbal reports as data. Cambridge, MA: MIT Press.

Fenrich, P. (1997). Practical guidelines for creating instructional multimedia applications. Forth Worth, PA: The Dryden Press.

Fletcher, D. (1989). The effectiveness and cost of interactive videodisc instruction. Machine-Mediated Learning, 3, 361-385.

Fletcher, D. (1990). (Report No. IDA Paper P-2372). Alexandria, VA: Institute of Defense Analyses.

Fletcher-Flinn, C. M., and Gravatt, B. (1995). The efficacy of computer assisted instruction (CAI):A meta-analysis. Journal of Educational Computing Research, 12, 219-242.

Frame, J. D. (1995). Managing Projects in Organizations:How to make the best use of time, techniques, and people. San Francisco: Jossey-Bass Publishers.

Frame, J. D. (1996). The new project management : tools for an age of rapid change, corporate reengineering, and other business realities . San Francisco: Jossey-Bass Publishers.

Gagné, R. M., and Merrill, M. D. (1992). Integrative goals for instructional design. Educational Technology Research and Development, 38(1), 23-30.

Gardner, H. (1984). Frames of mind. New York: Basic Books.

Gardner, H. (1985). The mind's new science:A history of the cognitive revolution. New York: Basic Books.

Gardner, H. (1993). Multiple intelligences:The theory in practice. New York: Basic Books.

Gist, T. M. M., Lorenzen, R., Schmidt, S., Boudot, J., and Fuller, R. (1988-1989). The Air Force Academy Instructor Workstation (IWS):II Effectiveness. Journal of Educational Technology Systems,285-295.

Glass, G. V., McGaw, B., and Smith, M. L. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage Publications.

Gleitman, H. (1991). Psychology. New York: W.W. Norton and Company, Inc.

Goleman, D. (1995). Emotional intelligence:Why it can matter more than IQ. New York: Bantam Books.

Gretes, J. A., and Green, M. (1994). The effect of interactive CD-ROM/digitized audio courseware on reading among low-literate adults. Computers in the Schools, 11(2), 27-42.

Gulliksen, J. (1996). Designing for usability: Domain specific human-computer interfaces in working life. Unpublished doctoral dissertation, Acta University of Uppsala, Uppsala. 

Hair, J. F., Anderson, R. E., Tatham, R. L., and Black, W. C. (1998). Multivariate data analysis. Upper Saddle River, NJ: Prentice Hall.

Hammond, N. (1993). Learning with hypertext:Problems, principles, and prospects. In C. McKnight, A. Dillon, and J. Richardson (Eds.), Hypertext:a psychological perspective (pp. 51-69). Chichester, England: Ellis Horwood.

Hannafin, M. J. (1989). Interaction strategies and emerging instructional technologies: Psychological perspectives. Canadian Journal of Educational Communications, 18(3), 167-179.

Harlow, S. D., and Johnson, D. L. (1998). An epistemology of technology. Educational Technology Review, Spring/Summer(9), 15-19.

Hartely, S. S. (1977). Meta-analysis of the effects of individually paced instruction in mathematics.Vol. 38(4003A), University Microfilms No. 77-29, 926. 

Havice, W. L. (1994). Attitudes, achievement, and reactions of college students toward integrated media presentation in teaching of computer information system. Unpublished doctoral dissertation, Kansas State University, Kansas. 

Heinich, R., Molenda, M., and Russell, J. D. (1993). Instructional media and the new technologies of instruction (4th ed. ed.). New York: Macmillan.

Heller, and Martin. (1995). A media taxonomy. IEEE Multimedia, (Winter), 36-45.

Hess, G. J. Jr. (1994). Strategies for integrating computer-based training in college music theory courses. ED-MEDIA 94 - World Conference on Educational Multimedia and Hypermedia .

Iuppa, N. V. (1984). A practical guide to interactive video design. White Plains, NY: Knowledge Industry Publications, Inc.

Jacobson, M. J., and Spiro, R. J. (1995). Hypertext learning environments, cognitive flexibility, and the transfer of complex knowledge:An empirical investigation. Journal of Educational Computing Research, 12(4), 301-333.

Janda, K. (1993). Teaching American Government in an Electronic Classroom. 85th Annual Meeting of the American Political Science Association .

Johnson, J. V., and Johansson, G. (1991). The psychosocial work environment: Work organization, democratization, and health. Amityville, NY: Baywood Publishing Company, Inc.

Johnston, R. (1995). The Effectiveness of Instructional Technology:A Review of the Research. Proceedings of the Virtual Reality in Medicine and Developers' Exposition Cambridge, MA: Virtual Reality Solutions, Inc.

Jonassen, D. H. (1985). Interactive lessons designs: A taxoomy. Educational Technology, 25(6), 7-17.

Jonassen, D. H. (1990). Semantic network elicitation: Tools for structuring hypertext. In C. Green, and R. McAleese (Eds.), Hypertext: State of the art . Oxford: Intellect Books Ltd.

Jonassen, D. H., and Grabinger, R. S. (1989). Problems and issues in designing hypertext/hypermedia for learning. Designing Hypermedia for learning . New York: Springer-Verlag.

Jonassen, D. H., Hennon, R. J., Ondrusek, A., Samouilova, M., Spaulding, K. L., Yueh, H. P., Lit, T., and Nouri, V. (1997). Certainty, determinism, and predictability in theories of instructional design:Lessons from science. Educational Technology, 37(1), 27-34.

Jones, T. H., and Paolucci, R. (1998). The learning effectiveness of educational technology:A call for further research. Educational Technology Review, Spring/Summer(9), 10-14.

Katz, L. (1992). Essentially multimedia: An explanation of interactive laserdisk and optical technology. The Canadian Multi Media Magazine, 1(1), 18-20.

Kearsley, G., Lynch, W., and Wizer, D. (1995). The effectiveness and impact of on-line learning in graduate education. Educational Technology, 35(6), 37-42.

Keller, J. M. (1983). Motivational design of instruction. Reigeluth (Ed.), Instructional design theories and models:An overview of their current status (pp. 383-434). NJ: Hillsdale.

Kemp, J. E., Morrison, G. R., and Ross, S. M. (1994). Designing effective instruction. Englewood Cliffs, NJ: Merrill/Prentice Hall.

Khalili, A., and Shashaani, L. (1994). The effectiveness of computer applications:A meta-analysis. Journal of Research on Computing in Education, 27(1), 48-61.

Kieras, D. E., and Bovair, S. (1983). The role of a mental model in learning how to operate a device . Technical Report No. 13, Department of Psychology, University of Arizona, Tucson, AZ, UARZ/DP/TR-83/ONR-13.

Kieras, D. E., and Bovair, S. (1984). The role of a mental model in learning to operate a device. Cognitive Science, 8, 255-273.

Kinzie, M. B., Strauss, R., and Foss, J. (1993). The effects of an Interactive Dissection simulation on the performance and achievement of high school biology students. Proceedings of Selected Research and Development Presentations at the Convention of the Association of for Educational Communications and Technology sponsored by the Research and Theory Division .

Kohls, L. R. (1981). Developing intercultural awareness. Washington DC: SIETAR.

Kozma, R. B. (1991). Learning with media. Review of Educational Research, 61(2), 179-211.

Kozma, R. B. (1994). Will media influence learning?Reframing the debate. Journal of Educational Technology Research and Development, 42(2), 7-19.

Kulik, C. C., and Kulik, J. A. (1986). Effectiveness of computer-based instruction in colleges. AEDS Journal , 19, 81-108.

Kulik, C. C., and Kulik, J. A. (1990). Conventional and newer statistical methods in meta-analysis. Annual Meeting of the American Educational Research Association .

Kulik, C. C., and Kulik, J. A. (1991). Effectiveness of computer-based instruction:An updated analysis. Computers in Human Behavior, 7, 75-94.

Kulik, C. C., Kulik, J. A., and Shwalb, B. J. (1986). The effectiveness of computer-based adult education:A meta-analysis.Journal of Educational Computing Research, 2(2), 235-252.

Kulik, J. A. (1983). Synthesis of research on computer-based instruction. Educational Leadership, 41(1), 19-21.

Kulik, J. A. (1985). Consistencies in findings on computer-based education. American Educational Research Association .

Kulik, J. A. (1987). Computer-based instruction:What 200 evaluations say. Paper presented at the Annual Convention of the Association for Educational Communication and Technology .

Kulik, J. A. (1994). Meta-Analytic Studies of Findings on Computer-Based Instruction. E. L. Baker, and H. F. O'Neil (Eds.), Technology Assessment in Education and Training (pp. 9-33). Hillsdale, NJ: Laurence Erlbaum Associates.

Kulik, J. A., Bangert, R., and Williams, G. (1983). Effects of computer-based teaching on secondary students. Journal of Educational Psychology, 75, 19-26.

Kulik, J. A., and Kulik, C. C. (1987). Review of recent research literature on computer-based instruction. Contemporary Educational Psychology, 12, 222-230.

Kulik, J. A., and Kulik, C. C. (1989). Meta-analysis:Historical origins and contemporary practice. Annual Meeting of the American Educational Research Association .

Kulik, J. A., Kulik, C. C., and Bangert-Drowns, R. L. (1985). Effectiveness of computer-based education in elementary schools. Computers in Human Behavior, 1, 59-74.

Kulik, J. A., Kulik, C. C., and Cohen, P. A. (1980). Effectiveness of computer-based college teaching:A meta-analysis of findings. Review of Educational Research, 50, 525-544.

Lashley, K. S. (1960). In search of the engram. F. A. Beach, D. O. Hebb, C. T. Morgan, and A. W. Nissen (Eds.), The neuropsychology of Lashley:Selected papers of K. S. Lashley (pp. 478-507). New York: McGraw-Hill.

Lave, J. (1988). Cognition in Practice:Mind, mathematics, and culture in everyday life. Cambridge, UK: Cambridge University Press.

Leidner, D., and Jarvenpaa, S. L. (1993). The Information Age Confronts Education:Case Studies on Electronic Classrooms. Information Systems Research, 4(1), 24-54.

Leidner, D., and Jarvenpaa, S. L. (1995). The Use ofInformation Technology to Enhance Management School Education:Theoretical View. MIS Quarterly, September, 265-291.

Leonard, W. H. (1992). A comparison of student performance following instruction by interactive videodisc versus conventional laboratory. Journal of Research in Science Teaching, 29(1), 93-102.

Levie, W. H., and Lentz, R. (1982). Effects of text illustrations: a review of research. Educational Communication and Technology Journal, (30), 195-232.

Liao, Y. C. (1998). Effects of Hypermedia versus traditional instruction on students' achievement:a meta-analysis. Journal of Research on Computing in Education, 30(4), 341-359.

Liu, M., and Reed, W. M. (1995). The effects of hypermedia assisted instruction on second language learning. Journal of Educational Computing Research, 12(2), 159-175.

Marks-Tarlow, T. (1995). Creativity inside out:Learning through multiple intelligences. Reading, MA: Addison-Wesley.

Marquardt, M. J., and Kearsley, G. (1999). Technology-based learning. Boston: St. Lucie Press.

Marsh, E. G. (1999) Instructional Design [Web Page]. URL http://www.bamaed.ua.edu/ail601/development.htm [1999, October 1]. 

Martin, E. D., and Rainey, L. (1993). Achievement and Attitude in a Satellite-Delivered High School Science Course. The American Journal of Distance Education, 7(1).

Mayer, R. E. and. A. R. B. (1991). Animations need narrations:An experimental test of a dual-coding hypothesis. Journal of Educational Psychology, 83, 484-490.

McClure, P. A. (1996). Technology Plans and Measurable Outcomes. Educom Review, 31(3), 29-30.

McCoy, L. P. (1994). Decisions in interential statistics with HyperCard:Design and field test. Computers in the Schools, 10(1/2), 69-77.

McLellan, H. (1996). 'Being Digital':Implications for Education. Educational Technology , Nov-Dec, 5-20.

McLuhan, M., and Fiore, Q. (1967). The medium is the message. New York: Bantam Books.

Merrill, M. D. (1983). Component display theory. C. M. Reigeluth (Ed.), Instructional-design theories and models:An overview of their current status (pp. 282-333). Englewood Cliffs, NJ: Erlbaum.

Moore, J. E., and Burke, L. A. (1997). A Conceptual Framework for Research in IS/IT Instruction. Proceedings of the International Academy for Information Management 12th Annual Conference.

Moore, M. G., and Kearsley, G. (1996). Distance Education. Belmont, California: Wadsworth Publishing Company.

Najjar, L. J. (1996). Multimedia Information and Learning. Journal of Educational Multimedia and Hypermedia, 5, 129-150.

Newby, T. J., Stepich, D. A., Lehman, J. D., and Russell, J. D. (1996). Instructional Technology for Teaching and Learning:Design Instruction, Integrating Computers, and Using Media. Englewood Cliffs, NJ: Prentice Hall.

Nugent, G. (1982). Pictures, audio, and print:Symbolic representation and effect on learning.Educational Communication and Technology Journal, 30, 163-174.

Nygren, E., Johnson, M., Lind, M., and Sandblad, B. (1992). The art of the obvious: Automatically processed components of the task of reading frequently used documents. Implications for task analysis and interface design. In Proceedings of Human Factors in Computing Systems (pp. 235-239). Monterey, California: ACM.

Olivier, T. A., and Shapiro, F. (1993). Self-efficacy and computers. Journal of Computer-Based Instruction, 20(3), 81-84.

Overbaugh, R. C. (1995). The efficacy of interactive video for teaching basic classroom management skills to preservice teachers. Computers in Human Behavior, 11(3/4), 511-527.

Pace, W., Smith, P., and Mills, G. (1991). Human resource development. Englewood Cliffs, N.J.: Prentice Hall.

Paivio, A. (1986). Mental representations.A dual-coding approach. New York: Oxford University Press.

Palmiter, S., and Elkerton, J. (1991). An evaluation of animated demonstrations for learning computer-based tasks. CHI'92 conference proceedings Addison-Wesley.

Park, I., and Hannafin, M. J. (1993). Empirically-based guidelines for the design of interactive multimedia. Educational Technology Research and Development, 41(3), 63-85.

Pask-McCartney, C. (1989). A discussion about motivation. Proceedings of Selected Research Presentations at the Annual Convention of the AECT (p. ERIC Document Reproduction ). 

Piaget, J., and Inhelder, B. (1969). The Psychology of the Child. New York: Basic Books.

Polson, P., and Lewis, C. (1990). Theory-based design for easily learned interfaces. Human-Computer Interaction, 5, 191-220.

Preece, J., Rogers, Y., Sharp, H., and Benyon, D. (1994). Human-computer interaction. New York, NY: Addison-Wesley.

Reeves, T. C. (1993). Evaluating Interactive Multimedia. D. M. Gayesky (ed), Multimedia for learning, development, application, evaluation (pp. 97-112). Englewood Cliffs, NJ: Educational Technology.

Reeves, T. C. (1998). 'Future Schlock,' 'The Computer Delusion,' 'The End of Education':Responding to Critics of Educational Technology. Educational Technology, September-October, 49-53.

Reigeluth, C. M. (1983). Instructional-design theories and models:An overview of their current status. Englewood Cliffs, NJ: Lawrence Erlbaum.

Roberts, M. R. (1993). A Comparison in the Effectiveness of the Delivery of an Interactive Computer-Assisted Instruction Module to a Traditional Lecture/Lab Delivered Module. Unpublished doctoral dissertation, West Virginia University, Morgantown, WV. 

Roblyer, M. (1992). Computers in education. G. Bitter (Ed.), Macmillan encyclopedia of computers . New York: Macmillan.

Roblyer, M. (1993). Why use technology in teaching:Making a case beyond research results. The Florida Technology in Education Quarterly, 5(4), 7-13.

Roblyer, M., Edwards, J., and Havriluk, M. A. (1997). Planning and Implementation for Effective Technology Integration. Integrating Educational Technology into Teaching (pp. 27-53). Upper Saddle River, NJ: Merrill, Prentice Hall.

Roblyer, M. D., Castine, W. H., and King, F. J. (1988). Assessing the impact of computer-based instruction. Computers in the Schools, 5(3-4).

Rojewski, J. W., Gilbert, J. P., and Hoy, C. A. (1994). Effects of a hypertext computer program on academic performance in a special education course for nonmajors. Teacher Education and Special Education, 17(4), 249-259.

Salomon, G. (1984). Television is "easy" and print is "tough":The differential investment of mental effort in learning as a function of perceptions and attributions. Journal of Educational Psychology, 76, 647-658.

Sandbald, B., Lind, M., and Nygren, E. (1991). Cognitive work environments problems. Uppsala: Uppsala University.

Schunk, D. H. (1991). Learning theories:An educational perspective. Englewood Cliffs, NJ: Merrill/Prentice Hall.

Schwier, R. A., and Misanchuk, E. R. (1993).Interactive multimedia instruction. Englewood Cliffs, NJ: Educational Technology Publications.

Seel, N., and Winn, W. D. (1997). Research on media and learning:Distributed cognition and semiotics. R. D. Tennyson, F. Schott, N. Seel, and S. Dijkstra (Eds.), Instructional design:International perspectives, Vol. 1:Theory and research (pp. 293-326). Mahwah, NJ: Lawrence Erlbaum Associates.

Shambaugh, R. N., and Magliaro, S. G. (1997). Mastering the possibilities:A process approach to Instructional Design. Boston: Allyn and Bacon.

Sheldon, E. J. (1995). CAI/multimedia approach to farm tractor and machinery safety certification. Unpublished doctoral dissertation, Purdue University.

Shiffrin, R. M., and Dumais, S. T. (1981). The development of automatism. In. Hillsdale, NJ: Erlbaum.

Shih, Y. F., and Alessi, S. M. (1996). Effects of text versus voice on learning in multimedia courseware . Journal of Educational Multimedia and Hypermedia, 5(2), 203-218.

Sims, R. (1997). Interactivity: A forgotten art? Computers in Human Behavior, 13(2), 157-180.

Sims, R. (1994). Seven levels of interactivity: implications for the development of multimedia education and training. InProceedings of the Asia-Pacific Information Technology in Training and Education (APITITE) Conference Brisbane, Qld.: APITITE.

Smith, S. G., Jones, L. L., and Waugh, M. L. (1986). Production and evaluation of interactive videodisc lessons in laboratory instruction. Journal of Computer-Based Instruction, 13(4), 117-121.

Souder, W. F. (1993). The Effectiveness of Traditional vs. Satellite Delivery in Three Management of Technology Master's Degree Programs. The American Journal of Distance Education, 7(1).

Spiro, R. J., Feltovich, P. J., Jacobson, M. J., and Coulson, R. L. (1991). Cognitive flexibility, constructivism, and hypertext:Random access instruction for advanced knowledge acquisition in ill-structured domains. Educational Technology, 31(5), 24-33.

Spiro, R. J., and Jehng, J. (1990). Cognitive flexibility and hypertext:Theory and technology for the nonlinear and multidimensional of complex subject matter. D. Nix, and R. Spiro (Eds.), Cognition, education, and multimedia:Exploring ideas in high technology (pp. 163-205). Hillsdale, NJ: Lawrence Erlbaum Associates.

Steinberg, E. R. (1991). Computer-Assisted Instruction:A Synthesis of Theory, Practice, and Technology. Hillsdale, New Jersey: Lawrence Erlbaum Associates.

Stemler, K. L. (1997). Educational characteristics of multimedia:A literature review. Journal of Educational Multimedia and Hypermedia, 6(3/4).

Summers, J. (1990-1991). Effect of interactivity upon student achievement, completion intervals, and affective perceptions. Journal of Educational Technology Systems, 19(1), 53-57.

Sweany, N. D., McManus, T. F., Williams, D. C., and Tothero, K. D. (1996). The use of cognitive and metacognitive strategies in a hypermedia environment. Proceedings of the EdMedia Conference Boston, MA.

Sweeters, W. (1994). Multimedia electronic tools for learning. Educational Technology, 34(5), 47-52.

Tabar, C. R. (1990). Computer-assisted interactive video instruction: An alternative to lecture method for nutrition education in baccalaureate nursing. Unpublished doctoral dissertation, University of Cincinnati, Cincinnati, Ohio. 

Tennyson, R. D., and Rasch, M. (1988). Linking cognitive learning theory to instructional prescriptions. Instructional Science, 17, 369-385.

Tennyson, R. D., and Milt Neilsen. (1998). Complexity Theory:Inclusion of the affective domain in an interactive learning model for instructional design. Educational Technology, November-December, 7-12.

Tergan, S.-O. (1997). Multiple views, contexts, and symbol systems in learning with hypertext/hypermedia:A critical review of research. Educational Technology, July-August, 5-18.

Thamhain, H. J. (1991). Developing Project Management Skills. Project Management Journal, XXII(3), 39-44.

Thuring, M., Hannemann, J., and Haake, J. (1995). Hypermedia and cognition: Designing for comprehension. Communications of the ACM, 38(8), 57-66.

Tolhurst, D. (1995). Hypertext, Hypermedia, Multimedia Defined? Educational Technology, March-April, 21-26.

Tomlinson, H., and Henderson, W. (1995). Computer Supported collaborative learning in schools:A distributed approach. British Journal of Educational Technology , 26(2), 131-140.

Toro, M. A. (1995). The effects of HyperCard authoring on computer-related attitudes and Spanish language acquisition. Computers in Human Behavior, 11(3/4), 633-647.

Tulving, E. (1983). Elements of episodic memory. London: Oxford University Press.

Vaill, P. (1996). Learning as a Way of Being:Strategies for Survival in a World of Permanent White WaterJossey-Bass Publishers.

Vaughan, T. (1996). Multimedia:Making It Work (3rd. ed.). Berkeley, CA: Osborne McGraw-Hill.

Volker, R. (1992). Applications of constructivist theory to the use of hypermedia. Proceedings of Selected Research Presentations at the Annual Convention of the AECT .

Walker, N., Jones, J. P., and Mar, H. H. (1983). Encoding processes and the recall of text. Memory and Cognition, 11, 275-282.

Waterworth, J. A. (1992). Multimedia interaction with computers: human-factors issues. Chichester, SUSSEX: Ellis Horwood Ltd.

Welsh, T. M., Murphy, K. P., Duffy, T. M., and Goodrum, D. A. (1993). Accessing elaborations on core information in a hypermedia environment. Educational Technology Research and Development, 41(2), 19-34.

Wirth, I., and Amos, S. J. (1996). Distance learning for project management. Transactions of AACE International,1-6.


[1]The determination of topic complexity is based on Bloom’s taxonomy of learning objectives (1956).