Abstract

This study investigated the use of scaffolds in problem-based hypermedia. Three hundred and twelve undergraduate students enrolled in a computer literacy course worked in project teams to use a problem-based, hypermedia program focused on designing a personal computer. The program included content scaffolds, metacognitive scaffolds, or no scaffolds. Results revealed that posttest scores for students who received content scaffolds were significantly higher than those who received metacognitive scaffolds. Type of scaffolds also had a significant impact on student attitudes. Findings have implications for the design and delivery of problem-based, hypermedia. Content scaffolds can direct student attention to important information and encourage understanding. However, considerations should be given to the difficulty of the task, the time allotted to solve the problem, and other demands students face in a problem-based, hypermedia environment. USING SCAFFOLDS IN PROBLEM-BASED HYPERMEDIA Problem-based learning (PBl) is an instructional approach used to prepare students to become better problem solvers for today's information society. an essential characteristic of PBl is the use of a problem to focus and anchor learning. PBl emphasizes active knowledge building while students solve problems rather than exposing them to discipline knowledge before problem solving 222 Su and Klein ible and reusable knowledge, accumulate problem solving skill, gain selfdirected learning ability, and generate high intrinsic motivation Research has been conducted to evaluate the effectiveness of PBl or compare it with conventional instructional approaches on various learning outcomes. norman and Schmidt (1992) reviewed experimental studies on PBl and found that it helped students transfer more concepts and integrate them into problems more effectively than those who did not participate in PBl. Their review also suggested that students' intrinsic interests and selfdirected learning skills appeared to be enhanced in PBl. However, they reported that PBl may decrease immediate knowledge acquisition. a recent meta-analysis by Dochy, Segers, Bossche, and Gijbels (2003) reviewed two major PBl outcomes -declarative information and knowledge application -in 43 empirical studies. Results revealed a robust positive effect for PBl on knowledge application but a slightly negative effect for PBl on declarative information. While students in PBl gained less knowledge immediately after instruction, they recalled more knowledge on delayed retention tests. While much of the research on PBl has been conducted in classroom settings, studies have also investigated PBl supported by computer technology, especially hypermedia technology. Hypermedia can be used as an "exploration tool to support early unstructured thinking on a problem when many disconnected ideas come to mind" Researchers studying hypermedia-based PBL have identified some of its benefits to learners. Pedersen (2003) developed a hypermedia-based PBL program and investigated student motivation during its implementation. Results indicated that the program significantly promoted intrinsic motivation. PBl students felt more challenge, perceived that they had more control over their learning program, and collaborated more with their classmates, than students in the regular classroom. 223 Using Scaffolds in Problem-based Hypermedia While hypermedia-based PBl can increase student motivation and learning, various problems exist when it is applied in real settings. Disorientation and increased cognitive load are two major problems associated with the richness, variety and freedom of hypermedia Scaffolding shapes the way students interact in a PBl environment by imposing additional structure to their learning Researchers have proposed different categorizations for scaffolds. Saye and Brush (2002) grouped them into two types based on their flexibility. Soft scaffolds refer to dynamic and situational supports that require teachers to continuously diagnose learners learning situation and provide them with just enough support in a timely manner. Hard scaffolds are static supports that can be predicted and planned in advance based on anticipated student difficulties during learning. Hannafin, Land, and Oliver (1999) categorized hard scaffolds into four types -conceptual, strategic, procedural and metacognitive scaffolds. azevedo, Cromley, it has been suggested that PBl students are likely to focus exclusively on problem solutions without paying enough attention to content knowledge Content scaffolds will mainly benefit students' understanding towards content knowledge instead of overall problem-solving performance Simons and Klein in addition to scaffolding, student prior knowledge plays a key role in hypermedia PBl. While advocates of open-ended learning environments such as hypermedia-based PBL suggest that scaffolds may reduce some of the difficulties students have in these settings, research does not clearly show which scaffolds most benefit learners. The main purpose of the current study was to examine the effect of content scaffolds and metacognitive scaffolds on problem-solving performance, knowledge acquisition and student attitudes during the implementation of a hypermedia-based PBl lesson. Prior knowledge was also examined because other studies have shown that it is related to student outcomes in PBl. 226 Su and Klein METHOD Participants & Design The participants in this study were 312 undergraduate college students enrolled in 20 sections of a computer literacy course at a large southwestern university in the united States. The computer literacy course was offered through the College of education at the university as a general studies elective. Demographic data showed that 63% of participants were female, 36% were freshmen, 28% were sophomores, 24% were juniors, and 12% were seniors. most participants were from a non-computer major and 42% were education majors. a 3 × 2 factorial research design was used in this study. The independent variables were types of scaffolds (no scaffolds, content scaffolds, metacognitive scaffolds) and prior knowledge (high versus low). Dependent measures included problem solving performance, posttest achievement, attitudes, and time solving the problem. additional data sources included student navigation patterns and how teams approached the problem solving task. The crossing of the factors of the independent variables resulted in the following six treatment groups: (1) low prior knowledge students with no scaffolds, (2) high prior knowledge students with no scaffolds, (3) low prior knowledge students using content scaffolds, (4) high prior knowledge students using content scaffolds, (5) low prior knowledge students using metacognitive scaffolds, and (6) high prior knowledge students using metacognitive scaffolds. Materials The materials in this study included a hypermedia-based PBl lesson, content and metacognitive scaffolds, a teacher guide, and a test of prior knowledge. These materials are described below. Hypermedia-based PBL Lesson. a web-delivered, hyper-media-based PBl program called All You Need is a Screwdriver was used in this study. The content was part of the curriculum of the computer literacy course. at the center of the program was an ill-structured problem scenario which required student teams to design a functional desktop computer for home use within a limited budget. using this problem as a context, directions were provided and resources were organized into a hypermedia database to facilitate student learning. The program was designed and developed by the principal investigator based on guidelines for open-ended learning environments The program included six sections. Students could navigate to each one by selecting a link on a menu bar located at the top of each screen. The InUsing Scaffolds in Problem-based Hypermedia 227 troduction opened the program; it was designed to gain students' attention and inform them of the purpose of the lesson. The Directions introduced the four main units in the lesson and provided information about how grades would be assigned and how much time was allotted for the lesson. The Project unit described the problem solving task. Project teams were told that they would design a home computer within a limited budget. They were also told that their team was required to submit a drawing displaying selected computer components and show lines as cables to connect components together. The Procedure to Follow gave students a list of ten steps to complete while working on the project. it was designed to explain the steps of problem-based learning. The Learning Resources included a collection 28 articles adapted from the internet introducing computer parts, explaining how they work and suggesting what to consider when shopping for them in the warehouse. The Hardware Warehouse simulated an online shopping site listing five to fifteen models for each computer part. Detailed technology specifications and price were also listed for each model. Three version of the PBl program were developed corresponding to the scaffolding conditions under study. No Scaffolds. Participants in this treatment group received only the instructional materials described above. These students had no access to scaffolds throughout the PBl lesson and were used as the control group in this study. Content Scaffolds. Three content scaffolds were provided to students as hard copies in this treatment condition: (1) a warm-up sheet, (2) a notetaking sheet, and (3) a project template. Students were cued in Procedure to Follow to use these required scaffolds. The purpose of the warm-up sheet was to introduce students to basic computer hardware and prepare them for the project. This scaffold guided students to read two introductory articles in the Learning Resources section. it contained two questions about computer parts and their functions and two questions about devices and their connections to ports, socket, and slots on a motherboard. One example of a question on the warm-up sheet is "list main components of a typical desktop computer." The note-taking sheet directed student attention to key terms and principles in the content. This content scaffold also contained 18 short-answer questions asking students to describe the functions of certain computer parts or explain technology specifications. One example of a question on the 229 Using Scaffolds in Problem-based Hypermedia note-taking sheet is "How do you tell if a CPu can be plugged into a motherboard?" The project template provided a sample layout for the computer design drawing and listed elements students should address when making purchasing decisions. Metacognitive Scaffolds. Three metacognitive scaffolds were provided to students in this treatment group: (1) a project planning sheet (2) an information collection log, and (3) a project reflection sheet. They were designed to provide students with guidance in planning, monitoring, and evaluating, three metacognitive skills required in open-ended learning environments (land, 2000). Students were cued in the Procedure to Follow to use these required scaffolds. The project planning sheet was developed to guide project teams to identify their goals, define the problem area, and identify available resources. The design of this scaffold was guided by the six-step problem solving approach advocated by eisenberg and Berkowitz (1990). The information collection log was designed to facilitate the metacognitive skill of monitoring. The scaffold explicitly asked students to write down tasks assigned by the team on a sheet and make notes to information they collected during their individual research. land Teacher Guide. a teacher guide was used to standardize implementation of the program across the 20 sections of the course. This guide provided step-by step directions to help the teacher introduce the unit, form problemsolving teams, announce important issues, collect problem solutions, and administer the posttest and attitude survey. Prior knowledge test. The prior knowledge test was a 10-question multiple-choice test designed to measure students' prior knowledge of computer hardware. it was used to block students by ability to ensure equality across the three treatment conditions. Data Sources Data sources included an achievement posttest to measure individual knowledge acquisition, a team project to measure problem solving performance, a student attitude survey, a measure of time on the problem solving 230 Su and Klein task, student navigation patterns, and a student interview. each of these data sources are described below. Posttest. The posttest contained 20 multiple-choice items to measure individual student learning on declarative knowledge and factual information learning. it contained eight questions on the meaning of technology specifications, six questions on functions of computer parts, three questions on connections among computer parts and directions of the data flow. It also included one question on the component compatibility, one question on the usage purpose, and one question on selecting computer components. The test was administered online after each team finished the group project. Two examples of questions on the posttest are "Which of the following component is "a must" for a personal computer?" and "Which of the following items does NOT have a positive impact on the CPU's performance?" The posttest was tested in four pilot tests conducted prior to implementation of the study. item analysis was performed on data collected from each pilot to examine the difficulty index and discrimination index. A few items were removed, while others were modified to better align with the computer-building project and to relate more closely to knowledge students may have acquired during problem-solving. Problem-solving performance. a rubric was developed by the principal investigator and was used to evaluate problem-solving performance. it was worth a total of 65 points and measured if the computer designed by each project team was functional and met performance and budget requirements. each computer component was assessed separately in the rubric. There were a total of thirteen components assessed using the criteria of performance, compatibility and connectivity. The rubric also measured if the overall budget fell within the range. To ensure the rubric validly measured the computer building task, it was reviewed be a graduate student majoring in engineering who had experience building computers. inter-rater reliability was determined by calculating the correlation between two sets of scores from two independent graders on ten projects. Result indicated that the correlation was .99 (p < .001). Attitude survey. The attitude survey had 16 questions designed to investigate student perceptions toward the PBl program, the support provided by the program and their team. These survey items were likert-type which included a 5-option scale ranging from "strongly disagree" to "strongly agree." The survey also included two open-ended questions asking students what they liked most about the program and what could be done to improve it. The reliability of this attitude survey was .89. Time on task. Students were asked to write down the time when their team started the project and the time when they finished it on their final project sheets. Student interviews. a sample of students was randomly selected from the three treatment groups to conduct interviews. an interview protocol contained questions to probe how student teams approached the problem solving project, for example, "What did your group mostly consider when making decisions?" Procedures This study was conducted in computer labs during regular class time of the computer literacy course. each class section met once per week for a three-hour time block and had an enrollment of 19 to 24 students. each student had access to a computer. each intact class section was randomly assigned to one of the three groups. Two weeks before the treatment, the prior knowledge test was administered to all students. A one-way ANOVA conducted on pretest scores indicated no significant difference between treatment groups F(2, 309) = .51, p > .05. in each group, the students were divided into two prior knowledge levels (high and low) by the overall median pretest score. Furthermore, students in each class section were randomly assigned to project teams prior to implementing the treatments. On the day of the study, the instructor assigned each student to one of the project teams based on the team list given by the lead researcher. each team consisted of three to four students. Students were then directed to the instructional program website. each project team was given one sheet of blank paper on which to draw their computer design. For classes in the two scaffold conditions, each project team received an envelope containing hard copies of the appropriate scaffold sheets and was told they were required to use those sheets. Two hours after students started their projects, the instructor collected the computer drawing from each team and directed students to the website where they took the individual posttest and attitude survey. a sample of students from each treatment then participated in an interview. Data Analysis Two separate 3 × 2 ANOVAs were carried out on posttest scores and navigation paths to detect if scaffolds and prior knowledge had an impact on the two dependent variables. Two separate one-way ANOVAs were performed on group project performance and time on task to identify if there was any significant difference among scaffolding types on the two variables. Attitude survey and interview data were also analyzed with appropriate methods. 232 Su and Klein RESULTS Posttest Achievement The first research question investigated the effect of scaffolds and prior knowledge on individual posttest achievement. A 3 × 2 ANOVA revealed significant main effects for type of scaffolding [F(2, 306) = 3.22, p < .05, partial ή 2 = .02] and for prior knowledge [F(1, 306) = 5.90, p < .05, partial ή 2 = .02]. The ANOVA did not show a significant interaction between scaffolds and prior knowledge. a follow-up Tukey HSD test indicated that posttest scores for students in the content scaffolding treatment were significantly higher (M = 11.89) than scores for students in the metacognitive scaffolding treatment (M = 10.82). Cohen's f statistic yielded an effect size estimate of .14 which corresponds to a small effect. Students with high prior knowledge performed significantly better on the posttest (M = 11.72) than those with low prior knowledge (M = 10.88). Cohen's f statistics yielded an effect size estimate of .29 for prior knowledge which indicates a medium effect.