Monthly Archives: February 2014
We did case studies in EDTECH 562 so we can see how evaluation is done in the real world. How are statistics used to validate what was done during the research process? Here is an example of one of the case studies I completed while taking the class.
EDTECH 562: Module 4 Case Study
Submit to Module 4: Case Study
Please read the Module 4 Case Study file: Li, Q. (2010). Inquiry-based learning and e-mentoring via videoconference: A study of mathematics and science learning of Canadian rural students. Educational technology research and development. 58(6), 729-753.
EDTECH 562: Module 4 Case Study
Your Name: Melissa Getz
1. Research question:
How does providing eighth grade math students living in a rural setting an opportunity to interact with people who do research allow for a more authentic experience, thereby increasing students achievement and interest in math and science?
According to the paper, the research questions they asked are:
- How does the experience in an IBLE affect rural students’ learning of math and science? Specifically,
- Does the overall learning experience in an IBLE environment improve rural students’ achievement in mathematics as demonstrated in test scores?
- In what ways does the overall learning experience in an IBLE environment impact rural students’ affective development in math and science?
- What are the challenges of establishing an IBLE environment in a rural context?
2. Research strategy used:
Before bringing the students into the activities, the adults did a bit of planning. As a team, they created projects for the students to do with eMentors. They identified overarching themes, by focusing on the overarching questions:
How does understanding multiple perspectives shape the way we live in the world? In what ways does diversity shape our understanding?
After identifying the themes, they brainstormed project ideas and designed the project structure.
The formation of the inquiry projects was based on these three questions:
- What are the curriculum topics that need the most attention?
- What topics will engage students?
- How can they match eMentors to students so that students benefit the most from their interactions with the eMentors.
Following the planning, they implemented an action plan that involved the students interacting with the eMentors and completed the project by doing the post-tests and student interviews.
There were two control groups (41 students) and one experimental group (26 students) whose post-test scores were compared. The research group also did a pre-test so that changes between the beginning of the project and the end of the project could be measured. Nine of the students in the experimental group were personally interviewed to collect evidence of students’ attitudes about the experience.
The research team used both quantitative analysis and qualitative data. The quantitative analysis was generated twice:
1. is there a statistical difference in post-test scores between the control and the treated group?
2. is there a statistical difference in pre-test and post-test scores for the treatment group?
Interviews were conducted with nine students in the treatment group so as to not disrupt their courses too much. All nine students were interviewed alone or in pairs three times during the project. They felt the number adequately covered the population because the students were chosen based on having representation from a variety of academic backgrounds as well as having a small enough group with which to develop trust and confidence between the researchers and the students.
3. Independent variable(s):
Independent variables are the ones the researchers manipulate. That is a definition for independent variable which I translate to mean the researchers are choosing a variable that can allow for output as a result of doing the experiment. For example, if they chose temperature, it would influence the experiment in a way that causes there to be output that is specific to the temperature of the experiment. Or time can be an independent variable because as it happens something else changes. The independent variable itself does not give us information that is used in the statistical analysis, but the output it can cause is used. The output also comes from dependent variables that depend on the independent variable to know how to behave.
In this situation, there is the variable of time because we have pre-tests and post-tests. The output on the pre and post tests depends on the experimental timing- had the students done the inquiry lab with the scientists as support or not? The tests themselves would also be an independent variable because the student responses to the tests gives us data- the student responses are a dependent variable that relies on the test to provide an output. An independent variable here also involves if the students interacted with an eMentor or not. We decided who worked with the eMentor and the output we will be measuring is the students’ gain in interest in math because they worked with an eMentor. The students’ opinions are dependent on whether or not they had access to an eMentor.
There is also the variable that we are working with children. Their output is a dependent variable- it is not predictable and is based on their doing the math that was in the assignment.
This also brings up another independent variable which probably should have been listed first because it is the main difference between what happens to the experimental and the control groups: who gets to work with the scientists? Which group of kids gets the eMentoring?
4. Dependent variable:
Dependent variables give us the output. They react to whatever is happening in the experiment and it gives us our data. In this experiment we have a few different dependent variables, all of which are the result of student output. The student’s reactions to the pre and post test questions depends on their prior knowledge or what they learned by doing the projects. We also have student reactions to the interview questions. The interview questions were chosen by the researchers which makes them independent variables, however the unknown result of them is what the children are going to say. The children’s responses are based on their experiences in the eMentoring project as well as how the questions were designed to elicit a response.
5. Data analysis/statistical analysis:
Our research hypothesis is that there is a difference between students’ achievement on the post-tests. The null hypothesis, therefore, would be that there is no significant difference between the students’ scores on the post-tests. We are accepting the null hypothesis here: there is no statistical difference in the two groups of student scores on the post-test.
T tests indicated there was no statistical difference between the control group’s post-test scores and the experimental group’s scores. The only scores that could be compared between these two groups (ones with an eMentor and ones without) are the ones at the end of the unit because the control group did not do the pre-test. Table 1 shows that the significance value is larger than 0.05: 0.056 with a t value of 59.03. That t value also seems quite large compared to the t values that came from our data analysis with the data sets in our assignment for this unit. It may be possible the t value is related to the N, which was 66. I have not done enough of these tests to know if the t value means as much as the sig value being as large as it is. This sig value of 0.056 means there are 5.6 opportunities, almost 6, in 100 that there is no significant difference between the mean test score values of two groups. There is a high chance the mean test scores are the same. The 0.056 is falling in the confidence interval instead of the critical region. If the sig value, p, had been smaller than 0.05, then we would have said there was a statistical difference in post-test scores between the two groups because there is a very, very small chance the mean of the test scores would not be the same. If the mean of the test scores were not the same, then we would be accepting the research hypothesis: there is a significant difference between students’ achievement on the post- tests.
The means of the post-test scores were too close for the effect of an eMentor to cause there to be a significant difference between the achievement of the control and the eMentor group. They conducted an independent t-test on the final grades because they had two sample groups for these scores: control group and the ones that had access to eMentors.
A paired-sample T-test between the pre- and post-tests did show a statistical significance in the scores between the pre- and post-tests. According to their results, student achievement was statistically significant in terms of improving by doing the IBLE project. The statistics, t(25) =3.54, p=0.002 tells me they did a test with 25 degrees of freedom, N-1, the t value coming from their statistics program and a significant value of 0.002, which they are calling p in the expository part of the paper. Table 2 shows the results of the paired sample t-test.
Since, however, the final test results were not statistically different between the control and the treated group, it may be an artifact of how the pre and post tests were designed, more than an indication of the influence of an inquiry approach to learning the material.
They took the student responses and used codes to categorize the types of responses they received. Once they had numeric codes, they could manipulate the qualitative data, the student responses, in a way that let them put a number on how much the IBLE environment had an impact on the students. They came up with a value of 82% using an inter-rater agreement (p.739).
They also analyzed the students survey responses to determine if there was
- Improved engagement and motivation
- Broadened understanding of the relevancy of math and science in students’ lives
- Increased awareness of roles and careers in math and science
6. Results and outcomes:
Enough of a difference was found that this research should continue to be funded. Even though on the final post-test both the experimental and the control groups’ scores did not show enough variability to be significant, there was evidence that the experimental group’s change in achievement from the pre-test to the post-test was significant. It seems like the pre-test and the post-test were not identical. They say, “But the results above between treatment and control group indicated that this change might be caused by changing of test items.”
The group would like to extend this to be a longitudinal study, similar to the one they did with urban students. They also don’t know yet if this study will have long-term effects. They do not have the right instruments because they don’t exist yet. They do not have a reliable way to continue to track these students beyond this classroom experience.
Some students reported that their interaction with the eMentors increased their own confidence in math and science because the researchers and eMentors did use the students for their input on what was to be studied. Unlike traditional learning that goes from the teacher to the student without student input, this collaborative environment included students in on the lesson plans, or the direction of the project.
In their conclusion they assert that the continuous input from an eMentor is a significantly different paradigm than one where guest lectures give momentary input that is not directed to individual students, but rather to an entire group. A guest lecturer’s presence is also temporary, not allowing for follow-up questions from the students once they have had a chance to struggle with the content a bit more. The eMentor is also significantly important because there is a limit to how much the students can interact with their teacher or use the teacher as a subject matter expert the same way the eMentors can fulfill that role.
They also expressed how students moved their role from that of an information recipient to that of an information seeker. As students became more engaged with the project, they took the initiative to do research online and found a government agency to whom they could write letters based on the research they did in the project on bear habitats.
The researchers did not institute their own content based assessments so the pre and post-tests with which they had to use to collect quantitative data were not necessarily designed in a way to be useful for research purposes. It sounded like in the end they were not happy that they were forced to only use teacher designed summative assessments. They identified a few other challenges they hope to not face the next time they do a similar study, which will require them to choose their teacher and school partners wisely. (Personally I recommend they see how UC Berkeley professors use the local schools because they choose their locations so that they don’t have the same challenges these researchers faced. I know I always gave UC created assessments in addition to my own and did not actually use the UC assessments for the students’ content grades. But now I’m rambling on about me which is not what this article is about. )
Principles for Effective Online Teaching
Boise State University
February 21, 2012
1. Teacher expectations for students are clearly stated.
One of the most frustrating things students face is not understanding what they are expected to do. I intentionally gave ambiguous assignments in the face to face classroom because I wanted students to be creative with their final products. I often hesitated to give examples because I feared I would get clones of the example and often when I did give an example that is exactly what happened. One advantage to me giving ambiguous assignments in a face to face classroom, though, is that my students knew they would see me in person at least three times a week. I made myself available at lunch and after school to help them figure out how to do the project. There were many afternoons that I would be in the library with them, sitting next to them, to help them figure out how to complete the assignment that involved technology. In the online environment, I can have office hours, but physically stalking down a student and inviting him to collaborate with me is not realistic. Another advantage to the face to face setting is that I can personally hand a student a piece of paper. In the online environment, unless we are in a synchronous setting where I push the document to the student or have created a recorded session that allows for documents to spontaneously show up in students’ faces, there is no guarantee a student will even find the syllabus or other structural document.
There are several publications that support the need for online courses and online instruction to be highly organized and to have clear expectations readily available. Southern Regional Education Board (SREB)’s document, Standards for Quality Online Teaching (2006) (abbreviated here as SQOT) states, “Teacher expectations are also a significant factor in how much and how well students learn.” In my principle, not only do teachers have to have high expectations, they need to communicate them. SQOT (2006) goes on to mention how important it is for a syllabus to be available online that not only includes the teacher’s expectations, but grading criteria and any other pertinent course organization (p.5). Palloff and Pratt (2007) describe their beginning documents which include “a syllabus, guidelines, and the accepted rules of ‘netiquette’” (p. 115). In addition, since they use discussion forums in their online courses, they state at the beginning of the course how they expect “a minimum of two substantive posts per week per participant” (Palloff & Pratt, 2007, p. 115). The National Education Association (NEA) Guide to Online High School Courses (n.d) also recommends a pre-class orientation or a specific introduction to help ease students into the mechanics of taking a course online. It also stresses the need for all necessary materials for the courses to be easily available to students (p. 21). Finally, SREB’s Standards for Quality Online Courses (2006) (abbreviated here as SQOC) wants course requirements to be consistent with course goals, “representative of the scope of the course, and are clearly stated” (p. 4).
2. Multiple modes are used to transmit information such that they overlap and there is some redundancy.
To cater to different learning styles, teachers need to use a variety of activities and ways of communicating information so that students can take ownership of it. In the online classroom, this means that in some situations, an audio file needs to accompany a text file. In other situations, it is not enough to assign students to do a scripted lab; demonstration video needs to accompany the lab so students can at least see how to do the lab safely. If information is very important, there should be at least two ways it is communicated to students. Even a Voki to communicate an afterthought can be effective because it is eye-catching and will let students know that something has changed or has been updated. If students are not catching all of the assignments and how to complete them, then the teacher needs to find a different, perhaps redundant, way to communicate what the assignments are and how students can be successful.
Publications supporting the use of multiple modes of transmitting information include SREB’s documents: Standards for Quality Online Teaching (SQOT) (2006) and Standards for Quality Online Courses (SQOC) (2006). In SQOT (2006), they say, “The teacher differentiates instruction based on students’ learning styles and needs and assists students in assimilating information to gain understanding and knowledge” (p. 4). The differentiation can use synchronous and asynchronous tools including multimedia and visual resources (p.3) that let the instructor “adapt and adjust instruction to create multiple paths to learning objectives” (SQOT, 2006, p.6). To support the idea that a variety of modes are used to communicate information, the SQOC document (2006) elaborates on how the online course provides “opportunities for interaction and communication student to student, student to instructor, and instructor to student” (p.4). “Courses must utilize technology that enables the teacher to customize each student’s learning experience through tools and formats such as video, interactive features, resources and links to related information” (SQOC, 2006, p.1). If these features restate course objectives or instructions on how to complete assignments, as long as the information does not contradict itself, the more modes of communication, the better. For students who can easily acquire and process information, the instructor can advise them through a path in the course that has fewer redundant resources. It should not be a scavenger hunt where a student needs to access all of the communication methods to get a complete picture; instead it should be a complete picture that has various points emphasized in alternative ways. SQOC (2006) also points out the need for courses to meet “universal design principles, Section 508 standards and W3C guidelines to ensure access for all students” (p.7). This means the teacher may have to “adjust the scope and sequence of instruction to meet students’ academic and learning needs” (SQOC, 2006, p.2). In addition, “the design reflects a clear understanding of student needs and incorporates varied ways to learn and multiple levels of mastery of the curriculum” (SQOC, 2006, p.4). To communicate expectations, “each lesson includes a lesson overview, content and activities, assignments and assessments to provide multiple learning opportunities for students to master the content” (SQOC, 2006, p.4). To further differentiate instruction, “the course makes maximum use of the capabilities of the online medium and makes resources available by alternative means; e.g., video, CDs and podcasts” (SQOC, 2006, p.7).
3. The learning process is dynamic. Students interact with information, ideas, and each other in addition to receiving static methods like taped lectures or videos.
In the face to face classroom, there are multiple ways to have students interact with information, although in many classes the lecture is used as the primary conduit of content. Lectures are pretty static- they often do not involve any student interaction or participation. In contrast, inquiry based assignments in the classroom engage the students to ask questions or to take control of some aspect of the learning environment. Online inquiry can look like the discussions in forums if students are asking questions that other students are answering. Having students create “new” ideas because of their interaction with online discussions, creating wikis with others, taping a Jing presentation to demonstrate what they have learned, or creating an animation to explain a concept are far more interactive than listening to the teacher talk. Rice (2012) suggests activities like “question generation, hypothesis development, and defense of ideas” in collaborative environments (p. 31). The online learning environment needs to be more student-centered than the one in a typical face to face classroom. Online, something other than the teacher is the source of information. According to Palloff and Pratt (2007), “the responsibility for learning falls to the learners in a learner-centered approach and the focus is on learning, not on grades” (p.106).
Other publications reinforce the idea that online learning needs to be student-centered and active (Rice, pp. 74-78; SQOT, pp. 4 and 6). The teacher serves as a “guide on the side, an instructor, a group process facilitator, and an advisor.” The guide gently encourages students to participate in what may seem like new educational circumstances, the instructor ensures what happens in the classroom runs smoothly, the group process facilitator monitors student interactions online, and the advisor is not only a stable adult for the children, but also serves as a resource parents can rely upon for direction on how to maneuver the online school setting (Rice, 2012, pp. 74-78).
The SQOC (2006) document emphasizes:
The course uses learning activities that engage students in active learning; provides students with multiple learning paths to master the content based on student needs; reflects multicultural education and is accurate, current, and free of bias. (p.4)
The teacher engages students in learning activities that address a variety of learning styles and preferences… The course provides opportunities for students to engage in higher-order thinking, critical reasoning activities and thinking in increasingly complex ways. (p.5)
By providing such opportunities, “information literacy and communication skills are incorporated and taught as an integral part of the curriculum”(SQOC, 2006, p. 2). On page 14, the NEA Guide to Online High School Courses (n.d) outlines similar expectations for online classrooms.
The course makes maximum use of the online medium by incorporating primary source materials, media, outside experts, and resources beyond the geography and culture of the students’ brick-and-mortar classroom experience.
They give examples of how to make courses interactive. Students utilize “a variety of media and resources” as they “learn from multiple viewpoints.” Their lessons may include online discussions, lab experiments, designing projects, writing critiques of what they read, writing lab reports, using multimedia to create a product, or having online group activities to teach collaboration skills (NEA, n.d., pp. 14-15).
Palloff and Pratt (2007) elaborate on why a variety of teaching techniques and student-centered activities are necessary. They urge instructors to be facilitators who use “a variety of learning activities and demonstrate instructional methods other than lecturing. Draw out creativity, innovativeness, and ideas in a collaborative manner” (Palloff & Pratt, 2007, pp. 109-110). The facilitator allows “students to explore the course material… without restriction” (Palloff & Pratt, 2007, p110). Students may make comments about the reading material in discussion forums where the content is further explained or clarified. They go on to elaborate on other researchers’ ideas of how online learning is facilitated such that the students are actively involved with the process of creating curricular material. This perspective is called a “teaching presence” and it often involves students facilitating discussion forums (p.111) or providing ideas on surveys administered during the course.
4. Safe, sociable, and legal practices are demonstrated and are taught to students.
Safe practices include what information to disclose or not disclose in an online environment. Sociable practices include using academic language and informing students of typical netiquette expectations. The online course can contain a social forum where students can be more casual with their language. This reinforces the idea that in the content forums, more appropriate word choices are necessary (Palloff & Pratt, 2007, pp.112-115). Legal practices include teaching students about how to appropriately use information they find online including copyright laws, what constitutes plagiarism, and how to evaluate information found online to ensure it is accurate and reliable (SQOC, 2006, pp.3-4; SQOT, 2006, p.5). If needed, the instructor can create a set of Acceptable Use Policies (AUP) if the school does not already have students signing one upon registration (Rice, 2012, p.242; SQOT, 2006, p.5). The NEA Guide to Online High School Courses (n.d.) refers to this type of knowledge as “Information Literacy,” whereas Rice (2012) calls it “Digital Literacy” (p.241-242). NEA’s publication (n.d.) encourages instructors to have students complete lessons that teach them how to evaluate appropriate online resources, how to judge the credibility of what is published online, and how to maintain respect for copyright or other intellectual property they would like to use for their own projects (p. 15).
5. The teacher has been an online student in at least two different settings.
In order for teachers to understand the frustrations that online students can have, they need to walk the walk themselves. Future online teachers need to experience the different ways that instructors can organize discussion forums, due dates, documents, expectations, or other mechanical qualities of the online setting so that they can be aware of some of the circumstances that can cause student frustration. The more exposure, the better; however, if I’m going to write a principle that leads to a standard, I want to quantify that there needs to be at least two experiences with different instructors. Rice (2012) recommends online teachers have “online course experience as a learner” (p.41). Both SREB (2006) and NEA (n.d.) want online instructors to have had at least one experience with being an online student and/or some training on how to be an online teacher (SQOT, 2006, p.6; SQOC, 2006, p.8; NEA, n.d., p17).
6. Assessment includes more analysis than determining the students’ ability for taking multiple choice tests.
Assessment could include contributions to a discussion forum, a video they make individually or in teams to demonstrate content in the class, an animation showing how something works, writing in a journal or blog, doing a vlog, creating a wiki by oneself or as a group, or photographing a physical project made at home. Not only does the instructor evaluate student work, but the students will often do self-assessments. Rice (2012) emphasizes that online learning environments should promote a process and not just an end product. These “online instructional strategies should encourage the use of a variety of metacognitive activities for assessment including feedback, self-reflection, and self-explanation” (Rice, 2012, p. 33). SQOC (2006) and SQOT (2006) want to see instructors create and utilize assessment that justifies the content, how it is administered to the students, shows what the students do with the information, and reports how they perceive their intellectual gain (SQOC, 2006, pp. 2 and 6; SQOT, 2006,pp. 5 – 8). NEA (n.d) on p. 19 suggests online high school courses’ assessments include:
- Contributions and responses to online discussions
- Completion of online assignments
- Portfolio submissions
- Special projects and/or presentations
- Creation of authentic products
- Tests and quizzes
For more advanced courses like those for college or other adult levels, small group collaboration or case study analysis that yields a collaboratively written paper or project can serve as an alternative assessment to a fact-based test (Palloff & Pratt, pp.115-116 ).
7. Curriculum is modified on a regular basis by the person teaching the course. The class is not merely static lessons developed with the idea students will have to adapt themselves to the curriculum. The content and its delivery needs flexibility that allows for individualized instruction and assessment when needed. The teacher is given the power to change the curriculum, activities, and assessments based on the information s/he gets from students and their progress. Resources supporting these ideas include Standards for Quality Online Teaching, 2006, p. 7; Standards for Quality Online Courses, 2006, p.8; NEA, n.d., p. 22; and Palloff and Pratt, 2007, p.115.
National Educational Association. (n.d.). Guide to Online High School Courses. Retrieved from http://www.nea.org/assets/docs/onlinecourses.pdf
Paloff, Rena M. & Pratt, Keith. (2007). Building Online Learning Communities: Effective Strategies for the Online Classroom. San Francisco: Jossey-Bass.
Rice, Kerry. (2012). Making the Move to K-12 Online Teaching: Research Based Strategies and Practices. Boston: Pearson.
Southern Regional Education Board. (2006). Standards for Quality Online Teaching. Retrieved from http://publications.sreb.org/2006/06T02_Standards_Online_Teaching.pdf
Southern Regional Education Board. (2006). Standards for Quality Online Courses. Retrieved from http://publications.sreb.org/2006/06T05_Standards_quality_online_courses.pdf
This was a project in EDTECH 521:
The assignment: Use the framework provided in the article as a template and identify community building strategies for use in your own classroom. Provide a rationale for each. I don’t usually like to put minimum and maximum counts on assignments, but some of you will ask, so let me suggest that you include between 5 and 10 strategies. What I’m most interested in is that you think carefully about how you will incorporate strategies for building community into your online classes.
My response: I am writing this chart to match up with a hypothetical class on high school chemistry. It is partly based on a class I created while taking classes at Merritt College in the last year.
The table won’t fit here so you’ll have to see the document as it is stored in my Google Drive: Community Strategies