EDTECH 562: Introduction to Statistics for Educational Technologists, Case Study
We did case studies in EDTECH 562 so we can see how evaluation is done in the real world. How are statistics used to validate what was done during the research process? Here is an example of one of the case studies I completed while taking the class.
EDTECH 562: Module 4 Case Study
Submit to Module 4: Case Study
Please read the Module 4 Case Study file: Li, Q. (2010). Inquiry-based learning and e-mentoring via videoconference: A study of mathematics and science learning of Canadian rural students. Educational technology research and development. 58(6), 729-753.
EDTECH 562: Module 4 Case Study
Your Name: Melissa Getz
1. Research question:
How does providing eighth grade math students living in a rural setting an opportunity to interact with people who do research allow for a more authentic experience, thereby increasing students achievement and interest in math and science?
According to the paper, the research questions they asked are:
- How does the experience in an IBLE affect rural students’ learning of math and science? Specifically,
- Does the overall learning experience in an IBLE environment improve rural students’ achievement in mathematics as demonstrated in test scores?
- In what ways does the overall learning experience in an IBLE environment impact rural students’ affective development in math and science?
- What are the challenges of establishing an IBLE environment in a rural context?
2. Research strategy used:
Before bringing the students into the activities, the adults did a bit of planning. As a team, they created projects for the students to do with eMentors. They identified overarching themes, by focusing on the overarching questions:
How does understanding multiple perspectives shape the way we live in the world? In what ways does diversity shape our understanding?
After identifying the themes, they brainstormed project ideas and designed the project structure.
The formation of the inquiry projects was based on these three questions:
- What are the curriculum topics that need the most attention?
- What topics will engage students?
- How can they match eMentors to students so that students benefit the most from their interactions with the eMentors.
Following the planning, they implemented an action plan that involved the students interacting with the eMentors and completed the project by doing the post-tests and student interviews.
There were two control groups (41 students) and one experimental group (26 students) whose post-test scores were compared. The research group also did a pre-test so that changes between the beginning of the project and the end of the project could be measured. Nine of the students in the experimental group were personally interviewed to collect evidence of students’ attitudes about the experience.
The research team used both quantitative analysis and qualitative data. The quantitative analysis was generated twice:
1. is there a statistical difference in post-test scores between the control and the treated group?
2. is there a statistical difference in pre-test and post-test scores for the treatment group?
Interviews were conducted with nine students in the treatment group so as to not disrupt their courses too much. All nine students were interviewed alone or in pairs three times during the project. They felt the number adequately covered the population because the students were chosen based on having representation from a variety of academic backgrounds as well as having a small enough group with which to develop trust and confidence between the researchers and the students.
3. Independent variable(s):
Independent variables are the ones the researchers manipulate. That is a definition for independent variable which I translate to mean the researchers are choosing a variable that can allow for output as a result of doing the experiment. For example, if they chose temperature, it would influence the experiment in a way that causes there to be output that is specific to the temperature of the experiment. Or time can be an independent variable because as it happens something else changes. The independent variable itself does not give us information that is used in the statistical analysis, but the output it can cause is used. The output also comes from dependent variables that depend on the independent variable to know how to behave.
In this situation, there is the variable of time because we have pre-tests and post-tests. The output on the pre and post tests depends on the experimental timing- had the students done the inquiry lab with the scientists as support or not? The tests themselves would also be an independent variable because the student responses to the tests gives us data- the student responses are a dependent variable that relies on the test to provide an output. An independent variable here also involves if the students interacted with an eMentor or not. We decided who worked with the eMentor and the output we will be measuring is the students’ gain in interest in math because they worked with an eMentor. The students’ opinions are dependent on whether or not they had access to an eMentor.
There is also the variable that we are working with children. Their output is a dependent variable- it is not predictable and is based on their doing the math that was in the assignment.
This also brings up another independent variable which probably should have been listed first because it is the main difference between what happens to the experimental and the control groups: who gets to work with the scientists? Which group of kids gets the eMentoring?
4. Dependent variable:
Dependent variables give us the output. They react to whatever is happening in the experiment and it gives us our data. In this experiment we have a few different dependent variables, all of which are the result of student output. The student’s reactions to the pre and post test questions depends on their prior knowledge or what they learned by doing the projects. We also have student reactions to the interview questions. The interview questions were chosen by the researchers which makes them independent variables, however the unknown result of them is what the children are going to say. The children’s responses are based on their experiences in the eMentoring project as well as how the questions were designed to elicit a response.
5. Data analysis/statistical analysis:
Our research hypothesis is that there is a difference between students’ achievement on the post-tests. The null hypothesis, therefore, would be that there is no significant difference between the students’ scores on the post-tests. We are accepting the null hypothesis here: there is no statistical difference in the two groups of student scores on the post-test.
T tests indicated there was no statistical difference between the control group’s post-test scores and the experimental group’s scores. The only scores that could be compared between these two groups (ones with an eMentor and ones without) are the ones at the end of the unit because the control group did not do the pre-test. Table 1 shows that the significance value is larger than 0.05: 0.056 with a t value of 59.03. That t value also seems quite large compared to the t values that came from our data analysis with the data sets in our assignment for this unit. It may be possible the t value is related to the N, which was 66. I have not done enough of these tests to know if the t value means as much as the sig value being as large as it is. This sig value of 0.056 means there are 5.6 opportunities, almost 6, in 100 that there is no significant difference between the mean test score values of two groups. There is a high chance the mean test scores are the same. The 0.056 is falling in the confidence interval instead of the critical region. If the sig value, p, had been smaller than 0.05, then we would have said there was a statistical difference in post-test scores between the two groups because there is a very, very small chance the mean of the test scores would not be the same. If the mean of the test scores were not the same, then we would be accepting the research hypothesis: there is a significant difference between students’ achievement on the post- tests.
The means of the post-test scores were too close for the effect of an eMentor to cause there to be a significant difference between the achievement of the control and the eMentor group. They conducted an independent t-test on the final grades because they had two sample groups for these scores: control group and the ones that had access to eMentors.
A paired-sample T-test between the pre- and post-tests did show a statistical significance in the scores between the pre- and post-tests. According to their results, student achievement was statistically significant in terms of improving by doing the IBLE project. The statistics, t(25) =3.54, p=0.002 tells me they did a test with 25 degrees of freedom, N-1, the t value coming from their statistics program and a significant value of 0.002, which they are calling p in the expository part of the paper. Table 2 shows the results of the paired sample t-test.
Since, however, the final test results were not statistically different between the control and the treated group, it may be an artifact of how the pre and post tests were designed, more than an indication of the influence of an inquiry approach to learning the material.
They took the student responses and used codes to categorize the types of responses they received. Once they had numeric codes, they could manipulate the qualitative data, the student responses, in a way that let them put a number on how much the IBLE environment had an impact on the students. They came up with a value of 82% using an inter-rater agreement (p.739).
They also analyzed the students survey responses to determine if there was
- Improved engagement and motivation
- Broadened understanding of the relevancy of math and science in students’ lives
- Increased awareness of roles and careers in math and science
6. Results and outcomes:
Enough of a difference was found that this research should continue to be funded. Even though on the final post-test both the experimental and the control groups’ scores did not show enough variability to be significant, there was evidence that the experimental group’s change in achievement from the pre-test to the post-test was significant. It seems like the pre-test and the post-test were not identical. They say, “But the results above between treatment and control group indicated that this change might be caused by changing of test items.”
The group would like to extend this to be a longitudinal study, similar to the one they did with urban students. They also don’t know yet if this study will have long-term effects. They do not have the right instruments because they don’t exist yet. They do not have a reliable way to continue to track these students beyond this classroom experience.
Some students reported that their interaction with the eMentors increased their own confidence in math and science because the researchers and eMentors did use the students for their input on what was to be studied. Unlike traditional learning that goes from the teacher to the student without student input, this collaborative environment included students in on the lesson plans, or the direction of the project.
In their conclusion they assert that the continuous input from an eMentor is a significantly different paradigm than one where guest lectures give momentary input that is not directed to individual students, but rather to an entire group. A guest lecturer’s presence is also temporary, not allowing for follow-up questions from the students once they have had a chance to struggle with the content a bit more. The eMentor is also significantly important because there is a limit to how much the students can interact with their teacher or use the teacher as a subject matter expert the same way the eMentors can fulfill that role.
They also expressed how students moved their role from that of an information recipient to that of an information seeker. As students became more engaged with the project, they took the initiative to do research online and found a government agency to whom they could write letters based on the research they did in the project on bear habitats.
The researchers did not institute their own content based assessments so the pre and post-tests with which they had to use to collect quantitative data were not necessarily designed in a way to be useful for research purposes. It sounded like in the end they were not happy that they were forced to only use teacher designed summative assessments. They identified a few other challenges they hope to not face the next time they do a similar study, which will require them to choose their teacher and school partners wisely. (Personally I recommend they see how UC Berkeley professors use the local schools because they choose their locations so that they don’t have the same challenges these researchers faced. I know I always gave UC created assessments in addition to my own and did not actually use the UC assessments for the students’ content grades. But now I’m rambling on about me which is not what this article is about. )