Category Archives: 5.2 Criterion-referenced Measurement
5.2 Criterion-referenced Measurement
5.2.1 Develop and apply criterion-referenced measures in a variety of SMET contexts.
5.2.2* Identify and collect appropriate data to support decision-making, short-and longterm,
for the school media program.
Relative Advantage of Instructional Software
When I can find software to use in class, everybody benefits. Obviously it means I don’t have to lecture that day, other than to explain how to use the software. Chances are the students will be more interested, especially if the software is fun to use. Unfortunately some of what I have had students use is less than ideal. For example, I had students do a webquest to learn about doing protein gels. I could have lectured, but it seemed better for them to see the animations. For this particular webquest, I gave them questions and links to various websites where they can find the answers. I learned that if I were to do this again, I may have to put the link to the website adjacent to the question it answers. As much as students like using computers, they don’t necessarily like to use them for research, or to find an answer that can’t be easily found in a Google search or a wiki.
To teach students how to analyze data by using software that gives them the opportunity to read graphs or the results of an experiment, is not as good as having them do it hands-on in the classroom, but it is better than them not getting any experience with the information. Unfortunately many of the virtual labs I have used with students are either so difficult that it takes me hours to figure them out, like Gizmos, or they are just point, click, and drag exercises that they actually end out being a waste of time. Until my abilities with creating software or using software to create lab scenarios gets better, I am afraid that if I use software with the kids, it is going to be written by somebody else.
Interested in what our textbook has to say, I started skimming through it. Sadly on page 77, they say, “Today, after more than 30 years of development and experimentation, there is less talk of computers replacing teachers…” which is actually an optimistic perspective. What is sad about it, is that from my experiences in the last 4 years, it is not true. Computers and scripted curriculum are replacing teachers. There are companies who are making lots of money by replacing the teachers that used to be in the classroom by replacing them with virtual teachers. These virtual teachers will often have a load of 200 students per day from whatever states they have a credential to teach in. While I realize this post is supposed to be about how educational software and technology tools help the classroom teacher, I feel the need to point out the disparity that exists between a classroom teacher and a virtual teacher. Software IS replacing the classroom teacher. I know this because I taught kids in Delaware who did not have a classroom teacher. The software and I replaced whoever should have been the classroom teacher when the school was restructured. For my Pennsylvania kids, I was their teacher, even though I never met them in person, and live 2000 miles away. I did not actually ever teach them anything. I tried to tutor them if they would stay focused enough during a tutoring session to let me explain things to them, but even then, I had some kids who were not used to the idea of being responsible for their learning. This is not at all what I meant this blog post to turn out as so I will curtail my digression on how bad virtual schools are at this point, but I do want to point out that in my presentation of tutorials, drill and practice, and other categories of instructional software, this is not the same software being used in virtual schools. The software links I am presenting for this post are stuff that I either used when I was in the classroom, or would use should I ever get back in a classroom. (The later seeming further and further away from possibility, but you never know. So far using a wheelchair rocks using a walker, and if I upgrade to a power wheelchair, who knows what my limits will be?)
Robolyer and Doerling point out on page 78 that “instructional software packages are developed for the sole purpose of supporting instruction and/or learning.” It is important they differentiate between technology that is merely a tool, technology that is replacing the teacher, and technology that supports the teacher. Granted, they are not acquiescing that software is replacing teachers, but trust me, it is. They go on to elaborate which types of software can allow for directed and / or constructivist approaches. Naturally, as the students are given more control of the software environment, the more constructivist it can be. For example, having students build a website gives them more freedom than merely doing a webquest where they go hunting for answers to questions. (I have had students do both.) I see a parallel between paper and equipment lessons and computer software ones. The tutorials and drill and kill are like the worksheets or notes I used to print out on paper for the kids to use. Simulations are like cookbook labs. Problem solving scenarios are like inquiry based labs. At the moment, I don’t have a parallel for instructional games, unless doing a Jeopardy review or having kids make board games qualifies as an instructional game.
In chapter 3, Robolyer and Doerling give advice on how to select good examples of software in each category. In addition they elaborate the pros and cons of each type. Many teachers scoff at having any rote memorization types of drill and kill, whether it is a worksheet or a computer program. It is comforting to see that I am not the only one who finds value in having students practice specific types of problems repeatedly. I am currently tutoring an algebra 2 student, and while preparing for her winter final, it became pretty chaotic with so many different problems to figure out. One thing I started to notice, however, is that what was becoming more important than getting the right answer, was learning how to evaluate the situation to determine which technique best solves each problem. We may never recognize we are factoring a binomial in the real world, but learning how to be calm while sorting through our resources and evaluating them is a skill both my student and I will benefit from knowing.
Tutorials are my favorite type of programs to create because I love learning how to use Articulate Storyline. I took the BSU class on Flash, and it was pretty much a nightmare. I used Articulate’s free 30 day download for two classes, and became hooked. Fortunately I have significant support from my husband and family, so I was able to purchase Storyline. Flash will integrate with Storyline so I may do some flying numbers in Flash to bring in to a Storyline project, but otherwise I think I am stuck on doing the “explanation screen” way of trying to help students with various science topics. I have not created many tutorials, but you are welcome to see what I have done at www.getzguides.com. For my students who were enrolled in virtual classes because they were at a treatment center, my guides were a way they could get additional support for the classes if a live tutor was not available. Robolyer and Doerling point this out on page 88, tutorials are useful for instruction when no teachers are available. You may be surprised by how many students are taking classes that don’t have a readily available teacher. It is for these students I write my tutorials.
I am a huge fan of physics simulations. Even making apps with Corona or other simple programs lets you use physics. Even though I did not figure out how to make an app by coding in lua for one of my BSU classes, I did come to appreciate how physics can easily be integrated into simple software programs. As much as I am addicted to Minecraft, it is odd how they only have physics apply to two types of blocks. Then again, because they suspend the laws of physics, students can easily make three dimensional representations of objects when building in creative mode. Redstone mimics electronics and minecarts can travel based on gravity, so Minecraft is not completely void of physics. The redstone and use of minecarts on trails can give kids an opportunity to participate in something a teacher created, therefore making it a simulation or game, or they can create their own situations which would fall into the problem solving category.
I am torn when it comes to digital dissection because I know I truly learned more about animals by dissecting them, than if I had just gone through a point and click way of learning body parts. I wonder, though, how necessary it is to kill so many animals just for tenth grade dissections. Our book quotes from studies that showed digital vs physical manipulation does not seem to matter in terms of what information students retain (Roblyer & Doerling, 2013, p.91). For many teachers, the benefits of no set-up or clean-up, less costly equipment once the software is acquired, unless its license has to be renewed annually, and less supervision needed during the class period, outweigh the negative perception that what the students are doing is not actually real. The American Chemical Society (ACS), and the National Science Teachers Association (NSTA) have come out against virtual labs. Even the College Board will not accept credits in classes where students did a virtual equivalent of a lab. (Robyler & Doerling, 2013, p. 93). This means students will be doing PCR and running agarose gels for their AP biology lab, instead of imagining the bands migrating through the gel.
The last two categories, Instructional Game Software and Problem-Solving Software, are more difficult for me to see in the science context. The book recognizes Geometer’s Sketchpad, which is a very good program. It helps make geometry more spatially available. They also mention Spore as a game for studying evolution. I can’t comment on Spore because I’ve never played it. I do have to say, though, that I did an internship for a nanotech company in Emeryville, and the folks who created Spore were either on our floor or above us. It was interesting to ride in the elevator with them. But I digress, once again…
You may notice in my presentation , instructional games and problem solving software have very few entries. Hopefully I will be able to add more links after I post my blog. Fortunately the book treats the last two categories like it did the first three by giving example scenarios, and pro/con lists. One possible con that struck me was the idea of having to choose software that can handle limited physical dexterity (Robyler & Doerling, 2013, p. 95). I don’t think many students like having me in class because I can find faults easily in student work, and I will mention it. It is not to be mean; I’m actually trying to be helpful. People who don’t have disabilities really have no clue what it is like to have some. Just ask me about how ludicrous some of the ADA adaptations are where I live, and I’d be glad to tell you how we need people with the disability to create the adaptive physical changes, or in the case of my classes, adaptive software. We used Minecraft as a game, and as a way to do problem solving when I took EDTECH 531. In 531, we created an example of how to use one of three software packages as an educational tool, and there were some lessons I could not physically do because of the way they were designed. I did not have the manual dexterity to click and drag fast enough. If you know how to contact me, and you want me to evaluate any website or program you create for its difficulty with my limitations, just ask. I happily volunteer my eyes, hands, and defective brain as a testing environment.
In 531, I was incredibly impressed with how Minecraft (MC) can be used to simulate many social studies situations. I thought of a few ways it could be used with science, and I plan to make quests in 3dGameLab that have students use Minecraft to look at some science concepts. I feel like Minecraft is predictable enough that you can act like a scientist, and evaluate the game in survival mode as if one is going through the scientific method. I wish I qualified for minecraftedu so I could create scenarios that have students go mining for organic and inorganic resources. I can do that with regular MC, but it will be much more difficult to control student access to specific areas, and to protect blocks. The possible lessons in Second Life are also amazing, but from what little I’ve experienced, they are not on the level of games or problem solving. I can see World of Warcraft being used for problem solving because that is what you have to do continuously- the first problem being how to play the doggone game. I felt that way with Minecraft, too. I think any of these software programs that are easily intimidating at first are actually really good tools for students to learn resilience, endurance, and perseverance. I was a MC misfit when I first started playing it. I later became addicted to it. The book makes a distinction between doing problem solving software activities merely for the sake of learning how to problem solve. (Robyler & Doerling, 2013, p. 97). I can totally see using software for that purpose, at least until someone figures out how to create something that can be open ended enough for students to be able to make mistakes and therefore be able to learn from them
One thing that should be in any of the interactive software games is a chance for failure. When click and drag scenarios are too predictable, students won’t be challenged and will complete the activity because they are required to, and not necessarily because they are enjoying what they want to learn. We need to be careful, though, to not build in failures that students will take too strongly or too personally. I still don’t know where I am going to fit into education in my next stages. I’m hoping it will involve creating quest based courses in 3dGameLab that other teachers will want to use. If I can figure out how to turn a quest or a course into how to problem solve something in science, other than an easily predictable physics or genetics lab, I will be ecstatic.
Roblyer, M. D., & Doering, A. H. (2013). Integrating educational technology into teaching [6th edition].
Dr. Thompson had us “practice” writing a request for proposal (RFP) document. In this document we had to plan how to execute helping Far West Laboratory with their need to educate their school clients.
In EDTECH 523 we created a synchronous presentation and were required to watch other presentations as well. To focus on other people’s presentations, we were asked to create an evaluation guide that we could fill out during the presentations. After the groups finished, we emailed our presentation form to the respective group. I’m connecting to my evaluation instrument by having a link to it here as it was uploaded to WordPress. The other link is to it living in my Google Doc area. Finally, I did a copy and paste of the document and put it at the end.
Purpose of this assignment: Create an evaluation instrument for evaluating the quality and appropriateness of synchronous instruction.
Yes/No or N/A
|Introduction to the lesson:Could be a poll on the screen or an activity on the whiteboard- just has to be something students can do to amuse themselves until the session starts|
|Presenter reviews classroom controls|
|Audio and/or video enabled for participants|
|Students do something in the classroom other than just sit there.|
|Presenter records session|
|Students are encouraged to raise hands if they have questions|
|A copy of the ppt slides if applicable|
|Web links are available|
|Presenter remembers to turn off recording|
|Polls for formative assessment|
|Polls to keep students engaged|
|Student expectations shared|
|Questions for students to answer that they will turn in later|
|Shares files students can download as take-away info|
|Presenter provides files students can download so they can follow the lesson as it happens|
|Ppt slides are uploaded to a share screen or are shown from presenter’s computer as a shared screen.|
|Students are used to help advance the lesson|
|Encourages student-student interactions.|
|Encourages student-content interactions.|
|Students use whiteboards or the equivalent to answer questions.|
|Formative assessment is done so the presenter can see if the lesson was successful|
|Student product is used somewhere in the lesson to either evaluate student work or to give an example of what can be done at a student’s level|
|Presenter is organized so that there are not unnecessary lulls|
|Presenter speaks clearly|
|Presenter speaks slowly enough|
|Presenter speaks loud enough|
|Some sort of closure|
|You can tell there was a reason this worked better as a synchronous lesson than an asynchronous one|
For Module 5 your summary should include information about the following:
- Complete the required reading and review materials provided on synchronous tools and strategies.
- Develop a lesson to be delivered synchronously using appropriate instructional strategies.
- Practice delivering your lesson.
- Create an evaluation instrument for evaluating the quality and appropriateness of synchronous instruction.
- Post the completed evaluation instrument and your reflections to the Evaluation of Synchronous Instruction forum in the MAIN discussion board.
- Please, also submit your completed evaluation instrument to the appropriate link in the Module 5 activities for grading.
- Submit discussion idea and/or lead a discussion.
- Participate in ongoing discussions. Apply critical thinking and questioning strategies to your discussion posts.
Module 5 was fun because it got me back in the Adobe Connect classroom which is always a challenging environment to be in because it is so convoluted. Preparing for Bret and my synchronous lesson has also been fun because it is collaborating with someone who is intelligent and knows more about our topic than I do. This means I get to learn something while doing this project and have someone who can patiently handle my questions when I get lost. In contrast, since I bought the Adobe eLearning suite in the fall, I have Adobe Presenter which takes PowerPoint files and uploads them to the Adobe Connect server. It was a nice opportunity to refamiliarize myself with the software.
My evaluation instrument was somewhat incomplete so I revised it as I reviewed the synchronous sessions. Interestingly, when I evaluated the Photoshop lesson presentation I realized I had not planned on evaluating something that was not necessarily like something listed in chapter 6. Bret and I have been trying to figure out how to adapt what we want to do to fit one of the suggested activities in chapter 6 and still be within 10 minutes. I think we have a clue what to do and I hope everybody who wants to be a part of our audience will join us and the other presenters on Thursday, May 3. It was also a relief to see that there will be at least six people in our audience. We were not sure what type of audience we could count on so we can now plan breakout rooms and student activities better.
It makes sense that chapter 5 in the Learning in Real Time text is applicable to Module 6 since it covers formative assessment, how to integrate it into the class, and how to pick up on non-visual body language. Since that chapter may need to be a part of the Module 6 reflection, I won’t go into more depth here. I will share a few ideas about chapters 4 and 6, though. For chapter 4, I found their analysis of various online teaching settings to be accurate based on what I’ve experienced. I used to IM with my students when they had a quick question and at times we would mosey on over to our virtual office to use the whiteboard for further explanation. Explaining dimensional analysis through IM can be done, but using a whiteboard is much easier. Although I have never taught or taken a class in a MUVE, I think that is what second life is like so I expect I will experience it when I take the class that has us use second life. Although I’ve attended webcasts, I’ve never led one. Although I have had a few online teaching opportunities, I’ve never actually been able to hold an online lecture or class session for students who were expected to be physically present. My current teaching situations are one-on-one and my previous one had live sessions as optional features for the students. I could do them as often as I wanted to, but they were never required to attend a session. I hope that one day I do get to have a real online class with real students that will be “my” students that I get to usher from one lesson to the next. The activities in chapter 6 will be very useful once I have the setting and the bodies with which I can practice.
Module 6 is where we teamed up with a partner to plan a live presentation for our peers using the Adobe Connect software. This is the reflection I wrote after Bret and I did our presentation, which was a fantastic experience.
Module 6 reflection
The readings and how they are reflected in our presentation:
Chapter 5 of the book was my favorite chapter. Even though Bret and I scoured through chapter 6 figuring out what type of interaction was possible and feasible, chapter 5 contained stuff that had tangible meaning for me at this time. In our presentation I played the role of behind the scenes host. I tried to calm people’s worries if they were expressed in the chat area during the presentation. I made it to one of the breakout rooms to help them get started with their conversation and let them know that they were doing great by writing on the notes screen. I also let them know they could use audio and video cams in the breakout room without bothering others. Before we pulled people out of the rooms, we sent the 20 sec warning that you were going to have your reality change. For the anticipated review of what went on in the groups, I pulled up the notes screens so they could be seen by everybody and therefore not be left out of any discussion. We also planned for a parting gift, which apparently did not download for some people. I have no clue why that didn’t work because we put the documents in there correctly. I also hope that some people get to take the survey so they can see what a Google form can do and if they use the links at the end of the form, they can view the data as it comes in. I was glad to see some welcomed the idea of having a “parting gift.”
I did not get to enact all that was suggested in chapter 5, in part because I was not a solo presenter. Also, since we were doing a round-robin of classroom jumping, there really was not a way to be prepared enough to welcome people as they entered. I understand that it was difficult to get people in as guests and Bret and I learned that barrier early on. I think this is why he came in our room as a guest and had me turn him into a host. Somehow everybody was turned into a host so it did not matter that Bret did not enter as a presenter/host. In some ways, the software is too friendly by putting a cookie in our machine and not making us re-register for each room. That is why I used my Mac when I was a participant and my PC as the presenter. I anticipated quick room changes and knew I’d mess it up if I tried to enter the other rooms while using my PC because the PC is cookied. It is not reasonable to expect people to have 2 computers to do this lesson so we could not expect everybody who had already presented to be out of the presenter registration. I think that is why so many people showed up as hosts when they entered the room- their machines were cookied and it is tough to remove that status. I guess since I spend so much time trouble shooting things because I often find them difficult to maneuver through quickly, that it proved to be an asset for me to know the Mac would work fine in the guest position.
Bret and I also used the Mac as a guest computer when we prepared for our session. Since we could not talk very clearly when we were not in the same room, it was hard for one of us to be presenter and the other to be guest when we practiced. I signed in to our room as a guest from my Mac laptop so I could see what the guests would see during the presentation. That is what taught me how the breakout rooms work. I could tell that putting ‘Mel on the Mac’ in a breakout room did not stop “her” from being a part of what was happening in the main room until the “start breakouts” button was pushed. Part of my nervousness in the beginning of our presentation was being afraid everybody would let their curiosity get the better of them and they’d move themselves out of the breakout rooms before we started them. The plan was originally to keep people as guests because we did not want them to play with stuff that was already set up to go. Fortunately we are working with adults so my fears were unnecessary. Everybody behaved themselves as perfect students and none of our tricks got messed up before they were delivered.
The backchannel- Bret and I did not necessarily see eye to eye on the backchannel, but this was not my place to be the total control freak so I went along with our main chat area being a backchannel. I don’t know if Bret has ever participated in a backchannel chat during a real presentation. I’ve actually only done it once, and that was when it was being taught to me at an ASCD presentation last year. I wanted there to be a backchannel and a real chat area, but it would have been too chaotic in the short amount of time we had. We named it the backchannel anyway so people could see that if they had enough room on their screen area, they could have 2 chat windows during their presentations- one for real concerns and the other one to be social. I am biased toward letting people use presentations as a way to make friends because sometimes not everything that is said needs to be heard.
That was another place I was not able to communicate well enough to get it into Bret that he did not need to do a lengthy introduction to what an LMS is. In our last practice he did narrow it down to maybe 2-3 min of talking, but today he went for more than 3 I would guess. I know my patience started to wane and I came close to just sending out the polls while he was talking. If you think today’s presentation was long-winded, you should have seen it during our first practices. I respect Bret because he wanted people to learn something during our presentation and he really is an expert on today’s talk. That is one reason we did this topic; it is relevant to what we may do as teachers and Bret had to do something similar for people in his district. Plus it had so many components that let us expand it in ways that let us play with Adobe Connect.
Bret did a fantastic job of outlining our expectations and establishing the norms for our session. You may have noticed that he built it in to the beginning of the PPT slides. He designed the presentation slides and let me go crazy with Adobe Connect bells and whistles. We somewhat followed the suggestions given starting on page 84 where there is one main person up front and someone else behind the scenes. I did not do all of the logistics alone; Bret helped with setting up the 4 types of polls and how to space everything so it would be ready to be used when we needed it to be there. I took care of naming things in a way that would make sense to us and others, putting the exit survey in a website link pod, uploading files for the file share, and creating the exit survey in Google Forms. Since I bought the eLearning suite when I was taking 521 I wanted to play as much as I could with the software. For some reason I could not get Bret’s slides to upload correctly so he did a screen share for our presentation instead of it being a file he used from the EDTECH servers. It would actually be really cool if the eLearning suite was required instead of the other CS5 suite because then we could possibly have lessons on how to use Adobe Captivate. I’ve only played with it once, but that is something that would be an asset to know how to use for online teaching. Dreamweaver , Flash, and Photoshop are also a part of the eLearning Suite so if you get to make suggestions to the department, you would not be too out of line if you suggested having the department use the eLearning suite in the future.
Other people’s presentations:
Even though I tried to follow advice and looked at other people’s eval tools when I revised mine for tonight, I found what I thought was important was somewhat tangential to what happened. Since the presentations went so fast and I did not want to take time to watch the recordings, I had lots of gaps in my evaluation forms. Regardless of what it seems I did not learn, I found these things to be new to me and very useful:
- Students writing on whiteboards. I knew it could be done, but had not experienced it myself in Adobe Connect yet. Actually I don’t know if I knew there could be interactive whiteboards in Adobe Connect. Had I known, we may have set up a whiteboard for each breakout room instead of using notes windows to record student interactions.
- Students could format their notes screens. When we pulled up the Notes screen for group 1, they had done some formatting. That was so cool. I don’t know if anybody else noticed it, but it was neat to learn that students could take ownership of some of their output if Notes pods were used for collaboration.
- I am still not sure what Adam did so we could move things on the whiteboard. I may have to email him to see if he can tell me. Adam did the music lesson, didn’t he?
- I liked Barry’s equations on the board. I had not thought of being able to pre-arrange whiteboards for each student until I thought about how to use what he did in his lesson. I do not expect you have had a chance to read my feedback to him yet so I will also mention it here. If I knew who my students were that were going to show up, I could create a whiteboard for each student. They come to class and put up a problem on their whiteboard while they wait for others to arrive. Another way of doing it could be to “seed” the whiteboards with problems and assign the whiteboards to students as they arrive. They would put up their work so they could explain it to the rest of the class during the session.
- This sort-of ties in with what Janette and Earl did with the chat windows. Even though we followed directions and only wrote on the chat screen we were assigned to, I wonder if they could have been set up to be pre-assigned to students and restricted from others being able to write on them. I had not thought of using chat windows as a way to run small discussions. Watching that process was very useful.
- I liked how Chioma used the chat window for formative assessment- she kept us alert because she was asking questions that required feedback. Even though I was a little disoriented because her Adobe Connect window would not open on the Mac at first, I found her technique to be effective. It was quick and she could use online learner cues (p.82) to gauge participant interest.
- Travis and Kirkland were very creative by having a game be the final assessment. I also found it interesting how they assumed everybody should know how to do a screen-shot. Is that the level of our online students? Do they know all of these techniques? If I did not have Snag-It I would be at a loss for how to do screen shots and actually use them.
The only problem I had with the presentations, other than them going at a pace that was a bit too fast for me to be comfortable with the changing scenery, was that there were not enough of them. I thought we were excluded from the rest of the spreadsheet because we were not welcome in other sessions so I did not try to be a part of them. Now that I see how talented my peers are, I wish I had been. I learned something from everybody today. It did not matter if their presentation had been memorized, polished, perfect or not, everybody offered something unique that let me walk away with more than I had arrived with in my bag of tricks. Thank you for this opportunity.
Reflect on assessment of learning outcomes in online environments. Consider the following questions in your reflection:
- What are appropriate assessment strategies in synchronous and asynchronous delivery methods?
I think formative assessment is more readily done in synchronous sessions because the feedback is instantaneous. It could be done asynchrously, but the instructor won’t know what the students are thinking until the student remembered to turn in his/her assessment.
In both cases, written assessment where students analyze something can be effectively done.
- Does this look different than assessment in traditional classrooms? How and why?
I think it looks somewhat different online than in a traditional classroom because students who are afraid to volunteer an answer in the classroom will often speak up online. Even today, everybody participated in Chioma’s questions. She did not call on single students like what normally happens in the traditional classroom. This is one reason I want to be an online teacher and enjoy being an online student. I hate answering questions in verbal face to face discussions, but as you have seen, I am quite prolific online. I know I am not unique so I wanted to used online discussions to compliment the ones we did face to face. Once again, I assert that hybrid instruction is optimal because the learning environments are diverse and can cater to the diversity of our learners.
We did case studies in EDTECH 562 so we can see how evaluation is done in the real world. How are statistics used to validate what was done during the research process? Here is an example of one of the case studies I completed while taking the class.
EDTECH 562: Module 4 Case Study
Submit to Module 4: Case Study
Please read the Module 4 Case Study file: Li, Q. (2010). Inquiry-based learning and e-mentoring via videoconference: A study of mathematics and science learning of Canadian rural students. Educational technology research and development. 58(6), 729-753.
EDTECH 562: Module 4 Case Study
Your Name: Melissa Getz
1. Research question:
How does providing eighth grade math students living in a rural setting an opportunity to interact with people who do research allow for a more authentic experience, thereby increasing students achievement and interest in math and science?
According to the paper, the research questions they asked are:
- How does the experience in an IBLE affect rural students’ learning of math and science? Specifically,
- Does the overall learning experience in an IBLE environment improve rural students’ achievement in mathematics as demonstrated in test scores?
- In what ways does the overall learning experience in an IBLE environment impact rural students’ affective development in math and science?
- What are the challenges of establishing an IBLE environment in a rural context?
2. Research strategy used:
Before bringing the students into the activities, the adults did a bit of planning. As a team, they created projects for the students to do with eMentors. They identified overarching themes, by focusing on the overarching questions:
How does understanding multiple perspectives shape the way we live in the world? In what ways does diversity shape our understanding?
After identifying the themes, they brainstormed project ideas and designed the project structure.
The formation of the inquiry projects was based on these three questions:
- What are the curriculum topics that need the most attention?
- What topics will engage students?
- How can they match eMentors to students so that students benefit the most from their interactions with the eMentors.
Following the planning, they implemented an action plan that involved the students interacting with the eMentors and completed the project by doing the post-tests and student interviews.
There were two control groups (41 students) and one experimental group (26 students) whose post-test scores were compared. The research group also did a pre-test so that changes between the beginning of the project and the end of the project could be measured. Nine of the students in the experimental group were personally interviewed to collect evidence of students’ attitudes about the experience.
The research team used both quantitative analysis and qualitative data. The quantitative analysis was generated twice:
1. is there a statistical difference in post-test scores between the control and the treated group?
2. is there a statistical difference in pre-test and post-test scores for the treatment group?
Interviews were conducted with nine students in the treatment group so as to not disrupt their courses too much. All nine students were interviewed alone or in pairs three times during the project. They felt the number adequately covered the population because the students were chosen based on having representation from a variety of academic backgrounds as well as having a small enough group with which to develop trust and confidence between the researchers and the students.
3. Independent variable(s):
Independent variables are the ones the researchers manipulate. That is a definition for independent variable which I translate to mean the researchers are choosing a variable that can allow for output as a result of doing the experiment. For example, if they chose temperature, it would influence the experiment in a way that causes there to be output that is specific to the temperature of the experiment. Or time can be an independent variable because as it happens something else changes. The independent variable itself does not give us information that is used in the statistical analysis, but the output it can cause is used. The output also comes from dependent variables that depend on the independent variable to know how to behave.
In this situation, there is the variable of time because we have pre-tests and post-tests. The output on the pre and post tests depends on the experimental timing- had the students done the inquiry lab with the scientists as support or not? The tests themselves would also be an independent variable because the student responses to the tests gives us data- the student responses are a dependent variable that relies on the test to provide an output. An independent variable here also involves if the students interacted with an eMentor or not. We decided who worked with the eMentor and the output we will be measuring is the students’ gain in interest in math because they worked with an eMentor. The students’ opinions are dependent on whether or not they had access to an eMentor.
There is also the variable that we are working with children. Their output is a dependent variable- it is not predictable and is based on their doing the math that was in the assignment.
This also brings up another independent variable which probably should have been listed first because it is the main difference between what happens to the experimental and the control groups: who gets to work with the scientists? Which group of kids gets the eMentoring?
4. Dependent variable:
Dependent variables give us the output. They react to whatever is happening in the experiment and it gives us our data. In this experiment we have a few different dependent variables, all of which are the result of student output. The student’s reactions to the pre and post test questions depends on their prior knowledge or what they learned by doing the projects. We also have student reactions to the interview questions. The interview questions were chosen by the researchers which makes them independent variables, however the unknown result of them is what the children are going to say. The children’s responses are based on their experiences in the eMentoring project as well as how the questions were designed to elicit a response.
5. Data analysis/statistical analysis:
Our research hypothesis is that there is a difference between students’ achievement on the post-tests. The null hypothesis, therefore, would be that there is no significant difference between the students’ scores on the post-tests. We are accepting the null hypothesis here: there is no statistical difference in the two groups of student scores on the post-test.
T tests indicated there was no statistical difference between the control group’s post-test scores and the experimental group’s scores. The only scores that could be compared between these two groups (ones with an eMentor and ones without) are the ones at the end of the unit because the control group did not do the pre-test. Table 1 shows that the significance value is larger than 0.05: 0.056 with a t value of 59.03. That t value also seems quite large compared to the t values that came from our data analysis with the data sets in our assignment for this unit. It may be possible the t value is related to the N, which was 66. I have not done enough of these tests to know if the t value means as much as the sig value being as large as it is. This sig value of 0.056 means there are 5.6 opportunities, almost 6, in 100 that there is no significant difference between the mean test score values of two groups. There is a high chance the mean test scores are the same. The 0.056 is falling in the confidence interval instead of the critical region. If the sig value, p, had been smaller than 0.05, then we would have said there was a statistical difference in post-test scores between the two groups because there is a very, very small chance the mean of the test scores would not be the same. If the mean of the test scores were not the same, then we would be accepting the research hypothesis: there is a significant difference between students’ achievement on the post- tests.
The means of the post-test scores were too close for the effect of an eMentor to cause there to be a significant difference between the achievement of the control and the eMentor group. They conducted an independent t-test on the final grades because they had two sample groups for these scores: control group and the ones that had access to eMentors.
A paired-sample T-test between the pre- and post-tests did show a statistical significance in the scores between the pre- and post-tests. According to their results, student achievement was statistically significant in terms of improving by doing the IBLE project. The statistics, t(25) =3.54, p=0.002 tells me they did a test with 25 degrees of freedom, N-1, the t value coming from their statistics program and a significant value of 0.002, which they are calling p in the expository part of the paper. Table 2 shows the results of the paired sample t-test.
Since, however, the final test results were not statistically different between the control and the treated group, it may be an artifact of how the pre and post tests were designed, more than an indication of the influence of an inquiry approach to learning the material.
They took the student responses and used codes to categorize the types of responses they received. Once they had numeric codes, they could manipulate the qualitative data, the student responses, in a way that let them put a number on how much the IBLE environment had an impact on the students. They came up with a value of 82% using an inter-rater agreement (p.739).
They also analyzed the students survey responses to determine if there was
- Improved engagement and motivation
- Broadened understanding of the relevancy of math and science in students’ lives
- Increased awareness of roles and careers in math and science
6. Results and outcomes:
Enough of a difference was found that this research should continue to be funded. Even though on the final post-test both the experimental and the control groups’ scores did not show enough variability to be significant, there was evidence that the experimental group’s change in achievement from the pre-test to the post-test was significant. It seems like the pre-test and the post-test were not identical. They say, “But the results above between treatment and control group indicated that this change might be caused by changing of test items.”
The group would like to extend this to be a longitudinal study, similar to the one they did with urban students. They also don’t know yet if this study will have long-term effects. They do not have the right instruments because they don’t exist yet. They do not have a reliable way to continue to track these students beyond this classroom experience.
Some students reported that their interaction with the eMentors increased their own confidence in math and science because the researchers and eMentors did use the students for their input on what was to be studied. Unlike traditional learning that goes from the teacher to the student without student input, this collaborative environment included students in on the lesson plans, or the direction of the project.
In their conclusion they assert that the continuous input from an eMentor is a significantly different paradigm than one where guest lectures give momentary input that is not directed to individual students, but rather to an entire group. A guest lecturer’s presence is also temporary, not allowing for follow-up questions from the students once they have had a chance to struggle with the content a bit more. The eMentor is also significantly important because there is a limit to how much the students can interact with their teacher or use the teacher as a subject matter expert the same way the eMentors can fulfill that role.
They also expressed how students moved their role from that of an information recipient to that of an information seeker. As students became more engaged with the project, they took the initiative to do research online and found a government agency to whom they could write letters based on the research they did in the project on bear habitats.
The researchers did not institute their own content based assessments so the pre and post-tests with which they had to use to collect quantitative data were not necessarily designed in a way to be useful for research purposes. It sounded like in the end they were not happy that they were forced to only use teacher designed summative assessments. They identified a few other challenges they hope to not face the next time they do a similar study, which will require them to choose their teacher and school partners wisely. (Personally I recommend they see how UC Berkeley professors use the local schools because they choose their locations so that they don’t have the same challenges these researchers faced. I know I always gave UC created assessments in addition to my own and did not actually use the UC assessments for the students’ content grades. But now I’m rambling on about me which is not what this article is about. )
This Communication Plan is supposed to cover:
- Routine Tasks
- Critical Thinking Prompts
- Management Issues and Strategies
- Online Discussion Forum Checklist/Rubric
- One original idea, category or thought
- Check discussion forums daily to see if there are new posts.
- Check email to see if students tried to contact me.
- See who has turned in assignments. If students who did not turn in assignments are on an IEP that requests they get additional nagging, nag them.
- Check to see what is coming up for due dates and post a reminder in the news forum or similar place.
- If there is a synchronous session about to happen, check my audio and video equipment to make sure they are working properly.
- Find a parent to call with good news.
- See if there are parents to call with less than happy news.
- Grade anything that needs to be graded.
- Write feedback to myself about how well things are working so I can note what needs to be changed the next time I teach this topic.
Critical Thinking Prompts:
Starting discussions with leads like:
- What do you think about…
- How would you determine…
- Why do you agree or disagree with…
- Evaluate famous person’s quote.
- Give feedback on the paper,- post a journal article or link to website for students to read
- Why is person’s ideas realistic, successful, or other adjective?
Discussion Board Strategies
- Have students start threads so that they can have a variety of places to share their “reply” type of responses
- I liked Adam’s suggestion of using a discussion forum as a way to do a jigsaw. Have questions already posted, students pick one question to answer, then they respond to something someone else said.
- Send students on a mission to find something online. It could be a picture, an explanation, or a specific website that gives specific information about a topic. They need to come back and share what they found to the rest of the group.
- Like we do here in the EDTECH program, students can post their unique projects to a forum for others to evaluate and to give feedback or suggestions for improvement.
- If there is a challenge question, you can have the forum set up so that you can’t read anybody else’s posts until you make one yourself. So let’s say there is a dilemma and you ask students to problem solve it. Each person needs to put up their own solution before they can read everybody else’s.
- I liked Glori’s idea of doing case studies. People would propose their recommendations for their case study. Then they would examine what everybody else put up to formulate a better idea or guide their peers toward more in depth thought.
- Ethical dilemma- students brainstorm how to solve an ethical dilemma. This is similar to a case study, but far less involved.
Management Issues and Strategies:
I am not excited about managing anything- my classroom management is pure dumb luck. I would kill my kids with kindness and make them feel too guilty to cause trouble for us. Discussion boards are not something I am looking forward to managing because I expect students to avoid them. So, to motivate students to post to boards or to continue posting, I can try:
- Positive feedback with words in the forum or through a personal email.
- Bribery with extra points as an incentive to just get students to be on the board.
- Ask students to talk about themselves. Let the discussion area be student-centric at least at first. Let students take ownership of the space before you squish their brains by having them expand their content knowledge in a forum.
- Use icebreakers. Our class came up with some amazing icebreakers to get students to share something about themselves. My peers did a wonderful job of organizing work we have done in the class and collected our icebreakers here.
- Do landscape style summary posts to recognize at least one contribution from each student up to that point.
- Although I will have a list of netiquette suggestions available in our first section, I want to invite students near the beginning of the class to share reasons why netiquette is important. I would also like to discuss what bullying looks like and why it is inappropriate. Some students may not realize that what they say is interpreted as bullying so I want to make sure students are aware of how to make our space safe. In this discussion I also want to include a review of what type of information is OK to put online, what information should not be shared, and the differences between where our class discussion happens and social media in general. Within this discussion I also want to point out why “I agree” or “Hello” posts are ineffective.
- In fact, we will have to have a forum where we set some ground rules, if I have not mandated them already. If this is my students’ first time in a discussion setting, we need to condition each other on to how to come to terms with too many posts to read. Part of the learning curve with online learning is to forgive yourself for not being able to read everybody’s posts. I want to discuss strategies with students about how to choose threads to open. I want to caution them about types of posts to make or not make if you want people to open your thread. I am an expert at turning people off in discussion forums so I know very clearly what to post to stop a discussion or truncate it before its time is exhausted. I can advise my students accordingly. We will have to have an introductory forum where we play with what to do and what not to do. I also want students to collaborate on strategies of what to pay attention to in our class’s LMS setting, what can be pushed aside until you have more time to spend reading, and what could possibly be ignored for a while without it causing too much trouble. (Number 7 is somewhat of a twist on what Palloff and Pratt say in chapter 4 of Building Online Learning Communities.)
- I want to have a survey always available for students to give anonymous feedback. I will address their feedback somewhere in the course so they can see that I am taking their suggestions seriously. If they know I care about their ideas and find them to be valid, maybe they will share more of them within the real forums?
- What is in chapter 7 will be very valuable. At the moment it is very difficult for me to hypothesize what my online classroom will be like because I have yet to experience an online classroom where I have freedom to make decisions like the ones that are described in many of the samples in chapter 7. I can make a list of what students can expect from me, though.
What students can expect from me, their instructor:
- Feedback from emails or phone calls within 24 hours.
- Someone who cares about students’ academic lives.
- Someone who expects students to make mistakes and to use those mistakes as learning opportunities.
- Someone who has high expectations and therefore will push every student to succeed. I will happily listen to challenges students face and will brainstorm with you ways to conquer currently perceived obstacles.
Online Discussion Forum Checklist/ Rubric:
Discussion Board Rubric
My discussion board rubric tool:
I am using the PBS rubric and Alexis Alexander’s rubrics to guide my discussion board rubrics. These rubrics rely on students being able to physically start a new thread when they reply to a general question. There is a diagram at the end that shows a graphic for how this type of forum can work.
What is expected in a post:
- Minimum of 50 words; no maximum
- Relates to the question posed or directly reflects what someone else said in response to the question
- Unique ideas or properly cited if not unique
- Proper grammar and usage of the English language
What could make up a post:
- Additional questions that expand the breadth of what has been said
- Personal anecdotes that relate to the question posed
- Quoted responses from the readings with your interpretation of what it means
- A respectful dissension of what someone else has said
- Discuss a related issue about which you would like feedback
- Provide an additional source that contributes to this topic that was not already provided in the course material. Tell us why you think this is appropriate.
|When did you post?||Your first response starts a thread within 4 days of the forum opening. Your follow up two posts are within two weeks of the forum opening. You are not limited to only doing two follow-up posts; this is the minimum required.||Your first response does not start a thread, but your posts stimulate discussion in the forum.
Your first response happens later than 4 days after the thread starts, but it still stimulates discussion.
|Your first response does not start a thread, nor do any of your responses stimulate discussion.
Your first response happens after 4 days after the thread starts and it does not stimulate discussion.
|What did you post?||Initial post and responses are on topic, demonstrate thorough understanding of it, and stimulate other people to think.||Initial post and/or responses are somewhat on topic, demonstrate some understanding of it, and/or stimulate some productive discussion.||Initial post and/or response are off topic, demonstrate faulty understanding, and/or do not stimulate any additional productive comments because of the quality of what you posted.|
|Usefulness of posts?||What you say contributes to other people’s schema. Others can use your ideas to generate their own, or to expand their thinking. You are able to get other people to consider a perspective that they may not have already thought about.||What you say does not stop discussion, although there may only be a few people who can relate to your suggestions.||Your posts are difficult to understand, do not provide concrete ideas others could use, or are very limited with their application.|
Diagram of how forum responses can be structured:
DiscussionParticipation is a pdf copy of the diagram.
Naturally there are more than three students in the course. The idea is that each person does an initial post before writing a response post. You continue to write response posts until you run out of ideas.
One original idea, category, or thought:
Teaching presence- is it possible?
Several education scientists have published their views on how discussion forums work. They analyze what students say, when they say it, how things are said, and anticipate the learning outcomes based on this information. There are three main presences that have been described: social presence, cognitive presence, and teaching presence. Much of our discussion in this class has focused on the social presence because that is the perspective our book, Paloff and Pratt (2007), uses when they explain how they see online teaching and learning will happen. To be thorough, they mention other presences even though they do not go as in depth with them. In 523 we had the opportunity to try out the teaching presence. Although I am usually an advocate of putting as much as possible into the hands of the students, I do not advocate having students lead discussions. I know that it is essential for those who want to be discussion leaders to be trained in how to facilitate a discussion and for some people in our class, this was their first opportunity. I bet it was amazing for them and they will have learned skills they did not realize could exist. At what expense is this done to the other students?
For this class, participating in the discussions was not a priority for me because I kept doing it wrong and therefore did not get the validation I am used to having in online classes that have active discussions. I think this is my twentieth online class so I arrogantly consider myself to be very experienced with how to maneuver in discussion forums. As you can see above, I am very much in favor of giving students the responsibility of starting threads. Within their thread, they own the pathway if they choose to respond to people who reply within that thread. I see it happen often in the 506 posts- we put up our image and people offer suggestions or ask questions. A dialogue forms between the one who started the thread and those who choose to volunteer their insights on the image that was posted. In a way the original poster can assume a teaching presence, although many of the replies to the original post are actually suggestions on what to improve and how to do it. In that way, a teaching presence can be seen in many of the posts.
In courses where the instructor oversees the path of the discussion, I rarely see a teaching presence allowed to happen by anybody except for the instructor. Even in some classes where students form their own threads, when the instructor posts in the threads, sometimes the discussion becomes one on one between the instructor and the person who started the thread. For me, I do not see that as being a productive use of time and space. Instead I think instructors should use email to directly address some of their questions designed to move a discussion forward that really only push one person to volunteer ideas. I see the instructor’s responsibility to help unify the group and interpret what others have said so that the ideas can get broader and more diverse. I am not convinced that students who have not had training in techniques that can be used to broaden discussions should be made the discussion leaders. Instead, I think students should be responsible for starting threads so they can have a microcosm of the discussion under their guidance. The instructor can still step in and offer engaging questions, but they do not have to be the only ones demonstrating a teaching presence.
Teaching presence actually has two “definitions”. Both have to deal with who is guiding a discussion, but one focuses on the teacher and the other focuses on the students. For the community of inquiry, the focus is on how well the students are able to guide the discussions. The research does not measure what a good job the instructor does of being the guide on the side. It measures how often or how well students step forward to cause inquiry to happen. In other words, are students posing the questions that guide future discussion or is the only one posting questions the instructor? Be careful if you decide to research the teaching presence because some people elaborate on how an instructor can manifest their presence in the discussions rather than how to get students to take leadership roles in the discussions. When students take on the leadership roles, then they are exhibiting a teaching presence. Chapter 8 in Palloff and Pratt elaborate on how to get students involved in the class. They have sections called, “Dialogue as Inquiry” (p. 170), “Encouraging Expansive Questioning” (p. 171), and “Sharing Responsibility for Facilitation” (p. 173). Although they don’t publicize that they are giving strategies for creating a Community of Inquiry or for getting students to develop a teaching presence, their ideas in chapter 8 align with those theories.
Articles or publications that elaborate on developing a teaching presence in students:
Garrison, Randy D. and Vaughan, Norman D. (2008) Blended Learning in Higher Education.
Community of Inquiry- a website including explanations and papers
Video that addresses how to establish the course such that students are aware of the instructor- I did NOT create this video; it is just here as a resource for anybody who may be using this edublog for ideas