Category Archives: Standard 5: Evaluation
Candidates demonstrate knowledge, skills, and dispositions to evaluate the adequacy of
instruction and learning by applying principles of problem analysis, criterion-referenced
measurement, formative and summative evaluation, and long-range planning. (from EDTECH 501)
Relative Advantage of Instructional Software
When I can find software to use in class, everybody benefits. Obviously it means I don’t have to lecture that day, other than to explain how to use the software. Chances are the students will be more interested, especially if the software is fun to use. Unfortunately some of what I have had students use is less than ideal. For example, I had students do a webquest to learn about doing protein gels. I could have lectured, but it seemed better for them to see the animations. For this particular webquest, I gave them questions and links to various websites where they can find the answers. I learned that if I were to do this again, I may have to put the link to the website adjacent to the question it answers. As much as students like using computers, they don’t necessarily like to use them for research, or to find an answer that can’t be easily found in a Google search or a wiki.
To teach students how to analyze data by using software that gives them the opportunity to read graphs or the results of an experiment, is not as good as having them do it hands-on in the classroom, but it is better than them not getting any experience with the information. Unfortunately many of the virtual labs I have used with students are either so difficult that it takes me hours to figure them out, like Gizmos, or they are just point, click, and drag exercises that they actually end out being a waste of time. Until my abilities with creating software or using software to create lab scenarios gets better, I am afraid that if I use software with the kids, it is going to be written by somebody else.
Interested in what our textbook has to say, I started skimming through it. Sadly on page 77, they say, “Today, after more than 30 years of development and experimentation, there is less talk of computers replacing teachers…” which is actually an optimistic perspective. What is sad about it, is that from my experiences in the last 4 years, it is not true. Computers and scripted curriculum are replacing teachers. There are companies who are making lots of money by replacing the teachers that used to be in the classroom by replacing them with virtual teachers. These virtual teachers will often have a load of 200 students per day from whatever states they have a credential to teach in. While I realize this post is supposed to be about how educational software and technology tools help the classroom teacher, I feel the need to point out the disparity that exists between a classroom teacher and a virtual teacher. Software IS replacing the classroom teacher. I know this because I taught kids in Delaware who did not have a classroom teacher. The software and I replaced whoever should have been the classroom teacher when the school was restructured. For my Pennsylvania kids, I was their teacher, even though I never met them in person, and live 2000 miles away. I did not actually ever teach them anything. I tried to tutor them if they would stay focused enough during a tutoring session to let me explain things to them, but even then, I had some kids who were not used to the idea of being responsible for their learning. This is not at all what I meant this blog post to turn out as so I will curtail my digression on how bad virtual schools are at this point, but I do want to point out that in my presentation of tutorials, drill and practice, and other categories of instructional software, this is not the same software being used in virtual schools. The software links I am presenting for this post are stuff that I either used when I was in the classroom, or would use should I ever get back in a classroom. (The later seeming further and further away from possibility, but you never know. So far using a wheelchair rocks using a walker, and if I upgrade to a power wheelchair, who knows what my limits will be?)
Robolyer and Doerling point out on page 78 that “instructional software packages are developed for the sole purpose of supporting instruction and/or learning.” It is important they differentiate between technology that is merely a tool, technology that is replacing the teacher, and technology that supports the teacher. Granted, they are not acquiescing that software is replacing teachers, but trust me, it is. They go on to elaborate which types of software can allow for directed and / or constructivist approaches. Naturally, as the students are given more control of the software environment, the more constructivist it can be. For example, having students build a website gives them more freedom than merely doing a webquest where they go hunting for answers to questions. (I have had students do both.) I see a parallel between paper and equipment lessons and computer software ones. The tutorials and drill and kill are like the worksheets or notes I used to print out on paper for the kids to use. Simulations are like cookbook labs. Problem solving scenarios are like inquiry based labs. At the moment, I don’t have a parallel for instructional games, unless doing a Jeopardy review or having kids make board games qualifies as an instructional game.
In chapter 3, Robolyer and Doerling give advice on how to select good examples of software in each category. In addition they elaborate the pros and cons of each type. Many teachers scoff at having any rote memorization types of drill and kill, whether it is a worksheet or a computer program. It is comforting to see that I am not the only one who finds value in having students practice specific types of problems repeatedly. I am currently tutoring an algebra 2 student, and while preparing for her winter final, it became pretty chaotic with so many different problems to figure out. One thing I started to notice, however, is that what was becoming more important than getting the right answer, was learning how to evaluate the situation to determine which technique best solves each problem. We may never recognize we are factoring a binomial in the real world, but learning how to be calm while sorting through our resources and evaluating them is a skill both my student and I will benefit from knowing.
Tutorials are my favorite type of programs to create because I love learning how to use Articulate Storyline. I took the BSU class on Flash, and it was pretty much a nightmare. I used Articulate’s free 30 day download for two classes, and became hooked. Fortunately I have significant support from my husband and family, so I was able to purchase Storyline. Flash will integrate with Storyline so I may do some flying numbers in Flash to bring in to a Storyline project, but otherwise I think I am stuck on doing the “explanation screen” way of trying to help students with various science topics. I have not created many tutorials, but you are welcome to see what I have done at www.getzguides.com. For my students who were enrolled in virtual classes because they were at a treatment center, my guides were a way they could get additional support for the classes if a live tutor was not available. Robolyer and Doerling point this out on page 88, tutorials are useful for instruction when no teachers are available. You may be surprised by how many students are taking classes that don’t have a readily available teacher. It is for these students I write my tutorials.
I am a huge fan of physics simulations. Even making apps with Corona or other simple programs lets you use physics. Even though I did not figure out how to make an app by coding in lua for one of my BSU classes, I did come to appreciate how physics can easily be integrated into simple software programs. As much as I am addicted to Minecraft, it is odd how they only have physics apply to two types of blocks. Then again, because they suspend the laws of physics, students can easily make three dimensional representations of objects when building in creative mode. Redstone mimics electronics and minecarts can travel based on gravity, so Minecraft is not completely void of physics. The redstone and use of minecarts on trails can give kids an opportunity to participate in something a teacher created, therefore making it a simulation or game, or they can create their own situations which would fall into the problem solving category.
I am torn when it comes to digital dissection because I know I truly learned more about animals by dissecting them, than if I had just gone through a point and click way of learning body parts. I wonder, though, how necessary it is to kill so many animals just for tenth grade dissections. Our book quotes from studies that showed digital vs physical manipulation does not seem to matter in terms of what information students retain (Roblyer & Doerling, 2013, p.91). For many teachers, the benefits of no set-up or clean-up, less costly equipment once the software is acquired, unless its license has to be renewed annually, and less supervision needed during the class period, outweigh the negative perception that what the students are doing is not actually real. The American Chemical Society (ACS), and the National Science Teachers Association (NSTA) have come out against virtual labs. Even the College Board will not accept credits in classes where students did a virtual equivalent of a lab. (Robyler & Doerling, 2013, p. 93). This means students will be doing PCR and running agarose gels for their AP biology lab, instead of imagining the bands migrating through the gel.
The last two categories, Instructional Game Software and Problem-Solving Software, are more difficult for me to see in the science context. The book recognizes Geometer’s Sketchpad, which is a very good program. It helps make geometry more spatially available. They also mention Spore as a game for studying evolution. I can’t comment on Spore because I’ve never played it. I do have to say, though, that I did an internship for a nanotech company in Emeryville, and the folks who created Spore were either on our floor or above us. It was interesting to ride in the elevator with them. But I digress, once again…
You may notice in my presentation , instructional games and problem solving software have very few entries. Hopefully I will be able to add more links after I post my blog. Fortunately the book treats the last two categories like it did the first three by giving example scenarios, and pro/con lists. One possible con that struck me was the idea of having to choose software that can handle limited physical dexterity (Robyler & Doerling, 2013, p. 95). I don’t think many students like having me in class because I can find faults easily in student work, and I will mention it. It is not to be mean; I’m actually trying to be helpful. People who don’t have disabilities really have no clue what it is like to have some. Just ask me about how ludicrous some of the ADA adaptations are where I live, and I’d be glad to tell you how we need people with the disability to create the adaptive physical changes, or in the case of my classes, adaptive software. We used Minecraft as a game, and as a way to do problem solving when I took EDTECH 531. In 531, we created an example of how to use one of three software packages as an educational tool, and there were some lessons I could not physically do because of the way they were designed. I did not have the manual dexterity to click and drag fast enough. If you know how to contact me, and you want me to evaluate any website or program you create for its difficulty with my limitations, just ask. I happily volunteer my eyes, hands, and defective brain as a testing environment.
In 531, I was incredibly impressed with how Minecraft (MC) can be used to simulate many social studies situations. I thought of a few ways it could be used with science, and I plan to make quests in 3dGameLab that have students use Minecraft to look at some science concepts. I feel like Minecraft is predictable enough that you can act like a scientist, and evaluate the game in survival mode as if one is going through the scientific method. I wish I qualified for minecraftedu so I could create scenarios that have students go mining for organic and inorganic resources. I can do that with regular MC, but it will be much more difficult to control student access to specific areas, and to protect blocks. The possible lessons in Second Life are also amazing, but from what little I’ve experienced, they are not on the level of games or problem solving. I can see World of Warcraft being used for problem solving because that is what you have to do continuously- the first problem being how to play the doggone game. I felt that way with Minecraft, too. I think any of these software programs that are easily intimidating at first are actually really good tools for students to learn resilience, endurance, and perseverance. I was a MC misfit when I first started playing it. I later became addicted to it. The book makes a distinction between doing problem solving software activities merely for the sake of learning how to problem solve. (Robyler & Doerling, 2013, p. 97). I can totally see using software for that purpose, at least until someone figures out how to create something that can be open ended enough for students to be able to make mistakes and therefore be able to learn from them
One thing that should be in any of the interactive software games is a chance for failure. When click and drag scenarios are too predictable, students won’t be challenged and will complete the activity because they are required to, and not necessarily because they are enjoying what they want to learn. We need to be careful, though, to not build in failures that students will take too strongly or too personally. I still don’t know where I am going to fit into education in my next stages. I’m hoping it will involve creating quest based courses in 3dGameLab that other teachers will want to use. If I can figure out how to turn a quest or a course into how to problem solve something in science, other than an easily predictable physics or genetics lab, I will be ecstatic.
Roblyer, M. D., & Doering, A. H. (2013). Integrating educational technology into teaching [6th edition].
Dr. Thompson had us “practice” writing a request for proposal (RFP) document. In this document we had to plan how to execute helping Far West Laboratory with their need to educate their school clients.
I think this is a reflection on what was done for the VoiceThread moderation:
Voice Thread moderation
How do you help students interact effectively in an online course?
A few of us have posted examples of how we help students interact effectively in our discussion areas. Although Chris has not held online discussions with students, she likes the small group approach. In addition she likes the idea of having students be facilitators. Bret confirms the importance of using multimedia and unique opportunities to engage a discussion. James also likes the idea of having student facilitators and freshness to the content, but cautions against overwhelming the students with too many new ideas or tools to learn. Sarah points out facilitator involvement is crucial, yet the facilitator needs to be careful to not become the discussion. Let the students be the discussion by finding a balance between facilitator input and student contributions. Earl stresses modeling effective communication so novice participants experience what they are expected to do in the discussion. So far our discussion has focused on group size, discussion format or setting, facilitator involvement that may involve student facilitators, and modeling what we expect of the participants. What other suggestions do you have about how to get students to interact in a discussion forum or even with wiki collaborations? We have a few more days left for this discussion, so please provide examples from your classroom, experiences with online learning, ideas from the readings, or unique perspectives you’ve acquired while in this or other online classes. What has motivated you to interact in our online courses?
How do you sustain online discussions?
A few people have shared aspects of online discussions including how the discussion is launched, what happens during the discussion, and how to prolong it. Kathryn stresses the importance of using open-ended questions to allow for freedom of expression of ideas. Bret cautions instructors to not assume that an open-ended question will guarantee student participation that stays focused on the topic. How would you build community building into the online discussion that may be more natural to create in a face to face environment? Sarah seeks the perfect balance between structure and flexibility. What suggestions do you have about how to create flexibility while still giving enough structure so students feel safe in the environment? Earl suggests extending discussions with hypothetical situations. With that in mind, what do you suggest we do to get people who have not posted to this discussion yet, to post to the discussion? Do the facilitators send out personal invitations to the discussion? Do we respect that for this discussion we are graduate students and therefore have the choice to participate or not? If you are working with adults like we are, but who may not be as comfortable with the online world as we are, how would you lure them to volunteer their ideas in an online discussion?
How do you keep a presence in online discussions without taking over the conversations?
So far, everybody recognizes the importance of having the facilitator being a part of the discussion. Kathryn points out that creating a social presence by providing feedback to participants. James suggests instructors target the posts that are not getting many responses by replying directly to those ideas in an attempt to stimulate discussion based on what is said in the neglected posts. Sarah’s audio file was not loading at the time I crafted this summary. Jessica recommends brevity while including leading questions to further the discussion. How do you suggest a facilitator follows these recommendations without overwhelming the discussion? How does one provide feedback, but not so much that it curtails further discussion?
How do you use online discussion in your blended courses?
So far we have heard a few ways people can use online discussion areas to allow for collaboration or submission of individualized perspectives on a topic. Glori has her students do mini-case studies by perusing the literature and deciding what they would incorporate into their own practice. Adam does a sort of jigsaw where he posts questions about themes in geography and allows students to self-select which ones they will respond to. Then he challenges students to visit a different theme and contribute to its thread. Bret confirms the uniqueness of using discussion forums as a review area for essay exams. He commends Adam for using the discussion area to stimulate student interactions and follows up with questions about procedures for doing such discussions. Kathryn suggested using the discussion areas as a place to do summarizing activities or for students to provide feedback on the course without having time constraints you can have in the face to face classroom. Sarah not only uses discussion areas as a place for students to brainstorm, but since it is out in the open, she can also give feedback and approve their ideas before students prematurely commit to a topic that may not be as fruitful as originally anticipated. What are some other concrete examples of ways you can engage students in an online forum?
In EDTECH 523 we created a synchronous presentation and were required to watch other presentations as well. To focus on other people’s presentations, we were asked to create an evaluation guide that we could fill out during the presentations. After the groups finished, we emailed our presentation form to the respective group. I’m connecting to my evaluation instrument by having a link to it here as it was uploaded to WordPress. The other link is to it living in my Google Doc area. Finally, I did a copy and paste of the document and put it at the end.
Purpose of this assignment: Create an evaluation instrument for evaluating the quality and appropriateness of synchronous instruction.
Yes/No or N/A
|Introduction to the lesson:Could be a poll on the screen or an activity on the whiteboard- just has to be something students can do to amuse themselves until the session starts|
|Presenter reviews classroom controls|
|Audio and/or video enabled for participants|
|Students do something in the classroom other than just sit there.|
|Presenter records session|
|Students are encouraged to raise hands if they have questions|
|A copy of the ppt slides if applicable|
|Web links are available|
|Presenter remembers to turn off recording|
|Polls for formative assessment|
|Polls to keep students engaged|
|Student expectations shared|
|Questions for students to answer that they will turn in later|
|Shares files students can download as take-away info|
|Presenter provides files students can download so they can follow the lesson as it happens|
|Ppt slides are uploaded to a share screen or are shown from presenter’s computer as a shared screen.|
|Students are used to help advance the lesson|
|Encourages student-student interactions.|
|Encourages student-content interactions.|
|Students use whiteboards or the equivalent to answer questions.|
|Formative assessment is done so the presenter can see if the lesson was successful|
|Student product is used somewhere in the lesson to either evaluate student work or to give an example of what can be done at a student’s level|
|Presenter is organized so that there are not unnecessary lulls|
|Presenter speaks clearly|
|Presenter speaks slowly enough|
|Presenter speaks loud enough|
|Some sort of closure|
|You can tell there was a reason this worked better as a synchronous lesson than an asynchronous one|
For Module 5 your summary should include information about the following:
- Complete the required reading and review materials provided on synchronous tools and strategies.
- Develop a lesson to be delivered synchronously using appropriate instructional strategies.
- Practice delivering your lesson.
- Create an evaluation instrument for evaluating the quality and appropriateness of synchronous instruction.
- Post the completed evaluation instrument and your reflections to the Evaluation of Synchronous Instruction forum in the MAIN discussion board.
- Please, also submit your completed evaluation instrument to the appropriate link in the Module 5 activities for grading.
- Submit discussion idea and/or lead a discussion.
- Participate in ongoing discussions. Apply critical thinking and questioning strategies to your discussion posts.
Module 5 was fun because it got me back in the Adobe Connect classroom which is always a challenging environment to be in because it is so convoluted. Preparing for Bret and my synchronous lesson has also been fun because it is collaborating with someone who is intelligent and knows more about our topic than I do. This means I get to learn something while doing this project and have someone who can patiently handle my questions when I get lost. In contrast, since I bought the Adobe eLearning suite in the fall, I have Adobe Presenter which takes PowerPoint files and uploads them to the Adobe Connect server. It was a nice opportunity to refamiliarize myself with the software.
My evaluation instrument was somewhat incomplete so I revised it as I reviewed the synchronous sessions. Interestingly, when I evaluated the Photoshop lesson presentation I realized I had not planned on evaluating something that was not necessarily like something listed in chapter 6. Bret and I have been trying to figure out how to adapt what we want to do to fit one of the suggested activities in chapter 6 and still be within 10 minutes. I think we have a clue what to do and I hope everybody who wants to be a part of our audience will join us and the other presenters on Thursday, May 3. It was also a relief to see that there will be at least six people in our audience. We were not sure what type of audience we could count on so we can now plan breakout rooms and student activities better.
It makes sense that chapter 5 in the Learning in Real Time text is applicable to Module 6 since it covers formative assessment, how to integrate it into the class, and how to pick up on non-visual body language. Since that chapter may need to be a part of the Module 6 reflection, I won’t go into more depth here. I will share a few ideas about chapters 4 and 6, though. For chapter 4, I found their analysis of various online teaching settings to be accurate based on what I’ve experienced. I used to IM with my students when they had a quick question and at times we would mosey on over to our virtual office to use the whiteboard for further explanation. Explaining dimensional analysis through IM can be done, but using a whiteboard is much easier. Although I have never taught or taken a class in a MUVE, I think that is what second life is like so I expect I will experience it when I take the class that has us use second life. Although I’ve attended webcasts, I’ve never led one. Although I have had a few online teaching opportunities, I’ve never actually been able to hold an online lecture or class session for students who were expected to be physically present. My current teaching situations are one-on-one and my previous one had live sessions as optional features for the students. I could do them as often as I wanted to, but they were never required to attend a session. I hope that one day I do get to have a real online class with real students that will be “my” students that I get to usher from one lesson to the next. The activities in chapter 6 will be very useful once I have the setting and the bodies with which I can practice.
Module 6 is where we teamed up with a partner to plan a live presentation for our peers using the Adobe Connect software. This is the reflection I wrote after Bret and I did our presentation, which was a fantastic experience.
Module 6 reflection
The readings and how they are reflected in our presentation:
Chapter 5 of the book was my favorite chapter. Even though Bret and I scoured through chapter 6 figuring out what type of interaction was possible and feasible, chapter 5 contained stuff that had tangible meaning for me at this time. In our presentation I played the role of behind the scenes host. I tried to calm people’s worries if they were expressed in the chat area during the presentation. I made it to one of the breakout rooms to help them get started with their conversation and let them know that they were doing great by writing on the notes screen. I also let them know they could use audio and video cams in the breakout room without bothering others. Before we pulled people out of the rooms, we sent the 20 sec warning that you were going to have your reality change. For the anticipated review of what went on in the groups, I pulled up the notes screens so they could be seen by everybody and therefore not be left out of any discussion. We also planned for a parting gift, which apparently did not download for some people. I have no clue why that didn’t work because we put the documents in there correctly. I also hope that some people get to take the survey so they can see what a Google form can do and if they use the links at the end of the form, they can view the data as it comes in. I was glad to see some welcomed the idea of having a “parting gift.”
I did not get to enact all that was suggested in chapter 5, in part because I was not a solo presenter. Also, since we were doing a round-robin of classroom jumping, there really was not a way to be prepared enough to welcome people as they entered. I understand that it was difficult to get people in as guests and Bret and I learned that barrier early on. I think this is why he came in our room as a guest and had me turn him into a host. Somehow everybody was turned into a host so it did not matter that Bret did not enter as a presenter/host. In some ways, the software is too friendly by putting a cookie in our machine and not making us re-register for each room. That is why I used my Mac when I was a participant and my PC as the presenter. I anticipated quick room changes and knew I’d mess it up if I tried to enter the other rooms while using my PC because the PC is cookied. It is not reasonable to expect people to have 2 computers to do this lesson so we could not expect everybody who had already presented to be out of the presenter registration. I think that is why so many people showed up as hosts when they entered the room- their machines were cookied and it is tough to remove that status. I guess since I spend so much time trouble shooting things because I often find them difficult to maneuver through quickly, that it proved to be an asset for me to know the Mac would work fine in the guest position.
Bret and I also used the Mac as a guest computer when we prepared for our session. Since we could not talk very clearly when we were not in the same room, it was hard for one of us to be presenter and the other to be guest when we practiced. I signed in to our room as a guest from my Mac laptop so I could see what the guests would see during the presentation. That is what taught me how the breakout rooms work. I could tell that putting ‘Mel on the Mac’ in a breakout room did not stop “her” from being a part of what was happening in the main room until the “start breakouts” button was pushed. Part of my nervousness in the beginning of our presentation was being afraid everybody would let their curiosity get the better of them and they’d move themselves out of the breakout rooms before we started them. The plan was originally to keep people as guests because we did not want them to play with stuff that was already set up to go. Fortunately we are working with adults so my fears were unnecessary. Everybody behaved themselves as perfect students and none of our tricks got messed up before they were delivered.
The backchannel- Bret and I did not necessarily see eye to eye on the backchannel, but this was not my place to be the total control freak so I went along with our main chat area being a backchannel. I don’t know if Bret has ever participated in a backchannel chat during a real presentation. I’ve actually only done it once, and that was when it was being taught to me at an ASCD presentation last year. I wanted there to be a backchannel and a real chat area, but it would have been too chaotic in the short amount of time we had. We named it the backchannel anyway so people could see that if they had enough room on their screen area, they could have 2 chat windows during their presentations- one for real concerns and the other one to be social. I am biased toward letting people use presentations as a way to make friends because sometimes not everything that is said needs to be heard.
That was another place I was not able to communicate well enough to get it into Bret that he did not need to do a lengthy introduction to what an LMS is. In our last practice he did narrow it down to maybe 2-3 min of talking, but today he went for more than 3 I would guess. I know my patience started to wane and I came close to just sending out the polls while he was talking. If you think today’s presentation was long-winded, you should have seen it during our first practices. I respect Bret because he wanted people to learn something during our presentation and he really is an expert on today’s talk. That is one reason we did this topic; it is relevant to what we may do as teachers and Bret had to do something similar for people in his district. Plus it had so many components that let us expand it in ways that let us play with Adobe Connect.
Bret did a fantastic job of outlining our expectations and establishing the norms for our session. You may have noticed that he built it in to the beginning of the PPT slides. He designed the presentation slides and let me go crazy with Adobe Connect bells and whistles. We somewhat followed the suggestions given starting on page 84 where there is one main person up front and someone else behind the scenes. I did not do all of the logistics alone; Bret helped with setting up the 4 types of polls and how to space everything so it would be ready to be used when we needed it to be there. I took care of naming things in a way that would make sense to us and others, putting the exit survey in a website link pod, uploading files for the file share, and creating the exit survey in Google Forms. Since I bought the eLearning suite when I was taking 521 I wanted to play as much as I could with the software. For some reason I could not get Bret’s slides to upload correctly so he did a screen share for our presentation instead of it being a file he used from the EDTECH servers. It would actually be really cool if the eLearning suite was required instead of the other CS5 suite because then we could possibly have lessons on how to use Adobe Captivate. I’ve only played with it once, but that is something that would be an asset to know how to use for online teaching. Dreamweaver , Flash, and Photoshop are also a part of the eLearning Suite so if you get to make suggestions to the department, you would not be too out of line if you suggested having the department use the eLearning suite in the future.
Other people’s presentations:
Even though I tried to follow advice and looked at other people’s eval tools when I revised mine for tonight, I found what I thought was important was somewhat tangential to what happened. Since the presentations went so fast and I did not want to take time to watch the recordings, I had lots of gaps in my evaluation forms. Regardless of what it seems I did not learn, I found these things to be new to me and very useful:
- Students writing on whiteboards. I knew it could be done, but had not experienced it myself in Adobe Connect yet. Actually I don’t know if I knew there could be interactive whiteboards in Adobe Connect. Had I known, we may have set up a whiteboard for each breakout room instead of using notes windows to record student interactions.
- Students could format their notes screens. When we pulled up the Notes screen for group 1, they had done some formatting. That was so cool. I don’t know if anybody else noticed it, but it was neat to learn that students could take ownership of some of their output if Notes pods were used for collaboration.
- I am still not sure what Adam did so we could move things on the whiteboard. I may have to email him to see if he can tell me. Adam did the music lesson, didn’t he?
- I liked Barry’s equations on the board. I had not thought of being able to pre-arrange whiteboards for each student until I thought about how to use what he did in his lesson. I do not expect you have had a chance to read my feedback to him yet so I will also mention it here. If I knew who my students were that were going to show up, I could create a whiteboard for each student. They come to class and put up a problem on their whiteboard while they wait for others to arrive. Another way of doing it could be to “seed” the whiteboards with problems and assign the whiteboards to students as they arrive. They would put up their work so they could explain it to the rest of the class during the session.
- This sort-of ties in with what Janette and Earl did with the chat windows. Even though we followed directions and only wrote on the chat screen we were assigned to, I wonder if they could have been set up to be pre-assigned to students and restricted from others being able to write on them. I had not thought of using chat windows as a way to run small discussions. Watching that process was very useful.
- I liked how Chioma used the chat window for formative assessment- she kept us alert because she was asking questions that required feedback. Even though I was a little disoriented because her Adobe Connect window would not open on the Mac at first, I found her technique to be effective. It was quick and she could use online learner cues (p.82) to gauge participant interest.
- Travis and Kirkland were very creative by having a game be the final assessment. I also found it interesting how they assumed everybody should know how to do a screen-shot. Is that the level of our online students? Do they know all of these techniques? If I did not have Snag-It I would be at a loss for how to do screen shots and actually use them.
The only problem I had with the presentations, other than them going at a pace that was a bit too fast for me to be comfortable with the changing scenery, was that there were not enough of them. I thought we were excluded from the rest of the spreadsheet because we were not welcome in other sessions so I did not try to be a part of them. Now that I see how talented my peers are, I wish I had been. I learned something from everybody today. It did not matter if their presentation had been memorized, polished, perfect or not, everybody offered something unique that let me walk away with more than I had arrived with in my bag of tricks. Thank you for this opportunity.
Reflect on assessment of learning outcomes in online environments. Consider the following questions in your reflection:
- What are appropriate assessment strategies in synchronous and asynchronous delivery methods?
I think formative assessment is more readily done in synchronous sessions because the feedback is instantaneous. It could be done asynchrously, but the instructor won’t know what the students are thinking until the student remembered to turn in his/her assessment.
In both cases, written assessment where students analyze something can be effectively done.
- Does this look different than assessment in traditional classrooms? How and why?
I think it looks somewhat different online than in a traditional classroom because students who are afraid to volunteer an answer in the classroom will often speak up online. Even today, everybody participated in Chioma’s questions. She did not call on single students like what normally happens in the traditional classroom. This is one reason I want to be an online teacher and enjoy being an online student. I hate answering questions in verbal face to face discussions, but as you have seen, I am quite prolific online. I know I am not unique so I wanted to used online discussions to compliment the ones we did face to face. Once again, I assert that hybrid instruction is optimal because the learning environments are diverse and can cater to the diversity of our learners.
We did case studies in EDTECH 562 so we can see how evaluation is done in the real world. How are statistics used to validate what was done during the research process? Here is an example of one of the case studies I completed while taking the class.
EDTECH 562: Module 4 Case Study
Submit to Module 4: Case Study
Please read the Module 4 Case Study file: Li, Q. (2010). Inquiry-based learning and e-mentoring via videoconference: A study of mathematics and science learning of Canadian rural students. Educational technology research and development. 58(6), 729-753.
EDTECH 562: Module 4 Case Study
Your Name: Melissa Getz
1. Research question:
How does providing eighth grade math students living in a rural setting an opportunity to interact with people who do research allow for a more authentic experience, thereby increasing students achievement and interest in math and science?
According to the paper, the research questions they asked are:
- How does the experience in an IBLE affect rural students’ learning of math and science? Specifically,
- Does the overall learning experience in an IBLE environment improve rural students’ achievement in mathematics as demonstrated in test scores?
- In what ways does the overall learning experience in an IBLE environment impact rural students’ affective development in math and science?
- What are the challenges of establishing an IBLE environment in a rural context?
2. Research strategy used:
Before bringing the students into the activities, the adults did a bit of planning. As a team, they created projects for the students to do with eMentors. They identified overarching themes, by focusing on the overarching questions:
How does understanding multiple perspectives shape the way we live in the world? In what ways does diversity shape our understanding?
After identifying the themes, they brainstormed project ideas and designed the project structure.
The formation of the inquiry projects was based on these three questions:
- What are the curriculum topics that need the most attention?
- What topics will engage students?
- How can they match eMentors to students so that students benefit the most from their interactions with the eMentors.
Following the planning, they implemented an action plan that involved the students interacting with the eMentors and completed the project by doing the post-tests and student interviews.
There were two control groups (41 students) and one experimental group (26 students) whose post-test scores were compared. The research group also did a pre-test so that changes between the beginning of the project and the end of the project could be measured. Nine of the students in the experimental group were personally interviewed to collect evidence of students’ attitudes about the experience.
The research team used both quantitative analysis and qualitative data. The quantitative analysis was generated twice:
1. is there a statistical difference in post-test scores between the control and the treated group?
2. is there a statistical difference in pre-test and post-test scores for the treatment group?
Interviews were conducted with nine students in the treatment group so as to not disrupt their courses too much. All nine students were interviewed alone or in pairs three times during the project. They felt the number adequately covered the population because the students were chosen based on having representation from a variety of academic backgrounds as well as having a small enough group with which to develop trust and confidence between the researchers and the students.
3. Independent variable(s):
Independent variables are the ones the researchers manipulate. That is a definition for independent variable which I translate to mean the researchers are choosing a variable that can allow for output as a result of doing the experiment. For example, if they chose temperature, it would influence the experiment in a way that causes there to be output that is specific to the temperature of the experiment. Or time can be an independent variable because as it happens something else changes. The independent variable itself does not give us information that is used in the statistical analysis, but the output it can cause is used. The output also comes from dependent variables that depend on the independent variable to know how to behave.
In this situation, there is the variable of time because we have pre-tests and post-tests. The output on the pre and post tests depends on the experimental timing- had the students done the inquiry lab with the scientists as support or not? The tests themselves would also be an independent variable because the student responses to the tests gives us data- the student responses are a dependent variable that relies on the test to provide an output. An independent variable here also involves if the students interacted with an eMentor or not. We decided who worked with the eMentor and the output we will be measuring is the students’ gain in interest in math because they worked with an eMentor. The students’ opinions are dependent on whether or not they had access to an eMentor.
There is also the variable that we are working with children. Their output is a dependent variable- it is not predictable and is based on their doing the math that was in the assignment.
This also brings up another independent variable which probably should have been listed first because it is the main difference between what happens to the experimental and the control groups: who gets to work with the scientists? Which group of kids gets the eMentoring?
4. Dependent variable:
Dependent variables give us the output. They react to whatever is happening in the experiment and it gives us our data. In this experiment we have a few different dependent variables, all of which are the result of student output. The student’s reactions to the pre and post test questions depends on their prior knowledge or what they learned by doing the projects. We also have student reactions to the interview questions. The interview questions were chosen by the researchers which makes them independent variables, however the unknown result of them is what the children are going to say. The children’s responses are based on their experiences in the eMentoring project as well as how the questions were designed to elicit a response.
5. Data analysis/statistical analysis:
Our research hypothesis is that there is a difference between students’ achievement on the post-tests. The null hypothesis, therefore, would be that there is no significant difference between the students’ scores on the post-tests. We are accepting the null hypothesis here: there is no statistical difference in the two groups of student scores on the post-test.
T tests indicated there was no statistical difference between the control group’s post-test scores and the experimental group’s scores. The only scores that could be compared between these two groups (ones with an eMentor and ones without) are the ones at the end of the unit because the control group did not do the pre-test. Table 1 shows that the significance value is larger than 0.05: 0.056 with a t value of 59.03. That t value also seems quite large compared to the t values that came from our data analysis with the data sets in our assignment for this unit. It may be possible the t value is related to the N, which was 66. I have not done enough of these tests to know if the t value means as much as the sig value being as large as it is. This sig value of 0.056 means there are 5.6 opportunities, almost 6, in 100 that there is no significant difference between the mean test score values of two groups. There is a high chance the mean test scores are the same. The 0.056 is falling in the confidence interval instead of the critical region. If the sig value, p, had been smaller than 0.05, then we would have said there was a statistical difference in post-test scores between the two groups because there is a very, very small chance the mean of the test scores would not be the same. If the mean of the test scores were not the same, then we would be accepting the research hypothesis: there is a significant difference between students’ achievement on the post- tests.
The means of the post-test scores were too close for the effect of an eMentor to cause there to be a significant difference between the achievement of the control and the eMentor group. They conducted an independent t-test on the final grades because they had two sample groups for these scores: control group and the ones that had access to eMentors.
A paired-sample T-test between the pre- and post-tests did show a statistical significance in the scores between the pre- and post-tests. According to their results, student achievement was statistically significant in terms of improving by doing the IBLE project. The statistics, t(25) =3.54, p=0.002 tells me they did a test with 25 degrees of freedom, N-1, the t value coming from their statistics program and a significant value of 0.002, which they are calling p in the expository part of the paper. Table 2 shows the results of the paired sample t-test.
Since, however, the final test results were not statistically different between the control and the treated group, it may be an artifact of how the pre and post tests were designed, more than an indication of the influence of an inquiry approach to learning the material.
They took the student responses and used codes to categorize the types of responses they received. Once they had numeric codes, they could manipulate the qualitative data, the student responses, in a way that let them put a number on how much the IBLE environment had an impact on the students. They came up with a value of 82% using an inter-rater agreement (p.739).
They also analyzed the students survey responses to determine if there was
- Improved engagement and motivation
- Broadened understanding of the relevancy of math and science in students’ lives
- Increased awareness of roles and careers in math and science
6. Results and outcomes:
Enough of a difference was found that this research should continue to be funded. Even though on the final post-test both the experimental and the control groups’ scores did not show enough variability to be significant, there was evidence that the experimental group’s change in achievement from the pre-test to the post-test was significant. It seems like the pre-test and the post-test were not identical. They say, “But the results above between treatment and control group indicated that this change might be caused by changing of test items.”
The group would like to extend this to be a longitudinal study, similar to the one they did with urban students. They also don’t know yet if this study will have long-term effects. They do not have the right instruments because they don’t exist yet. They do not have a reliable way to continue to track these students beyond this classroom experience.
Some students reported that their interaction with the eMentors increased their own confidence in math and science because the researchers and eMentors did use the students for their input on what was to be studied. Unlike traditional learning that goes from the teacher to the student without student input, this collaborative environment included students in on the lesson plans, or the direction of the project.
In their conclusion they assert that the continuous input from an eMentor is a significantly different paradigm than one where guest lectures give momentary input that is not directed to individual students, but rather to an entire group. A guest lecturer’s presence is also temporary, not allowing for follow-up questions from the students once they have had a chance to struggle with the content a bit more. The eMentor is also significantly important because there is a limit to how much the students can interact with their teacher or use the teacher as a subject matter expert the same way the eMentors can fulfill that role.
They also expressed how students moved their role from that of an information recipient to that of an information seeker. As students became more engaged with the project, they took the initiative to do research online and found a government agency to whom they could write letters based on the research they did in the project on bear habitats.
The researchers did not institute their own content based assessments so the pre and post-tests with which they had to use to collect quantitative data were not necessarily designed in a way to be useful for research purposes. It sounded like in the end they were not happy that they were forced to only use teacher designed summative assessments. They identified a few other challenges they hope to not face the next time they do a similar study, which will require them to choose their teacher and school partners wisely. (Personally I recommend they see how UC Berkeley professors use the local schools because they choose their locations so that they don’t have the same challenges these researchers faced. I know I always gave UC created assessments in addition to my own and did not actually use the UC assessments for the students’ content grades. But now I’m rambling on about me which is not what this article is about. )