Analysis of Cycle 3

I have spent the past week collecting and analyzing the data for my cycle 3 synchronous class. I am really pleased to have so much data available. With pre and posttest results, post-course survey data, and results from personal interviews, I feel pretty confident in my assessment.

So what is my assessment? I have little doubt that synchronous instruction can be just as effective as traditional classroom instruction. The feedback shows that students appreciate both the convenience and the fun of being online. And although the test scores were a little lower this time, they are still comparable to classroom results.

The technology worked extremely well given the wide range of experience. In fact, of the ten students I interviewed, none had ever taken an online synchronous course. Therefore, it is amazing we only had a few technical issues. I am still surprised with how fast students adapt to the technology. The technology itself is almost a non-issue and of little concern.

So from a positive perspective, the approach works. It allows for an efficient way to spread knowledge around the country (and potentially around the world). Our participation from 5 locations proves the tremendous reach and effectiveness of this method of instruction. Here are some other benefits:

• As already mentioned, the students like the approach and technology.
• The technology is at a point that it is dependable and easy to use.
• A flexible schedule with multiple and shorter sessions works very well for our audience.
• Once the students learn the technology, they often feel more comfortable and relaxed then being in a classroom. Because of this comfort level there may be even more participation, as long the course is designed properly.
• Synchronous instruction is very efficient in terms of cost, timeliness and access to expertise.
• A learner-centered approach is workable if properly planned. We used the last two sessions to have students problem solve troubleshooting scenarios and then present their approach online to the class. It worked great! In hind sight, I wish I had desinged even more learner-directed instruction for this topic, but it was a successful start.

Although, there were many positive aspects of the last class, we also had some real challenges. Personally, I feel there is still much to learn before this type of instruction begins to reach its full potential. This last session offered many lessons to me as a designer and researcher. Here a sample of what I learned:

• The instructor needs to be prepared and very familiar with the topic. This is even more critical with highly technical topics that our instructed online. Even with a learner-centered approach, the instructor need to be ready to step in and provide support.
• The material needs to be tailored to the audience. In our case, we had several participants who did not have the appropriate experience or technical knowledge required as a prerequisite.
• The learner's environment for taking the class must be quiet and free from distractions.
• With online learning, students need to be more disciplined in arranging their schedules to allow for sufficient self-study and class time. Supervisors should also be responsible in helping their staff set-aside time for training.

Despite these and other "bumps" in the road, I see a tremendous potential for synchronous instruction for a variety of topics.

Cycle 3 Course Analysis: Day 5

This was the final day of our course. To be honest, I am very relieved to have it over.

The fact the course was completed over 5 days really seemed exhausting and demanding from an administrative standpoint. This is something important to remember given this type of schedule.

I question whether one of our training groups has the ability to support this type of training. Since this was my project, I devoted a tremendous amount of time preparing and supporting the instructor. The fact that the instructor was still uncomfortable with the technology added even more demands on myself. There is no doubt that synchronous distance learning requires tremendous coordination and communication.

We finished today's class by having students present more troubleshooting scenarios. Again, this overall approach of handing the instruction over to the students worked very well in a synchronous environment. We ended the day by answering questions and completing the post-tests.

We had twelve students complete the course (of 17 enrolled). One of the twelve was a previous instructor of the course. Although he decided to not take the exams, he did participate in all the sessions. Most of those who did not complete the instruction had schedule conflicts, so this is not an area of great concern. It did concern me was the lack of consistent attendance of some students. Although each person had major work related conflicts that impacted attendence, the lack of participation must have impacted their learning.

I am still tabulating the scores, so more analysis will need to be completed later. I also have individual interviews set-up next week to gather feedback. It does appear from the initial scores I've seen, that this class will scored lower then the previous session. I will need to study this further and consider the feedback I receive from the class. I also sent-out a post-course survey that should also shed some light on the results. After assessing all of this data, I should have a clearer picture of the results.

What worked today:

• Again, the Subject Matter Expert really help answer questions and clarified concepts.
• We ended in plenty of time, and the students were able to complete the post-test.
• As always, the technology seemed to work great.

Improvements needed:

• This instructor needs to better understand the content and technology.
• We should have added a final exercise that would have allowed the students to review content. Having them deliver this exercise themselves would have worked very well.

Final thoughts:

I thought our effort to add learner-centered activities in the end worked well. The next step is to add this approach earlier in the course. The key to synchronous instruction is involvement. I didn't feel hit a high level of involvement during this cycle. One major factor was the instructor. He simply did not have the expertise to facilitate this course. I thought the instructor for our second cycle was more effective because he new the subject matter, and as a result could better facilitate discussion with the class.

Cycle 3 Course Analysis: Day 4

Today we had a portion of the class instructor led facilitation, and the remaining time devoted to student presentations.

My hope was that by turning the delivery over completely to the students, we would move toward a more student-centered approach, and move toward less dependence on the instructor. What was interesting was the contrast between both approaches. The students, especially for those delivering, performed very well and seemed to benefit from the whole experience.

In contrast, the instructor led piece seemed to drag along, and involvement was minimal. Even though the instructor attempted to ask more questions that would improve dialog, it was difficult to get the amount of discussion desired. I believe part of the problem was the instructors motivation today. He didn't feel well for one. He also had several stressful meetings planned for later in the day. During the student's presentations at the end, he even excused himself to attend another meeting.

On the positive side, the student-led presentations went very well. The students led the class through troubleshooting scenarios and explained the choices they made. In the end, my subject expert offered some very effective feedback. Tomorrow, we will spend more time having the students present. My hope is that we will again have some positive participation.

What worked today:

• The student led presentations.
  
• The support of the subject matter expert. He provided just the right amount of support and feedback.
  
• The timing and pace of the course seems appropriate today.
  

What should be improved:

• The instructor must be better prepared and appear confident throughout the presentation
  
• When minor changes are made to the course, it is not beneficial for the instructor to share the changes with the students. For example, he noted several times that we decided to leave out certain polling questions to save time and because they were not necessary. This probably distracted the students and hurt the perception of the course.

Cycle 3 Course Analysis: Day 3

Having our Subject Expert (SE) return to today's session made a real impact. It also reinforced my concern that the instructor may not have enough expertise to effectively deliver the content independently. This weakness seemed to hurt the interaction. The SE, on the other hand, seem to invite more participation and questions. When he was called upon, the interest from the students quickly increased.

So what is the lesson here? The online instructor should not only be an excellent facilitator, but he or she should have a sufficient level of expertise about the topic. This seems to be important even if a SE is available to assist.

We had 7 students missing today (of 16 enrolled). This was due to several work emergencies the pulled away many of the students. This lack of attendance might have also hurt the participation and momentum of the session.

Work worked well today:

• Our SE was able support much of the discussion and questions.
• We introduced next week's activity that will allow the students to lead work through a scenario and present to the group. Everyone seemed positive about the exercise.

Improvements needed:

• The instructor was moving through the content too quickly at times. This was the opposite of what happened with our previous instructor who too slow.
• As already mentioned, the instructor needs more expertise with the subject matter.

A final thought I had relates the different approaches and personalities of instructors. I chose the instructor for this cycle because was more energetic and engaging in his approach. However, he does lack subject knowledge and is less comfortable with the technology. My instructor from the previous cycle had a much stronger subject knowledge and seemed more relaxed instructing online. Unfortunately at times he seemed too passive when delivering. In the end, I appears that online instruction needs a little of both types.

Cycle 3 Course Analysis: Day 2

It is amazing the similarities between this course and my previous one (cycle 1). In both cases the primary concern is with the instruction. Both instructors were awkward with using the technology. With my current instructor, the uneasiness with the technology seems even greater. The fact that I am helping him more in managing the technology may even be hurting his learning curve. Is he becoming too dependent on me? Should I have let him struggle with the tools more from the beginning? I'm not sure.

My hope was that by supporting him with the technology, he would be free to focus on facilitation. What's interesting is that he does not seem as prepared in delivering content. The previous instructor seemed more comfortable with the subject matter. My current instructor seems to struggle with every question.

One surprise today was the absence of our subject expert (SE). He told me that he would be late, however, because of some urgent work issues, he had to miss the entire session. I quickly realized how dependent the instructor was on the SE. My instructor simply did not have answers for challenging questions. Does this mean I picked the wrong instructor? Is it too difficult depending on two key facilitators? I believe the answer is somewhere in the middle. I need an instructor who is prepared enough to handle content alone if needed. However, there is also a benefit in having a SE ready to support an experienced instructor.

I had another concern that was similar to the last course. We again have a wide range of abilities represented in the students. The fact that the instructor struggled with the content, and appeared to lacked confidence, probably isolated both beginners and experienced students. I decided to try something to remedy this situation. I asked my SE if he could come to the next class an half-hour earlier, and be available to answer questions. I also asked him to stay a half-hour after class to answer additional questions. I sent out an email today informing the students of this added service. My hope is that some will respond and use my SE's expertise.

We had 13 attend today (out of 17). Since several student were unavailable because of shift challenges, and one was on sick-leave due to back problems, I felt the turnout was acceptable.

Tomorrow will be interesting. I plan to introduce an activity that will allow the students to present their own chosen troubleshooting scenario. This we be a major step in making this course more learner-centered. The students will choose their own scenario and present to the group a strategy next week. I am very interested to see how this will work. It should take the focus and pressure off the instructor and allow the learner to take more control of the learning. It should be interesting.

Here is some quick thoughts about this session.

What worked:

• We had more interaction/questions today from some of the students.
• Despite having more questions, we still covered most of the content needed.
• We started on time and had all participants log-0n successfully (although one student had to use a loaner computer because his computer had problems downloading a needed plug-in).

Challenges:

• The instructor was not adequately prepared to deliver the content.
• The Subject Expert was unable to attend and support the instructor.
• The instructor was still learning the technology.
• There seems to be a wide variety of skill levels represented.

Cycle 3 Course Analysis: Day 1

For some reason, I was more nervous about the start of this cycle.

My previous cycle involved implementing our first synchronous course. Maybe it was because we considered the first course a pilot that I felt less stress. Calling our first effort a pilot took the pressure off trying to make the course mistake-proof and lowered my end-result expectations.

Of course, the anxiety this time may simple be caused by the start of a new OMET term and a demanding work schedule. One thing is for sure, I am still pleased to get started with my final cycle.

We had 14 students in attendance today. Considering we had 16 enrolled, I felt pleased with the exceptional turnout (similar results to our first cycle). We had one person unable to access the web-conferencing tool. Although he was able to listen to the audio portion, it was frustrating that he was unable to get online. I plan to meet with him early in the morning to work on the problem. Hopefully with the help of the web-conferencing technical support we will be able to solve the issue.

What went well:

• As I already mention, we had an excellent turnout.
• Nearly all of the participants logged-on early. This helped the course get started on-time.
• The move to an hour-and-a half session worked great (cycle 1 was one hour). This session feel less rushed and we finished on-time.
• We had more questions and discussion during this session compared to cycle 1.
• We had all 5 sites represented (compared to 3 for cycle 1). It was really exciting to see such a diversity of representation.
• Since both myself and the subject expert were in the same room as the instructor, we were able to provide more support and keep the pace moving. This helped the course go much smoother.
• Except for the one student with technical issues, all other students easily used the technology.

Improvements needed:

• This was the first online course for my instructor, so he was still awkward with the technology and the need to multi-task. We simply needed more preparation time.
• I need have a better plan in place to help those having technical issues. I felt unprepared for the one person who was unable to get online.

Overall, did I feel this was a successful start? Yes, considering we had an understandably nervous instructor. And as I stated earlier I was also nervous. It is difficult to move from a pilot to a regular session, with all the added expectations. The truth is, I feel this whole approach will continue to be a learning experience for years to come. The pilot may never end!

Student Prep. & Cycle Differences

As with my previous cycle, I have been investing time preparing students for the upcoming online session. There are similarities to the last class (cycle 1), but there are also differences.

For one, the class size is currently at 11 students. That is quite a reduction from the previous class of 16. Hopefully this adjustment will encourage more participation. I have also tried to better prepare students for the challenges of online learning. This has included recommendations for creating a more quiet environment and one free of distractions.

Finally, I have worked with a new instructor to prepare him for more dynamic online sessions. I will continue over the next few days to improve the course content, including adding an approach that is more learner centered.

There is one final difference worth noting. I will have 6 students from our Puerto Rico site. I'm not sure how having this new group will impact the results. Will their challenges in understanding English become a liability? Will their participation be more or less then that of other students? I don't know. It will be interesting to evaluate the effectiveness of this type of delivery for the Puerto Rican culture.