Teaching and Learning with Web Course Management Systems

Student Experiences in Web Course Management Systems

Gayna Stevens Credle (Center for the Advancement of Teaching)

Introduction

This report on survey data collected as part of the case study addresses student computer skills and agreement levels for statements related to student experiences.

Students in six courses used Web course management systems. Assignment of students to the groups occurred by enrollment in participating courses. Due to the formation of groups, the sample groups were non-random. The types of activities that provided students with learning experiences are listed in Table 1.

The Survey

The survey was created using items from the Current Student Inventory (CSI) developed by the Flashlight Project, which is a set of tools for the study of educational improvement with technology. Some items were edited to refer specifically to the WCMS technology. The survey questions are listed below. The complete pre- and post-course surveys are found in Appendix I.

The survey questions included:

Rate your ability to do each of the following: 
on the scale of 1 - 5 with 1 = No ability or knowledge to 5 = Expert.

Question 1: Send and receive e-mail

Question 2: Create a word-processed document

Question 3: Create or edit a Web page

Question 4: Send and receive files by way of the computer (over a modem,
the Internet/WWW as file attachments or ftp, etc.)

Question 5: Search for information on the Internet/World Wide Web

Question 6: Use the Web system for this course (eCollege, Blackboard,
IntraKal)

Indicate how strongly you agree or disagree with each of the following
statements:

(1) no basis for judgment (2) strongly disagree (3) disagree (4) agree
(5) strongly agree

Question 7: The technology to be used is appropriate for performing the
tasks required in this course.

Because of the way this course will use electronic communication:

Question 8: The assignments will be more interesting than in a regular
classroom.

Question 9: I will spend too much time learning to use the technology

Question 10: I think I will be better able to juggle my course work with
my work and home responsibilities

Because of the way this course will use electronic communication:

Question 11: I will communicate better/more, with other students in the
class

Because of the way this course will use electronic communication

Question 12: I will communicate better/more with the teacher in class

Because of the way this course will use electronic communication

Question 13: My grade was (will be) affected

Because of the way this course will use electronic communication

Question 14: I will be a better learner

Question 15: I will be at a disadvantage because I do not possess
adequate computer skills.

Question 16: Rate the experience of using a Web course management
system:

(1) Excellent (2) Good (3) Neutral (4) Bad (5) Very Bad

Question 17: Would you prefer to have more courses that use a WCMS?

Yes No

The variables studied for relationships were student computer skills and agreement levels for statements on the experience with WCMS. The survey also contains demographic information and short answer questions.

Methodology

At the start of the semester, students were afforded opportunities (three separate occasions) to attend training sessions. Two courses used class time in the computer lab to acquaint students with the particular system used in class. In the sessions, handouts with directions for each WCMS were distributed to students (these handouts are available upon request). Students also received individualized assistance. After the sessions, each student was asked to complete a pre-course survey. At the end of the semester, the students also completed post-course surveys.

Sample group

The majority of the students were undergraduates, the majority being sophomores and juniors. Two of the courses consisted of homogenous student samples, one course contained all graduate students and one course contained predominantly freshman students. In all the courses, at least 81% of the students were enrolled because the course was required (Table 2).

Background experiences of students prior to this study were difficult to ascertain. The survey might have had a design flaw, in that the questions were in two parts. Many students omitted the section or only responded to one part, the second half of the question. The first question referred to experience with WebBoard Conference, an electronic threaded bulletin board system. In the course with predominantly freshman students, the WebBoard Conference experience was given a low rating. Of the other remaining students to use the WebBoard Conference, a good to neutral experience rating was given. Overall, 41% of the total sample group had used WebBoard Conference. The survey revealed that approximately 38% of the sample group had been in courses in which e-mail was routinely used. Studentsi prior experience with e-mail averaged a neutral rating. In two courses, however, the experience was rated good and above.

Approximately 25% of the students had used a WCMS prior to this study. Overall prior WCMS experience received a neutral rating. For the final rating of this WCMS study experience, the WCMS experience was rated as good. At the time of this writing, post-course survey results for one of the courses are not available.

Findings

In the pre-course survey, no significant differences between the sample groups were evident for computer skills or agreement levels on experience questions (see Table 4 for a comparison of means). Overall, there was a significant difference in mean score comparisons between the pre- and post-course surveys for six survey items and one aggregated score. The aggregate score was derived from agreement levels for computer skills (questions 1-6 combined). There was a significant difference between groups, pre- and post-course, and for self-rating of computer skills (p=.025). When the measure of computer skills was calculated with the use of the WCMS omitted (question 6), the ANOVA result (p=.084) was not significant at the .05 level (see Table 5 for data).

Significant differences between pre and post group means were found for the following:

The ability to use the WCMS system in the course (p=.001) [question 17]
Student agreement on whether they felt communication was better with other students in the class because of the way the system was used (p=.003) [question 11]
The survey statement, i Because of the way this course used electronic communication, I communicate better/more, with the teacher in classi (p=.048) [question 13]
The statement, iBecause of the way this course used electronic communication, I did (will) spend too much time learning to use the technologyi (p=.041). [question 9]
The statements associated with student computer ability (skill) also resulted in significant ANOVA mean differences:
Send and receive files by way of the computer (over a modem, the Internet/WWW as file attachments) (p=.010) [question 4]
Search for information on the Internet/World Wide Web) (p=.052) [question 5]
Individual courses also displayed significant differences in mean scores for other items on the survey. However, subgroup analyses were not performed. Table 6 shows the mean comparisons of pre- and post-course survey items.

Conclusions

Differences in student responses could not be attributed to a particular Web course management system. Survey results suggest that differences in student responses may be related to prior experiences and student academic level. The initial student impressions of the systems for all groups were that it would not take much time to learn to use them. Although students initially strongly disagreed with the notion of spending too much time learning the systems, post-course surveys indicated a move towards agreeing that the amount of time learning the system was too much. The results also suggest that WCMS take more time to learn than most students considered appropriate. There appears to be an initial preference by freshman for a WCMS over WebBoard Conference. This conclusion is from student initial ratings of both systems for which they had newly experienced.

The use of the WCMS appears to increase student confidence in their computer skills. Post-course survey results indicate that most students agree they reached near expert user level for the WCMS used in courses. The change in computer skills was not completely due to students learning how to use the WCMS for the course.

Expectations for communication with others changed over time. At the start of the study, students were more optimistic about communication within a WCMS. However, post-course survey results indicate less agreement with the better communication statement than in the pre-course survey. Lower (towards disagree) than pre-course levels of agreement were also given for the post-course survey statement, "Because of the way this course used electronic communication, I communicate better/more, with the teacher in class." This may also be viewed as a lower expectation. Influences upon communication may also be attributed to network and server problems. Most of the systems experienced difficulty with chat features due to university firewall issues.

Data

The data for this study are available as a 10 page, 77 Kb PDF document.

This site contains documents in the Portable Document Format (PDF), which you can download and print for your reading convenience. There are a number of viewers which you may use to display PDFs. We recommend the free Acrobat Reader by Adobe.

<-- Prev. Section | Survey Data [PDF, 77Kb] -->