ANALYSIS OF ONLINE QUIZZES AS A TEACHING AND ASSESSMENT TOOL

Lorenzo Salas-Morera, Antonio Arauzo-Azofra, Laura García-Hernández

University of Córdoba, Escuela Politécnica Superior, Área de Proyectos de Ingeniería

Córdoba/Spain

Received November 2011
Accepted February 2012

Abstract

This article deals with the integrated use of online quizzes as a teaching and assessment tool in the general program of the subject Proyectos in the third course of Ingeniero Técnico en Informática de Gestión over five consecutive years. The research undertaken aimed to test quizzes effectiveness on student performance when used, not only as an isolated assessment tool, but also when integrated into a combined strategy, which support the overall programming of the subject. The results obtained during the five years of experimentation using online quizzes shows that such quizzes have a proven positive influence on students' academic performance. Furthermore, surveys conducted at the end of each course revealed the high value students accord to use of online quizzes in course instruction

Keywords -  Engineering education, educational technology, blended learning, online quizzes, e-assessment.

----------

1         INTRODUCTION

The actual implementation of the European Higher Education Area in Spain has led to deep changes in official regulations and university course catalogues, and to the inclusion of interactive teaching and learning methods. With the application of these methods, student responsibility in the learning process, and the need to acquire both content knowledge as well as specific and generic professional-related skills, has increased notably (Secretaría General de Universidades, 2009).

This paper presents the results of five consecutive years applying a combination of teaching strategies —discussion forums, group tutoring, collaborative learning, peer review and online quizzes (Salas-Morera, Cubero-Atienza, Redel-Macías, Arauzo-Azofra & García-Hernández, 2012) — and focuses on analyzing the effectiveness of online quizzes. In general, the following secondary objectives must be achieved in order to involve students in these teaching-learning methods (Salas-Morera, Berral-Yerón, Serrano-Gómez & Martínez-Jiménez, 2009; Lan & Yan, 2009): a) students must be motivated to work regularly and to follow the correct sequence of activities in the course programme; b) it is essential to monitor working time of students and check that it is compatible with the number of credits assigned to the subject, c) the teacher should have sufficient time to properly attend to all student requirements regarding forum, correcting papers, tutoring, etc.; d) the students must receive quick feedback on each of the learning activities that they do, and e) the students should interact proactively with each other and with the teacher.

In general, e-learning techniques involves a combination of teaching strategies (Lahwal, Amaimin & Al-Ajlan, 2009), such as assessment or self-assessment quizzes, discussion forums, and tutoring, that provide tools for further and collaborative learning (Nevgi, Virtanen & Niemi, 2006). In this context, the teacher’s role is critical (De Laat,& Lally, 2003), as the teacher must now move from being a mere transmitter of information (as in traditional educational systems) to becoming a content facilitator, technical consultant, and social animator —a more complex and laborious role (Jianhua & Yinjian, 2008). E-assessment has two key advantages: it substantially reduces teacher work time on assessments, and allows for rapid teacher feedback on student learning outcomes (Whitelock, 2009). Therefore, depending on the objectives to be met, e-assessment is expected to be highly efficient.

Furthermore, the use of online evaluation by means of a series of online quizzes (each one with little influence on the final evaluation) obtains better student academic results than using only standard exams as the learning assessment tool. Nevertheless, a set of precautions must be taken (Angus & Watson, 2009): a) online assessment methods should be effective substitutes for traditional ones; b) students should be at least indifferent to the use of these methods compared to traditional methods; and c) the online methods should achieve better results than traditional ones. Thus, it is highly beneficial to the learning process to use periodic quizzes as it compels students to work consistently and in accordance with other scheduled class activities (Savander-Ranne, Lundén & Kolari, 2008).

With regard to implementing methodological changes required to adapt to the European Higher Education Area, this paper proposes the following objectives:

·     Analyze the effectiveness of using online quizzes as a teaching and assessment tool

·     Test the influence of quizzes on student academic performance

·     Analyze workload that quizzes suppose for students, and quiz compatibility with the other planned activities

·     Solicit student opinion regarding utility of questionnaires

2         Methods

According to the ECTS system (Secretaría General de Universidades, 2009), the average student should be able to pass all subjects scheduled in an academic year with a total of between 1500 and 1600 working hours (in Spain) —equivalent to 25-30 hours per credit in a standard course of 60 credits. For this to be possible, it is crucial to establish a teaching methodology that integrates three key elements: a) is designed so that students acquire the knowledge and skills needed; b) student workload is consistent with subject objectives and credit allocation; and c) there is adequate information regarding methods, assessment criteria, and a detailed schedule of tasks.

Therefore, when scheduling activities, the first issue is to make sure students can complete the required tasks in the total working hours assigned to the subject. To achieve this goal, the students completed a series of weekly surveys to determine the appropriateness of the allocation of hours for each activity and its consistency with the overall objectives of the subject. Specifically, the subject Proyectos is worth 4.5 ECTS credits, and thus requires activities totalling between 112 and 135 hours of student work time. Proyectos deals with project methodology, organization, and management. The subject aims to strengthen student abilities in: organizing and planning; analysis and synthesis; applying knowledge in practice; troubleshooting; decision-making; oral and written communication in a professional context; adapting to new situations; and managing technical information.  It is expected that upon completing this subject, students will be able to organize the process for carrying out engineering projects, from managing technical documentation to implementing of solutions.

The Moodle platform was used as a tool to support teaching, specifically in the area of discussion forums and periodic quizzes. The general structure of the subject is as follows: classroom lectures with the whole group; practical classes in computer labs in groups of 25 students; participation of students in online forum; forum activity feedback through group tutoring; periodic online quizzes online, and feedback on quizz results through group tutorials. Overall student assessment is broken down in the following manner: Final examination in theory and problems (40%); participation in forum (10%); computer classroom activities (12.5%); group tutorials (12.5%); and quizzes (25%). The following sections of this article will focus exclusively on the methodology used in online quizzes.

The main purpose of online quizzes in Moodle is to encourage students to study regularly, and at a pace that is consistent with course requirements. The course is divided into three thematic blocks - TBI, TBII, and TBIII, respectively - and a total of six quizzes, evenly spread out during the semester. The topics covered in each quiz is as follows: 1) first half of TBI, 2) complete TBI block, 3) first half of TBII, 4) complete blocks TBI and TBII, 5) first half of TBIII and 6) complete blocks TBI , TBII y TBIII. Online quiz scheduling is designed to encourage students to work at the pace set by the teacher and in accordance with the other subject activities. The expectation is that by working at the pace set by the teacher, and in line with course objectives, students will display a more productive attitude in other subject activities. Each quiz is open for 4 days, and the total estimated time for completion, including review of concepts, is 2 hours. There are several sets of questions for each quiz, which are shared by all teachers in various subjects in the area of engineering projects. These sets of questions vary in degree of difficulty, and are assigned randomly to students in different questionnaires. Table 1 shows some examples of questions related to project planning (there is at least one correct answer).

Table 1. Some of the questions used in quizzes and statistical data provided by Moodle

After students have completed the two questionnaires for each block, the teacher gets from Moodle the statistics of students᾽ answers, analyzes the results (see last three columns of Table 1), and identifies the main weaknesses. This report contains the partial credit assigned to each answer, the number of students that chose each option, and the overall mark obtained. A 2-hour tutorial session is then scheduled with the entire group. During this session, concepts that produced the poorest quiz results are discussed with students in order to reinforce student understanding of them. Given the incremental nature of quizz content, students are required to periodically review concepts throughout the semester, which significantly improves students’ mastery of subject.

In addition to weekly surveys regarding student workload for each of the planned activities (which serve as feedback for activity scheduling for following academic year), other surveys regarding qualitative and quantitative aspect of the subject are completed by students at the end of each course. Moreover, we analysed the possible influence of questionnaires on the general student academic performance. The following section lists and discusses the results obtained.

3         REsults

3.1 Students' workload

After every week of the semester, a survey is programmed in Moodle about the time the students spent working in each one of the activities of the subject: preparation and completion of questionnaires, forum activity, studying, preparation of papers and reports, and tutoring assistance. Table 2 shows the average results obtained over five years compared to the working time initially allocated. It shows how, in the students' opinion, the overall amount of work performed substantially exceeds the assigned according to the course credits (4.5 ECTS x 25-30 hours/credit = 112.5 to 135). In the new organization, 6 ECTS credits have been allotted for the subject (150 hours of student work), which should remedy this situation.

Regarding the distribution of work over the semester (Figure 1), students spent a fairly uniform time in preparation for papers and quizzes during the whole semester, while personal study substantially increased when approaching the final exam. Although the forum is an assessable activity, it was the only activity in which students worked less time than programmed over the years of study. In fact, although not in significant number, some students completely renounced participating in the forum, despite the fact that debates are meant to enrich the content of classes and tutoring sessions to foster more successful group work. Thus it is clear that students, at least in part, are not prepared for collaborative learning that requires more active participation than in traditional classes.

Table 2. Average weekly workload of students in the five-years study.

3.2 Quizzes' results

Periodic quizzes via Moodle are intended to capture students' attention regarding the most relevant concepts of the subject and to link those concepts to other subject activities in order to increase participation and effectiveness of group tutorials and practical classes. As shown in Table 2, the average time spent on preparing and completing quizzes has been somewhat higher than projected in the teaching guide, which gives an idea of the importance that students give to this activity, which accounts for 25% of the final marks. Table 3 shows the overall results of the quizzes. The table shows irregular but consistently high participation and relatively high marks.

Fig. 1. Average distribution of student work along the 15 weeks of the semester.

A Ji² test was taken to ascertain quizzes’ possible positive influence on students’ final marks. The hypothesis (of the statistical test) was that students with marks of 6.5 or higher out of a maximum mark of 10, passed the theoretical part of the subject, and that there is a statistically significant relationship between the two variables. [Ji ² (1, N = 336) = 3.19, p = 0.07].

3.3  Students' opinions

Comprehensive surveys about the utility students attach to quizzes, or to participation in the forum activity, have been carried out at the end of each year under study to provide feedback for teaching guides to be used the following academic year. In response to the survey question regarding quizzes’ usefulness in keeping up with subject content, students gave quizzes a rating between 4.06 and 4.20 out of 5. This, together with high levels of participation and high average grades, shows that quizzes has been very helpful.

4         Conclusions

The integration of quizzes with other instructional activities in a teaching strategy has been very favourable. Students highly appreciated the role of the quizzes that, with moderate effort, helped keep students up with the subject, strengthened their involvement in other activities, and had a very positive impact on academic outcomes.

On the other hand, it has been proven that there is a statistically significant relationship between quizzes and other assessable activities, such as the theoretical part of the subject. Thus, although there are other activities in the subject, such as the forum, tutorials and group practices, overall academic success of those who made an average effort (more than 6.5 rating) on quizzes was very high.

Given the results of using quizzes, and their relationship to the mark for the theoretical section of the subject, it is reasonable to consider eliminating the theoretical section and substituting it with quizzes. However, there is no certainty regarding the usefulness of quizzes for evaluating certain competencies that must be reinforced, and could perhaps be better assessed in an overall exam. On the other hand, quizzes have the added advantage of guiding students through the learning process with moderate and continual effort. This advantage would probably be lost if the quizzes had greater weight in the final mark. Likewise, increasing the weight of quizzes in the final mark would have a significant repercussion on student study hours, which are already very high in comparison with the number of study hours anticipated in the initial work plan.

Table 3. Quizzes results for the five-years study.

REFERENCES

Angus, S. D., & Watson, J. (2009). Does regular online testing enhance student learning in the numerical sciences? Robust evidence from a large data set. British Journal of Educational Technology, 40(2), 255–272. http://dx.doi.org/10.1111/j.1467-8535.2008.00916.x

De Laat, M., & Lally, V. (2003). Complexity, theory and praxis: Researching collaborative learning and tutoring processes in a networked learning community. Instructional Science, 31, 7–39. http://dx.doi.org/10.1023/A:1022596100142

Jianhua, Z., & Yinjian, J. (2008). A case study on collaborative e-tutoring. Proceedings of the 2008 IEEE International Symposium on IT in Medicine and Education, 439-444.

Lahwal, F., Amaimin, M., & Al-Ajlan, A. (2009). Perception cultural impacts: Principles for trainer’s skills for e-learning. Proceedings of the NCM 2009 - 5th International Joint Conference on INC, IMS, and IDC. Article number 5331619, 986-993.

Lan, Y. F., & Yan, C. L. (2009). A practical approach to encourage students’ participation in asynchronous online discussions based on expectancy theory. Proceedings of the International Conference on Virtual Environments, Human-Computer Interfaces and Measurements Systems. Article number 5068907, 271-276.

Nevgi, A., Virtanen, P., & Niemi, H. (2006). Supporting students to develop collaborative learning skills in technology-based environments. British Journal of Educational Technology, 37(6), 937-947. http://dx.doi.org/10.1111/j.1467-8535.2006.00671.x

Salas-Morera, L., Berral- Yerón, J., Serrano-Gómez, I., & Martínez-Jiménez, P. (2009). An assessment of the ects in software engineering: A teaching experience. Proceedings of the IEEE Transactions on Education, 52 (1), 177-184.

Salas-Morera, L., Cubero-Atienza, A. J., Redel-Macías, M. D., Arauzo-Azofra, A., & García-Hernández, L. (2012). Effective use of e-learning for improving students’ skills. In R. Babo, & A. Azevedo (Eds.), Higher Education Institutions and Learning Management Systems: Adoption and Standardization (Chapter 14 pp. 292-314).  Information Science Reference (IGI Global).

Savander-Ranne, C., Lundén, O., & Kolari, S. (2008). An alternative teaching method for electrical engineering courses. Proceedings of the IEEE Transactions on Education, 51(4), 423-431. http://dx.doi.org/10.1109/TE.2007.912500

Secretaría General de Universidades. (2009). Guía del usuario del ECTS. Retrieved from  http://ec.europa.eu/education/lifelong-learning-policy/doc/ects/guide_es.pdf

Whitelock, D. (2009). Editorial: E-assessment: Developing new dialogues for the digital age. British Journal of Educational Technology, 40(2), 199–202. http://dx.doi.org/10.1111/j.1467-8535.2008.00932.x

 

 

AUTHOR BIOGRAPHY

Lorenzo Salas-Morera

He is a professor of University of Córdoba (Spain). His primary areas of research are intelligent user adaptive systems for automated facility layout design, projects management and educational technology. Since 2002, he has led several projects for the adaptation of teaching methodologies to the European Higher Education Area and, in particular, to the ECTS. He received the University of Córdoba Social Council award of Educational Innovation in 2008.

Antonio Arauzo-Azofra

He is currently assistant professor at the University of Córdoba (Spain). His primary areas of research are feature selection applied in classification problems for intelligent data mining in information systems and intelligent user adaptive systems for automated facility layout design. He is in active collaboration on several projects for the improvement of teaching methodologies adapted to the European Higher Education Area.

 

Laura García-Hernández

She is currently assistant professor at the University of Córdoba (Spain). Her research interests are in the fields of Facility Layout Design, Interactive Algorithms, Evolutionary Computation, Project Management and Educational Technology. She is in active collaboration on projects for the adaptation of teaching methodologies to the European Higher Education Area.

Published by OmniaScience (www.omniascience.com)

Journal of Technology and Science Education, 2012 (www.jotse.org)

Article's contents are provided on a Attribution-Non Commercial 3.0 Creative commons license. Readers are allowed to copy, distribute and communicate article's contents, provided the author's and Intangible Capital journal's names are included. It must not be used for commercial purposes. To see the complete licence contents, please visit http://creativecommons.org/licenses/by-nc/3.0/es/