Journal of Technology and Science Education

• Home
• About
• Login
• Archives
• Submissions
• Publication Fee
• Indexing & Statistics

Home > Vol 6, No 2 (2016) > García López

THE USE OF TECHNOLOGY IN A MODEL OF FORMATIVE ASSESSMENT

Department of Mathematics Applied to ICTs

Technical University of Madrid (Spain)

Received October 2015

Accepted December 2015

Abstract

This work describes a formative assessment model for a Mathematical Analysis course taken by engineering students. It includes online quizzes with feedback, a portfolio with weekly assignments, exams involving the use of mathematical software and a project to be completed in small groups of two or three students. The model has been perfected since 2009, and during the 2014-15 academic year the creation of a pilot online learning community was added. Based on Google+, it has been used for a peer assessment experiment involving student projects, among other uses.

 

 

Keywords – Formative assessment, Moodle quizzes, Social networks, Technology in assessment.

----------

 

 

 

1. 1. 1. Introduction

Adaptation to the European Higher Education Area presents a model of learning that is both student focused and competence based, which implies new rules for assessment. Competence is defined as “...a roughly specialized system of abilities, proficiencies, or skills that are necessary to reach a specific goal…” (Weinert, 2001). Along with specific competences in the subject matter, the degree programs include generic or multidisciplinary competences and thus the faculty is obliged to design activities that enable them to be developed and assessed. A good assessment model is essential for effective learning and must consider, along with the summative aspect (needed to assign a grade), a formative component that relies on feedback strategies, which may be assisted by technology.

The mathematics courses in the engineering degree programs tend to have extensive syllabi, which makes it difficult to manage class time and the students' work. Many professors, arguing a lack of time, are reluctant to innovate and work on competences. In the study conducted by García, García, Lías, Mahillo, Miñano and Pinero (2013) it was shown that a great majority of the mathematics courses in the IT degree programs at Spanish universities propose to assess generic competences with the same activities used to assess mathematics. This can be a valid approach, as long as the indicators for each competence are adequately defined and assessed. Both Díaz, García and de la Villa (2011) and Tuning Educational Structures in Europe (2006) demonstrate how normal mathematical learning activities help develop multidisciplinary competences, such as problem solving, communications skills, analysis and synthesis, and critical thinking. Each of these competences is related to one or more of the mathematical competences defined in the KOM Project (Niss & Højgaard, 2011) (Figure 1). Based on this relationship, it is possible to design activities to develop and assess both types of competences simultaneously.

Figure 1. Mathematical competences (KOM Project)

In the aforementioned study by García et al. (2013) on the models of assessment for the mathematics courses in the Computer Engineering degree programs, it was also observed that many courses incorporate the use of a mathematical software package for teaching and learning and in some way assess the exercises completed by the students, although few of them allow the use of software on exams. García, García, Martín, Rodríguez and de la Villa (2014) demonstrate the usefulness of integrating mathematical software in the different assessment activities. Moreover, the benefits of online assessment have also been identified in numerous works. Trenholm (2007) highlights the importance of its formative aspect, while Limniou and Smith (2014) experimentally verify the improvement resulting from the dynamic feedback and the possibility to make new attempts. Gikandi, Morrow and Davis (2011) present a report that indicates the potential of technology to:

• •Help clarify learning objectives and assessment criteria.

• •Facilitate the development of self-assessment activities.

• •Provide students with quality information and immediate feedback.

• •Facilitate communication between students and professors.

• •Offer the opportunity to correct deficiencies in previous instruction.

• •Facilitate data collection and processing for instructors.

The aim of this article is to show how the introduction of technology in a continuous assessment model facilitates the work of the instructor and favors the formative aspect of the assessment, thanks to quick feedback that promotes continuous improvement.

We present the assessment model used in the Mathematical Analysis course in the Higher Technical School of Computer Systems Engineering at the Technical University of Madrid. This continuous assessment model has been under development since the 2009-10 academic year, with the objective of fostering significant learning and assessing both specific mathematical and generic competences.

The article is organized as follows: first, the model is presented, describing the characteristics of the course and the different types of activities used for assessment, as well as the changes that have been introduced as the result of the experience gained. Next, some data from the last completed year (2014-15) are discussed. Finally, some conclusions and proposals are presented.

2. Description of the model

The assessment model that we have been applying, with successive improvements since the 2009-10 academic year, is based on the cornerstones of formative assessment:

• •The learning objectives and grading criteria must be clear and explicit for each activity.

• •Dynamic feedback positively affects learning and promotes communication between instructors and students.

As an online communication tool providing support for learning and assessment, we primarily use the Moodle platform, which provides (among others) the following advantages:

• •Easy access and use.

• •A variety of resources to manage instructional activities.

• •Institutional support.

• •It is widely used.

The Mathematical Analysis course is worth 6 ECTS credits. It is taught to different groups of around 70 students each, which are fully coordinated with one another and have a common Moodle environment. The specific mathematical competences are the usual ones addressed in a first-year calculus course for engineering students, although an attempt is made to emphasize those that are more closely related to computer science. The contents are divided into eight topics, which are organized in three modules. The classroom contact hours are divided between master classes with varying degrees of participation and practical lessons (in a classroom or laboratory setting), in which the students work to solve problems using the mathematical free and open code wxMaxima software. Below is a brief description of the assessment activities, including an explanation of how they are applied and how they have evolved over the years.

2.1 Online Moodle Quizzes (MQ)

For each of the eight topics, students are required to complete an MQ, designed by their professors according to the following protocol:

• •The learning results were defined and about ten indicators were established.

• •For each indicator, six or seven true/false questions were written, along with their corresponding feedback.

• •The MQ was drafted with one question per indicator, chosen randomly by the system from among the six or seven questions available.

The MQs are a learning tool that has a certain weight on the summative grade. Once the theoretical aspects of a topic have been explained, the students are given a couple of days to complete some trial self-assessments. A specific time is then indicated for everyone to connect online outside the classroom and complete the MQ (different for each student). The students are allowed two attempts, the best of which is counted. Each MQ has a weight of 0.5% on the final grade, as long as at least 80% of the questions are correctly answered. The total weight of the MQ on the final grade is 4%. In other words, if a student earns a grade (correctly answering 80% of the questions) on all 8 MQs, he/she would earn 0.4 of the 10 points of the final grade.

Over the years, as the result of the experience gained, certain improvements have been added. Feedback is no longer limited to merely indicating the correct answer, but rather to providing the justification or results on which it is based. Moreover, students receive feedback for both correct and incorrect answers (since it is possible to have a correct answer, but with faulty reasoning). As a complement, we have enlisted the students' help to create a glossary in Moodle, to provide quick access to the most important concepts. Last year, we limited to three the number of trial tests prior to each questionnaire, after observing that some students completed a large number of trial tests, but spent just a few minutes on each one, in an attempt to discover all the possible questions.

2.2 Portfolio

Throughout the semester, the students complete a series of short tasks, sometimes individually, sometimes as a team; some in the classroom with the professor's assistance, and others outside the classroom. All are returned corrected. The weight of this set of activities on the final grade has increased over the years, and they presently account for 21% of the final grade. These tasks include the writing of summaries, the creation of concept maps, the solving of exercises and problems, an occasional group workshop assignment and several ten minutes test about basic concepts. During the 2012-13 academic year, the requirement was introduced to have completed at least 70% of these tasks in order to pass the course through continuous assessment, as it is believed to be the most effective way to control attendance and class participation.

2.3 Small group project (SGP)

Working in small groups of two or three people, the students are required to complete an assignment, for which they are given a choice from among several different options, including:

• •Reading a mathematical document to learn a new concept, technique or algorithm on their own.

• •Using wxMaxima software to implement procedures that allow them to apply the technique studied to solve problems.

• •Solving a specific problem assigned to the group from a collection of different problems.

• •Drafting a report with the following sections: Summary, Introduction, Description of the method, Implementation and examples, Solution to the problem, Conclusions and Technical details.

• •Uploading the report and the wxMaxima files, implementing the procedures and calculations performed, to the Moodle platform.

The SGP is separate from the rest of the portfolio tasks in order to give it greater emphasis, as it is the first step towards the complete resolution of an engineering problem, and it enables the development of competences such as autonomous learning, teamwork and problem solving.

Each student spends an average of 10 hours on this project and it is worth 10% of the final grade. After the successive improvements, the assessment process presently consists of:

• •Correcting and grading the first version of the SGPs, according to a previously posted rubric and returning it to the students, along with suggestions for improvement.

• •Giving students the opportunity to improve their grades with a second version of the assignment, in which they correct their errors and implement the proposals for improvement.

• •The grade for the second version is weighted by that of an individual validation test administered in the classroom, which requires students to solve a problem similar to that of the project, using the tools that were practiced.

The two-stage correction is the product of experience. We have observed the effectiveness of giving a grade that the students know they can improve by a certain percentage by incorporating the suggested modifications.

The validation test was introduced in 2011, when it was observed that the teams were not distributing the work equally. As it is completed individually, the validation test detects those students who have done little work and intend to take advantage of the work done by their teammates. Students who do not demonstrate that they have achieved the objectives have their SGP grade reduced drastically, while those who show that they have met them improve their grade.

2.4 Mandatory guided practice

65% of the grade is based on exams. However, practice activities have been designed to foster significant learning. Extensive practice exercises, with clearly specified learning objectives, as well as theoretical questions, exercises and problems are provided for each of the three modules of the course. This activity is carried out throughout the instructional period of the module. In order to be able to take the corresponding exam, students must hand in the fully completed practice exercises, but it is not this work, rather the exam that will be graded. The exam is written by selecting some of the objectives and including exercises similar to those completed during the practice activities.

2.5 Pilot experience creating the APLICA_MATES online community

During the 2014-15 academic year, we created an online learning community based on Google+, in which student participation was voluntary. At the beginning, an open community was created with the aim of providing collaborative work tools, permitting the exchange of ideas and the communication of news and other activities, and making the students' work more relevant. But later, due to the dispersion of the members and the difficulties this poses when trying to distinguish the contributions made by our students, we redefined it as a private community. As an experiment, students were invited to share their SGPs on a voluntary basis, and these were assessed by some of their classmates. The assessment work was carried out after the semester had ended, and was recognized with one ECTS credit for miscellaneous activities. In this process, each student was assigned a SGP with a topic different from that he/she had completed, and they were given an assessment sheet that was similar to that used by the professors, but that included three questions related to an academic topic, in order to assess the assimilation of the concepts presented. The community has also been used to host a project competition, including the posting of proposals and an online vote between the two finalists.

Finally, it should be mentioned that, in fulfillment of legal requirements, in addition to the continuous assessment model, an alternative assessment consisting of a single exam is also offered, which students can request before October 30th. This is chosen by a minority of the students and has very poor results.

3. Data and results

Some data corresponding to the 2014-15 academic year are provided below. A total of 372 students were registered. The Mathematical Analysis course is common to the degree programs in Computer Engineering (CE), with 109 students, Software Engineering (SE), with 180 students, and Information Systems (IS), with 83 students. Of the 308 students who selected the default option (continuous assessment), 242 completed the minimum required tasks, and of them 143 passed the course. It should be pointed out that the efficiency rate for this course, traditionally considered to be “the hardest”, is currently better than that of other courses from the same semester (see Table 1, which presents the data from the university's semester report).

	CE	SE	IS
Mathematical Analysis	43.12	37.80	33.73
Introduction to Computers	14.91	14.95	5.68
Programming	42.34	44.23	32.14
Discrete Mathematics	51.46	58.05	30.95

Table 1. Efficiency rates for courses common to all three degree programs

With regard to the assessment model, it is well considered by the students, who expressed their approval on a survey to evaluate the instruction. The mean score for the assessment model used in the Mathematical Analysis course was 4.86 points (out of 6), while the mean for all courses was 4.54.

Participation in the MQs (see Figure 2) continued, but decreased throughout the course. The number of students who scored points makes it possible to identify the topics that are conceptually the most difficult. Even though the contribution to the final grade of this activity is small, its impact is significant, as more than 75% of the students with success on five or more MQs passed the course.

Figure 2. Results of the MQs

The group work (SGP), which is worth 10% of the final grade, was completed by 240 students. The assessment guidelines do not stipulate whether it is mandatory to complete the project in order to pass the course. However, only one student passed the course without having completed the project. The average grade for the written assignments was 6.08. After the individual validation test, 78 students improved their grades, while 142 students saw their grades go down. The SGP was passed by 116 students, of whom 91 passed the course.

At the end of the 2014-15 academic year, the private community APLICA_MATES had 77 members.  Student participation was not very active, possibly due to the initial problems. It cannot be said that the pilot project was a clear success, since participation was quite a bit lower than in the Moodle activities. A total of 18 SGPs were uploaded, and 17 students completed the assigned task of assessing an SGP. The responses to the three academic content questions on the assessment sheet were satisfactory, which indicates that assessing the SGPs has expanded knowledge, in addition to generic competences. The final grades assigned by the students (see Figure 3) differ little from those assigned by the professors. An analysis of variance shows (with a p-value of 0.9) that there are no significant differences.

Figure 3. Peer assessment

In the project competition, participants shared their proposals and used a survey format to vote online for one of two projects in the final round; in total, 44 community members participated. Figure 4 shows the results.

Figure 4. Electronic voting, completed by the APLICA_MATES community 

4. Conclusions

Over these years of successive innovations, we have observed that increasing the demand of the work throughout the semester has a positive effect on the results. For this ongoing work, timely feedback is vitally important, but this is difficult for the professor to provide, unless tasks are included that can be corrected quickly and/or automatically, with the help of technological tools.

With regard to the online learning community, we have witnessed some positive aspects: it facilitates peer assessment, gives relevance to the students' work and can be a suitable environment for certain portfolio tasks. But we have also encountered certain challenges: it is difficult to use different communication environments if they are not well integrated. The separate maintenance of the Moodle environment and the online social network complicates the work and forces those involved to use different tools.

Our current challenges are:

• •To find a way to enable students to make the most of online environments, publish their work on open platforms, use collaborative learning tools and improve their academic and social integration.

• •To improve and streamline feedback processes for both students and professors.

Firstly, we want to achieve a better integration of the online learning community in the Moodle environment and make the community more relevant. As a matter of fact, the number of members has increased to 117 in December 2015. Furthermore, we have recently offered the option to participate as a guest, as long as applicants can justify their interest.

Secondly, we are experimenting with the use of the application Socrative (http://www.socrative.com/) to replace some of the tasks that are carried out in the classroom with questions, problems and/or short exercises that are presented by the professor and answered by the students using their mobile telephones. Feedback in this case is immediate (for both students and the professor) and the method promotes class participation and discussion.

References

Díaz, A., García, A., & de la Villa, A. (2011). An example of learning based on competences: Use of Maxima in Linear Algebra for Engineers. The International Journal for Technology in Mathematics Education, 18(4), 177-181.

García, A., García, F., Lías, A., Mahillo, M.A., Miñano, R., & Pinero, R.M. (2013). Tratamiento de Competencias Genéricas en las Asignaturas de Matemáticas para Grados de Informática en las Universidades Españolas. Actas del II Congreso Internacional sobre Aprendizaje Innovación y Competitividad, CINAIC 2013, 375-380.

García, A., García, F., Martín, A., Rodríguez, G., & de la Villa, A. (2014). Changing assessment methods: New rules, new roles. Journal of Symbolic Computation, 61, 70-84. http://dx.doi.org/10.1016/j.jsc.2013.10.012

Gikandi, J.W., Morrow, D., & Davis, N.E. (2011). Online formative assessment in higher education: A review of the literature. Computers and Education, 57, 2333-2351. http://dx.doi.org/10.1016/j.compedu.2011.06.004

Limniou, M., & Smith, M. (2014). The role of feedback in e-assessments for engineering education. Education and Information Technologies, 19(1), 209-225. http://dx.doi.org/10.1007/s10639-012-9200-5

Niss, M., & Højgaard, T. (eds) (2011). Competencies and Mathematical Learning. IMFUFA 485-2011, Roskilde University.

Trenholm, S. (2007). An investigation of assessment in fully asynchronous online math courses. International Journal for Educational Integrity, 3(2), 41-55.

Tuning Educational Structures in Europe (2006). Approaches to teaching, learning and assessment in competences based degree programmes. Available online at:  http://www.unideusto.org/tuningeu/teaching-learning-a-assessment.html

Weinert, F.E. (2001). Concept of competence: A conceptual clarification. In D. Rychen & L. Salganik, Defining and Selecting Key Competencies, Seattle: Hogrefe and Huber.

---