Journal of Technology and Science Education

• Home
• About
• Login
• Archives
• Submissions
• Publication Fee
• Indexing & Statistics

Home > Vol 7, No 2 (2017) > Sánchez-Carmona

A GAMIFICATION EXPERIENCE TO IMPROVE ENGINEERING STUDENTS' PERFORMANCE THROUGH MOTIVATION

Universitat Autònoma de Barcelona (UAB) (Spain)

Received October 2016

Accepted March 2017

Abstract

The students' lack of motivation is a usual problem. The students value more the obtention of the degree than the developing of competences and skills. In order to fight this, we developed a gamification's experience based on merits and leaderboards. The merits are linked to the attainment of skills and competences that students usually do not appreciate. During the year 2016, we are carrying out a pilot involving 950 students at the Graduate of Engineering in Computer Science at the Autonomous University of Barcelona (UAB).

 

 

Keywords – Gamification, Merits, Leaderboards, Skills, Competences evaluation.

 

----------

 

1. 1. 1. Introduction

Engineering Professors face a problem increasingly widespread: even when the 76% of the students consider they feel interest or vocation for their studies, many of them do not seem to be sufficiently motivated (López, Alarcón, Rodríguez & Casado, 2014). This situation is accentuated in abstract subjects like algebra or calculus, because the students considered them to be uninteresting or even useless for their training. But the lack of motivation is not related only to the contents (Lee & Hammer, 2011), the goal of many students is to get a degree that allows them to enter the labor market (Smith-Robbins, 2011). For this reason, they do not adequately value the assimilation of knowledge, or the development of skills and competencies that they will need to develop their work. This situation also discourages teachers, leading to a vicious cycle that worsens the learning process and it is not easy to break. The most visible consequences of this situation are the low grades and the high dropout rate of these students. This communication presents a gamification experience (Deterding, Dixon, Khaled & Nacke, 2011) aimed at engineering students, with the objective of motivating them to improve their academic performance from the acquisition of knowledge, skills and abilities. This increase in the motivation of the students should involve, among other things, an increase of the attendance ratio and an improvement of their grades (Biggs, 2003).

2. Gamification at the University

Despite the increase of gamification as a trend in several areas, there are few examples of its usage under the scope of higher education. In Prieto Martín, Díaz Martín, Montserrat Sanz and Reyes (2014) a combination between gamification, flipped learning (Bergmann & Sams, 2012) and just-in-time-teaching achieved its goal of improving the performance of Biology students. In González González and Mora Carreño (2015), the authors present a gamification experience involving 100 students of one Computer Engineering's subject. This experience increased the attendance ratio and the motivation of the students, but it was applied only to a small part of the subject's agenda. In O'Donovan, Gain and Marais (2013), the authors transformed some of the subject's tasks into puzzles in order to engage the students to solve open problems and to trey alternative and original solutions. Similar experiences conducted with MSc students (Barata, Gama, Jorge & Gonçalves, 2013a/b) have been useful to increase their participation. These experiences have been also useful to discover some downsides of gamification, as the feeling of unfairness generated when rewarding quantity over quality, or the complains about the competitive approach of the course (instead of a cooperative one). The delivery of badges has been also used to incentivize high school students to participate in afterschool activities (Davis, 2015).

3. Description of the experience

The gamification experience revolves around the merit. A merit is an abstraction that recognizes that a student has demonstrated a certain level of competence in a particular skill. Each merit has an associated icon, a score (not all are equally important or difficult to obtain), a name and a description that identify and make them unique. For example, Figure 1 shows the merit "Jedi Master" associated with the subject "Information and Security", which is awarded to the students who pass the whole subject at their first try. Another example is the merit “good programmer”, related with the laboratory practices of the same subject, which is awarded to the students who deliver code that is not only functional, but well-structured and documented.

Figure 1.  Detail of one of the merits from the subject “Information and Security”

Merits can be defined in advance and presented to the students at the beginning of the course, so that they know what to do to get them and can plan to do so, or they may be secret (only its existence is announced, but not how to obtain them) to motivate students who meet Bartle's "explorer" profile (Bartle, 1996), allowing them to play inferring or guessing what to do to obtain them. Note that the gamification experience also offers motivational elements for the other profiles: the "killers" will want to beat their rivals and reach the top of the leaderboard; the "achievers" will want to get all the achievements, especially those that are difficult to obtain and are more rare or unique because few students obtain them; Finally, the gamification experience provides the "socializers" with an excuse to join the community or to discuss how to achieve this or that merit.

The students enter more deeply into the gamification experience as they obtain merits. To represent this progression, we have designed a three-level scale. The first level, called "chairwarmer" identify the students who have not obtained any merit, the second level, "one of many" identifies students who have achieved between one and five merit, finally, from the obtention of the fifth merit, the students reach the "future engineer" level. From the fifth merit obtained, it is considered that students have already entered enough the gamification experience and will continue participating although they cannot increase their level. In addition, the fact that the maximum level is relatively affordable for everyone provides a target to the segment of students who does not considers itself able to reach the first places of the leaderboard. All these students can target reaching the third level to differentiate themselves from those colleagues who does not put the slightest interest. Note that this gamification experience uses all the main gamification mechanics (collections, points, leaderboards, levels and feedback) identified in Cortizo Pérez, Carrero García, Monsalve Piqueras, Velasco Collado, Díaz del Dedo and Pérez Martín (2011).

The tool which drives the gamification experience is a web platform called TOP ENGINYERIA (http://top.uab.cat/). Through this platform, teachers can create and give merits to the students, and students can check the level achieved by each partner by consulting the merits' leaderboard. In this leaderboard, sorting is based on the scores of each student's merits, therefore, students are ordered according to their skills and abilities. It has a short feedback cycle, as may be updated whenever a new merit is given. This allows certain immediacy between performing an action (for example, deliver some optional exercises) and obtaining a reward (in this case, get the merit "practice makes perfection").

Figure 2.  Top 6 positions of the leaderboard, filtered by subject

Figure 2 shows a screenshot of this leaderboard. This leaderboard is tied to a reward system that rewards students based on their position at a particular time of year. The rewards tend to take the form of extra-curricular activities that expand the training of students, but they are not evaluable. For example, a tour to the servers of the faculty with the "Networks" professor, only for the first 25 students, or forwarding a job or paid internship offer to the first 15. This reward system provides an element of extrinsic motivation for that segment of students who does not obtain enough intrinsic motivation from competition and prestige for reaching and keeping the top of the leaderboard. However, it is important to note that it is not the central element of the experience, but only an additional incentive to that placed near the top. The experience is designed so obtaining these rewards should not be perceived as the students' ultimate goal (Deci & Ryan, 1985).

The last element of the gamification web platform is the student profile. The profile consists of a web page that displays a visual summary of the student's participation in the gamification experience. The profile shows principally the position at the leaderboard, a list of all the merits obtained, and the level it has reached. On one hand, this profile is useful for every student to know exactly what is its state in the gamification experience, how many merits needs to level up, which merits has already obtained and the value that they have, how is the student positioned in comparison with his friends, etc. Furthermore, students have the option of making their profile public, so that anyone can see it. This option allows them to share it on social networks in order to obtain recognition, or, for example, use it to support their resume when they want to enter the labour market, or simply to request a grant or apply to a business' practices. In Figure 3, we provide an example of a student's profile.

Figure 3. Example of the personal profile of a student

4. Assessing the impact of a pilot test

We have been conducting a pilot test during year 2016, involving 15/16's second semester and 16/17's first one. In this test, 950. students enrolled in eleven subjects of the Degree in Computer Engineering at the Autonomous University of Barcelona (UAB) may participate. All these students are part of the experience as they appear in the leaderboards and get merits, but are not required to participate nor receive any penalty for not doing so. At the time of writing this communication we do not have enough indicators to determine if the pilot has reached its goal. However, in order to assess the reception that this initiative is having among students participating, we have carried out two activities:

• •a qualitative analysis of the learning experience through surveys with the students and interviews with the teachers;

• •a quantitative analysis of the interaction statistics between the students and the web platform.

A total of 656 surveys were completed, the students participating in the pilot project have answered three questions in a five-Likert scale. The three questions referred to the student's level of participation in the gamification experience, the perceived usefulness and if they enjoyed participating on it. As in Bravo, Amante, Simo, Enache and Fernandez (2011), students have also expressed their opinion about a negative and a positive aspect of the experience via two open questions.

In all subjects, students are divided into two profiles based on their responses:

• •Favorable students, responding to all three Likert scale questions with values greater than or equal to 3 (neutral element) and mention the word "motivation" at the open question of the positive section; and

• •Unfavorable students responding to all three questions with values less than or equal to 3 and have mentioned "lack of time" or "no rewards" at the negative open question.

From the eight subjects surveyed, in four of them the number of favorable and unfavorable students has been very similar (with acceptance rates of between 42% and 54%). At the remaining four, the number of unfavorable students (around 60%) has doubled the favorable ones (around 30%).

Figure 4.  Amount of authentications at the internal area of TOP ENGINYERIA by day

Regarding the traffic generated on the web platform, Figure 4 shows the number of authentications per day taken during 15/16 second semester. As you can clearly see, there are a few peaks of interest during the first days that relate to the presentation of the platform to the students, and the first assignments of merits from the subjects with more students, then there is a valley that corresponds to the period of Easter holidays, and later retrieves similar values to those of the first half of the semester if we ignore the peaks produced the introduction of the experience. Needless to say, according to the usual pattern in the School of Engineering, after a holiday period, the same values of previous attendance and delivery of activities are never recovered, therefore, we consider a success that the amount of accesses is quite similar, because this indicates that students have not forgotten or ignored the gamification experience.

Note that an important part of the web platform is open to the public without having to authenticate (any visitor can see the merits and the first 50 positions in the leaderboards), so part of the received visit are not reflected in this chart. Regarding the most visited pages of the web platform, it is surprising that the most visited one is the personal profile, with 30% more visits than the internal leaderboard (which lists all students).

In addition, the fact that the public leaderboard has almost as many visits as the internal one indicates that the actual number of visits per day is approximately twice as shown in Figure 4. There are 322 of the 905 students that have accessed the web platform to view their profile, their merits or their position at the leaderboard. From these, only 27 have done so once. This means that almost 30% of students were interested enough to authenticate two or more times. There are eight students who were interested enough to access more than once a day (at the date of writing of this communication, this means 90 or more accesses), and there is a student that have accessed 185 times in 90 days. This data show that there is a part of the students, despite being very small, that feels really involved with the experience.

Figure 5.  The percentage of active students decreases linearly to their average grades

Figure 5 shows the percentage of active students (we consider a student as active if it has logged in at least two times) based on their average grades. It is quite remarkable to see how the participation of students who get better grades is very high, over 75% among students with average grade greater than or equal to 8. The participation of students with grades above 6 is also very positive (around 50%), the participation drops to 25% among students with a grade average around 5, and among those with a grade lesser than 4, participation is completely marginal. Assuming that it is impossible to motivate everyone, and that those with the worst grades are probably the students who attend less, spend fewer hours or take their studies a less serious, to provide an element of motivation that makes them do more (or make them better) to more than half the students averaging higher than 6 can also be considered a success.

Finally, it is worthy to note that some of the merits has not had among the students the kind of reception that was expected at the moment of their design, and this has ultimately resulted in the introduction of small changes in the designing process by teachers.

For example, a merit for the subject "Information and Security" which was awarded to the quickest students who realized an optional task, had generated a discontent among a group of students who wanted to have "all the merits" and felt that it was unfair to not qualify for that particular due to "schedule issues". As a result, the teachers of the subject changed the design of the merit to grant it to all students who complete the optional task well, and have eliminated similar time constraints from other designed merits.

Hidden merits, for their part, have not generated much interest in general, but in the cases where the teacher has provided a small hint during a lecture have managed to have an impact. In the subject "Advanced Internet Technologies" for the first time in four years, the web forum of the course was active during the Easter holidays as a result of a hidden merit awarded to the students that posted interesting interactions realized about a topic that was mentioned in the last lectures.

Maybe this kind of merit might be redesigned in the coming years to increase its impact among students. Likewise, since many students have mentioned the word "competitiveness" when referring to the gamification experience, it might be a good idea to design collaborative merits that require pairs or groups of students to work together to achieve them.

5. Conclusions

In order to improve the experience, we already have made some changes during this last months, i.e. the inclusion of a leaderboard filter by subject. This measure has been in great demand by students who are enrolled to less subjects than their classmates, and felt unable to access to the top of the leaderboard. In addition, as one of the more criticized things has been the lack of rewards, we will study the possibility of getting sponsors who are willing to give gifts to the winners, as well as the option to give extra points to the first students of every subject leaderboard, in order to give some extra incentives to the students who want to participate.

Besides, in order to try to reach as many students as possible, we will involve some subjects from the first semester, in order to produce an experience that lasts throughout the whole academic course. Overall, the participating teachers considered the experience as positive. Therefore, this experience will continue during the course 2016/17 with the eleven subjects of the Degree in Computer Engineering at the Autonomous University of Barcelona (UAB) that have participated this year, and it will be extended with the inclusion some other subjects. This way, we will continue to improve the experience and to study its impact on the student's performance.

References

Barata, G., Gama, S., Jorge, J,  & Gonçalves, D. (2013a). So Fun It Hurts - Gamifying an Engineering Course. Proceedings of the Foundations of Augmented Cognition: 7th International Conference. Las Vegas, NV, USA. https://doi.org/10.1007/978-3-642-39454-6_68

Barata, G., Gama, S., Jorge, J., & Gonçalves, D. (2013b). Engaging Engineering Students with Gamification. Proceedings of the 5th International Conference on Games and Virtual Worlds for Serious Applications (VS-GAMES), Poole, 2013, 1-8. https://doi.org/10.1109/VS-GAMES.2013.6624228

Bartle, R. (1996). Hearts, Clubs, Diamonds, Spades: Players who suit MUDs. The Journal of Virtual Environments, 1(1). Retrieved from: http://www.mud.co.uk/richard/hcds.htm

Bergmann, J., & Sams, A. (2012). Flip your classroom: Reach every student in every class every day. San Diego, CA: International Society for Technology in Education.

Biggs, J. (2003). Teaching for Quality Learning at University (2nd ed.). Milton Keynes, United Kingdom: Open University Press. ISBN: 0335242758.

Bravo, E., Amante, B., Simo, P., Enache, M., & Fernandez, V. (2011). Video as a new teaching tool to increase student motivation. IEEE Global Engineering Education Conference (EDUCON), Amman, 638-642. https://doi.org/10.1109/EDUCON.2011.5773205

Cortizo Pérez, J.C., Carrero García, F.M., Monsalve Piqueras, B., Velasco Collado, A., Díaz del Dedo, L.I., & Pérez Martín, J. (2011). Gamificación y docencia: lo que la universidad tiene que aprender de los videojuegos. Actas de las VIII Jornadas Internacionales de Innovación Universitaria. Madrid. Retrieved from: http://hdl.handle.net/11268/1750

Davis, K. (2015). Digital badges in afterschool learning: Documenting the perspectives and experiences of students and educators. Computers & Education, 88, 72-83. https://doi.org/10.1016/j.compedu.2015.04.011

Deci, E., & Ryan, R.M. (1985). Intrinsic motivation and self-determination in human behaviour. New York: Plenum.

Deterding, S., Dixon, D., Khaled, R., & Nacke, L. (2011). From game design elements to gamefulness: Defining gamification. Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments, Tampere Finland. ACM, 9-15. https://doi.org/10.1145/2181037.2181040

González González, C.S., & Mora Carreño, A. (2015). Técnicas de gamificación aplicadas en la docencia de Ingeniería Informática. Revista de Investigación en Docencia Universitaria de la Informática, 8(1). ISSN 1989-1199

Lee, J.J., & Hammer, J. (2011). Gamification in Education: What, How, Why Bother?. Academic Exchange Quarterly, 15(2), 1-5.

López Fernández, D., Alarcón Cavero, P.P., Rodríguez Sánchez, M., & Casado Fuente, M.L. (2014). Motivación en estudiantes de ingeniería: Un caso de estudio con teorías e instrumentos para su medida y desarrollo. REDU. Revista De Docencia Universitaria, 12(4), 343-376. Retrieved from: http://red-u.net/redu/index.php/REDU/article/view/637

O'Donovan, S., Gain, J., & Marais, P. (2013). A Case Study in the Gamification of a University-level Games Development Course. Proceedings of the South African Institute for Computer Scientists and Information Technologists Conference (SAICSIT). South Africa. https://doi.org/10.1145/2513456.2513469

Prieto Martín, A., Díaz Martín, D., Montserrat Sanz, J., & Reyes, E. (2014). Experiencias de aplicación de estrategias de gamificación a entornos de aprendizaje universitario. Revista de Investigación en Docencia Universitaria de la Informática, 7(2).  ISSN 1989-1199

Smith-Robbins, S. (2011). “This Game Sucks”: How to Improve the Gamification of Education. EDUCAUSE Review, 467(1), 58-59. Retrieved from: http://er.educause.edu/

---