Journal of Technology and Science Education

• Home
• About
• Login
• Archives
• Submissions
• Publication Fee
• Indexing & Statistics

Home > Vol 13, No 1 (2023) > Bienvenido-Huertas

ANALYSIS OF THE EFFECTIVENESS OF USING KAHOOT! IN UNIVERSITY DEGREES IN BUILDING ENGINEERING

1Department of Building Construction, University of Granada (Spain)
2Department of Building Construction II, University of Seville (Spain)

Received October 2022

Accepted December 2022

Abstract

Gamification is increasingly used for better content assimilation. However, it is hardly used in theoretical and technical courses related to building and architecture. This study focuses on assessing the use of gamification with Kahoot! to arouse students’ interest and to ease content assimilation. Experimentation was undertaken during the 2020/2021 and 2021/2022 academic years. All students enrolled in the course participated in experimentation: 67 and 65 students, respectively. Gamification sessions took place in the middle and at the end of each academic year (the week before partial exams). Students were also surveyed to assess their degree of satisfaction with gamification sessions. The results showed that gamification dynamics improved students’ performance in exams, obtaining a lower percentage of failures and increasing higher marks. Cluster analysis results also showed that students’ performance in Kahoot! is not a clear indicative of their performance in exams, so it is useful as a self-assessment tool. Moreover, most students positively assessed gamification sessions for better content assimilation. This study stresses the implementation potential of Kahoot! in teaching dynamics in university degrees in architecture and building engineering.

 

Keywords – Kahoot!, University education, Architecture and building, Performance, Student opinion.

To cite this article:

Bienvenido-Huertas, D., Rubio-Bellido, C., & León-Muñoz, M.Á. (2023). Analysis of the effectiveness of using Kahoot! in university degrees in building engineering. Journal of Technology and Science Education, 13(1), 288-300. https://doi.org/10.3926/jotse.1984

 

----------

1. 1. 1. Introduction

Games have always been used as an entertainment source in constant evolution (Sailer, Hense, Mayr & Mandl, 2017) due to a combination of several factors, including concentration, creativity, socialization, and reasoning (Dias, 2017; Reeves & Read, 2009; Yee, 2006). There are factors not related to fun that contribute to the fact that games are an entertainment source. Hence, games could be used for various purposes, not just for entertainment. As a result, the concept of gamification, i.e., the use of games in various contexts, was created (Deterding, Dixon, Khaled & Nacke, 2011). These techniques could be used in different scopes, such as commercial (Zichermann & Cunningham, 2011) and educational scopes (Kapp, 2012; Villagrasa, Fonseca, Redondo & Duran, 2014).

The use of games in education aimed to remove the barriers between students and teachers. Master class is among the most used dynamics in classroom (Finkel, 2000). This model is widely used in all educational levels, including university education (mainly in theoretical courses). However, this model has many detractors because of the lack of communication between teachers and students (Weimer, 2002). Traditional teaching could become monotonous and be lack of student participation. Previous studies have shown that low student motivation implies both worse exam performance and an uncomfortable environment in classroom (Baker, D’Mello, Rodrigo & Graesser, 2010; Liu, Bridgeman & Adler, 2012). However, students actively participating usually learn more and have better performance (Butler, 1992; Murray, 1991; Williams & Williams, 2011).

New technologies today give the possibility of changing education from several perspectives (Hinojo‑Lucena, Aznar-Díaz, Cáceres-Reche, Trujillo-Torres & Romero-Rodríguez, 2020). Current students, highly digitalized, increasingly demand new technologies in class. In this regard, teacher-student interaction could be improved by using new technologies in class through game competitiveness (Wang, 2015). Consequently, gamification approaches in teaching aim to increase student participation and commitment (de Marcos, Garcia-Cabot & Garcia-Lopez, 2017; Hamari, Koivisto & Sarsa, 2014).

However, gamification tools not just improve student commitment, but are useful to ease knowledge transfer from teachers to students. University course content could be complex, thus implying that students loss their interest and affecting their academic performance. Gamification technologies could improve the learning process in these disciplines. Both teachers’ combined and interdisciplinary work and the development of specific software is crucial to improve student participation (Priyaadharshini, NathaMayil, Dakshina, Sandhya & Bettina-Shirley, 2020). Games should be adapted to course content. For this purpose, defining educational goals to be met by games could imply greater student participation and strengthen the knowledge acquired in theoretical sessions (Moncada & Moncada, 2014). Increasing difficulty in games could therefore lead to greater student participation and interest as challenge and overcoming are included (Priyaadharshini et al., 2020). As a result, these techniques could imply greater attendance to classes (del Cerro, 2015) and that games are not considered as evaluation tests, but as an additional dynamics (Fernández-Mesa, Olmos-Peñuela & Alegre, 2016).

Research studies on the advantages of gamification in teaching are recently increasing, showing its huge potential to improve the learning process in university education (Daubenfeld & Zenker, 2015; Day-Black, 2015; Giannetto, Chao & Fontana, 2013; Guenaga, Arranz, Florido, Aguilar, de Guinea, Rayón et al., 2013; Jordine, Liang & Ihler, 2014; Llorens-Largo, Gallego-Durán, Villagrá-Arnedo, Compañ-Rosique, Satorre‑Cuerda & Molina-Carmona, 2016; Mathrani, Christian & Ponder-Sutton, 2016; Pertegal-Felices, Jimeno-Morenilla, Sánchez-Romero & Mora-Mora, 2020). Likewise, their great potential in both other education levels (Prieto, Palma, Tobías & León, 2019) and online teaching (Toma, Diaconu & Popescu, 2021) have been shown. Most of these studies were conducted with university students of humanities and computer science.

In architecture teaching there are various gamification approaches according to the thematic block: gamified strategies in teachings on urban design are based on city creation (Fonseca, Navarro, Villagrasa, Villagrasa, Villagrasa, Valls et al., 2017), whereas in teachings on architectural projects games are based on model creation (Ulargui-Agurruza, de Miguel-García, Montenegro-Mateos & Mosquera-González, 2018). However, few studies focus on gamification techniques in more theoretical thematic blocks in university degrees in architecture, thus implying a lack of student motivation in university degrees in architecture, specifically considering its interdisciplinary character.

This study therefore suggests including gamification in the course Building Systems I of the university degree in building engineering of the University of Seville during two academic years (2020/2021 and 2021/2022). Gamification was based on a question-and-answer game, together with content explanations to strengthen the tested knowledge. The Kahoot! application was used. Moreover, student’s degree of satisfaction was assessed through surveys. This experience aims to show the possibility of including these techniques in more complex theoretical and less creative teachings in university degrees in architecture and building engineering.

2. Methodology

2.1. Case Study

Gamification sessions were implemented in the course Building Systems I of the university degree in building engineering of the University of Seville. This university degree was first implemented in 2009/2010 and is divided into four academic years, with 60 ECTS each.

The course Building Systems I has the greatest failure rate. In quantified data, performance data have recently shown a dropout rate of more than 20%, and failure rates of 40% of students attending exams (Rubio-Bellido, León-Muñoz, Canivell, Martínez-Rocamora & Bienvenido-Huertas, 2021).

The course Building Systems I focuses on most systems in residential buildings, developing all theoretical, technical, and normative content of the following systems: water supply, domestic hot water (DHW), evacuation plumbing, electricity, telecommunications, ventilation, acclimatization, gas, protection against fires, and lifts. Sections were divided into theoretical and practical classes. The difficulty level of the course is expected to be high, so students have difficulties in assimilating content successfully. This aspect has been already discussed in other studies related to courses of the area. Bienvenido-Huertas (2020a,b) focused on the difficulties in energy audit courses. These studies were based on internationalization of postgraduate courses in building engineering; however, their results showed the difficulty level of technical courses and the need for establishing measures to solve these barriers.

New approaches were therefore included in the course to avoid a low student performance rate. Including gamification dynamics could improve this aspect. The experimentation was undertaken during the 2020/2021 and 2021/2022 academic years. All students enrolled in the course participated in the experimentation: 67 and 65 students, respectively.

2.2. Implementation of Gamification Sessions with Kahoot!

Gamification took place with Kahoot!, a free use application to create interactive question-answer games (Ismail & Mohammad, 2017; Rodríguez-Fernández, 2017). This tool has recently become important. Its success is based on its operation dynamics, like the TV-game Who wants to be millionaire? There are questions with different possible answers that should be answered in a certain time. This aspect, combined with a nice and coloring interface and the constant update of the participant ranking with fun messages, has contributed to its use in class. Students need an electronic device with Internet access to participate (Zhang & Yu, 2021). Researchers have recently analyzed the advantages of its use in class as this tool improves student participation and performance, among other aspects (Bicen & Kocakoyun, 2018; Dolezal, Posekany, Motschnig & Pucher, 2018; Kay & LeSage, 2009; Plump & LaRosa, 2017; Wang & Lieberoth, 2016).

Gamification sessions took place in the middle and at the end of each academic year, i.e., the week before partial exams. The course included two exams: (i) E1, which evaluated student content knowledge on water supply installations, DHW, and evacuation plumbing; and (ii) E2, which evaluated student content knowledge on electricity, telecommunications, HVAC, and the remaining systems. Games were made up of 10 questions, and a time between 120 and 180 s was applied to each answer according to difficulty. After knowing students’ answers, the teacher explained the correct one briefly.

The methodology was based on students’ satisfaction and motivation, as well as on their academic performance improvement, as in other studies (Fuster-Guilló, Pertegal-Felices, Jimeno-Morenilla, Azorín‑López, Rico-Soliveres & Restrepo-Calle, 2019). Firstly, student performance both in Kahoot! sessions and exams were analyzed in detailed. Dividing content into thematic blocks of building systems provided the individual performance in each issue. It is worth stressing that students were grouped with the k-means algorithm for data analysis (Hartigan & Wong, 1979). The silhouette index was used as indicative of group quality (Kaufman & Rousseeuw, 1990). Secondly, students were surveyed to assess their degree of satisfaction with gamification sessions. These surveys were anonymous and voluntary and were conducted at the end of the gamification sessions. To make it easier for students, the survey included few questions (Table 1). Questions were based on Kahoot! usefulness assessment (Questions 1 and 2), organization improvement (Question 3), error origin (Question 4), and a free-answer question (Question 5). Questions 1 and 2 included answers based on the Likert scale, whereas the other included specific answers for each question.

Question		Answer
Question 1	Do you think Kahoot! sessions have been useful to understand certain aspects better?	A1	Strongly disagree
		A2	Disagree
		A3	Neutral
		A4	Agree
		A5	Strongly agree
Question 2	Do you think Kahoot! sessions have been useful for exam preparation?	A1	Strongly disagree
		A2	Disagree
		A3	Neutral
		A4	Agree
		A5	Strongly agree
Question 3	When do you consider Kahoot! sessions should be conducted?	A1	Some days before the exam
		A2	At the end of each lesson or thematic area
		A3	The week before the exam
		A4	Two weeks before the exam
		A5	Through the virtual campus to answer when I am free
Question 4	You consider that the errors in Kahoot! sessions were based on:	A1	The time to answer was short
		A2	Questions were difficult
		A3	Problems with Internet connection
		A4	I did not understand the content of the question
		A5	Letters were two small
Question 5	Regarding the usefulness of Kahoot! sessions, choose the options you consider as appropriate:	A1	I would recommend them for future students
		A2	They will be useful to pass the exam
		A3	I would recommend them for courses like Building Systems I
		A4	I think they are a waste of time
		A5	I think they waste time instead of explaining course content

Table 1. Format of the student survey conducted through Google Forms

3. Results and Discussion

3.1. Student Performance with Gamification

Firstly, student performance was assessed through the gamification sessions performed during the 2020/2021 and 2021/2022 academic years. This analysis focused on their performance during both Kahoot! sessions and exams. As mentioned in Section 2., Kahoot! sessions were conducted twice during the course, adjusting the content to the exam that would be taken a week later. Different student performance was obtained in gamification sessions. Fig 1 shows the percentage of correct answers. The analysis was performed by dividing the answers into thematic blocks. Results were different. In E1, student performance was low in most thematic blocks, with distributions that presented values in Q3 of up to 70% of correct answers and with medians equal or lower than 50% in most thematic blocks. Null performance of some students was also found. In E2, similar results were obtained, with values in Q2 between 30 and 50%. Moreover, distributions with values in Q1 of 0% of correct answers were obtained. These results showed low student performance some days before the exam, an expected aspect based on the bad results obtained in the exams in previous academic years. Student performance assessment during Kahoot! sessions was complemented with the average time spent to answer (Figure 2). They took long time to answer questions. The average time in E1 was 93 s in the 2020/2021 academic year, and 76 s in the 2021/2022 academic year, whereas the average time in E2 was 110 and 55 s in the 2020/2021 and 2021/2022 academic years, respectively. Students spent more than 50% of the average time to answer. However, most students answered before the time run out, so these time performance records could be appropriate for the posterior exam performance.

Figure 1. Percentage of correct answers in each thematic block in Kahoot! sessions

Figure 2. Average value and typical deviation in the time taken by students to answer in Kahoot!

Performance results therefore proved that students did not answer Kahoot! issues correctly, although the time taken to answer each question was often appropriate. As these sessions aimed to improving both content knowledge and exam performance, the next step consisted in comparing exam results with Kahoot! performance. Thematic blocks were also divided (Figure 3). Student exam performance significantly improved. In E1, increase values related to correct answers were obtained in the course distributions as follows: (i) in water supply installations, increase values of 24.5% were obtained in Q1, 32% in Q2, and 49.5% in Q3, with maximum values of 78%; (ii) in DHW installation, increase values of 0% were obtained in Q1, 25% in Q2, and 50% in Q3, with maximum values of 75%; and (iii) in evacuation plumbing, increase values of 9% was obtained in Q1, 33% in Q2 and, 44% in Q3, with maximum values of 66%. Similar improvements were obtained in E2: (i) in electricity and telecommunications, increase values of 51% were obtained in Q1, 60% in Q2, and 62% in Q3, with maximum values of 80%; (ii) in HVAC systems, increase values of 14% were obtained in Q1, 31% in Q2, and 60% in Q3, with maximum values of 86%; and (iii) in the remaining systems, increase values of 20% were obtained in Q1, 41.5% in Q2, and 60% in Q3, with maximum values of 100%. Distributions values showed that the percentage of correct answers in the exam improved. Most students achieved better exam performance, compared to Kahoot! sessions. Improvement values of 0% were obtained in some cases (e.g., Q1 in the performance distribution in DHW installations) due to some students’ performance in Kahoot! sessions, so the percentage of correct answers did not vary in exams. However, this value was obtained in a small part of students; most of them improved their performance.

Figure 3. Increase percentage of correct answers in the final exam, compared to Kahoot! sessions

The improvement achieved by Kahoot! sessions was also shown by comparing the students’ result in the course assessment. The results obtained in the two academic years in which experimentation was conducted (2020/2021 and 2021/2022) were compared with those from the previous academic year (2018/2019). The results of the 2019/2020 academic year were not used as it coincided with the lockdown in Spain, so student performance was not representative of the common tendency of the course. Gamification sessions improved students’ marks: 22.5% of students failed in the 2018/2019 academic year, as against 4.5% in 2020/2021 and 0% in 2021/2022. In addition, marks also increased, particularly in upper second-class marks (between 7 and 8.9) and first-class marks (between 9 and 10). The percentage of students that obtained upper second-class marks was 8.4% in 2018/2019, as against 32.8% in 2020/2021 and 23.1% in 2021/2022. As for first-class marks, 0% of students obtained this mark in 2018/2019, as against 6% in 2020/2021 and 4.6% in 2021/2022. These results therefore showed that student performance significantly improved. Content knowledge was strengthened through gamification and provided students with a perspective of the exam. Likewise, these sessions took place in a nice and participatory environment. Nonetheless, gamification did not remove the percentage of withdrawal from the course. This percentage increased in comparison with the 2018/2019 academic year. The reason was based on the date when gamification took place, i.e., sessions took place in the middle and at the end of course, thus implying that some students withdrew from the course the first teaching weeks. This aspect should be assessed in the next academic years by implementing gamification sessions more often in class.

 

Figure 4. Comparison of the percentage of students according to the marks obtained
in the academic year before the gamification experience (2018/2019) with that obtained
in the experimentation academic years (2020/2021 and 2021/2022)

Finally, student course performance was analyzed through the clusters obtained according to students’ profile in Kahoot! sessions. As indicated in Section 2.2., a cluster analysis was performed to assess the similarity among students according to their performance (points in Kahoot, average answer time, and percentage of correct answers). A total of 3 different clusters were obtained. The silhouette index was always between 0 and 1 (Table 2), an indicative of cluster quality. Cluster 1 corresponded to students with both varied performance in Kahoot! sessions and slower answers, and Cluster 2 corresponded to students with quicker answers. Cluster 3 corresponded to students with better marks and better answer time in Kahoot!. Assessing clusters’ marks was useful to find that distributions were coincident among clusters (Figure 5). Thus, even in Cluster 3 (students with the best performance in Kahoot!), marks were like those of the students from the other two clusters. Student performance in Kahoot! was not a clear indicative of their exam performance. Students with low marks in Kahoot! obtained high marks in exams and very similar to that obtained by students with high points. Thus, Kahoot! sessions were a useful tool to assimilate course content better. They were also a self-assessment tool for students to pass exams.

Cluster	Silhouette index
1	0.27
2	0.32
3	0.25

Table 2. Silhouette index obtained by clusters

Figure 5. Assessment of the marks obtained in the final exams of the course
by students that belonged to the 3 clusters

3.2. Results of Students’ Assessment Survey

After analyzing student performance, their assessment was also analyzed. Anonymous and voluntary surveys were conducted through Google Forms after Kahoot! sessions. Quick surveys made up of 5 questions were configured to encourage students to complete them. Each question had five possible answers, described in Table 1. Figure 6 shows the percentage of answers given by students in each option. The first question (Kahoot! effectiveness for content knowledge) was positively assessed in the Likert scale, with agree and strongly agree assessments between 23.1 and 76.9%. Question 2 (Kahoot! effectiveness for exam preparation) obtained similar results: agree and strongly agree assessments between 15.4 and 84.6%. Thus, students positively valued Kahoot! sessions to strengthen knowledge acquired in the course and for exam preparation.

The remaining were free-answer questions. Question 3 focused on the date when Kahoot! sessions should be conducted. Answers varied, although the greatest percentages corresponded to the first answer options: (i) days before the exam, with values of 69.2% in 2020/2021 and 25% in 2021/2022; (ii) at the end of each thematic block, with values of 15.4% and 47%, respectively; and (iii) the week before the exam, with a value of 15% in both academic years. The other two options (two weeks before the exam and through the virtual campus when they are free) obtained answer percentages lower than 8%. Students therefore preferred that sessions take place as in the experimentation (days before the exam), although some students also indicated the possibility of conducting these sessions more often. This aspect could be interesting to reduce students’ dropout rate, as discussed in Figure 4.

The percentages obtained in Question 4 (error origin) varied in all answer options. Nevertheless, students chose the answers related to the short time given to answer, the difficulty of questions, and the fact of not understanding the question, whereas connection problems was chosen between 1.9 and 7% of students. In Question 5 (assessment of Kahoot! session usefulness), most students valued their effectiveness, indicating that they would recommend them for future students and that they would be useful to prepare exams. In addition, they would recommend them in similar courses. Only 2.4% of students of the 2021/2022 academic year indicated that they were a waste of time.

Figure 6. Percentage of answers given by students in assessment surveys

Thus, students positively assessed gamification sessions. To complete the analysis, the relations among the answers given by students were assessed to detect behavior patterns. Figure 7 shows the matrix of frequencies of students’ answers. This matrix shows the responses in the polytomous variables in each pair of question. There were tendencies between students’ answers, with the answers given in Questions 1, 2 and 3 clearly related among them. This aspect was also observed when assessing the contingency coefficient among the polytomous variables. Some coefficients were greater than 0.5. Correlations among Questions 1-2, 1-3, and 1-5 can be also stressed. Likewise, there was certain correlation between Questions 2-3, 2-5, and 3-4. Students’ answers for Kahoot! session assessment were expected to be related to their programming/duration. Likewise, the Kahoot! session assessment was independent of the problems related to Kahoot! performance. These results therefore showed that students’ assessment followed clear answer patterns related to the effectiveness of using gamification sessions in class. This assessment was not related to student performance as answers were not related to the errors during the sessions. Student performance in Kahoot! Sessions was low; however, students were not discouraged by these dynamics, but the opposite. Based on the results obtained in both student performance and survey assessment, using this tool to improve teaching and dynamics in highly theoretical courses in university degrees in architecture and building engineering, as in Building Systems courses, has been effective.

Figure 7. Matrix with the frequencies found in the polytomous variables of the survey. FF: fringe frequency

Figure 8. Contingency coefficient among the polytomous variables of the survey

4. Conclusions

Students could be less interested in technical courses in university degrees in architecture and building engineering, such as Building Systems, thus making content assimilation something of a challenge. This study aimed to address the possibilities of using gamification with Kahoot! to arouse students’ interest and to ease content assimilation. The course Building Systems I of the university degree in building engineering of the University of Seville was selected to assess the gamification experimentation during two academic years (2020/2021 and 2021/2022). Gamification dynamics improved both theoretical course content assimilation and exam performance. Low performance in Kahoot! sessions were compensated by greater exam performance. The percentage of failures was lower than that of previous academic years, and the number of students with higher marks increased. These marks were independent of student performance in Kahoot! sessions as the clusters obtained through the cluster analysis showed a coincidence in mark distributions. These results are like those obtained in studies where Kahoot! has been implemented in various university degrees. The satisfactory results obtained in university degrees in architecture and building engineering encourage to implement these dynamics in class more often due to the difficulty of students for assimilating content. The profile of these university degrees is interdisciplinary (technical and artistic), so content assimilation has been something of a challenge. The improvement of student performance has been useful to prove the effectiveness of these dynamics.

Students’ assessment of gamification sessions should also be stressed. Students positively assessed them, considering that they are highly useful to assimilate course content and to prepare exams. The possibility of using these tools in other courses of the university degree was also assessed. The correlation analysis showed that satisfactory answers were independent of error origins in Kahoot!. Even students that indicated that they had no time to answer or that questions were complex positively assessed gamification sessions. This aspect is consistent with the results from scientific literature as students were satisfied with dynamics that allow them to play a more active role in class.

To conclude, the results of this study have shown the great potential of using Kahoot! in teaching dynamics in university degrees in architecture and building engineering. These dynamics will imply a more active students’ role in class, better content assimilation, and more satisfactory results in exams. Nonetheless, these results have not addressed the problems of the course. The dropout rate of the course was not reduced by gamification sessions as sessions were conducted in the middle and at the end of the course, so some students withdrew from the course in the first weeks. Implementing more often Kahoot! sessions could avoid this aspect and could be related to the assessment given by some students, who suggested that Kahoot! sessions should be more often conducted.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

References

Baker, R. Sj., D’Mello, S.K., Rodrigo, M.M.T., & Graesser, A.C. (2010). Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive–affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies, 68(4), 223-241. https://doi.org/10.1016/j.ijhcs.2009.12.003

Bicen, H., & Kocakoyun, S. (2018). Perceptions of students for gamification approach: Kahoot as a case study. International Journal of Emerging Technologies in Learning, 13(2). https://doi.org/10.3991/ijet.v13i02.7467

Bienvenido-Huertas, D. (2020a). Comparative analysis of the experiences of internationalization of the teaching content of the energy audit subject in university master’s degree: years 2018/2019 and 2019/2020. In Innovación docente e investigación en ciencias, ingeniería y arquitectura (127-136). Dykinson, S.L.

Bienvenido-Huertas, D. (2020b). Evaluating the possibilities of integrating game-based learning in energy audit subjects in university master’s degrees. In Innovación docente e investigación en ciencias, ingeniería y arquitectura (137-145). Dykinson, S.L.

Butler, J.A. (1992). Use of teaching methods within the lecture format. Medical Teacher, 14(1), 11-25. https://doi.org/10.3109/01421599209044010

Daubenfeld, T., & Zenker, D. (2015). A game-based approach to an entire physical chemistry course. Journal of Chemical Education, 92(2), 269-277. https://doi.org/10.1021/ed5001697

Day-Black, C. (2015). Gamification: An Innovative Teaching-Learning Strategy for the Digital Nursing Students in a Community Health Nursing Course. ABNF Journal, 26(4).

de Marcos, L., Garcia-Cabot, A., & Garcia-Lopez, E. (2017). Towards the social gamification of e‑learning: A practical experiment. International Journal of Engineering Education, 33(1), 66-73.

del Cerro, G. (2015). Aprender jugando, resolviendo: diseñando experiencias positivas de aprendizaje. XII Jornadas Internacionales de Innovación Universitaria Educar Para Transformar: Aprendizaje Experiencial (237-244). http://hdl.handle.net/11268/4334

Deterding, S., Dixon, D., Khaled, R., & Nacke, L. (2011). From game design elements to gamefulness: defining “gamification”. Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments (9-15). https://doi.org/10.1145/2181037.2181040

Dias, J. (2017). Teaching operations research to undergraduate management students: The role of gamification. International Journal of Management Education, 15(1), 98-111. https://doi.org/10.1016/j.ijme.2017.01.002

Dolezal, D., Posekany, A., Motschnig, R., & Pucher, R. (2018). Effects of introducing a game-based student response system into a flipped, person-centered classroom on object-oriented design. International Conference on Web-Based Learning (132-139). https://doi.org/10.1007/978-3-319-96565-9_13

Fernández-Mesa, A., Olmos-Peñuela, J., & Alegre, J. (2016). Pedagogical value of a common knowledge repository for Business Management courses. @Tic. Revista D’Innovació Educativa, 0(16). https://doi.org/10.7203/attic.16.8044

Finkel, D.L. (2000). Teaching with your mouth shut. Boynton/Cook Publishers Inc.

Fonseca, D., Navarro, I., Villagrasa, S., Villagrasa, S., Villagrasa, S., Valls, F. et al. (2017). Sistemas de Visualización Gamificados para la mejora de la Motivación Intrínseca en Estudiantes de Arquitectura. IV Congreso Internacional Sobre Aprendizaje, Innovación y Competitividad (CINAIC 2017). https://doi.org/10.26754/cinaic.2017.000001_043

Fuster-Guilló, A., Pertegal-Felices, M.L., Jimeno-Morenilla, A., Azorín-López, J., Rico-Soliveres, M.L., & Restrepo-Calle, F. (2019). Evaluating Impact on Motivation and Academic Performance of a Game‑Based Learning Experience Using Kahoot. Frontiers in Psychology, 10(December), 1-8. https://doi.org/10.3389/fpsyg.2019.02843

Giannetto, D., Chao, J., & Fontana, A. (2013). Gamification in a social learning environment. Proceedings of the Informing Science and Information Technology Education Conference (195-207). https://doi.org/10.28945/1806

Guenaga, M., Arranz, S., Florido, I.R., Aguilar, E., de Guinea, A.O., Rayón, A. et al. (2013). Serious games for the development of employment oriented competences. IEEE Revista Iberoamericana de Tecnologias del Aprendizaje, 8(4), 176-183. https://doi.org/10.1109/RITA.2013.2285024

Hamari, J., Koivisto, J., & Sarsa, H. (2014). Does gamification work?--a literature review of empirical studies on gamification. 2014 47th Hawaii International Conference on System Sciences (3025-3034). https://doi.org/10.1109/HICSS.2014.377

Hartigan, J.A., & Wong, M.A. (1979). Algorithm AS 136: A k-means clustering algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics), 28(1), 100-108. https://doi.org/10.2307/2346830

Hinojo-Lucena, F.J., Aznar-Díaz, I., Cáceres-Reche, M.P., Trujillo-Torres, J.M., & Romero-Rodríguez, J. (2020). Sharenting: Internet addiction, self-control and online photos of underage children//Sharenting: Internet addiction, self-control and online photos of underage children. Comunicar, 28(64), 97-108. https://doi.org/10.3916/C64-2020-09

Ismail, M., & Mohammad, J. (2017). Kahoot: A Promising Tool for Formative Assessment in Medical Education. Education in Medicine Journal, 9(2), 19-26. https://doi.org/10.21315/eimj2017.9.2.2

Jordine, T., Liang, Y., & Ihler, E. (2014). A mobile-device based serious gaming approach for teaching and learning Java programming. 2014 IEEE Frontiers in Education Conference (FIE) Proceedings (1-5). https://doi.org/10.1109/FIE.2014.7044206

Kapp, K.M. (2012). The gamification of learning and instruction: game-based methods and strategies for training and education. John Wiley & Sons. https://doi.org/10.1145/2207270.2211316

Kaufman, L., & Rousseeuw, P.J. (1990). An introduction to cluster analysis. John Wiley and Sons.

Kay, R.H., & LeSage, A. (2009). Examining the benefits and challenges of using audience response systems: A review of the literature. Computers & Education, 53(3), 819-827. https://doi.org/10.1016/j.compedu.2009.05.001

Liu, O.L., Bridgeman, B., & Adler, R.M. (2012). Measuring learning outcomes in higher education: Motivation matters. Educational Researcher, 41(9), 352-362. https://doi.org/10.3102/0013189X12459679

Llorens-Largo, F., Gallego-Durán, F. J., Villagrá-Arnedo, C.J., Compañ-Rosique, P., Satorre-Cuerda, R., & Molina-Carmona, R. (2016). Gamification of the learning process: lessons learned. IEEE Revista Iberoamericana de Tecnologias Del Aprendizaje, 11(4), 227-234. https://doi.org/10.1109/RITA.2016.2619138

Mathrani, A., Christian, S., & Ponder-Sutton, A. (2016). PlayIT: Game based learning approach for teaching programming concepts. Journal of Educational Technology & Society, 19(2), 5-17.

Moncada, S.M., & Moncada, T.P. (2014). Gamification of learning in accounting education. Journal of Higher Education Theory and Practice, 14(3), 9.

Murray, H.G. (1991). Effective teaching behaviors in the college classroom. Higher Education: Handbook of Theory and Research, 7, 135-172.

Pertegal-Felices, M.L., Jimeno-Morenilla, A., Sánchez-Romero, J.L., & Mora-Mora, H. (2020). Comparison of the effects of the Kahoot tool on teacher training and computer engineering students for sustainable education. Sustainability (Switzerland), 12(11), 1-12. https://doi.org/10.3390/su12114778

Plump, C.M., & LaRosa, J. (2017). Using Kahoot! in the classroom to create engagement and active learning: A game-based technology solution for eLearning novices. Management Teaching Review, 2(2), 151‑158. https://doi.org/10.1177/2379298116689783

Prieto, M.C., Palma, L.O., Tobías, P.J.B., & León, F.J.M. (2019). Student assessment of the use of kahoot in the learning process of science and mathematics. Education Sciences, 9(1). https://doi.org/10.3390/educsci9010055

Priyaadharshini, M., NathaMayil, N., Dakshina, R., Sandhya, S., & Bettina-Shirley, R. (2020). Learning analytics: Game-based Learning for Programming Course in Higher Education. Procedia Computer Science, 172(2019), 468-472. https://doi.org/10.1016/j.procs.2020.05.143

Reeves, B., & Read, J.L. (2009). Total engagement: How games and virtual worlds are changing the way people work and businesses compete. Harvard Business Press.

Rodríguez-Fernández, L. (2017). Smartphones y aprendizaje: el uso de Kahoot en el aula universitaria. Revista Mediterránea de Comunicación, 8(1), 181-189. https://doi.org/10.14198/medcom2017.8.1.13

Rubio-Bellido, C., León-Muñoz, M., Canivell, J., Martínez-Rocamora, A., & Bienvenido-Huertas, D. (2021). Implementation of the subject Building Installations I during confinement period: facts and results. 5th International Conference of Educational Innovation in Building CINIE 2021 (83-84).

Sailer, M., Hense, J.U., Mayr, S.K., & Mandl, H. (2017). How gamification motivates: An experimental study of the effects of specific game design elements on psychological need satisfaction. Computers in Human Behavior, 69, 371-380. https://doi.org/10.1016/j.chb.2016.12.033

Toma, F., Diaconu, D.C., & Popescu, C.M. (2021). The use of the kahoot! learning platform as a type of formative assessment in the context of pre-university education during the covid-19 pandemic period. Education Sciences, 11(10). https://doi.org/10.3390/educsci11100649

Ulargui-Agurruza, J., de Miguel-García, S., Montenegro-Mateos, N., & Mosquera-González, J. (2018). El juego de la ciudad. Una nueva estrategia docente para Proyectos Arquitectónicos. VI Jornadas Sobre Innovación Docente En Arquitectura (111-123). https://doi.org/10.3926/jotse.395

Villagrasa, S., Fonseca, D., Redondo, E., & Duran, J. (2014). Teaching case of gamification and visual technologies for education. Journal of Cases on Information Technology (JCIT), 16(4), 38-57. https://doi.org/10.4018/jcit.2014100104

Wang, A.I. (2015). The wear out effect of a game-based student response system. Computers & Education, 82, 217-227. https://doi.org/10.1016/j.compedu.2014.11.004

Wang, A.I., & Lieberoth, A. (2016). The effect of points and audio on concentration, engagement, enjoyment, learning, motivation, and classroom dynamics using Kahoot. European Conference on Games Based Learning, 20.

Weimer, M. (2002). Learner-centered teaching: Five key changes to practice. John Wiley & Sons.

Williams, K.C., & Williams, C.C. (2011). Five key ingredients for improving student motivation. Research in Higher Education Journal, 121-123.

Yee, N. (2006). The psychology of massively multi-user online role-playing games: Motivations, emotional investment, relationships and problematic usage. In Avatars at work and play (187-207). Springer. https://doi.org/10.1007/1-4020-3898-4_9

Zhang, Q., & Yu, Z. (2021). A literature review on the influence of Kahoot! On learning outcomes, interaction, and collaboration. Education and Information Technologies, 26(4), 4507-4535. https://doi.org/10.1007/s10639-021-10459-6

Zichermann, G., & Cunningham, C. (2011). Gamification by design: Implementing game mechanics in web and mobile apps. O’Reilly Media, Inc.

---