Journal of Technology and Science Education

• Home
• About
• Login
• Archives
• Submissions
• Publication Fee
• Indexing & Statistics

Home > Vol 12, No 1 (2022) > Parra-González

ELABORATION AND VALIDATION OF THE SCALE TO MEASURE
THE EXPERIENCE ON GAMIFICATION IN EDUCATION (EGAMEDU)

1Department Research Methods and Diagnosis in Education, University of Granada (Spain)
2Department Didactics and School Organization, University of Granada (Spain)

Received October 2021

Accepted December 2021

Abstract

Nowadays, we talk about the use of gamification in education, an active methodology that consists of the use of mechanics, design or game structures in class. When this type of methodology is used, the effort is rewarded, and it is used as a motivating tool in class. However, there is no valid or well-structured instrument to measure gamification properly in education. Aim of this study is to develop and validate an instrument to measure the experience of gamification in educational contexts (EGAMEDU) as a valuable tool of diagnosis for the teaching staff to guide their teaching practices to include this methodology. A sample of 401 participants is used for the validation of the questionnaire related to education and gamified experiences. The results demonstrate good validity indices and a factorial structure according to the one proposed in the theory.

 

Keywords – Validation, Educational innovation, Gamification, EGAMEDU.

To cite this article:

Parra-González, M.E., Segura-Robles, A., Moreno-Guerrero, A.J., & López-Belmonte, J. (2022). Elaboration and validation of the scale to measure the experience on gamification in education (EGAMEDU). Journal of Technology and Science Education, 12(1), 217-229. https://doi.org/10.3926/jotse.1505

 

----------

1. 1. 1. Introduction

The massive incursion of Information and Communication Technology (ICT) in all spheres of our lives has led to a radical shift in how we produce, consume, and distribute information and knowledge (Hainey, Westera, Connolly, Boyle, Baxter, Beeby et al., 2013).

As a result, it seems that more and more teachers are moving away from traditional teaching methods, as there has been a proliferation of active teaching methods. The so-called active methodologies are those that mainly require the student to be active to build their learning (Berenguer, 2016; De Vargas, 2006; Gayá, 2016; López, 2015; Moreno-Guerrero, Soler-Costa, Marín-Marín & López-Belmonte, 2021). Thus, the student is encouraged to work to achieve the objective he pursues (Pintrich & Schunk, 2006), and emphasis is given on the student rather than the teacher during learning. In this context, learning focuses on how and what the student learns, with the teacher as a guide or companion to this learning, who is there to assist the student if necessary (Cruz-Miguel, Rodriguez-Resendiz, Garcia-Martinez, Camarillo-Gomez & Perez-Soto, 2019; Garduno-Aparicio, Rodriguez-Resendiz, Macias-Bobadilla & Thenozhi, 2018; Gorrostieta, Vargas-Soto, Zuñiga-Aviles, Rodríguez-Resendiz, Tovar-Arriaga, 2015; Gutiérrez, Reséndiz,  Santibáñez, Bobadilla, 2015).

One of the active methodologies on the rise is gamification (López-Belmonte, Parra-González, Segura-Robles & Pozo-Sánchez, , 2020), which consists of using elements, designs or structures of games in non-playful contexts (Bruder, 2015; Deterding, Dixon, Khaled & Nacke, 2011; Deterding, Sicart, Nacke, O’Hara & Dixon, 2011).

On the other hand, students are better involved in their learning when they are motivated, and this is where gamification becomes a good methodological tool to motivate students in the teaching-learning process (Banfield & Wilkerson, 2014; Kapp, 2012; Zichermann & Cunningham, 2011).

According to Ortiz-Colón, Jordán and Agreda (2018), the main thing is that involving and motivating students with game-based learning materials promotes learning with a solid social component that produces a very beneficial methodology. On the other hand, we must be careful when implementing this type of methodologies in the classroom, since their continued and repetitive use can cause adverse effects on the engagement and motivation of students. This type of tool, therefore, must be adapted over time to maintain a high engagement to the task (Lavoue, Monterrat, Desmarais & George, 2019).

Thus, the most recent research (Quintero, Jiménez & Area, 2018; Segura-Robles, Fuentes-Cabrera, Parra‑Gonzále & López-Belmonte, 2020) shows that the use of gamification improves motivation and cooperative learning, and even students become more involved and actively participate in the activities. By improving interest and motivation, in addition to increasing motivation, it also reduces dropout rates (Area‑Moreira & González, 2015), which is an issue of concern today. The term gamification is being used increasingly, although sometimes it is used indiscriminately. The study by Robson outlines the definition of gamification and proposes an initial framework based on key psychological theories, including the theory of self-determination and intrinsic and extrinsic motivation (Robson, Plangger, Kietzmann, McCarthy & Pitt, 2016). It is argued that this method is not new since games and elements from them have been used in teaching and learning for years, especially to develop imagination (Piaget, 1962; Malone, 1980; Malone & Lepper, 1987).

The difference between the use of the game as a finished and concrete product (Foncubierta & Rodríguez, 2016) through which didactic content may be presented and what gamification is itself, based on didactic content that combines game elements and design must be noted (Seaborn & Fels, 2015). The active motivation of students should be kept by creating rankings and awarding medals based on effort rather than achievement, as in traditional methodology. Zichermann and Cunningham (2011) highlight the use of medals and the establishment of rankings as two of the mechanics of the use of gamification in education.

One of the leisure activities is playing video games, and in education, it is fashionable to use digital games and video games to acquire knowledge (Contreras, 2016). Since its use provides a number of benefits, it is also used in companies or marketing to improve user experiences and commitment to the activity (Korn & Schmidt, 2015; Simpson & Clem, 2008). As defended by Deterding, Sicart et al. (2011), gamification is a term that includes the use of elements of video games to improve the user experience and commitment to non-recreational services and applications. According to these authors, gamification can be applied in areas other than educational ones, such as business, ecology, and marketing. Given their high social component, these games allow you to play again later, check the rankings, and even socialize (Marín‑Marín, Campos-Soto, Navas-Parejo & Gómez-García, 2022).

As students are active in the learning process, they also have to learn from their interaction with others, and through realistic learning, students will be able to transfer the knowledge acquired (Bernal & Martínez, 2009). In this way, two fundamental characteristics of learning are promoted, such as the sociability of learning, which explains that social interaction and interactivity are necessary for learning to take place, since through technological tools, skills are promoted and expanded, called learning communities, which promotes distance learning. Despite the benefits of gamification as stated above, the literature also contains evidence on the alteration of the intrinsic motivation of students when conducting training activities through gamification (Hinojo-Lucena, Gómez-García, Marín-Marín & Romero‑Rodríguez, 2021). This motivation can be diminished, which can affect learning outcomes (Lavoué et al., 2019). In recent years, gamification is gaining great projection in the educational field. All of this is justified in the innumerable training experiences that have been conducted under this teaching methodology and also by the results achieved in each of them, as reflected in the literature. Likewise, gamification is experiencing integration with other teaching models and techniques such as flipped learning and project-based learning, among the most prominent. This reflects a formative evolution with the purpose of adapting the instructive actions to the new teaching and learning models of a digital era, in which the student is the main protagonist (Pozo‑Sánchez, López-Belmonte, Fuentes-Cabrera & López‑Núñez, 2020).

Palacios and Medrano (2007) highlight the importance of creating or adapting already validated scales for gamified experiences to other contexts since there were no scales for measuring gamification in education in Spanish. This work presents a tool built with dual purposes, on the one hand, to offer a diagnostic tool for work skills and decision making, and on the other, to guide teachers in making methodological decisions in teaching-learning processes by choosing gamification as a methodology. (Ayén, 2017; Erenli, 2013; Kapp, 2012; Lee & Hammer, 2011; Pintrich & Schunk, 2006; Quintero et al., 2018; Wang, 2015; Zichermann & Cunningham, 2011). In this study, we constructed and validated a scale to measure gamification in education (EGAMEDU).Several studies have made use of the GAMEX scale. Among these studies is that of Anguas-Gracia, Subiron-Valera, Anton-Solanas, Rodríguez-Roca, Satustegui-Dorda and Urcola-Pardo (2021) in which they analyzed an escape room with nursing students in the Faculty of Health Sciences. The results indicated that gamification is a valid tool for the acquisition of professional competences in higher education. Another study is that developed by López-Belmonte, Segura-Robles, Fuentes-Cabrera and Parra-González (2020), which analyzed the application of an escape room with students enrolled in the Master’s Degree in Teacher Training program. During the study, a group that developed an expository pedagogical experience was compared to an experimental group that developed a gamified pedagogical experience. The students in the experimental group obtained better results than the students in the control group. Parra-González, Segura-Robles and Gómez-Barajas (2020) developed another study in which the gamification experience of 255 students and teachers in Andalusia in the area of Physical Education was analyzed. The results indicated that gamification facilitates the development of creative thinking, a reduction in negative affect and an improvement in student collaboration. Márquez-Hernández, Garrido-Molina, Gutiérrez-Puertas, García-Viola, Aguilera-Manrique and Granados-Gámez (2019) developed another study in which the GAMEX scale was adapted and validated for nursing students. According to the research, the adapted version showed good reliability and validity.

2. Methodology

An instrumental design has been used to conduct this research (Parra-González, López-Belmonte, Segura‑Robles & Moreno-Guerrero, 2021). Instrumental designs are used mainly to analyze the psychometric properties of instruments in order to validate them (Ato, López-García & Benavente, 2013).

Although there are instruments for assessing specific experiences, such as flipped learning (Fuentes‑Cabrera, López-Belmonte, Parra-González & Morales-Cevallos, 2020), the tools to assess the effects of gamified experiences focus on other areas, such as the GAMEX scale (Gameful Experience in Gamification) or scale of gamified experience developed and validated in English by Eppmann, Bekk and Klein (2018) in the field of marketing. GAMEX is an actual and validated tool developed to measure gameful experiences. This makes it necessary to adapt and propose a new concrete instrument for the educational field.

The proposed scale is composed of 8 main dimensions that measure participants’ experiences in gamification. The first six dimensions are adapted from the GAMEX scale (fun, attention, creative thinking, activation, absence of negative effect and control), with different modifications in its items for adaptation to the purely educational environment. The last two dimensions are proposed based on the literature (Socialization and learning). Gamification is being used as a learning tool in different areas and subjects and for the development of collaborative attitudes and behaviors and autonomous study (Caponneto, Earp & Ott, 2014; Ratnawati, Sukamto, Ruja & Wahyuningtyas, 2020). This collaborative work produces, almost inevitably, an increase in the socialization of the subjects that make up the work teams (Montaner-Marco, 2019). Thus, socialization stands out as one of the fundamental axes in scientific studies on gamification (De Sousa-Borges, Durelli, Reis & Isotani, 2014). Furthermore, the literature about gamification has shown its potential in learning improvements (Kirillov, Vinichenko, Melnichuk, Melnichuk & Vinogradova, 2016). According to other studies, students who took lecture sessions that were gamified had a more positive perception of their learning performance than those who took lectures without gamification (Morillas-Barrio, Munoz-Organero & Sanchez Soriano, 2016). For all this reasons, it is necessary to add two specific dimensions that value these aspects of the gamified process.

In short, with the construction of this scale, it is possible to cover all the elements that must be measured when using this methodology. In principle, what researchers want to know and measure through the dimensions are the following:

• •Fun: This dimension measures the degree to which the user had fun with the activity and liked the gamification project or experience. 

• •Attention: In this dimension, we want to know how much attention the user considers they have given to the game and other activities outside the gamified activity. 

• •Creative thinking: The degree of imagination, creativity or experimentation that the user claims to have had or practiced in the development of the experience is measured. 

• •Activation: The degree of activity, emotion, or nervousness that the user considers having experienced during the gamification experience is measured. 

• •Absence of negative effect: The gamification process is also examined to determine whether or to what degree users have felt frustration, annoyance, or stress. 

• •Control: In this dimension, we wanted to know the degree of autonomy with respect to the rest or the self-confidence that the user developed or has during the gamified experience. 

• •Socialization: Within this dimension, we were most interested in finding out about the degree of socialization among participants, how they interacted with each other, and how they worked together during the gamification process. 

• •Learning: Furthermore, in this dimension, we wanted to know how the user learns through this methodology, such as if they have learned new things, if they have practiced, or even if they have invested effort in the process. The data collection process followed previous studies (Segura‑Robles, Moreno-Guerrero, Parra-González & López-Belmonte, 2021), in which an online platform (Limesurvey®) was used to facilitate access to and the creation of a data matrix. This allows us to access participants more quickly, for example, contact them through social networks or using email. For the subsequent collection, the same platform generated an ordered data matrix that can be loaded into any statistical analysis software. A proprietary equation modeling tool AMOS, version 23, and the SPSS tool, version 24, were used for this analysis. The items corresponding to the dimension “absence of negative effect” must be rotated before they can be analyzed since they are written in a negative manner, so they must be interpreted in reverse. All the participants who have collaborated in the development of this research are, in one way or another, related to the university’s educational environment. Likewise, all the questionnaires analyzed indicated that respondents had or participated in gamified experiences during the past year.71% of the participants in the study are university students with different degrees, while 29% of them are teachers (as main preview degree). The total number of participants who completed the questionnaire was 415, although, after an in-depth review and various previous analyses, 14 of them were discarded for different reasons (lack of a completed item, questionnaire with fictitious values, etc.). As a result, 401 participants were included in the final sample. 

3. Results

3.1. Expert Validation

In developing the definition of what is meant to be measured, generated, or selected or specified the formats of the elements and the scoring system, expert judgment is essential (Berk, 1990; Delgado-Rico, Carretero-Dios & Ruch, 2012).

To validate the questionnaire, a protocol based on the questionnaire was prepared and sent to five expert doctors who have conducted research on educational phenomena and active methodologies. In this protocol (Figure 1), the objectives of the research and the information collection technique used were explained. The questionnaire itself was also attached.

Experts were asked to analyze each item’s relevance and adequacy, as recommended by Hernandez-Nieto and Pulido (2014); recognizing the relevance of the item when it allows the researcher to collect information relevant to the study objective and when it is focused on the respondents. For this, they received a scale from 1 to 5 (1 being the minimum relevance or adequacy and 5 the maximum). In the event that the valuation of an item was low (1 or 2), they could propose its modification, deletion or substitution; additionally, a space was also provided for a recommendation. All the instructions were explained in the protocol that was sent to the experts.

After analyzing the responses of the experts, the mean assessment of the items obtained a high mean (4.4) score, as well as a low standard deviation (0.70) in adequacy and relevance. Furthermore, these experts revealed a favorable and coinciding opinion (Fleiss’s Kappa = 0.85; Kendall’s W = 0.82). In addition, these specialists offered observations to improve the design, presentation and writing of certain items. Among these observations, some experts doubted the relevance of the control dimension in the final questionnaire.

Figure 1. Fragment of the expert assessment scale used

3.2. Exploratory Study

Initially, the Bartlett Sphericity Test will be conducted to determine whether the subscales are interdependent. Contrastingly, the KMO (Kaiser-Meyer Olkin Measure of Sampling Adequacy) criterion is used to determine the sample’s sufficiency (Kaiser, 1974).

The KMO measure of sampling adequacy was 0.915, and the significance in the Bartlett test was less than 0.01, confirming that this type of analysis can be performed (Alias, Ismail & Sahiddan, 2015; Hair, Black, Babin & Anderson, 2014).

Next, an exploratory factor analysis was conducted with varimax rotation. Factor analysis is commonly used in the fields of psychology and education (Hogarty, Hines, Kromrey, Ferron & Mumford, 2005), and is considered a reliable analysis method in self-filling tests (Thompson, 2005). Factor analysis is a multivariate statistical procedure that has multiple uses.

The results are presented in the form of a matrix of rotated components, and are shown in Table 1. Following the recommendation of Tabachnick and Fidell (2007), we used the restrictive value of 0.40 in the load factors as a cut-off point to rethink or permanently delete the item.

	Component
	1	2	3	4	5	6
Fun
Item 1			0.786
Item 2			0.860
Item 3			0.821
Item 4			0.568			0.418
Attention (ATTE)
Item 5					0.815
Item 6					0.828
Item 7					0.760
Creativity (CREA)
Item 8						0.737
Item 9						0.779
Item 10						0.753
Negative Effect Absence (NEA)
Item 11				0.848
Item 12				0.861
Item 13				0.870
Item 14				0.571
Socialization (SOC)
Item 15		0.725
Item 16		0.802
Item 17		0.803
Item 18		0.761
Learning (LEAR)
Item 19	0.729
Item 20	0.746
Item 21	0.793
Item 22	0.728
Item 23	0.736
Control (CON)
Item 24		0.256			0.233
Item 25	0.303				0.313
Activation (ACT)
Item 25	0.186
Item 26			0.201
Item 27	0.303				0.313
Item 28			0.313			0.213

Table 1. Obtained components and load factors.

Following the results obtained and after consulting with the experts again, it was decided to eliminate the Control component (items 24 and 25), as factor loadings that do not exceed the value 0.30 cannot be considered relevant or sufficient (Lloret-Segura, Ferreres-Traver, Hernández-Baeza & Tomás-Marco, 2014). It was also not possible to place it clearly in any previous factor. The same was true for the activation factor, where the items can be considered unreliable. After a new round of consultation with the expert committee, their elimination was considered for the analyses and the final scale.

3.3. Validation Study

The validity and consistency of the questionnaire were evaluated in a second phase once an agreement was achieved between the expert judgment and the statistical results. The KMO test and Bartlett’s sphericity test were used again. The KMO measure of sampling adequacy was 0.911, and the significance in the Bartlett test was less than 0.001. Once the initially proposed structure was confirmed, the confirmatory factor analysis (CFA) was used to check its dimensionality.

Using the classic literature indicators, the values obtained in the evaluation of the fit of the model indicated a good fit in all of them. The Chi-square value was χ2 = 303.17; gl = 201; p = .003. As this value is very sensitive to the sample size and does not have a clear interpretation (Kline, 2005), it should be combined with other indices of goodness of fit of the model that are less susceptible to sample size (Byrne, 2012), such as the index of normalized fit (NFI), Tucker Lewis index (TLI), comparative fit index (CFI), fit increment index (IFI), and root mean square error of approximation (RMSEA).

Table 2 illustrates the values obtained for the model, in comparison with those proposed by the scientific literature, showing values consistent with a model with an excellent fit (Cangur & Ercan, 2015). To determine whether the relationships between the variables of the estimated model in the analyses adequately reflect the relationships collected or observed in the data matrix, adjustment values are shown. (Weston & Gore, 2006).

Likewise, different analyses were conducted to know the validity of the model. In this case, Cronbach’s alpha (α) is calculated, a classic statistic in the scientific literature (Heo, Kim & Faith, 2015). This statistic must be complemented with other indicators since studies have shown its weakness when used in isolation are different (Dunn, Baguley & Brunsden, 2013). Composite Reliability (CR), Average Variance Extracted (AVE), and Maximum Shared Variance (MSV), with acceptable values exceeding 0.7 for HR and 0.50 for AVE (Hair & Gómez-Suárez, 2010), while MSV must show values lower than AVE (Henseler, Ringle & Sarstedt, 2014). Results show reliable reliability indices, obtaining a score of 0.925 for the total of the scale. Likewise, no convergent validity problems AVE> 0.5 or discriminant validity, MSV> AVE are observed (Table 3).

	CMIN/DF	NFI	NNFI /TLI	CFI	IFI	RMSEA
Proposed model	1.419	0.921	0.971	0.975	0.975	0.043
Literature	Between 1-3	>=0.90	>=0.90	>=0.90	>=0.90	=<0.08

Table 2. Goodness of fit indices for the proposed model and cuts marked by the literature.

	α	CR	AVE	MSV	Apr	Div	AEN	So	Crea	Aten
LEAR	0.907	0.904	0.653	0.520	0.808
FUN	0.895	0.888	0.670	0.370	0.578*	0.819
NEA	0.894	0.832	0.567	0.087	0.267*	0.264*	0.753
SOC	0.924	0.926	0.758	0.520	0.721*	0.609*	0.295*	0.871
CREA	0.888	0.896	0.742	0.459	0.677*	0.569*	0.234*	0.653*	0.862
ATTE	0.899	0.858	0.669	0.357	0.583*	0.486*	0.072	0.421*	0.597*	0.818

Table 3. Reliability and validity indexes.

After demonstrating the statistical strength of the questionnaire, different comparisons of means were made in relation to two of the variables collected. No significant differences (p values> 0.05) were observed in regards to gender (Table 4) during the development and measuring of the different factors with the questionnaire.

The scores obtained by teachers and students for each of the dimensions of the questionnaire were analyzed (Table 5). According to the data, the dimensions analyzed do not show any significant differences, as with the previous comparison obtaining a p value > 0.05 for all of them.

	Gender	M	F	t	p
Learning	Woman	21.02	1.966	-0.290	0.162
	Man	20.87
Fun	Woman	17.46	2.531	0.178	0.113
	Man	17.53
Negative Effect Absence	Woman	16.57	0.375	-0.335	0.541
	Man	16.43
Socialization	Woman	17.14	0.232	-0.054	0.631
	Man	16.94
Creativity	Woman	12.46	0.037	-0.470	0.847
	Man	12.44
Attention	Woman	11.18	0.256	-1.212	0.613

Table 4. Student’s t test results according to the gender of the participants

	Type	M	F	t	p
Learning	Teacher	18.02	0.956	-0.390	0.122
	Student	20.27
Fun	Teacher	16.47	1.521	0.294	0.013
	Student	18.56
Negative Effect Absence	Teacher	17.87	1.393	-0.441	0.631
	Student	17.93
Socialization	Teacher	16.18	0.297	-0.046	0.081
	Student	15.34
Creativity	Teacher	16.47	0.237	-0.698	0.840
	Student	16.44
Attention	Teacher	13.15	0.863	-2.312	0.130

Table 5. Student’s t test results according to the type of participants

4. Discussion and Conclusions

Validation and adaptation of the different scales available in the scientific literature based on predetermined objectives always yields interesting results for scientific knowledge itself, either by proving its efficacy or by rejecting it. Often, researchers are unable to access the different designs and instruments associated with other areas or fields when these types of studies are ignored (Palacios & Medrano, 2007).

The main objective of this study was the creation, adaptation, and validation of a measurement instrument, specifically a scale, which was capable of measuring the results around different dimensions obtained after being immersed in some gamified educational experience. It was necessary to adapt and validate the scale to a more educational setting, as it is the base scale on which it works reliably. Both dimensions and items of analysis are based on the original scale of the marketing field, and both dimensions created for EGAMEDU are based on a review of the literature and the gamified experiences identified through the information search process, analyzing what variables should be known and measured when we mention gamification. The construction of the scale allows us to measure all the elements that must be considered when using this methodology, so that learning can take place and benefits can be obtained by the user, which in the case of education are students or teachers who are trained in the methodology, if appropriate (Ayén, 2017; Erenli, 2013; Kapp, 2012; Lee & Hammer, 2011; Piaget, 1962; Pintrich & Schunk, 2006; Quintero et al., 2018; Wang, 2015; Zichermann & Cunningham, 2011). At first, there were eight dimensions; however, after the validation and reliability analyses, six dimensions remained that allowed us to measure what we wanted to know and measure through this scale.

The reliability indices and goodness of fit indices of EGAMEDU show that it is a useful instrument for collecting information on gamified experiences. In this case, the increase in items to form two new dimensions typical of current educational processes (learning and socialization) and the elimination of the control and activity dimension has caused, at least within the purely educational field, an increase in the originally proposed goodness-of-fit values and indices.

5. Limitations, Future Work, and Practical Implications

A major limitation of this study is the selection of the sample since it is primarily confined to a university environment, so other educational levels are not included. Another type of complementary statistical analysis can be carried out to verify the validity of the construct, as well as extending the sample to other educational levels and geographical contexts in the future. This research has different implications of a practical nature. This scale could be used as a diagnostic instrument to analyze the levels of learning, fun, absence of negative effect, socialization and interaction, creativity, and attention to student learning with respect to gamification and other aspects that are crucial for effective learning to occur. On the other hand, it is also useful to guide teachers on how to conduct educational practice by implementing the use of gamification as a methodology within the teaching-learning processes with the results obtained. EGAMEDU can also analyze the different dimensions with the use of gamification or any other type of methodology used with the students and compare the results found as a diagnostic tool and as a guide. All this will serve the selection and decision-making of the methodological process of teachers.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

No funding was received for this research.

References

Alias, R., Ismail, M., & Sahiddan, N. (2015). A Measurement Model for Leadership Skills Using Confirmatory Factor Analysis (CFA). Procedia - Social And Behavioral Sciences, 172, 717-724.
https://doi.org/10.1016/j.sbspro.2015.01.424

Anguas-Gracia, A., Subiron-Valera, A.B., Anton-Solanas, I., Rodríguez-Roca, B., Satustegui-Dorda, P.J., & Urcola-Pardo, F. (2021). An evaluation of undergraduate student nurses’ gameful experience while playing an escape room game as part of a community health nursing course. Nurse Education Today, 103, 1-7. https://doi.org/10.1016/j.nedt.2021.104948

Area-Moreira, M., & González, C. (2015). De la enseñanza con libros de texto al aprendizaje en espacios online gamificados. Educatio Siglo XXI, 33(3), 15. https://doi.org/10.6018/j/240791

Ato, M., López-García, J., & Benavente, A. (2013). Un sistema de clasificación de los diseños de investigación en psicología. Anales De Psicología, 29(3). https://doi.org/10.6018/analesps.29.3.178511

Ayén, F. (2017). ¿Qué es la gamificación y el ABJ? Iber: Didáctica de las ciencias sociales, geografía e historia, 86, 7‑15.

Banfield, J., & Wilkerson, B. (2014). Increasing student intrinsic motivation and self-efficacy through gamification pedagogy. Contemporary Issues in Education Research, 7(4), 291-298. https://doi.org/10.19030/cier.v7i4.8843

Berenguer, C. (2016). Acerca de la utilidad del aula invertida o flipped classroom. In Tortosa, M., Grau, S.. & Álvarez, J. (Eds.), XIV Jornadas de redes de investigación en docencia universitaria. Investigación, innovación y enseñanza universitaria: enfoques pluridisciplinares (1466- 1480). Alicante, España: Universitat d’Alacant.

Berk, R. (1990). Importance of Expert Judgment in Content-Related Validity Evidence. Western Journal Of Nursing Research, 12(5), 659-671. https://doi.org/10.1177/019394599001200507

Bernal, M.C., & Martínez, M.S. (2009). Metodologías activas para la enseñanza y el aprendizaje. Revista de Pedagogía Saberes y quehaceres del pedagogo, 14, 101-106.

Bruder, P. (2015). Game on: gamification in the classroom. Education Digest, 80(7), 50-56.

Byrne, B.M. (2012). Structural Equation Modeling with Mplus: basic concepts, applications, and programming. New York: Taylor & Francis Group. https://doi.org/10.4324/9780203807644

Cangur, S., & Ercan, I. (2015). Comparison of Model Fit Indices Used in Structural Equation Modeling Under Multivariate Normality. Journal Of Modern Applied Statistical Methods, 14(1), 152-167.
https://doi.org/10.22237/jmasm/1430453580

Caponneto, I., Earp, J., & Ott, M. (2014). Gamification and education: a literature review [Conference presentation]. European conference on games-based learning 8. Berlín, University of Applied Sciences.

Contreras, R. (2016). Digital games and gamification applied to education. RIED. Revista Iberoamericana de Educación a Distancia, 19(2), 27-33. https://doi.org/10.5944/ried.19.2.16143

Cruz-Miguel, E.E., Rodriguez-Resendiz, J., Garcia-Martinez, J.R., Camarillo-Gomez, K.A., & Perez-Soto, G.I. (2019). Field-programmable gate array-based laboratory oriented to control theory courses. Computer Applications In Engineering Education, 27(5), 1253-1266. https://doi.org/10.1002/cae.22149

De Sousa-Borges, S., Durelli, V.H.S., Reis, H.M., & Isotani, S. (2014). A systematic mapping on gamification applied to education. In Proceedings of the 29th Annual ACM Symposium on Applied Computing (216-222). https://doi.org/10.1145/2554850.2554956

De Vargas, E. (2006). La situación de enseñanza y aprendizaje como sistema de actividad: el alumno, el espacio de interacción y el profesor. Revista Iberoamericana de Educación, 39(4).

Delgado-Rico, E., Carretero-Dios, H., & Ruch, W. (2012). Content validity evidence in test development: An applied perspective. International Journal of Clinical and Health Psychology, 12(3), 449-459.

Deterding, S., Dixon, D., Khaled, R., & Nacke, L. (2011). From game design elements to gamefulness: defining gamification. In Proceedings of the 15th International Academic MindTrek Conference (9-15). https://doi.org/10.1145/2181037.2181040

Deterding, S., Sicart, M., Nacke, L., O’Hara, K., & Dixon, D. (2011). Gamification: using game-design elements in non-gaming contexts. In Proceedings of the 2011 Annual Conference of Human Factors in Computing Systems (2425-2428). https://doi.org/10.1145/1979742.1979575

Dunn, T., Baguley, T., & Brunsden, V. (2013). From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation. British Journal of Psychology, 105(3), 399-412. https://doi.org/10.1111/bjop.12046

Eppmann, R., Bekk, M., & Klein, K. (2018). Gameful Experience in Gamification: Construction and Validation of a Gameful Experience Scale [GAMEX]. Journal of Interactive Marketing, 43, 98-115. http://doi.org/10.1016/j.intmar.2018.03.002

Erenli, K. (2013). The impact of gamification: Recommending education scenarios. International Journal of Emerging Technologies in Learning, 8(1), 15-21. https://doi.org/10.3991/ijet.v8iS1.2320

Foncubierta, J.M., & Rodríguez, C. (2016). Didáctica de la gamificación en la clase de español. Madrid: Edinumen.

Fuentes-Cabrera, A., López-Belmonte, J., Parra-González, M.E., & Morales-Cevallos, M.B. (2020). Diseño, validación y aplicación de un cuestionario para medir la influencia de factores exógenos sobre la eficacia del aprendizaje invertido. Psychology, Society, & Education, 12(1), 1-16. https://doi.org/10.25115/psye.v0i0.2334

Garduno-Aparicio, M., Rodriguez-Resendiz, J., Macias-Bobadilla, G., & Thenozhi, S. (2018). A Multidisciplinary Industrial Robot Approach for Teaching Mechatronics-Related Courses. IEEE Transactions on Education, 61(1), 55-62. https://doi.org/10.1109/TE.2017.2741446

Gayá, V. (2016). Enseñar a pensar: Las nuevas metodologías colocan al alumno como parte activa de su propio aprendizaje. El siglo de Europa, 1177, 36-37.

Gorrostieta, E., Vargas-Soto, E., Zuñiga-Aviles, L.A., Rodríguez-Resendiz, J., Tovar-Arriaga, S. (2015). Mechatronics methodology: 15 years of experience. Ingenieria e Investigacion, 35(3), 107-350. https://doi.org/10.15446/ing.investig.v35n3.47543

Gutiérrez, C.A.G., Reséndiz, J.R., Santibáñez, J.D.M., Bobadilla, G.M. (2014). A model and 352 simulation of a _ve-degree-of-freedom robotic arm for mechatronic courses. IEEE Latin America 353 Transactions, 12(2), 78-86. http://doi.org/10.1109/TLA.2014.6749521

Hainey, T., Westera, W., Connolly, T., Boyle, L., Baxter, G., Beeby, R. et al. (2013). Students’ attitudes toward playing games and using games in education: Comparing Scotland and the Netherlands. Computers & Education, 69, 474-484. https://doi.org/10.1016/j.compedu.2013.07.023

Hair, J., & Gómez-Suárez, M. (2010). Análisis multivariante. Madrid: Prentice Hall.

Hair, J., Black, W., Babin, B., & Anderson, R. (2014). Multivariate data analysis. NJ: Prentice Hall.

Henseler, J., Ringle, C., & Sarstedt, M. (2014). A new criterion for assessing discriminant validity in variance-based structural equation modeling. Journal Of The Academy Of Marketing Science, 43(1), 115-135. https://doi.org/10.1007/s11747-014-0403-8

Heo, M., Kim, N., & Faith, M. (2015). Statistical power as a function of Cronbach alpha of instrument questionnaire items. BMC Medical Research Methodology, 15(1), 1-9. https://doi.org/10.1186/s12874-015-0070-6

Hernandez-Nieto, R., & Pulido, J. (2014). Instrumentos de Recoleccion de Datos: Validez y Confiabilidad. Editorial Academica Espanola.

Hinojo-Lucena, F.J., Gómez-García, G., Marín-Marín, J.A., & Romero-Rodríguez, J.M. (2021). Gamificación por insignias para la igualdad y equidad de género en Educación Superior. Prisma Social, 35, 184-198.

Hogarty, K., Hines, C., Kromrey, J., Ferron, J., & Mumford, K. (2005). The Quality of Factor Solutions in Exploratory Factor Analysis: The Influence of Sample Size, Communality, and Overdetermination. Educational And Psychological Measurement, 65(2), 202-226. https://doi.org/10.1177/0013164404267287

Kaiser, H. (1974). An index of factorial simplicity. Psychometrika, 39(1), 31-36. https://doi.org/10.1007/bf02291575

Kapp, K.M. (2012). The Gamification of learning and Instruction. San Francisco: John Wiley.

Kirillov, A.V., Vinichenko, M.V., Melnichuk, A.V., Melnichuk, Y.A., & Vinogradova, M.V. (2016). Improvement in the Learning Environment through Gamification of the Educational Process. International Electronic Journal of Mathematics Education, 11(7), 2071-2085.

Kline, R.B. (2005). Methodology in the social sciences. Principles and practice of structural equation modeling. New York: Guilford Press.

Korn, O., & Schmidt A. (2015). Gamification of Business Processes: Re-designing Work in Production and Service Industry. Procedia Manufacturing, 3(1), 3424-3431. https://doi.org/10.1016/j.promfg.2015.07.616

Lavoué, E., Monterrat, B., Desmarais, M., & George, S. (2019) Adaptive Gamification for Learning Environments. IEEE Transactions on Learning Technologies, 12(1), 16-28. https://doi.org/10.1109/TLT.2018.2823710

Lee, J.J., & Hammer, J. (2011). Gamification in Education: What, How, Why Bother. Academic Exchange Quaterly, 15(2).

Lloret-Segura, S., Ferreres-Traver, A., Hernández-Baeza, A., & Tomás-Marco, I. (2014). El análisis factorial exploratorio de los ítems: una guía práctica, revisada y actualizada. Anales De Psicología, 30(3), 1151-1169. https://doi.org/10.6018/analesps.30.3.199361

López-Belmonte, J., Parra-González, M.E., Segura-Robles, A., & Pozo-Sánchez, S. (2020). Scientific Mapping of Gamification in Web of Science. European Journal of Investigation in Health, Psychology and Education, 10, 832-847. https://doi.org/10.3390/ejihpe10030060

López-Belmonte, J., Segura-Robles, A., Fuentes-Cabrera, A., & Parra-González, M.E. (2020). Evaluating Activation and Absence of Negative Effect: Gamification and Escape Rooms for Learning. International Journal of Environmental Research and Public Health, 17(7), 1-12. https://doi.org/10.3390/ijerph17072224

López, J. (2015). Experiencia docente mediante la metodología de Aprendizaje Basado en Problemas. Ikastorratza, e-Revista de didáctica, 14, 72-85.

Malone, T.W., & Lepper, M.R. (1987). Making learning fun: A taxonomy of intrinsic motivations for learning. Aptitude, learning, and instruction, 3, 223-253.

Malone, T.W. (1980). What makes things fun to learn? Heuristics for designing instructional computer games. In Proceedings of the 3rd ACM SIGSMALL Symposium and the First SIGPC Symposium on Small Systems (162-169). https://doi.org/10.1145/800088.802839

Marín-Marín, J.A., Campos-Soto, M.N., Navas-Parejo, M.R., & Gómez-García, G. (2022). Gamification as a Motivational and Socio-educational Resource in Classrooms with Students at Risk of Social Exclusion. In: De la Prieta F., Gennari, R., Temperini, M., Di Mascio, T., Vittorini, P., Kubincova, Z. et al. (Eds.), Methodologies and Intelligent Systems for Technology Enhanced Learning. 11th International Conference. MIS4TEL 2021. Lecture Notes in Networks and Systems, 326. Cham: Springer. https://doi.org/10.1007/978-3-030-86618-1_19

Márquez-Hernández, V.V., Garrido-Molina, J.M., Gutiérrez-Puertas, L., García-Viola, A., Aguilera‑Manrique, G., & Granados-Gámez, G. (2019). How to measure gamification experiences in nursing? Adaptation and validation of the Gameful Experience Scale. Nurse Education Today, 81, 34-38. https://doi.org/10.1016/j.nedt.2019.07.005

Montaner-Marco, J. (2019). Gamification Strategies as Socialization Tools in the Context of Pediatric Hospitalization. Extended Abstracts of the Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts (35-39). https://doi.org/10.1145/3341215.3356328

Moreno-Guerrero, A., Soler-Costa, R., Marín-Marín, J., & López-Belmonte, J. (2021). Flipped learning and good teaching practices in secondary education. Comunicar, 68, 107-117.
https://doi.org/10.3916/C68-2021-09

Morillas-Barrio, C., Munoz-Organero, M., & Sanchez Soriano, J. (2016). Can Gamification Improve the Benefits of Student Response Systems in Learning? An Experimental Study. IEEE Transactions on Emerging Topics in Computing, 4(3), 429-438. https://doi.org/10.1109/TETC.2015.2497459

Ortiz-Colón, A., Jordán, J., & Agreda, M. (2018). Gamificación en educación: una panorámica sobre el estado de la cuestión. Educaçao E Pesquisa, 44, 1-17. https://doi.org/10.1590/s1678-4634201844173773

Palacios, S., & Medrano, M. (2007). Elaboración y validación de la escala de dominios de valores televisivos (EDVT). Revista De Investigación Educativa, 25(2), 403-420.

Parra-González, M.E., López-Belmonte, J., Segura-Robles, A., & Moreno-Guerrero, A.J. (2021). Spanish Adaptation and Validation of the Teaching and Learning Experiences Questionnaire. International Journal of Environmental Research and Public Health, 18, 3518. https://doi.org/10.3390/ijerph18073518

Parra-González, M.E., Segura-Robles, A., & Gómez-Barajas, E.R. (2020). Assessing Gamified Experiences in Physical Education Teachers and Students. IJERI. International Journal of Educational Research and Innovation, (13), 166-176. https://doi.org/10.46661/ijeri.4595

Piaget, J. (1962). Play, dreams and imitation in childhood. New York: W.W. Norton & Co.

Pintrich, P.R., & Schunk, D.H. (2006). Motivación en contextos educativos. Teoría, investigación y aplicaciones. Madrid: Pearson Prentice Hall.

Pozo-Sánchez, S., López-Belmonte, J., Fuentes-Cabrera, A., & López-Núñez, J.A. (2020). Gamification as a Methodological Complement to Flipped Learning–An Incident Factor in Learning Improvement. Multimodal Technologies and Interaction, 4(2), 1-13. https://doi.org/10.3390/mti4020012

Quintero, L., Jiménez, F., & Area, M. (2018). Más allá del libro de texto. La gamificación mediada con TIC como alternativa de innovación en Educación Física. Retos: nuevas tendencias en educación física, deporte y recreación, 34, 343-348. https://doi.org/10.47197/retos.v0i34.65514

Ratnawati, N., Sukamto, S., Ruja, I., & Wahyuningtyas, N. (2020). “Defense of the Ancients”, Gamification in Learning: Improvement of Student’s Social Skills. International Journal of Emerging Technologies in Learning (IJET), 15(7), 132-140. https://doi.org/10.3991/ijet.v15i07.13221

Robson, K., Plangger, K., Kietzmann, J., McCarthy, I., & Pitt, L. (2016). Game on: Engaging customers and employees through gamification. Business Horizons, 59(1), 29-36. https://doi.org/10.1016/j.bushor.2015.08.002

Seaborn, K., & Fels, D. (2015). Gamification in theory and action: A survey. International Journal of Human‑Computer Studies, 74, 14-31. https://doi.org/10.1016/j.ijhcs.2014.09.006

Segura-Robles, A., Fuentes-Cabrera, A., Parra-González, M.E., & López-Belmonte, J. (2020). Effects on Personal Factors Through Flipped Learning and Gamification as Combined Methodologies in Secondary Education. Frontiers in Psychology, 11, 1-8. https://doi.org/10.3389/fpsyg.2020.01103

Segura-Robles, A., Moreno-Guerrero, A.J., Parra-González, M.E., & López-Belmonte, J. (2021). Adaptation and Validation of the Motivated Strategies for Learning Questionnaire for Spanish Adolescents. European Journal of Investigation in Health, Psychology and Education, 11, 142-153.
https://doi.org/10.3390/ejihpe11010012

Simpson, E., & Clem, F. (2008). Video Games in the Middle School Classroom. Middle School Journal, 39(4), 4-11. https://doi.org/10.1080/00940771.2008.11461638

Tabachnick, B.G., & Fidell, L.S. (2007). Using multivariate statistics (5th ed.). Boston, MA: Pearson Education.

Thompson, B. (2005). Exploratory and confirmatory factor analysis. Washington, DC: American Psychological Association.

Wang, A.I. (2015). The wear out effect of a game-based student response system. Computers & Education, 82, 217-227. https://doi.org/10.1016/j.compedu.2014.11.004

Weston, R., & Gore Jr., P.A., (2006). A Brief Guide to Structural Equation Modeling. The Counseling Psychologist, 34(5), 719-751. https://doi.org/10.1177/0011000006286345

Zichermann, G., & Cunningham, C. (2011). Gamification by Design: Implementing Game Mechanics in Web and Mobile Apps. Cambridge: O’Reilly Media.

---