Journal of Technology and Science Education

• Home
• About
• Login
• Archives
• Submissions
• Publication Fee
• Indexing & Statistics

Home > Vol 12, No 2 (2022) > Sánchez Ramírez

DESIGN AND VALIDATION OF AN ASSESSMENT RUBRIC
OF RELEVANT COMPETENCIES FOR EMPLOYABILITY

Universidad Internacional de La Rioja (Spain)

Received July 2021

Accepted March 2022

Abstract

This article highlights the importance of promoting relevant competencies for employability in vocational training students, while considering the demands of the globalized world. The objective was to design and validate an assessment rubric of relevant competencies for employability. Seven competencies were selected: problem-solving, teamwork, adaptive capacity, communication, creativity, leadership and decision-making. A rubric was designed in which three command levels were established: low, medium and high, along with their respective indicators. A content validation process was also used by means of expert judgement. Ten (10) expert judges were selected to carry out a quantitative and qualitative validation in three stages and the indicators were modified until Aiken’s V coefficients of ≥ 0.80 (p = 0.05) were obtained for all indicators of the established competencies. It is concluded that the rubric is valid enough to assess relevant competencies for employability in vocational training students.

 

Keywords – Assessment rubrics, Content validity, Competencies, Employability.

To cite this article:

Sánchez-Ramírez, J.M., Íñigo-Mendoza, V., Marcano, B., & Romero-García, C. (2022). Design and validation of an assessment rubric of relevant competencies for employability. Journal of Technology and Science Education, 12(2), 426-439. https://doi.org/10.3926/jotse.1397

 

----------

1. 1. 1. Introduction

The implementation of the EU Convergence process in the Spanish University System has brought about a change in teaching and has allowed models to be established which are more in line with the educational needs of students and appropriate to social demands (Alonso-Sáez & Arandia-Loroño, 2017), shifting from teaching-based models to learning-based models (Gargallo, Jiménez, Martínez, Jiménez-Beut & Pérez-Pérez, 2017). Progress must be made towards methodological proposals with education plans aimed at acquiring competencies, emphasizing those which improve employability (Olo, Correia & Rego, 2022; Mahajan, Gupta & Misra, 2022). This change in the education system is particularly significant in Higher Education and in Vocational Training, as these prepare students for the world of work. In fact, one way of facilitating the transition from education to the labour market is promoting and developing skills within education systems in order to guarantee that students obtain the competencies required in the labour market (Ministerio de Educación y Formación Profesional, 2018). Consequently, assessment systems are required for these competencies which also make it possible to direct the education process in order to meet the labour demands of society.

Based on this research, the design and validation of an assessment rubric aims to contribute to this process, which may be used by teachers who are committed to vocational training, in addition to promoting competencies for the employability of their students. The analysis of prior studies on competencies for employability and business reports on labour demands in society in the 21st century are used as a basis. Some research carried out previously in this area is set out below.

2. Frame of Reference

2.1. Competency-based Education for Employability

The Instituto Nacional de Evaluación Educativa (2013), in its report on the Programme for the International Assessment of Adult Competencies (PIAAC), assesses the cognitive and workplace skills needed for individuals to participate in society. The level of cognitive competencies, reading comprehension and writing, numeracy and mathematical problem-solving in technology-rich environments is analysed, in addition to the level of social competencies, collaboration, communication and planning. Results show that the level of said competencies has a positive influence on employability and boosts career development, therefore leading to higher salaries. Gal and Tout (2014) have found that there is a positive correlation between PISA 2009 and PIAAC 2012 scores., which supports the need to begin working on any competencies that make it easier to access the labour market in Secondary Education in order to continue in Vocational Training and University Studies.

The need to include competencies in syllabuses is reflected in education legislation, Royal Decree 1105/2014 of 26 December (BOE, 2015), which stipulates the basic curriculum for Compulsory Secondary and Higher Education, which sets out competencies as the binding element of the curriculum. This is because all parts of the curriculum and activities planned by teachers must be aimed at acquiring the aforementioned competencies. This signifies a change in the approach to the traditional teaching practice, mainly aimed at passing on knowledge which must be taken in by students, towards practices that allow students to be able to be practical, integrating knowledge to implement strategies which mean they can efficiently solve the problems faced in their day-to-day lives.

Organic Law 5/2002 (BOE, 2002), on qualifications and Vocational Training, defines professional qualification as the set of professional competencies which are important for employment and which may be acquired by means of modular education or other forms of education, as well as through work experience. In addition to acquiring the appropriate competencies of each vocational training qualification, the development of personal and social competencies, which are also called transversal competencies, must be encouraged. These facilitate students with lifelong ongoing learning, and make it easier for them to develop their careers within companies and adapt to performance with the technological resources that evolve and shape the conditions on the labour market (Campaña-Jiménez, Gallego-Arrufat & Muñoz-Leiva, 2019; Fahrenbach, 2022).

Nowadays, the learning objectives have changed, however the importance of education in knowledge must be recognised. At present, as a result of knowledge being put into practice, a series of professional capacities and skills must be developed (Silva & Mazuera, 2019; Fahrenbach, 2022). There is a difference between what students learn and what is really expected for them to know when they start a job for which they have been educated (Cotronei-Baird, 2020; Olo et al., 2022; Mahajan et al., 2022; Senan & Sulphey, 2022). The labour market calls for students to be capable of solving complex problems in which they must make decisions and come up with creative solutions and efficiently report the results, while carrying out collaborative work in which they must take on various roles that will prepare them to manage a team of workers, with adaptive capacity and who are also digitally literate. In short, students must be educated in competencies which allow them to develop self-directed learning and make it easier for them to join the labour market, overcoming the differences between the education acquired in classrooms and the real demands for competencies of job positions and which are required by employers. This is due to the fact that qualifications are nowadays becoming less valuable due to the differences between the training offered for specific skills compared to the capacities for lifelong learning (Chamorro-Premuzic & Frankiewicz, 2019; Boud, 2020).

Furthermore, it is recommended to promote education in accordance with the needs of the labour market, through vocational training and university studies, encouraging vocational studies from secondary education, making up-to-date information on the necessary skills that must be worked on available to anyone involved (OECD, 2017).

Relevant competencies for employability are any transversal competencies requiring skills that may be used in various activities regardless of the profession or job position. These involve cognitive, emotional and behavioural factors and are necessary in order to work in any professional and work environment (Zarta & Trujillo, 2020; Fernández, 2018; Navia-Osorio, 2017). Various studies highlight the following as transversal competencies for employability: communication, capacity to solve problems, teamwork, creativity, initiative and the capacity to adapt to new situations, as well as planning and organisation, and effective use of technology (Fahrenbach, 2022; Canossa, 2019; Alarcón, 2018; Assessment and Teaching of 21st Century Skills (ATCS21), 2009; EPYCE, 2018; Hassall, Joyce, Arquero & Donoso, 2003; Institute of Museum and Library Services, (n.d.); McLeish, 2002; Robles, 2012; Van Loo & Toolsema, 2005). Over 50% of the 2,000 employers surveyed by Manpower Group mentioned problem-solving, collaboration, customer service and communication as the most valuable skills (Przytuła, 2018). Human resources experts also refer to the aforementioned competencies as soft skills, which are known as interpersonal skills too. These are skills which must start to be practised in secondary education, by means of active methodologies and through experience, laying the foundations in order for these skills to continue being developed in later education, causing students to reflect on their own talents and skills and the development of the autonomous learning capacity. Employers very much favour the capacity to learn or autonomous learning and to adapt to changes above qualifications obtained through studying, bearing in mind that the rapid advance of technology is a driving force for change in the work environment. In fact, the consultancy firm TIC Cognizant has published two reports including a total of 42 jobs which do not exist yet, but will appear in the next ten years and for which no specific education can therefore be provided (Davis, 2018).

Educational institutions must be a true reflection of the competency approach taken in vocational training and by the European Higher Education Area which support the promotion of employability, seeking the best synergy possible between initial training and the labour market (Llanes, Figuera & Torrado, 2017; Fahrenbach, 2022). The demands of the labour market clearly require a paradigm shift, as students invest a great deal of time and money in higher and professional education with the main objective of becoming more employable and being valuable contributors to the economy.

In this respect and using the aforementioned studies and/or reports as a reference in this study, the following have been selected as key competencies for employability: capacity to solve problems, to work in a team, adaptive capacity, communication, leadership, decision-making and creativity. Education in these competencies requires syllabuses to be developed that include active methodologies and tools to be designed so they can be assessed.

2.2. Assessment of Relevant Competencies for Employability

In accordance with student-centred learning models, learning assessment must go beyond simply evaluating results and it has to be fully built into the teaching processes. Any change in education methodology must be accompanied by a change in assessment, as teaching, learning and assessing are inseparable processes and a change in one of them must affect the others. Assessment must be an integrated process based on concepts, experiences and values developed by students and which determines what students study, how they do so and the approach they take to their learning (Ibarra-Sáiz & Rodríguez-Gómez, 2019; Ibarra-Sáiz, Rodríguez-Gómez, Boud, Rotsaert, Brown, Salinas-Salazar et al., 2020). Real and high-quality assessment must be closely linked to the learning results specified in the syllabus and must guarantee that students are able to demonstrate knowledge, skills and attitudes or behaviours, that is, knowing what to say, do and be, which is mainly encouraged in formative assessment processes and is supported by final assessment (Reyes, Díaz, Pérez, Marchena & Sosa, 2020; Boud, 2020; De la Orden & Pimienta, 2016). In short, assessment must make it possible for the command level of the competencies obtained by students to be measured at the end of an educational stage or process given that, as has been indicated previously, employers value students who have developed skills and competencies through learning tasks. Traditional assessment systems are seldom able to demonstrate these skills. A real assessment must measure the competencies of students, in addition to giving relevant guidance on their integration into the world of work (Canossa, 2019; Brown, 2015; Akbari, Nguyen,  McClelland & Van Houdt, 2022; Knight & Drysdale, 2020).

The assessment of competencies for employability requires assessment tools to be designed whose content is closely related to what students learn in schools and which makes it possible for the capacities they have acquired in the education process to be visible through the classroom experience. The tools used to assess the competencies acquired by students are often rubrics, as stated by García-Valcárcel,  Hernández, Martín and Olmos (2020). A rubric is a matrix which includes the assessment indicators for each competency to be assessed, in addition to the criteria set to determine the achievement level for each indicator. Designing rubrics is a complex process which requires the appropriate indicators to be selected for the competency to be assessed and the command levels to be established for the chosen indicators in order to guarantee high‑quality, valid and reliable assessment (Guzmán-Cedillo, Lima-Villeda & Meza‑Cano, 2017; Fuentes-Cabrera, López-Belmonte, Parra-González & Morales-Cevallos, 2020).

On the other hand, in order for any assessment tool to be able to be used for the purpose for which it has been created, it must undergo a content validation process (Robles & Rojas, 2015). The concept of content validity refers to the fact that the indicators of an assessment tool must be relevant and representative of the construct being measured and this is generally determined by means of expert judgment. This technique consists of asking for the opinion of individuals with experience in the subject to be evaluated and who are therefore considered as experts and qualified to provide information, evidence, judgment and assessment. Their opinion makes it possible to determine whether the items of a measurement tool are relevant and representative of the dimensions set out in the construct being assessed (Galicia, Balderrama & Navarro, 2017; Escobar-Pérez & Cuervo-Martínez, 2008). By using this technique, it is possible to establish the extent to which the tool really measures what is intended to be measured and whether it may be used for the purpose for which it was designed. Furthermore, it also allows researchers to do a thorough analysis of the tool, which gives rise to decisions being made in relation to what must be changed, removing any irrelevant indicators from the tool and adjusting and redefining any indicators which are required in order to make the tool more relevant, coherent and, in short, which improve the quality of the tool (Cabero-Almenara & Llorente‑Cejudo, 2013).

In this respect, the objective of this work is to design an assessment tool which makes it possible to determine the achievement or performance level in any competencies which are selected for being considered the most important in improving the employability of students. In the designed tool, three achievement levels for each competency are proposed together with the indicators that make it possible for students to be awarded one of these levels. In addition, the designed tool has undergone a review, improvement and validation process carried out by a group of experts in order to determine the content validity, evaluating the degree of relevance, suitability and clarity in the wording of each one of the indicators which describe the aforementioned competencies.

3. Method

3.1. Design of the Tool

In order to assess the employability competencies in vocational training students, a rubric was designed including the competencies which had been established previously considering the reference sources and prior research.

When designing the tool, the structure of command levels established for language learning (Instituto Cervantes, 2002) and the Digcom digital competence portfolio (INTEF, 2017) was used. Three levels of competency command were established (A: beginner, B: intermediate, C: expert). As described in the Common European Framework of Reference for Languages (2002), these levels are set out in ascending order to indicate the command that a student has in a certain competency, in such a way that the person being assessed may only be at one of these levels. Three levels were established (A, B and C), and six proposed levels were rejected (A1, A2, B1, B2, C1 and C2) in order to facilitate the validation process and subsequent use by the judges. In line with this model, indicators were designed which describe the behaviours that may be observed in students for every command level in each one of the competencies established to assess employability. Between 2 and 5 indicators were prepared for every command level regarding each one of the seven competencies to be assessed (please see indicators in Table 1).

Problem-solving
A	Identifies and understands problems	Recognises the problems that arise in class.
		Only focuses on aspects that affect him/her directly.
B	Analyses and solves problems	Makes good decisions in unforeseen circumstances in order to achieve specific goals.
		Looks for different solutions.
C	Prevents problematic situations and solves complex problems	Offers new means to solve a problem.
		Copes very well with solving complex problems.
Teamwork
A	Collaborates when asked to do so	Brings new ideas to the group.
		Collaborates with the rest of the team in order to achieve common goals.
		Shares the necessary information with the group to carry out tasks.
B	Gets involved in the team	Offers to help other members of his/her group when seeing they are overloaded.
		Gets involved in class debates by preparing them, offering ideas and respecting the opinions and ideas of other classmates.
		Is able to leave behind personal goals in favour of group goals.
		Does what the rest of the group expects from him/her in suitable time with the appropriate quality.
C	Works towards the cooperation of the team	Motivates his/her classmates by recognising their merit.
		Takes on different roles within his/her team.
		Acts as a mediator when differences of opinion emerge between team members.
		Shares both the achievements and failure in teamwork.
		Encourages cooperation with other teams.
Adaptive capacity / flexibility
A	Adapts to and take in changes	When a result must be achieved, he/she is capable of analysing various options.
		Easily fits into different work groups.
		Accepts changes as a challenge.
B	Contributes to change	It is easy for him/her to change the way in which he/she works in order to achieve his/her goals as a student.
		Proposes various solutions to solve a problem.
C	Encourages and handles changes	Is capable of handling a change in several activities, in addition to supporting other classmates.
		Drives change in various activities.
		Able to develop new approaches by integrating internal and external opinions.
Communication
A	Passes on verbal or written information appropriately	Is capable of issuing clear and organised messages.
		Structures messages in a logical way.
		Knows how to attract and keep the attention of other classmates or teachers.
B	Communicates, knows how to listen and be receptive	Makes sure that he/she understands what others say.
		Is capable of properly transmitting complex ideas, information or instructions.
		Checks whether he/she has been correctly understood by the other person.
		Takes the appropriate amount of time to make statements.
		Chooses the most suitable means and ways to communicate according to the situation, message and the recipient.
C	Communicates efficiently in particularly complex situations	Willingly accepts the opinion or feedback of other classmates or teachers and he/she bears these in mind in order to prepare his/her next statements.
		Does high-impact presentations in public.
		Is capable of adapting his/her style of presenting to people from different cultures and countries.
		Manages to convince others based on his/her speech, proposal or project.
Creativity
A	Has an interest and takes a comprehensive approach	Is curious and interested to discover and learn new things.
		Able to see situations from different perspectives.
		Analyses his/her own ideas in order to maximize and improve results.
B	Generates ideas fluently	Uses various techniques to create ideas (e.g., brainstorming, raising questions, making connections, etc.).
		Is able to create new ideas or objects with little information or limited material.
C	Redefines and creates original ideas for a specific purpose	Displays originality and inventiveness in individual or group work.
		Likes and is easily able to implement new ideas or new projects.
		Is capable of redefining ideas for new purposes.
Leadership
A	Coordinates and manages the team activity	Makes sure that the members of his/her team work with the appropriate guidelines or rules.
		Assigns responsibilities to members of the group based on the capacities of each person.
		Informs other classmates of any decisions that may affect them.
		Creates a positive group environment.
B	Contributes to improve the performance and development of his/her team	Delegates according to the skills of the members of his/her team.
		Gives both positive and negative feedback to other classmates.
		Facilitates the participation of other classmates in decision-making.
		Develops a cooperative atmosphere.
		Takes responsibility for both the achievements and the mistakes of the team.
		Provides appropriate support, advice and communication to improve performance.
C	Manages complex teams	Successfully manages complex and diverse work groups.
		Works with his/her team to solve their problems on time.
		Takes charge in various ways based on the level of maturity of the members of his/her team.
		Makes any differing opinions or positions be listened to and respected.
		Manages to get people with different viewpoints to commit to the decisions made.
Decision-making
A	Supported decision-making without setting any specific criteria	Talks with the teacher in order to make decisions.
		Does not question the criteria used to set a matter in motion before a decision has been made.
		Makes decisions motivated by the emotional factor, seen to be more impulsive than rational.
B	Decision-making with criteria when alternatives are put forward	Makes just enough decisions when trying to choose between various alternative solutions to a problem.
		Considers all factors of the different alternatives.
C	Chooses the most suitable option in anticipation of the consequences	Anticipates the circumstances in order to make decisions.
		Reduces the emotional burden involved in delicate decisions.
		Chooses the option that he/she considers most appropriate in a justified way.

Table 1. Competency indicators for employability in accordance with command level

3.2. Validation of the Tool

3.2.1. Selection of Experts

Once the tool had been designed and prepared, it underwent a content validation process by means of expert judgment (Salazar-Gómez, Tobón & Juárez-Hernández, 2018; Lima-Rodríguez, Lima-Serrano, Ponce-González & Guerra-Martín, 2015; Fuentes-Cabrera et al., 2020). 10 judges were invited to make both a quantitative and qualitative assessment of each one of the indicators. Of the 10 judges selected, 5 were male and 5 were female. Four (4) of the judges work in the business and organisational environment, while the other 6 are lecturers in the field of education. In order to select these judges, criteria were considered such as: experience in the organisational environment, holding of senior positions, talent management, doctor of education, assessment specialist, and researcher in digital and professional competencies, which make it possible to observe the expert competency (Cabero & Barroso, 2013).

3.2.2. Validation Process

The judges were asked to make both a quantitative and a qualitative assessment of each one of the indicators established in order to assess the levels of competencies for employability. Three assessment criteria were therefore set for each indicator: the degree of clarity in the wording, the suitability of the indicator to assess a competency and the relevance of this indicator to measure the command level. Each one of the criteria was assessed on a 4-point quantitative Likert scale, in which 1 is understood as the least suitable for the indicator according to the given criteria and 4 is considered the most suitable. In turn, the judges were invited to make any comments that they considered appropriate in relation to the indicators and how they may be improved. In order to carry out this assessment, the validation tool was prepared with the specifications shared electronically to make the response process and subsequent processing of data easier. (See validation tool in https://zenodo.org/record/3838989#.Xsb5z8DtaUl).

3.2.3. Statistical Analysis

Once all the responses have been received from the judges, in order to establish the content validity of the rubric, data were gathered and the average value for each competency and the recommended Aiken’s V coefficient was calculated in order to determine the content validity (Escurra, 1988; Salazar-Gómez et al., 2018; Arifin, Kartono & Surpriyadi, 2018). Aiken’s V coefficient, whose value ranges between 0 and 1, is calculated using the formula:

Where: V is the agreed index, s is the difference between the value given by each judge and the lowest validity score (1), c is the highest validity score (4) and n the number of descriptors. The coefficient takes values of between 0 and 1. A value of 0.8 was taken as minimum acceptance criteria (Salazar-Gómez et al., 2018). Likewise, the qualitative responses of the judges were considered, expressed by means of the comments and observations regarding the indicators. These comments were taken into account in order to modify the indicators and improve them. This process was repeated twice more until suitable values were obtained in Aiken’s V statistics, which are recommended to be equal to or greater than 0.80 (≥ 0.80; p = 0.05) when there are 10 judges to accept one item (Escurra, 1988).

4. Results

As has been indicated previously, Aiken’s V coefficient was used for statistical analysis, with items being accepted with values equal to or greater than 0.80 (≥ 0.80; p = 0.05) (Escurra, 1988). For qualitative analysis, the comments and suggestions of the judges were considered to improve the clarity, suitability and relevance of competency indicators. The validation process was carried out in three stages. The statistical results are shown in Table 2. In the first stage, the only competency that was agreed on by the judges in the three aspects assessed (clarity, suitability and relevance) was that of leadership, with an average of over 3.5 and a value exceeding 0.8 in Aiken’s V. In the other competencies, the results did not reach the minimum acceptance level according to Aiken’s V statistics, as the majority of competencies obtained coefficients lower than 0.80 (Escurra, 1988; Salazar-Gómez et al., 2018). Any indicators which were not agreed on or for which proposals had been made to improve them were modified. In total, 16 indicators distributed throughout all of the competencies, with the exception of leadership, were reviewed while taking into account the results of the qualitative assessment (Qualitative results table in https://zenodo.org/record/3838989#.Xsb5z8DtaUl) expressed by the experts in open-ended questions, and a second assessment was subsequently carried out by the aforementioned.

	1st validation
	Average Clarity	Average Suitability	Average Relevance	Aiken’s V Clarity	Aiken’s V Suitability	Aiken’s V Relevance
Problem-solving	3.43	3.17	3.34	0.66	0.67	0.72
Teamwork	3.73	3.72	3.7	0.84	0.78	0.85
Adaptive capacity	3.69	3.69	3.74	0.8	0.73	0.83
Communication	3.77	3.83	3.88	0.77	0.86	0.88
Creativity	3.67	3.51	3.63	0.8	0.74	0.71
Leadership	3.67	3.7	3.9	0.81	0.84	0.93
Decision-making	3.45	3.4	3.6	0.71	0.76	0.8
	2nd validation
Problem-solving	3.67	3.58	3.75	0.75	0.67	0.92
Teamwork	3.75	3.67	3.87	0.83	0.87	0.87
Adaptive capacity	3.62	3.94	3.87	0.69	0.94	0.87
Communication	3.95	3.87	3.87	0.96	0.87	0.87
Creativity	3.87	3.87	4	0.87	0.87	1
Leadership	3.73	3.83	3.93	0.87	0.9	0.93
Decision-making	3.25	3.93	3.87	0.69	0.94	0.87
	3rd validation
Problem-solving	3.60	3.89	3.72	0.81	0.89	0.89
Teamwork	3.83	3.78	3.8	0.89	0.85	0.86
Adaptive capacity	3.67	3.54	3.58	0.85	0.85	0.87
Communication	3.97	3.88	3.89	0.97	0.97	0.97
Creativity	4.00	4.00	4.00	1.00	1.00	1
Leadership	3.93	3.97	3.98	0.96	0.89	0.98
Decision-making	3.54	3.42	3.5	0.81	0.83	0.80

Table 2. Average values and Aiken’s V for every competency selected in each one of the three validations

In the second stage of validation, an appropriate consensus was reached by the judges in the teamwork, communication and creativity competencies as regards the clarity, suitability and relevance of the indicators established to assess them, which is demonstrated by the increase in averages (≥ 3.75) and in Aiken’s V statistics (≥ 0.83). As for the items relating to problem-solving, the judges indicated that there was a lack of clarity in the wording and that the suitability must be improved. Items regarding adaptive capacity and decision-making must only improve in terms of clarity. The relevant modifications were made based on the observations and comments of the judges, giving rise to a third stage of validation. In this stage, five items were rejected for not obtaining the required percentage agreed on.

Subsequently, a third validation was carried out which gave rise to a final improvement in the wording of the indicators and which resulted in these being accepted. The average values and Aiken’s V statistics for each dimension are shown in Table 2. Aiken’s V reaches values of above 0.8 in all competencies, with the creativity and communication competencies obtaining the highest values (1 and 0.97, respectively) in the three criteria proposed for assessment.

Lastly, as a result of the analysis of agreements and of the comments and suggestions of the judges, 21 items or indicators of the competencies were modified in order to meet the three criteria of clarity, suitability and relevance and as agreed on by the judges, with the rubric finally being left with 68 indicators for 7 competencies to assess employability. (See final rubric in https://zenodo.org/record/3838989#.Xsb5z8DtaUl).

5. Discussion and Conclusions

Assessing competencies for employability is a complex process due to the number of elements that influence the job possibilities of graduates. There are personal and also social, economic and political elements which at the same time affect potential employers and their demands for human talent (Mahajan et al, 2022; Olo et al., 2022; Cotronei-Baird, 2020). Hence the lack of unanimity in determining the competencies for employability (Fahrenbach, 2022). Nonetheless, several authors and studies carried out on this subject do come to an agreement in relation to some of these competencies (Zarta & Trujillo, 2020; Canossa, 2019; Fernández, 2018; Przytuła, 2018; Navia-Osorio, 2017; Robles, 2012; McLeish, 2002) and we use this as a basis for the selection of the competencies that we propose.

A rubric is presented in this study for the assessment of seven competencies selected as being relevant for employability, which include the capacity to solve problems, to work in a team, adaptive capacity, communication, leadership, decision-making and creativity. Indicators have been designed for every competency, allowing three achievement levels to be determined for each one. The process and the results are summarized in this visual thinking (Figure 1).

Figure 1. Visual thinking which summarizes the study carried out

The use of a rubric to assess competencies for employability becomes a support tool for real assessment and it has been confirmed that this favours the expression of competencies for professional activity (Boud, 2020), as well as the inclusion of different people (Akbari et al., 2022). Having assessment tools in vocational training which are also open, stress-free, adapted to the style of learning of students, as indicated by Ibarra-Sáiz et al. (2020), Boud, (2020) and Knight and Drysdale (2020), may make it more likely for graduates to adapt to the emerging economy. This is particularly significant in the midst of the volatility which has marked the global economy following the COVID-19 pandemic, as is also highlighted by Mahajan et al., (2022) and Akbari et al. (2022). However, the benefits offered by this rubric to assess competencies for employability are not enough to overcome the problems regarding the differences between syllabuses and corporate demands (Senan & Sulphey, 2022; Olo et al., 2022; Mahajan et al., 2022). Curriculums and didactic methodologies in schools would therefore have to be studied in more detail.

We insist that having an assessment rubric for competencies acquired or developed as a result of an education process makes education more valuable, as this is considered the most relevant tool when it comes to assessing the process, as indicated by García-Valcárcel et al. (2020). The rubric designed meets the demand for assessment of competencies which are expected from those who have completed vocational training, including adaptive capacity, finding solutions, teamwork, communication, problem‑solving, leadership and creativity, etc. These coincide with the competencies which are most highly demanded by employers, as stated by Navia-Osorio (2017), Fernández (2018), Canossa (2019) and Zarta and Trujillo (2020) and the Manpower Group report (Przytuła, 2018). Throughout design, the inclusion of indicators for each competency, as well as various levels of development in each indicator were guaranteed in order to speed up the assessment process for teachers, in line with Guzmán-Cedillo et al. (2017).

By means of a content validation process, with which it is ensured that the rubric contains a sample of indicators that are representative of all possible conducts intended to be measured (Galicia et al., 2017, Lima-Rodríguez et al., 2015; Escobar-Pérez & Cuervo-Martínez, 2008) the validity of the aforementioned was guaranteed as an assessment tool. This was achieved by means of the quantitative and qualitative assessment of 10 judges with expert competency, as stated by Cabero and Barroso (2013), in the subject under assessment, which made it possible to improve both the formulation of indicators and the appropriateness and relevance of these. Likewise, in accordance with Escurra (1988), the number of judges selected is considered to be suitable for content validation. Thus meeting the standards set out for the content validity process, as recommended in the literature and which may be confirmed in studies such as those carried out by Galicia et al. (2017), García-Valcárcel et al. (2020); Cabero-Almenara and Llorente‑Cejudo (2013), Robles and Rojas (2015), Escobar-Pérez and Cuervo-Martínez (2008).

With the validation process made up of three stages as explained in the previous section, in which Aiken’s V correlation coefficients of over 0.8 were finally obtained, it may be confirmed that the proposed rubric has validity guarantees to assess competencies for employability. Adaptation is established by the agreement among experts in the assessment of indicators which were set out to assess the competencies for employability until they were clear, appropriate and relevant enough, as recommended by Salazar‑Gómez et al. (2018) and Lima-Rodríguez (2015)

The validation of this rubric makes it possible to provide the educational community with an efficient tool to be able to assess any transversal competencies of students which are important for employability. Having assessment tools for these types of competencies may be as important as assessing the competencies defined in the curriculum themselves, as emphasized by authors such as García-Valcárcel et al. (2020) and Llanes et al. (2017).

By means of this study, the need to assess these competencies is met, from a pedagogical perspective as part of the formative and summative assessment of those who have completed various vocational training and university courses, as recommended by Ibarra-Sáiz and Rodríguez-Gómez (2019), Reyes et al. (2020) and Brown (2015). A tool is therefore provided to assess competencies in higher education students which are required for integration into the world of work.

Limitations of this study may include the fact that it does not consider other competencies which are also relevant for employability, such as digital competencies, which are becoming more essential every day for career development, particularly in new professions as stated by Fahrenbach (2022). It is therefore recommended to include this type of competency in the assessment for employability. On the other hand, as regards the validation of the proposed tool, the analysis of agreement among judges may be reinforced with other statistical indicators, as recommended by Fuentes-Cabrera et al. (2020): Fleiss’ kappa to guarantee agreement and Kendall’s W to find out the extent of agreement.

As action plans in the field of education emerging from this work, it is proposed that this rubric be passed on to both teachers who are training and qualified teachers, and its use be encouraged. This would be one way to keep the link between specific technical competencies and soft skills at the centre of education. In addition, it would be an opportunity to overcome the differences between vocational training syllabuses and employability, as is also recommended by Olo et al. (2022) and Senan and Sulphey (2022). Likewise, it is suggested that vocational training students be informed of this rubric as a self-assessment tool which helps them to focus their efforts on self-improvement of the competencies which are considered relevant for employability. The circulation and implementation of the rubric presented will make it possible to support the education processes of professionals of the immediate future, as well as monitor and assess its use for teachers, graduates and employers, particularly in a world after the COVID-19 pandemic, in which the skills and competencies put forward in this study, also called soft skills, have been put to the test.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

References

Akbari, M., Nguyen, H.M., McClelland, R., & Van Houdt, K. (2022), Design, implementation and academic perspectives on authentic assessment for applied business higher education in a top performing Asian economy. Education + Training, 64(1), 69-88. https://doi.org/10.1108/ET-04-2021-0121

Alarcón, R. (2018). La formación para el trabajo y el paradigma de formación por competencias. Calidad en la educación, 0(16).

Alonso-Sáez, I., & Arandia-Loroño, M. (2017). 15 años desde la Declaración de Bolonia. Desarrollo, situación actual y retos del Espacio Europeo de Educación Superior. Revista Iberoamericana de Educación Superior, 8(23), 199-213 https://doi.org/10.22201/iisue.20072872e.2017.23.256

Arifin, Z., Kartono, K., & Surpriyadi, S. (2018). Development of authentic assessment instrument for performance in learning Mathematics in linear program. Journal of Educational Research and Evaluation 7(2), 153-161. https://doi.org/10.15294/jere.v7i2.24884

Assessment and Teaching of 21st Century Skills (ATCS21) (2009). 21st Century Skills Project. Universidad de Melbourne. Available at: http://www.atc21s.org/about.html (Accessed: January 2020).

BOE (2002). Ley Orgánica 5/2002, de 19 de junio, de las cualificaciones y de la Formación Profesional. Available at: https://www.boe.es/buscar/act.php?id=BOE-A-2002-12018 (Accessed: June 2019).

BOE (2015). Real Decreto 1105/2014, de 26 de diciembre, por el que se establece el currículo básico de la Educación Secundaria Obligatoria y del Bachillerato. Available at: http://www.boe.es/boe/dias/2015/01/03/pdfs/BOE-A-2015-37.pdf (Accessed: July 2019).

Boud, D. (2020). Retos en la reforma de la evaluación en educación superior: una mirada desde la lejanía. Revista Electrónica de Investigación y Evaluación Educativa (RELIEVE), 26(1). https://doi.org/10.7203/relieve.26.1.17088

Brown, S. (2015). Evaluación auténtica: utilizar la evaluación para ayudar a los estudiantes a aprender. Revista Electrónica de Investigación y Evaluación Educativa (RELIEVE), 21(2) https://doi.org/10.7203/relieve.21.2.7674

Cabero, J., & Barroso, J. (2013). La utilización del juicio de experto para la evaluación de TIC: el coeficiente de competencia experta. Bordón, 65(2), 25-38. https://doi.org/10.13042/brp.2013.65202

Cabero-Almenara, J., & Llorente-Cejudo, M.C. (2013). La aplicación del juicio de experto como técnica de evaluación de las tecnologías de la información (TIC). Eduweb: Revista de Tecnología de Información y Comunicación en Educación, 7(2), 11-22.

Campaña-Jiménez, R, Gallego-Arrufat, M., & Muñoz-Leiva, L. (2019). Estrategias de enseñanza para la adquisición de competencias en formación profesional: perfiles de estudiantes. Educar, 55(1), 203-229. https://doi.org/10.5565/rev/educar.876

Canossa, H. (2019). ART Habilidades blandas en el estudiantado universitario y la empleabilidad en Costa Rica. Revista Académica Arjé, 2(2), 5-13.

Common European Framework Of Reference For Languages: Learning, Teaching, Assessment (2002) Council of Europe. Language Policy Unit, Strasbourg.   www.coe.int/lang-CEFR

Cotronei-Baird, V.S. (2020) Academic hindrances in the integration of employability skills development in teaching and assessment practice. Higher Education 79, 203-223. https://doi.org/10.1007/s10734-019-00405-4

Chamorro-Premuzic, T., & Frankiewicz, B. (2019). Does Higher Education Still Prepare People for Jobs? Harvard Business Review. [Blog] Available at: https://hbr.org/2019/01/does-higher-education-still-prepare-people-for-jobs (Accessed: February 2020).

Davis, E. (2018). 21 more jobs of the future. A guide to getting and staying employed through 2029. Center for the future of work - Cognizant. Available at: https://www.cognizant.com/futureofwork/whitepaper/21-more-jobs-of-the-future-a-guide-to-getting-and-staying-employed-through-2029 (Accessed: March 2020).

De la Orden, A., & Pimienta, J.H. (2016). Instrumento para determinar los tipos de evaluación utilizados por los profesores universitarios. Revista Electrónica de Investigación Educativa, 18(2), 40-52.

EPYCE (2018). Posiciones y competencias más demandas. Available at: http://marketing.eae.es/prensa/_EPyCE2017.pdf (Accessed: June 2021).

Escobar-Pérez, J., & Cuervo-Martínez, A. (2008). Validez de contenido y juicio de expertos: una aproximación a su utilización. Avances en Medición, 6(1), 27-36.

Escurra, L. (1988). Cuantificación de la validez de contenido por criterio de jueces. Revista de psicología, 6(1), 103-111. https://doi.org/10.18800/psico.198801-02.008

Fahrenbach, F. (2022). A design science approach to developing and evaluating items for the assessment of transversal professional competences. Education + Training, 64(1), 21-40.
https://doi.org/10.1108/ET-03-2020-0056

Fernández, B. (2018). Competencias laborales y de empleabilidad en la educación vocacional. Calidad en la Educación, 27, 36-51. https://doi.org/10.31619/caledu.n27.217

Fuentes-Cabrera, A., López-Belmonte, J., Parra-González, M.E., & Morales-Cevallos, M.B. (2020). Diseño, validación y aplicación de un cuestionario para medir la influencia de factores exógenos sobre la eficacia del aprendizaje invertido. Psychology, Society and Education, 1(12) https://doi.org/10.25115/psye.v10i1.2334

Gal, I., & Tout, D. (2014). Comparison of PIAAC and PISA Frameworks for Numeracy and Mathematical Literacy. OECD Education Working Papers, No. 102, OECD Publishing, Paris, https://doi.org/10.1787/5jz3wl63cs6f-en

Galicia, L., Balderrama, J., & Navarro, R. (2017). Validez de contenido por juicio de expertos: propuesta de una herramienta virtual. Apertura, 9(2), 42-53. https://doi.org/10.32870/ap.v9n2.993

García-Valcárcel, A., Hernández, A., Martín, M., & Olmos, S. (2020). Validación de una rúbrica para la evaluación de trabajos fin de máster. Profesorado, Revista de Currículum y Formación del Profesorado, 24(2), 72‑96. https://doi.org/10.30827/profesorado.v24i2.15151

Gargallo, B., Jiménez, M., Martínez, N., Jiménez-Beut, J., & Pérez-Pérez, C. (2017). Métodos centrados en el aprendizaje, implicación del alumno y percepción del contexto de aprendizaje en estudiantes universitarios. Educación XX1, 20(2). https://doi.org/10.5944/educxx1.19036

Guzmán-Cedillo, Y.I., Lima-Villeda, D.N., & Meza-Cano, J.M. (2017). Diseño y confiabilidad de una rúbrica para evaluar infografías didácticas. Enseñanza & Teaching, 35(2), 17-36. https://doi.org/10.14201/et20173521736

Hassall, T., Joyce, J., Arquero Montano, J.L., & Donoso Anes, J.A. (2003). The vocational skills gap for management accountants: The stakeholders' perspectives. Innovations in Education and teaching International, 40(1), 78-88.

Ibarra-Sáiz, M., Rodríguez-Gómez, G., Boud, D., Rotsaert, T., Brown, S., Salinas-Salazar, M.L. et al. (2020). El futuro de la evaluación en la educación superior. Revista Electrónica de Investigación y Evaluación Educativa (RELIEVE), 26(1), 1-6. https://doi.org/10.7203/relieve.26.1.17323

Ibarra-Sáiz, M., & Rodríguez-Gómez, G. (2019). Una evaluación como aprendizaje. In Patricio-Royo, J., Fernández, A., & Fernández, I. (Eds.), Cartografía de la buena docencia universitaria. Un marco para el desarrollo del profesorado basado en la investigación (175-196). Narcea.

Institute of Museum and Library Services (n.d.). Museums, Libraries, And 21st Century Skills. [Museos, bibliotecas y habilidades del siglo XXI]. Available at: https://www.imls.gov/issues/national-initiatives/museums-libraries-and-21st-century-skills/definitions (Accessed: May 2019).

Instituto Cervantes (2002). Marco común europeo de referencia para las lenguas: aprendizaje, enseñanza y evaluación. Centro Virtual Cervantes. Instituto Cervantes. Available at: https://cvc.cervantes.es/ensenanza/biblioteca_ele/marco/cvc_mer.pdf (Accessed: June 2019).

Instituto Nacional de Evaluación Educativa (2013). Programa internacional para la evaluación de competencias de la población adulta (PIAAC). Available at: http://www.educacionyfp.gob.es/inee/dam/jcr:761d480f-1ccf-479c-95f5-bee8694cce8c/presentacionpiaac.pdf (Accessed: June 2019).

INTEF (2017). Marco común de competencia digital docente octuble 2017. Available at: http://aprende.intef.es/sites/default/files/2018-05/2017_1020_Marco-Com%C3%BAn-de-Competencia-Digital-Docente.pdf (Accessed: June 2019).

Knight, G.L., & Drysdale, T.D. (2020). The future of higher education (HE) hangs on innovating our assessment–but are we ready, willing and able? Higher Education Pedagogies, 5(1), 57-60. https://doi.org/10.1080/23752696.2020.1771610

Lima-Rodríguez, J., Lima-Serrano, M., Ponce-González, J., & Guerra-Martín, M. (2015). Diseño y validación de contenido de rúbricas para evaluar las competencias prácticas en estudiantes de Enfermería. Educación Médica Superior, 29(1), 119-133.

Llanes, J., Figuera, P., & Torrado, M. (2017). Desarrollo de la empleabilidad y gestión personal de la carrera de graduados en Pedagogía. Revista Española de Orientación y Psicopedagogía, 28(2), 46-60. https://doi.org/10.5944/reop.vol.28.num.2.2017.20118

Mahajan, R., Gupta, P., & Misra, R. (2022). Employability skills framework: a tripartite approach. Education + Training, 64(3), 360-379. https://doi.org/10.1108/ET-12-2020-0367

McLeish, A. (2002). Employability skills for Australian small and medium sized enterprises. Australia: Commonwealth Department of Education, Science and Training. Available at: https://www.voced.edu.au/content/ngv%3A3781 (Accessed: July 2021).

Ministerio de Educación y Formación Profesional (2018) Panorama de la educación Indicadores de la OCDE. Informe español. Available at: http://www.educacionyfp.gob.es/inee/dam/jcr:2cba4aaa-4892-40d7-ac8b-00efbc95b8a2/Panorama%20de%20la%20Educacion%202018_final.pdf (Accessed: July 2019).

Navia-Osorio, Y. (2017). Diseño de un instrumento de evaluación de las competencias transversales de los aprendices del nivel tecnológico del Centro de Servicios y Gestión Empresarial del Servicio Nacional de Aprendizaje - SENA, en Medellín. Revista Facultad de Ciencias Contables, Económicas y Administrativas (FACCEA), 7(2), 194-203.

OECD (2017). Getting Skills Right: Spain. Paris: OECD Publishing. https://doi.org/10.1787/25206125

Olo, D., Correia, L., & Rego, C. (2022). How to develop higher education curricula towards employability? A multi-stakeholder approach. Education + Training, 64(1), 89-106. https://doi.org/10.1108/ET-10-2020-0329

Przytuła, S. (2018). Global Labor Market Trends and Their Significance for the Future Employees’ Competences. Journal of Intercultural Management, 10(4), 5-38. https://doi.org/10.2478/jim-2018-0023

Reyes, C.I., Díaz, A., Pérez, M., Marchena, M., & Sosa, F. (2020). La evaluación del aprendizaje: Percepciones y prácticas del profesorado universitario. Profesorado. Revista de currículum y formación del profesorado, 24(1), 136-162. https://doi.org/10.30827/profesorado.v24i1.8449

Robles, M. (2012). Executive perceptions of the top 10 soft skills needed in today’s workplace. [Percepción de los ejecutivos sobre las 10 principales habilidades sociales necesarias en el lugar de trabajo actual.] Business Communication Quarterly, 75(4), 453-465. https://doi.org/10.1177/1080569912460400

Robles, P., & Rojas, M.D.C. (2015) La validación por juicio de expertos: dos investigaciones cualitativas en Lingüística aplicada. Revista Nebrija de Lingüística Aplicada, 18, 124-139. https://doi.org/10.26378/rnlael918259

Salazar-Goméz, E., Tobón, S., & Juárez-Hernández, L. (2018). Diseño y validación de una rúbrica de evaluación de las competencias digitales desde la socioformación. Apuntes Universitarios, 8(3). https://doi.org/10.17162/au.v8i3

Senan, N.A.M., & Sulphey, M.M. (2022), Construction and validation of the employability questionnaire for accounting graduates, Education + Training, 64(1), 141-159. https://doi.org/10.1108/ET-04-2021-0152

Silva, W.H., & Mazuera, J.A. (2019). ¿Enfoque de competencias o enfoque de capacidades en la escuela? Revista Electrónica de Investigación Educativa, 21(17), 1-10. https://doi.org/10.24320/redie.2019.21.e17.1981

Van Loo, J.B. & Toolsema, B. (2005). The empirical determination of key skills from an economic perspective. Education Economics, 13(2), 207-221. https://doi.org/10.1080/09645290500031421

Zarta, J.B., & Trujillo, J.S. (2020). Análisis de los factores para la empleabilidad en los profesionales del Espinal-Tolima. Revista Innova ITFIP, 6(1), 9-28. https://doi.org/10.54198/innova06.01

---