Journal of Technology and Science Education

• Home
• About
• Login
• Archives
• Submissions
• Publication Fee
• Indexing & Statistics

Home > Vol 12, No 3 (2022) > Puertas

ASSESSMENT STRATEGIES IN MIXED LEARNING APPLIED
IN IT ENGINEERING AND MATHEMATICS STUDIES

Facultat de Matemàtiques i Informàtica, Universitat de Barcelona (Spain)

Received April 2022

Accepted July 2022

Abstract

During the past COVID pandemic crisis, teachers of any level must adapt their teaching methodologies to an online learning environment. One of the fears most aspects was the course assessment. Nonetheless, a lot of teachers were attracted to using new tools, that previously exists on their online platforms. In Universitat de Barcelona learning was developed in a mixed form after the first COVID crisis, doing some activities online and others, like final semester exams, in person. During this phase, we carried out a questionnaire among students and teachers to know their opinion on assessment tasks during these semesters when online learning was more important than ever before. The results we have recollected show that professors and students do not always share the same point of view about the usefulness of different assessment strategies or how they are related to competence and skills achievements.

 

Keywords – Mixed learning, Higher education, Assessment strategies, Competencies, IT engineering, Mathematics.

To cite this article:

Puertas, E. (2022). Assessment strategies in mixed learning applied in IT engineering and mathematics studies. Journal of Technology and Science Education, 12(3), 567-577. https://doi.org/10.3926/jotse.1708

 

----------

1. Theoretical Framework

Assessment has traditionally been associated with a finalist objective, with the focus being the final product or result. This is aligned with a unilateral, transmissive approach to education, where the teacher is the provider of knowledge, and the student is a passive recipient. However, there have been some significant changes in the approach to higher education and the curricular design of teaching plans and assessments. The competency-based curriculum changes the roles of both the teacher and the students. On the one hand, students are active participants in the learning process, providing adequate solutions to some challenges or problems set in a specific context, by combining their knowledge, skills, and attitudes. On the other hand, teachers no longer transmit knowledge; instead, they guide learners through the process to develop a set of competencies.

As a result, assessment in competency-based approach designs should be about performance and not only about outputs; it should be authentic and realistic (Villarroel & Bruna, 2014). This implies significant changes in how the assessment is implemented. In competency-based assessment, several tools and agents should be involved (Ibarra-Saiz, Rodríguez-Gómez & Boud, 2020), using reflection on the knowledge to solve a problem or make some decisions (Gulikers, Bastiaens & Kirschner, 2004). The role of the teacher in assessment is not only to certify a certain level of competence development or attainment of objectives, which is known as Assessment of Learning. The teacher also uses assessment to guide the students and help them learn, promoting their autonomy and self-reflection at the same time, which is known as Assessment for Learning (Stiggins, Arter, Chappuis & Chappuis, 2002). This implies an active role for both the teacher and the students. Assessment of learning and for learning are not exclusive. The accreditive and formative purposes can occur simultaneously, but learning should be more important for teachers than marks.

Digital tools can be used for assessment purposes, in both Assessment of Learning and for Learning. Tools are a resource to foster learning, not the other way around. Technology itself does not transform the learning process. Teachers must focus on the use of the digital tool in the learning and assessment process, which should ultimately be related to promoting learning (Spector, 2013). That is, the teacher shall reflect on the purpose of the activity, how the tool can be a facilitator for the assessment and learning process, when it is adequate to implement, how it can be combined with other elements, how it can help provide feedback and promote reflection, etc.

The reason why teachers usually select some digital tools over others is mainly related to their availability in their institution or ease of use, not so much because of their potential to strengthen the more active and committed role of students to be strengthened (García-Peñalvo, Corell, Abella-García & Grande, 2020). Interdisciplinary teams of computer engineers and educators would be needed to achieve tools at the service of interaction, content creation, peer feedback, and so on beyond the use of tools for communication (on many occasions practically unidirectional) and content verification (through questionnaires).

During the past COVID pandemic crisis, teachers at any level must adapt their teaching methodologies to the online learning environment. In Universitat de Barcelona, likewise anywhere, one of the fears most aspects was the course assessment. Nonetheless, a lot of teachers were attracted to using news tools, that previously exists on their online platforms. Therefore, it was a good opportunity to know the opinion among students and teachers about assessment tasks during the COVID-19 pandemic when online learning was more important than any time before

2. Methodology

At Universitat de Barcelona, during the beginning of the Covid crisis world, in a first phase, all the teaching was exclusively online. Then a second phase started on the next course, where learning was developed in a mixed form, doing some activities online and others like final semester exams in person. This phase lasted until the classes have been able to be face to face again at our university. In the middle of this second face, we carried out a questionnaire among students and teachers to know their opinion about assessment tasks during these phases where online learning was more important than any time before.

The questionnaire aimed to know more about the satisfaction regarding the evaluation activities, the purposes of the evaluation, the frequency of use of certain evaluation strategies, and the usefulness of these strategies for the evaluation of transversal competencies carried out in mixed teaching environments from the point of view of students and teachers as well. The evaluation strategies are such that they can also be evaluated from an online e-learning platform such as moodle. In Table 1 the different strategies are listed from 3.1 to 3.12 items.

Therefore, based on the theoretical framework analyzed, a voluntary and anonymous online survey (using Google Forms) was created and administered in March 2021. It was sent to professors and teachers from different faculties.

2.1. Sample

The number of answers to the questionnaire on our faculty was around 100 undergraduate students in Mathematics and IT Engineering out of the 500 enrolled (20%) and 11 professors from the Department of Mathematics and Computer Science out of the 40 (27.5%) who teach, most of them full-time, in the IT Engineering and Mathematics degrees. Teachers have a background in mathematics, physics, or computer science.

2.2. Procedure

The questionnaire comprises a series of closed questions with answers to be scored on a scale of 1 to 5 to which the range Don’t know/No answer has been added, as well as a closed multiple-choice question, to investigate their perceptions. These questions can be grouped into four different aspects, the overall level of satisfaction, degree of agreement, frequency of use, and usefulness. Listed in Table 1 are several questions regarding those aspects and their possible answers.

Questions		Answers
1. Overall level of satisfaction with the evaluation activities carried out in mixed teaching environments		A scale from 1-Little to 5-A lot
2. Degree of agreement with 4 purposes of the evaluation	2.1. Identify the needs of the student	A scale of 1 - Completely disagree to 5 - Completely agree
	2.2. Identify their level of competence
	2.3. Guide their learning
	2.4. Certify learning
3. Frequency of use of 12 evaluation strategies, in mixed teaching environments	3.1. Self-assessment activities	A scale from 1-Never to 5-Always
	3.2. Peer assessment activities
	3.3. Objective tests (true/false, multiple choice, pairings...)
	3.4. Short-responses tests
	3.5. Long response tests, development
	3.6. Oral tests (individual, group, presentations...)
	3.7. Work and projects
	3.8. Reports/ Memoirs/ Practice notebooks
	3.9. Execution tests of real and/or simulated tasks
	3.10. Attitude scales to collect values or social skills
	3.11. Observation techniques with records, checklists...
	3.12. Portfolio or learning folder
4. Usefulness of these strategies for the evaluation of transversal competences, in mixed teaching environments. Select the transversal competences for which each one of the previous evaluation strategies could be useful for its evaluation, referring to the transversal competences incorporated in the UB graduate profiles (2008)	4.1. Self-assessment activities	A scale from 1-Little to 5-A lot.
	4.2. Peer assessment activities	Closed Multiple Choice Question:
	4.3. Objective tests (true/false, multiple choice, pairings...)	C1: Ethical commitment (critical and self-critical capacity, ethics)
	4.4. Short answer tests
	4.5. Long response tests, development	C2: Capacity for learning and responsibility (ability to learn in new situations, analysis, synthesis, and global visions)
	4.6. Oral tests (individual, group, presentations...)
	4.7. Works and projects	C3: Teamwork (collaboration with others in interdisciplinary teams)
	4.8. Reports/memoirs/practice notebooks
	4.9. Execution tests of real and/or simulated tasks	C4: Creative and entrepreneurial capacity (search and integrate new knowledge and design new projects)
	4.10. Attitude scales to collect values or social skills
	4.11. Observation techniques with records, checklists...	C5: Sustainability (assess the social and environmental impact of actions in its field)
	4.12. Portfolio or learning folder	C6: Communication capacity (comprehension and expression with specialized language of the discipline)
5. Usefulness of specific technological tools during mixed teaching.	5.1. Assistance	A scale from 1-Little to 5-A lot.
	5.2. Database
	5.3. BB Collaborate
	5.4. Consultation
	5.5. Interactive content
	5.6. Survey
	5.7. Forum
	5.8. Glossary
	5.9. Book
	5.10. Lesson
	5.11. SCORM package
	5.12. Page
	5.13. Questionnaire
	5.14. Feedback
	5.15. Workshop
	5.16. Task
	5.17. Wiki
	5.18. Chat
6. Frequency in which some statements have happened in mixed teaching	6.1. Assessment activities are productive (creative, application, design, decision making, etc.).	A scale from 1-Never to 5-Always
	6.2. Assessment activities are consistent with the goals of the degree and the aims of the subjects.
	6.3. Students participate in the definition and understanding of the objectives of the task.
	6.4. Students participate in the definition and understanding of the criteria for evaluating the task.
	6.5. Students participate in self-assessment activities.
	6.6. Students participate in peer assessment activities.
	6.7. Students can integrate the feedback received in future assignments or versions of the same assignment.
	6.8. Students can reflect on the feedback received.
	6.9. Students can use the support of various technological tools to give and receive feedback.

Table 1. Online Survey Questions

2.3. Data Analysis

After a descriptive exploration of the quantitative data collected, with the GraphPad Prism software package, to see how the results behave (sample distribution) and how they are distributed (min., max., M, Me, σ), a mean comparison analysis for unpaired data by parametric treatment (Student’s t-test) was performed, after checking its normal distribution, to detect possible significant differences between student and faculty perceptions.

3. Results

From the results we have collected we selected the indicators that better reflect better the difference between student and faculty perceptions about how and why assessments in University courses have to be done.

3.1. Satisfaction

Both students and professors were asked, “What is your overall satisfaction with the assessment activities that teachers have performed in mixed teaching environments?” The results shown in Table 3 show that student satisfaction is below the middle point (3), less in the case of Mathematics students, who are more used to using face-to-face assessments. Regarding the use of technological tools, students and professors were asked “What is your overall level of satisfaction with the use of technology tools for competency assessment?”. The results in Table 3 show that student satisfaction is near the middle point (3). However, teacher satisfaction is higher than the middle point.

	Assessment activities Satisfaction
Mathematics Students	2.64
IT Engineering Students	2.93
Teachers	3.36

Table 2. Assessment activities satisfaction

	Technological tools Satisfaction
Mathematics Students	2.90
IT Engineering Students	2.89
Teachers	3.55

Table 3. Technological tools satisfaction

3.2. Comparatives

In this section, we will focus on the comparison between results for the same question between teachers and the students of the two degrees.

3.2.1. Motivation for Assessment

When students are asked about the motivation for the assessment, they coincide in the same perception, regardless of the grade they are taking. Teachers and students rank “certify learning” as the topmost motivation for doing assessment tasks. For teachers, assessment is also important for “identifying the level of competence development” of students. However, this does not seem to be as important for Mathematics or IT Engineering students. In Figure 1 we can see the comparison between the three collectives.

Figure 1. Motivation for assessment

3.2.2. Strategies for Assessment

When we ask students and teachers which strategies for assessment, they think are more useful (see Table 1 for more details), we saw some important differences. For example, IT Engineering students perceive peer assessments, oral tests, and projects as very useful, whereas teachers perceive tests and projects as more useful. Overall, projects are perceived as the most useful assessment strategy. In figure 2 we can see the comparison between the three collectives.

Figure 2. Degree of the usefulness of different assessment strategies

3.2.3. Competencies Relating to Different Assessment Strategies.

We ask students and professors about the degree of competencies achieved according to different assessment strategies. Only the most related strategies in our studies are considered here, those strategies from 3.1 to 3.8 in Table 1. Table 4 shows the assessment strategies and competencies considered for our analysis.

Assessment Strategies

3.1 Self-assessment activities

3.2 Peer assessment activities

3.3 Objective tests (true/false, multiple choice, pairings...)

3.4 Short-responses tests

3.5 Long response tests, development

3.6 Oral tests (individual, group, presentations...)

3.7 Works and projects

3.8 Reports/memoirs/practice notebooks

Table 4. Assessments strategies related to IT Engineering and Mathematics studies

Figure 3 shows the different points of view between teachers and students about the competencies achieved relating to the different assessment strategies. We can see that the vision of the students, regardless of what degree they are studying, is quite similar. Minor changes exist, for example, in the Learning Ability: IT Engineering students consider works and projects more relevant than the rest, whilst Mathematics students think that tests are more relevant.

On the contrary, teachers have a very different vision about the competencies their students develop when they are using different assessment strategies. For example, with more than 80%, they consider long answer tests to be very relevant for learning ability competency. Another accentuated difference is regarding self-assessment activities, where teachers think it is relevant for Learning Ability competences. Instead, for students, it is more relevant for the competency in ethical commitment. Some activities are not well known by students or teachers, such as peer evaluation activities, which are usually mistaken for group activities. As for other activities, for instance, oral activities (team or individual presentations, oral tests…) students found them more relevant for assessing competencies than their professors. Taking into account these results, further research could promote the convergence of the perspectives of both teachers and students about the assessment of competencies.

Figure 3. Competencies achieved relating to different assessment strategies

3.2.4. The Usefulness of Different Technological Tools

Additionally, we ask students and professors about the usefulness of specific technological tools during mixed teaching. Figure 4 shows the degree of usefulness of different technological tools from the point of view of students and teachers.

Some tools were not useful or even used at all. However other tools became very popular and useful, such as BB collaborate, which was the tool used to do videoconferences at the University of Barcelona. The forums were also found to be useful both for students and teachers. Feedback and questionnaires were perceived as more useful for teachers than for students. In general, IT Engineering students found the technological tools more useful than Mathematics students.

Figure 4. Degree of the usefulness of different technological tools

3.2.5. Perception of the Frequency that Certain Events Have Given in Mixed Teaching

Finally, students and teachers were asked about the frequency in which some statements have happened in mixed teaching. In figure 5 we can see the frequency that certain events have given in mixed teaching according to the point of view of teachers and students.

The biggest discrepancies between students and teachers are related to (1) assessment activities being productive or not, (2) whether assessment activities are coherent with the goals of the Degree and the aims of the subjects and (3) whether students participate in the definition and understanding of the objectives of the task.

Figure 5. The frequency that certain events have given in mixed teaching

3.3. Discussion

Seeing the results obtained from the questionnaires, we can extract some interesting information.

The degree of satisfaction both with the technological tools used and with the evaluation processes is moderate and leaves a clear margin for improvement, but is, in general, higher among teachers.

Projects are the most valued learning and evaluation proposals. Proposing projects that integrate learning, authentic and significant seems to be a good strategy for evaluation in contexts of face-to-face teaching and hybrid or online (Baughan, 2020; Hou, Chang & Sung, 2007)

The increased use of tools such as BB Collaborate may be due to the confinement being unexpected. There was little time to adapt teaching to a true online instructional design and the sessions planned for face-to-face teaching were replicated (García-Peñalvo et al., 2020).

The fact that questionnaires have been considered less useful by students and yet continue to be a frequent instrument is something that has also been verified and that, in part, is due to the lack of evaluative literacy of teachers (Ayalon & Wilkie, 2020).

It has been found that IT Engineering students found technological tools more useful than Mathematics students. This may be due perhaps to the nature of each of the disciplinary fields and the profiles of the students (Winstone, Balloo & Carless, 2020)

The least valued, in general, are the assessment strategies participated in by the student body, both in the appropriation of the criteria, as well as in self-assessment processes, which abounds in what the literature has already pointed out (Wiliam, 2011; O’Donovan, den Outer, Price & Lloyd, 2019; Van der Kleji, Adie & Cumming, 2019).

When asking about what type of assessment strategies are more associated with the development of competencies, perhaps the item has been misunderstood and part of the teaching staff has indicated what type of assessment strategies would make it possible to know better if a certain competency is available, with a specific purpose. more credible. In this sense, projects are associated with creativity or exhibitions with oral communication. The challenge, however, lies not so much in how to validate whether a certain competence has been developed, but rather in using the evaluation, with educational and training purposes, to promote competence in all students throughout the process (Cano, 2019).

The transversal competence of the UB “Learning and responsibility”, which contains the ability to learn to learn, has been the most valued by the teaching staff. This may respond to a perhaps simple vision, which interprets that any learning activity proposed promotes learning-to-learn. Against this perspective, there are numerous contributions (Henderson, Ajjawi, Boud & Molloy, 2019; Tai, Ajjawi, Boud, Dawson & Panadero, 2018) that are committed to an explicit design with activities expressly designed for this purpose that encourage learning-to-learn. This means accompanying the learning activities with reflective processes about the steps taken, activated learning, and promoting metacognitive strategies that allow them to become aware and replicate, when appropriate, the learning processes.

4. Conclusions

Facing the results, we get from the questionnaire we can extract some conclusions.

First, perceptions about assessment strategies are quite different between students and teachers. Teachers seem that understand better the motivation of assessments in the subjects than students, regardless of the kind of degree. However, teachers and students appreciated the use of technological tools during mixed teaching, but students seem more critical of its usefulness than teachers.

Therefore, with a glimpse of these results, we can determine some implications for improving the learning experience in our degrees. First, university teachers should strengthen evaluative literacy and/or teaching skills related to evaluation. It could also be interesting to strengthen pedagogical training in general to ensure a better understanding of the implications of competency-based designs.

From the point of view of curricular design, it should be good to betting on teaching plans that articulate varied but sustainable forms of evaluation. The assessment experiences in which students can live must be sufficiently diverse. However, teaching coordination is necessary to avoid fragmented activities of little significance, which the student body only fulfils by adjusting to the qualification requirements.

Finally, from the institutional point of view, it should be good to evaluate what type of platforms are better to choose for online teaching, according to curricular design and teachers’ teaching skills.

Declaration of Conflicting Interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

This work is part of the project:

“Analysis of evaluation practices in mixed teaching environments aimed at the development of transversal skills” (REDICE20-2083) financed by the research section of the IDP of the University of Barcelona.

This contribution has been made within the framework of the project:

“Análisis de los efectos de la provisión de feedback soportado por tecnologías digitales de monitoreo sobre las competencias transversales” (Referencia PID2019-104285GB-I00). Ministerio de Ciencia e Innovación (MICINN). Agencia Estatal de Investigación

References

Ayalon, M., & Wilkie, K.J. (2020). Developing assessment literacy through approximations of practice: Exploring secondary mathematics pre-service teachers developing criteria for a rich quadratics task. Teaching and Teacher Education, 89, 103011. https://doi.org/10.1016/j.tate.2019.103011

Baughan, P. (2020). On Your Marks: Learner-focused Feedback Practices and Feedback Literacy. Adanve HE.

Biggs, J. (2003). Aligning teaching for constructing learning. Higher Education Academy, January, 1-4.

Cano, E. (2019). Presentación del monográfico: Evaluación por Competencias en la Educación Superior: Buenas Prácticas ante los Actuales Retos. Revista Iberoamericana de Evaluación Educativa, 12(2), 5-8. Available at: https://revistas.uam.es/index.php/riee/issue/view/931 https://doi.org/10.15366/riee2019.12.2.003

García-Peñalvo, F.J., Corell, A., Abella-García, V., & Grande, M. (2020). Online assessment in higher education in the time of COVID-19. Education in the Knowledge Society, 21, 1-26. https://doi.org/10.14201/eks.23013

Gulikers, J.T.M., Bastiaens, T.J., & Kirschner, P.A. (2004). A five-dimensional framework for authentic assessment. ETR&D, 52, 67-86. https://doi.org/10.1007/BF02504676

Henderson, M., Ajjawi, R., Boud, D., & Molloy, E. (2019). The impact of feedback in higher education: Improving assessment outcomes for learners. In The Impact of Feedback in Higher Education: Improving Assessment Outcomes for Learners. https://doi.org/10.1007/978-3-030-25112-3

Hou, H.T., Chang, K.E., & Sung, Y.T. (2007). An analysis of peer assessment online discussions within a course that uses project-based learning. Interactive Learning Environments, 15(3), 237-251. https://doi.org/10.1080/10494820701206974

Ibarra-Sáiz, M.S., Rodríguez-Gómez, G. & Boud, D. (2020). Developing student competence through peer assessment: the role of feedback, self-regulation and evaluative judgement. High Educ, 80, 137-156. https://doi.org/10.1007/s10734-019-00469-2

O’Donovan, B.M., den Outer, B., Price, M., & Lloyd, A. (2019). What makes good feedback good? Studies in Higher Education, 46(2), 318-329. https://doi.org/10.1080/03075079.2019.1630812

Spector, J.M. (2013). Emerging educational technologies and research directions. Journal of Educational Technology & Society, 16(2), 21-30.

Stiggins, R., Arter, J., Chappuis, J., & Chappuis, S. (2002). Assessment for learning. Education Week, 21(26), 30-33.

Tai, J., Ajjawi, R., Boud, D., Dawson, P., & Panadero, E. (2018). Developing evaluative judgement: enabling students to make decisions about the quality of work. Higher Education, 76(3), 467-481. https://doi.org/10.1007/s10734-017-0220-3

Van der Kleij, F.M., Adie, L.E., & Cumming, J.J. (2019). A meta-review of the student role in feedback. International Journal of Educational Research, 98, 303-323. https://doi.org/10.1016/j.ijer.2019.09.005

Villarroel, V., & Bruna, D. (2014). Reflexiones en torno a las competencias genéricas en educación superior: Un desafío pendiente. Psicoperspectivas, 13(1), 23-34. https://doi.org/10.5027/psicoperspectivas-Vol13-Issue1-fulltext-335

Wiliam, D. (2011). What is assessment for learning? Studies in Educational Evaluation, 37(1), 3-14. https://doi.org/10.1016/j.stueduc.2011.03.001

Winstone, N.E., Balloo, K., & Carless, D. (2020). Discipline-specific feedback literacies: A framework for curriculum design. Higher Education, 83, 57-77. https://doi.org/10.1007/s10734-020-00632-0

---