Developing computer‑assisted formative feedback in the light of resource theory: a case on heat concept

DEVELOPING COMPUTER-ASSISTED FORMATIVE FEEDBACK
IN THE LIGHT OF RESOURCE THEORY: A CASE ON HEAT CONCEPT

Ahmad Suryadi , Sentot Kusairi

Department of Physics, Faculty of Mathematics and Natural Sciences, Universitas Negeri Malang (Indonesia)

Received September 2020

Accepted April 2021

Abstract

Every student needs feedback to understand a scientific concept deeply. However, with a large number of students in each class, individual feedback is a challenge for teachers in Indonesia. Although there have been many studies conducted to maximize the giving of personal feedback, studies conducted by ensuring the consistency of a student’s knowledge of a concept before providing feedback are rarely disclosed. This study aims to develop computer-assisted formative feedback (CAFF) based on students’ cognitive resources. Besides, this study also intends to determine student perceptions related to the use of CAFF. This current study involved two experts as advisers in the development process and 31 high school students who acted as participants in the pilot phase. CAFF could identify the students’ cognitive resources and then provided interactive feedback to students based on it. Therefore, it will be beneficial to associate students’ prior knowledge with scientific concepts. Instead of giving feedback by showing whether or not students’ answers are correct, displaying the exact concept text, or providing general instructions to the right concept, CAFF presents feedback by showing students’ cognitive resources then followed by the interactive explanation associated with specific cognitive resources. Students shared positive perceptions about the program. Cognitive resource diagnoses and forms of feedback in the CAFF model can be further developed on more nuanced topics.

 

Keywords – Heat, Formative feedback, Computer, Resource theory.

To cite this article:

Suryadi, A., & Kusairi, S. (2021). Developing computer‑assisted formative feedback in the light of resource theory: A case on heat concept. Journal of Technology and Science Education, 11(2), 343-356. https://doi.org/10.3926/jotse.1100

 

----------

    1. 1. Introduction

The issue of feedback in learning has received considerable critical attention. There have been several related studies about the importance of feedback in some subjects, for example, in Mathematics (García‑López & García-Mazarío, 2016), Biology (Lookadoo, Bostwick, Ralston, Elizondo, Wilson, Shaw et al., 2017), Chemistry (Van Horne, Curran, Smith, VanBuren, Zahrieh, Larsen et al., 2018), English academic writing (Ma, 2018), Physics (Đorić, Lambić & Jovanović, 2019), and other subjects. Some studies showed that providing feedback is essential to motivate students to learn (Li, Wong, Yang & Bell, 2020; Nurfitri, Rozimela & Refnaldi, 2019). Furthermore, feedback not only can improve the quality of learning but also it can inform what students need and what students have to do (Clarke & Boud, 2018; Nahadi, Firman & Farina, 2015; Van der Schaaf, Donkers, Slof, Moonen-van Loon, van Tartwijk, Driessen et al. 2017). Nahadi, Firman and Farina (2015) reported that students became enthusiastic, motivated, and more active in learning after being given feedback. As a result, feedback would make students learn a topic in-depth (Fui & Lian, 2018).

Despite the vital role of feedback, teachers often face specific difficulties when giving students feedback. One of the problems is related to the nuance of students’ behavior. According to Carvalho, Santos, Conboy and Martins (2014), the ways of students receive feedback depend on their individual and situational characteristics. Therefore, teachers need to consider this situation when they want to provide feedback because the interaction between students and teachers is vital in the learning process (Asikainen, Blomster & Virtanen, 2018; Nguyen, Cannata & Miller, 2018). Specifically, in Indonesia, a significant challenge in presenting feedback is in terms of the number of students in each class. The number of students in each class is relatively large, so the teacher will find it difficult if they have to interact with the students individually. Therefore, it needs an effective way to deliver appropriate feedback to a large number of students.

In response to the challenges mentioned above, media tools are needed to provide feedback. Goldin, Narciss, Foltz and Bauer (2017) revealed Information technology gives a wide variety of feedback types and strategies to be provided in digital learning atmospheres. Information technology allows support of students’ retention (Nottis, Prince, Vigeant, Nelson & Hartsock, 2009), students’ motivation (Anita, Darmawan & Kartika, 2017), and students’ interest (Maeng, 2017). By using a computer, many researchers believed that feedback could be delivered personally and widely (Irianti, Dharmono & Mahrudin, 2020; Kusairi, 2020; Maier, Wolf & Randler, 2016).

There have been several studies about the ways to provide feedback in learning. Bhagat, Subheesh, Bhattacharya and Chang (2017) recommended that the optimization of feedback with the help of computers would improve students’ conceptual understanding. A study by Van der Schaaf, Donkers, Slof, Moonen-van Loon, van Tartwijk, Driessen et al. (2017) found that the use of e-portfolios increases the quality of personal feedback, which, in turn, affects students’ competency. Afify (2018) also identified the benefit of mobile devices to provide feedback immediately to improve students’ learning outcomes and attitudes. In a similar vein, Kusairi (2020) used a web-based isomorphic test to provide immediate feedback to the students. Most studies that have been conducted provide feedback by utilizing technological advances. However, feedback presented by associating the consistency of students’ pieces of knowledge and the phenomenon explained is still rarely done.

Students’ needs in learning can be based on prior conceptual understanding. One theory that explains the benefits of students’ previous understanding of learning is resource theory (Hammer, 2000). According to resource theory, students’ understanding is not always wrong (Hammer & Elby, 2003), but it is very context-dependent (Richards, Jones & Etkina, 2018) and productive (diSessa, 2018). Resource theory believes that the learning process should be directed at aligning students’ prior understanding with the appropriate context. For instance, Hammer and Elby (2003) reported that students activate different resources when reasoning about heat phenomena in nuance conditions. This view is different from misconception theory. Misconception theory does not allow a student to have a different explanation with scientists (Docktor & Mestre, 2014). Therefore, the student’s misconceptions should be deleted and replaced with scientific concepts (Wood, Galloway, Hardy & Sinclair, 2014). However, some research showed that replacing students’ misconceptions with scientific theories is very difficult (Afra, Osta & Zoubeir, 2009; Başer, 2006; diSessa, 2018).

In Physics, heat is a crucial concept for students. Heat is a topic taught in senior high school in the Indonesian context. The primary competency of this topic is students must possess the ability to analyze the effects of heat in everyday life. A student learns this topic from elementary school until senior high school. However, many studies showed that students fail to understand heat concepts (Alwan, 2011; Coca, 2013; Silung, Kusairi & Zulaikah, 2017). For example, students sometimes have difficulty in understanding heat equations/formulas (Silung et al., 2017). In addition, they also often use inconsistent explanations for the same phenomenon (Yeo & Zadnik, 2001). As such, students need guidance to associate their prior understanding with a scientific phenomenon that they explain.

Based on the aforementioned evidence, CAFF in physics learning should be developed by using resource theory. This study aims to develop CAFF based on resource theory. This current study also aims to know students’ perceptions on the use of CAFF.

2. Theoretical Background and Literature Review

2.1. Resource Theory

Student prior knowledge is an important foundation of the learning process. One theory that explains the use of students’ initial understanding of learning is resource theory (Hammer, 2000). According to resource theory, knowledge of students consists of chunks of smaller grain sizes, which are not necessarily accumulated into a more significant concept (Docktor & Mestre, 2014). This grain size knowledge of students is always related to the scientific phenomenon in their daily life. This theory has the potential to understand how students think and learn (Gire, Manogue, Henderson, Sabella & Hsu, 2008). Resources in concept understanding theory are seen as pieces of fundamental knowledge that can be activated alone or together with other resources used by students to reason on certain physical problems (Richards et al., 2018).

Resources theory was born because of the inconsistency of misconceptions in explaining student reasoning in various contexts. Misconception theory views that students’ knowledge that is not following the concepts of scientists must be eliminated. Meanwhile, resource theory sees students’ knowledge to be dynamic and should be adjusted to the context not completely eliminated. Students activate several resources when reasoning about certain phenomena (Harrer, Flood & Wittmann, 2013). For example, Hammer and Elby (2003) provided a description related to students’ understanding when given problems about pieces of ice wrapped in cloth. Some students activate the resource that “softness means warm” by associating it with materials such as jackets and gloves that feel warm when worn. With this resource, students understand that ice wrapped in a cloth quickly melts. Some students activate the “blocking” resource because it connects this problem with the use of cloth when handling a hot pan.

Several studies related to student cognitive resources have been conducted. Loverude (2015) conducted a case study of 8 participants and found that students do not experience consistent difficulties. Reasoning elements such as conservation are productively used in one context but not in another context. Sabo, Goodhew and Robertson (2016) mapped student resources on energy concepts through open-ended questions. Each question was made in two formats, story format, and diagram format. They found five themes related to student resources in the concept of energy, which is considered capable of contributing to the teacher’s pedagogical content knowledge. In addition, in a more complex matter, solar cells Richards et al. (2018) investigated how students combine various elements of resources in explaining a phenomenon. They found that the student explanation scheme for the concept of solar cells is composed of associations of various types of resources. As a relatively new topic, the implementation of resource theory is ‘traditionally’ implemented in the classroom

In short, resource theory is a relatively recent approach when it comes to considering a change in the concept of students. While not as common as the misconception theory, some evidence suggests that this paradigm is capable of providing an understanding of how students face difficulties in scientifically describing a phenomenon. Until now, studies have typically been carried out in which the teacher directly assists the cognitive resources of the students. The use of this frame in the context of aided computers is still rare.

2.2. Formative Feedback

Feedback is a fundamental part of pedagogy theory (Handley & Williams, 2011). Providing feedback that suits students’ needs is a challenge for a teacher. Feedback not only forms as an assessment of student performance but also contains a description of the things students need (Sardareh, 2016). Formative feedback has a corrective purpose, which means that it should (a) validate whether the student’s answer is correct or incorrect and (b) provide the learner with knowledge about the correct response (either instructive or facilitative) in its simplest form (Shute, 2007). Formative feedback can be presented in several ways, such as peer-peer formative feedback (Gikandi & Morrow, 2016), formative feedback based on multi-tier questions (Maier et al., 2016), or template-based natural language generation mechanism feedback (Perikos, Grivokostopoulou & Hatzilygeroudis, 2017).

Gikandi and Morrow (2016) conducted a case study within a New Zealand university. The study included 17 people; some data was gathered using web analyses, recorded public discussions, and interviews. The study sought to determine how peer-peer formative feedback in an online course was facilitated. The results suggest formative peer feedback encourages the involvement and constructive interaction of successful learners. Their results also confirm that mechanisms for dialogic peer formative feedback support help for learning and self-regulation. In our view, this research performed at university has ample knowledge of a subject so that other students can comment on their peer works and provide feedback.

Maier et al. (2016) stated that the effect of feedback on multi-tier questions in boosting learning performance is unclear. On evolutionary adaptations, they developed computer-assisted formative experiments for a teaching unit. They implement three kinds of feedback. The first type has feedback in every item and conceptual explanation after the all multi-tier answered. The second type provides feedback in every multi-tier item without reason at the end of the multi-tier question set, and the third without feedback. Their study revealed that the interpretation and use of feedback by the students is related to intrinsic motivation and self-reported ratings. Feedback on the study comes from a variety of responses to multi-tier questions. However, the student uprising identified does not necessarily be transferred to the context of a similar problem, since each tier diagnoses a different component of the issue.

Perikos et al. (2017) mentioned that logic is a central topic of a course on Artificial Intelligence (AI), which contains a range of sub-themes. One of the challenges is to convert natural language sentences into the formulation of first-order logic (FOL). They designed and fitted the NLtoFOL program with a powerful assist and feedback framework. The mechanism can provide assistance if requested before a response is submitted. The student feedback assessment shows that complete feedback sequences lead to higher learning gains than sequences consisting of flag input and bottom-out hints only (n = 226), and that standardized, template-based input sequences are comparable to the usefulness of problem-specific hints provided by human tutors (n = 120).

In particular, Oker, Pecune and Declercq (2020) tested human-computer interactions with 22 kids using an embodied conversational agent platform. Children did some numeracy tasks that were provided by two separate simulated officers. One participant received only verbal input, the other mixed facial gestures and verbal feedback focused on actual muscle contractions. Children considered the bimodal agent more empathic and provided much more correct responses. For consistency, there is a strong association between accuracy and mean reaction times.

In conclusion, formative feedback can help students understand a concept. Nevertheless, the model of providing feedback is still a challenge. The use of computers in presenting feedback is offered as a solution by several researchers. Nevertheless, the studies that exist give the most feedback after students answer incorrectly on just one question. Feedback does not consider the consistency of answering students, so there is still a possibility that the feedback given is not following what students need. Therefore, the diagnostic question model developed in this study is an isomorphic question model in which two questions have different contexts but require the same conceptual understanding in answering them.

3. Method

3.1. Procedures

The procedure in this study was divided into two parts, namely, need analysis and formative feedback construction. We defined the terms of the learning in the first process, need analysis, which covers the rationale of learning problems, student characteristics, task characteristics, content, and goal setting. We also performed a literature review to collect information on the cognitive resources of the students in the context of heat. Based on the nuance cognitive resources list, we confirmed the list by conducting an interview with a small sample subject. All the information through this process used in the next process. The next step was constructing the CAFF. In this process, we developed the CAFF and tested it for senior high school students. The CAFF construction process was started by determining the contents, methods, and approaches that will be applied. After the prototype had been done, we offered the product to two experts. All input from experts was followed to enhance product quality. The product was then tested to senior high school students. The last step was the disseminating stage, where the CAFF was presented as one of the dissemination sections in an educational forum of researchers. The forum discussed educational research on learning multimedia development, which was held on 27-28 August 2019.

3.2. Participants

Two senior lecturers and 31 senior high school students (aged 16-17 years) agreed to participate in this research. The two senior lecturers recently undertook research on multimedia and resources. During the development stage, researchers discussed all matters with the experts. According to Merriam and Tisdell (2015), peer review and expert judgment maintain the trustworthiness of the study. High school students also participated during the CAFF field study. When they participated, the students enrolled in temperature and heat subject.

3.3. Data Collection

Data were collected in the form of qualitative data and quantitative data. The qualitative data were obtained based on the results of the literature study and the multimedia validation by two experts. First, a literature study was conducted to gather some information about alternative conceptions about heat concept. These alternative conceptions from previous research were considered in developing some questions with appropriate feedback. Second, a qualitative data source was the validation of two experts. The data was used to determine the lack of multimedia that had been developed. We also interviewed some students about their perceptions of multimedia. All of the suggestions from the experts were used to improve the quality of the product.

Quantitative data was collected to know student perception. The data was obtained from a questionnaire by measuring the students’ perceptions of multimedia. The instrument used was a questionnaire with a Likert scale (strongly agree, agree, have not decided, disagree, and strongly disagree). The questionnaire contains three aspects, namely the conceptual understanding, the multimedia display, and the usefulness in learning. First, the questionnaire would provide information about students’ perceptions related to the feedback content understanding. Second, the questionnaire item also informed about students’ knowledge of multimedia display (such as font style, color, multimedia button, so forth). Finally, some questionnaire items informed about students’ perceptions of how multimedia helps the students to learn about the heat concept.

4. Results and Discussion

4.1. Need Analysis

The first step was carried out by a needs analysis. The topic of temperature and heat was provided for 11 grade senior high schools in the 2013 Curriculum of Indonesia. In the Indonesian curriculum, the basic competency related to the cognitive domain that students should have is “analyzing the effect of heat and heat transfer, which includes the thermal characteristics of a material, capacity, and heat conductivity in daily life.” Some studies showed that students often have difficulties in learning temperature and heat material (Coca, 2013; Hitt & Townsend, 2015; Silung et al., 2017; Yeo & Zadnik, 2001) even though they have studied it in formal schools. The problem at a formal school is limited learning time. Therefore, the limited interaction between student and teacher affects the student’s understanding. Students need additional interactive learning material. These learning resources are not only accessible at school but can also be accessed at home independently. One solution that can be applied is multimedia.

4.2. Construction of Interactive Multimedia

The next step was to build multimedia in the light of cognitive resource theory. The multimedia design was started with determining learning objectives, determining the multimedia content, and designing the computer system to be used. The design of multimedia development based on resource theory is presented in Figure 1.

 

Figure 1. Design of interactive feedback through multimedia based on resource theory

The multimedia displays a home page first. It was created to attract attention and provide instructions regarding multimedia use. Furthermore, the identification of students’ cognitive resources was using isomorphic question - two multiple-choice questions with the same indicator (see Figure 2). After the student resource was identified, the multimedia provides feedback according to the students’ cognitive resources. At the same time, the cognitive resource represents a fragment of knowledge that is possibly activated alone or with other resources when the student was reasoning about physical phenomena (Richards et al., 2018). Every cognitive resource has a different explanation or feedback. The feedback was developed based on cognitive resource characteristics. The last phase was the confirmation of students’ understanding through a quiz. The learning process in multimedia is finished when students successfully answered the quiz correctly. If they make wrong answers, the multimedia informs students to repeat the learning process from the beginning.

Figure 2 shows a diagram of the student resource identification system. Each answer option has a code based on the type of cognitive resources - code A refers to option A, code B covers to option B, and code C refers to option C. Students who choose the same code on the two questions will be identified as the students’ cognitive resource, and the students proceed to the next stage. On the other hand, students who choose the different codes will be identified as a non-consistent category, which will automatically be asked to do a retest.

 

Figure 2. The identification stages of students’ cognitive resources

At the developing stage, a cognitive resource identification instrument was developed with two multiple-choice tests. The items used at the identification stage can be seen in Figure 3.

 

Figure 3. Display questions related to students’ cognitive resource identification

Students who choose A or C in the first question and choose A in the second question are identified using softness means warmth resource. This resource says that students think the soft material feels warm (Hammer & Elby, 2003). Students who chose D in the first question and chose B in the second question are identified using blocking resources. This resource means that students perceive that material blocks the heat transfer process (diSessa, 1993).

There are four possibilities of cognitive resource identification carried out, namely: softness (warm), blocking, conduction concept, and non-consistent. Students get feedback on the topic following the characteristics of their cognitive resources. The example of the feedback is shown in Figure 4.

The concept of heat is learned by students at some stages. First, the feedback is provided to the students by showing their cognitive resources - what students understand. Second, students are then guided by questions interactively and sequentially. Through the text and video feedback, students learn the heat concept. Finally, students take a quiz to check their understanding related to the heat concept.

 

(a)

 

(b)