Journal of Technology and Science Education

• Home
• About
• Login
• Archives
• Submissions
• Publication Fee
• Indexing & Statistics

Home > Vol 11, No 2 (2021) > Ayub

THE IMPACT OF DEVELOPING A BLENDED LEARNING SUB-SYSTEM ON STUDENTS ONLINE LEARNING ENGAGEMENT

Faculty of Information Technology, Maranatha Christian University (Indonesia)

Received December 2020

Accepted June 2021

Abstract

In this research, we show a development process of engagement sub-systems in a blended-learning management system and evaluate the impact of student interaction in the whole system. We develop special sub-systems for engagement purposes via forum, course rating, and class assignment modules. During the system development process, we employ continuous improvement methodology which helps to shorten the software delivery time without disturbing the overall operation. We evaluate the impact of engagement processes in terms of behavioral, emotional and cognitive aspects. Our evaluation results show that by employing the engagement sub-systems we have increased a 0.30 satisfaction point on average (1-5 Likert scale) for 11 evaluation survey questions distributed to 305 students during 2 times evaluation period. Another interesting finding from the surveys is that behavioral (discussion forum and attendance list sub-system) and cognitive (course rating sub-system) aspects have great influences for the students’ activities (class assignment sub-system) which finally has a great impact on their cognitive performances.

 

Keywords – Online learning, Learning course management system, Continuous improvement, Software development, Student engagement.

To cite this article:

Ayub, M., Toba, H., Wijanto, M.C., Parsaoran, R., Sani, A., & Hernita, Y.T. (2021). The impact of developing a blended learning sub-system on students online learning engagement. Journal of Technology and Science Education, 11(2), 556-568. https://doi.org/10.3926/jotse.1196

 

----------

1. 1. 1. Introduction

In a blended learning management system, students’ interactions are very important (Purarjomandlangrudi, Chen & Nguyen, 2016). To ensure the interactions between students and lecturers, special features need to be developed in the learning management system, such as course rating, discussion forum, and class assignment. In this way, lecturers can organize activities which will attract students during the learning process. On the other hand, the interactive facilities need to be fitted with the students’ abilities and adapting to their needs. These are also in line with strategies to improve student engagement in learning. Research by Taylor and Parsons (2011) defines the strategies into six categories: (1) Interaction, (2) Exploration, (3) Relevancy, (4) Multimedia & Technology, (5) Engaging & Challenging Instruction, and (6) Authentic Assessment for Learning. The discussion forum is an example of interaction, exploration, and engaging & challenging instruction. With discussion forums, students spend more time on peer interaction (Roll, Macfadyen & Sandilands, 2015). The course rating is also a support of interaction, exploration, relevancy, and engaging & challenging instruction. For the class assignment, this is an example which applies all of the strategies.

This research explores the impact of engagement sub-systems development of an online learning management system at the Faculty of Information Technology, Maranatha Christian University. This system is focused on a blended-learning ability which combines the face-to-face (synchronous process) and asynchronous processes. During this pandemic period, the synchronous process is mainly done by online meetings, and thus the context of blended learning could also be defined as the combination of synchronous and asynchronous processes in online activities entirely. The main research question is: what is the impact of an engagement sub-system for the student’s online behavioral, emotional, and cognitive aspects? In this research, we explore a quasi-experimental approach based on the quality evaluation strategy of technology-mediated learning (Henrie, Halverson & Graham, 2015) and evaluate students’ performances before and after the engagement sub-system is implemented in our online learning management system.

To show the students’ engagement process in our learning management system, we incorporate the quality evaluation strategy of technology-mediated learning (TML) as proposed in (Söllner, Bitzer, Janson & Leimeister, 2018). We adapt the process and structural quality measurements in our surveys, which are focused on these following aspects: IT process support, interactivity, learning group, quality of exercises, learning environment, and learning materials. Further, we also evaluate the impact of engagement processes in terms of behavioral, emotional, and cognitive aspects (Fredricks, Blumenfeld & Paris, 2004; Henrie et al., 2015). Behavioral engagement covers the behaviors to gain success in academics, such as participation, attendance, and task completion. Emotional engagement covers feelings from learning experience, such as interest, boredom, or discouragement. Cognitive engagement covers efforts to understand the material.

2. Research Methodology

2.1. Theoretical Underpinning and Contributions

We incorporate the quality evaluation strategy of technology drivers, known as technology-mediated learning (TML) system as proposed in (Söllner et al., 2018) to evaluate the students’ engagement process in our learning management system. We adapt the process and structural quality measurements in our surveys, which focus on these following aspects: IT process support, interactivity, learning group, quality of exercises, learning environment, and learning materials. These aspects are incorporated into our course learning system (CLS) engagement sub-systems development process.

Further, we also evaluate the impact of the engagement processes in terms of behavioral, emotional, and cognitive aspects (Fredricks et al., 2004; Henrie et al., 2015). Behavioral engagement covers the behaviors to gain success in academics, such as participation, attendance, and task completion. Emotional engagement covers feelings from learning experience, such as interest, boredom, or discouragement. Cognitive engagement covers efforts to understand the material. Our main contribution and novelty in this research is to show how the student engagement aspects are implemented and evaluated in a CLS which also incorporates technological quality strategies. Another contribution is to recommend practical implications based on our experiences which extend our recent approach in blended-learning with collaborative strategies.

2.2. Engagement Sub-Systems

The whole engagement sub-systems development process is executed in a combination of classical (waterfall) and agile software engineering methods (see Figure 1). The waterfall approach is completed during the initial development of the system in the year 2017 until the first semester of 2020. The main features during the initiation of development can be followed in our previous publications (Ayub, Toba, Wijanto, Yong & Wijaya, 2019). (Ayub, Toba, Wijanto & Yong, 2017), which are tournament and leader board features. In the midst of 2020, the system is enhanced with special features dedicated to facilitating students’ engagement, namely: Course Rating, Forum, and Class Assignment.

The main difference between these new facilities and the previously developed ones (Ayub et al., 2019; Ayub et al., 2017), is the ability of students to give suggestions directly during an online session which is supervised by two dedicated programmers to ensure the development phases. The programmers are also involved in the classroom as teaching assistants. In this way, it is expected that suggestions from the users will be developed, tested, released, and tuned in time (Barricelli, Cassano, Fogli & Piccinno, 2019; Barcellos, 2020). Especially, during this pandemic period, all activities are prepared and executed via online infrastructures, and thus rapid improvement of the whole system is needed to facilitate learning activities. Continuous improvement methodology as a state of the art in software engineering and development is used during the development. Together with the Continuous Integration/Continuous Delivery (CI/CD), they form the underlying concepts in the agile principle in modern software engineering (Demchenko, Surbiryala, Koulouzis, Shi, Liao & Gordiyenko, 2019). Important phases in CI/CD implementation are as follows: plan and measure, develop and test, release and deploy, and finally track and fine-tune (Arachchi & Perera, 2018). All the phases need to be completed in multi-cycle implementation. This will say that in each cycle, there are some software aspects that need to be improved.

The development of the sub-systems, testing phase, and deployment are executed in a GitHub environment following the user enhancements discussed during an online class or directly via the lecturer involved in the class. After the testing phase is acceptable during the agile sprint, the system in production will be updated (Arachchi & Perera, 2018; Monteiro, Lima, Venturini & de Oliveira, 2018; Demchenko et al., 2019). The sprint period from the planning phase until fine-tuning is set for 7 days which gives an ideal situation to evaluate the system in a weekly online session.

Figure 1. The overall methodology during the development of engagement sub-systems

Course Rating is a feature to evaluate each learning session in a course. First, the lecturer sets up a syllabus of a course for one semester. Using this feature, the students will be able to submit commentaries and their perceptions about the lecture contents for each learning session. The perceptions are entered by answering some questions about understanding of given material using a Likert scale. The lecturer is able to view the evaluation of the lectures by learning sessions, or by courses. Figure 2 shows the use case of the engagement Course Rating sub-system.

The forum feature allows students and lecturers to interact by posting comments or ideas about a topic. Discussion in the forum is started by the lecturer with posting a topic as a part of a thread. There is a feature to set up discussion groups for project-based learning, so the lecturer can evaluate how coordination in a project group is performed. Figure 3 shows an implementation example of a discussion forum between lecturer and students about a class assignment. Important issues are also shown by using a word cloud representation in each forum. The word cloud kept students’ focus on important concepts and provided topic-specific knowledge for students to search more with the Internet (Xie & Lin, 2018).

Figure 2. Use case design of interaction between lecturers, students, and faculty management
(PJS actor in the figure) in the course rating sub-system

Figure 3. An example of interactions in a discussion forum and important issues are represented as word clouds

The assignment is a feature to maintain assignments given in a course. In this feature, lecturers can give assignments to students and assess the assignment. Students can also download the assignment and submit the task through this feature.

Another interesting feature to ensure students’ engagement is by integrating the attendance list with the forum, course rating and class assignment. Each student who submits their assignment, participates in a discussion or gives a rating for a particular course content will be automatically recorded as an active student and included in the attendance list. These processes are repeated for all planned (syllabus) of a course. The students can view their attendance in courses as well as the lecturers. Forum and Course Rating have also the typical communication facilities between students, lecturers, and the faculty management team (written as PJS in the use case).

Figure 4 shows an example of the interaction between students and lecturers via the course rating interface. In each class session, students are encouraged to fill in their comments about the lecturer and assignment. The comments are submitted as a Likert-scaled survey, valued between 1-4. After the students fill in the comments, lecturers can view the perceptual opinion of a learning session. These opinions would be essential for the lecturers to improve the quality of the next sessions. We also anticipate that a lecturer cannot view specific opinions from a student, and thus can only view the overall performance shown as a bar chart.

An important aspect of learning activities is by giving feedback for submitted assignments. In our implementation, students as well as lecturers can provide comments for each submitted assignment. It is also possible to promote the comments to be viewed in the general forum discussion if the topic is considered important by the lecturer. By this mechanism, the whole class will have the opportunity to learn something from each other. The implementation of the assignment sub-system can be seen in Figure 5.

Figure 4. An example of interactions in course rating. Students can give opinions about a session and lecturers can view the opinions in a bar chart

Figure 5. Assignment sub-system which provides the possibilities of interaction by using comments as feedback to students and extending it as a discussion topic in a general discussion in a forum

2.3. System Evaluation

The evaluation of the system was conducted in two periods in Informatics undergraduate program study in Maranatha Christian University. The first period was in the short semester in the mid-term of 2020 (July – August 2020), involving 101 students from three courses as shown in Table 1. The second period was in the odd semester of 2020 (September – November 2020), involving 204 students from five courses as shown in Table 1. The participants of short semester utilized the system in eight weeks. During odd semester, utilization of the system was in seven weeks. The sampling method is a quasi-experiment.

Measuring student engagement in blended learning commonly uses surveys that apply quantitative items (e.g., Likert scale) and completed by students (Henrie et al., 2015). Quantitative self-report assesses the quality of students’ experience, such as participation in projects, and involvement in learning activities. Besides, quantitative observational measures assess the level of students’ engagement in learning, such as attendance, time spent doing the task, and task completion. Behavioral engagement can be performed through discussion forum, class assignment, and course rating. In discussion forum, student participation is shown by creating a word cloud or as a proof of attendance. In class assignment, a task submission can become a proof of attendance, as well as filling perception of a lecture in course rating. Cognitive engagement can be measured in course rating by evaluating the perception from students about a learning session (Fredricks et al., 2004).Lecturer provides comments as learning feedback for student work.

Semester	Course	#participants
		Male	Female	Total
Short	Ethics	34	8	42
Short	Statistics	21	5	26
Short	Advanced Web programming	32	1	33
Odd	Software engineering	43	5	48
Odd	Data base	22	4	26
Odd	Advanced Web programming	28	5	33
Odd	Strategic Algorithms	37	10	47
Odd	Computational Thinking	43	7	50

Table 1. The participants of the study

Number	Engagement Aspects (Henrie et al., 2015)	TML Aspects (Söllner et al., 2018)	Statement
S1	Emotional	IT process support	View of the home page is interesting.
S2	Emotional	IT process support	Courses page can be accessed from home page easily.
S3	Behavioral	Interactivity	Notification of Forum and Assignment features to remind the students about the task is helpful.
S4	Behavioral	Interactivity	Menu choices of the system can be access easily.
S5	Cognitive	Quality of exercise	Tournament feature for a quiz can be used easily.
S6	Behavioral	Interactivity	Feature for recording student’s attendance in Forum can be used easily.
S7	Cognitive	Learning materials	The syllabus of a course can be accessed easily.
S8	Cognitive	Learning materials	Course Rating feature facilitates the student to give an evaluation of a learning (lecture) session.
S9	Cognitive	Quality of exercise	Assignment feature to download assignment file can be accessed easily.
S10	Cognitive	Quality of exercise	Assignment feature to upload file can be accessed easily.
S11	Emotional	Learning group	Discussion feature in Forum can support students to understand lecture materials based on a textual conversation between lecturers and students.
S12	Behavioral	Learning environment	Choose 3 features which are very helpful.
S13	Behavioral	Learning environment	Choose the best features.
S14	Behavioral	Learning environment	Choose 3 features which are less helpful.
S15	Behavioral	Learning environment	Choose the worst features.
S16	Emotional	Learning environment	Write your suggestions to enhance the overall system.

Table 2. Questionnaires of the evaluation surveys

To evaluate utilization of the system, we also supervised a number of periodic questionnaires, i.e., during the midterm of the short semester in August 2020, and the midterm of the odd semester in November 2020. The detail of the questionnaires can be seen in Table 2. Questionnaire S1 to S11 use a 5-points Likert-scale, 1 for strongly disagree, 2 for disagree, 3 for neutral, 4 for agree, 5 for strongly agree. Questionnaire S12 to S15 are multiple-choice, and S16 is an open question. The main objective of S1 to S11 is to determine how related the facilitates to ensure students’ engagement and the perceived quality of the developed learning management system (Henrie, et al., 2015; Söllner et al., 2018). The other objective of S1 to S11 is to evaluate students’ satisfaction with the system. S12 and S13 is to select the most helpful feature, while S14 and S15 is to choose the most difficult feature. S16 is to obtain suggestions to improve the system.

Based on those questionnaires, we evaluate the most influential features and how significantly the system improved during evaluation in the short and odd semester periods. The influential features will show the preferred students’ engagement features, and the system’s improvement will show the effectiveness and qualities of the features observed by the students (Söllner et al., 2018). The features’ influence is measured by using Pearson Correlation Coefficient based on the Likert-scale from the surveys (Yang, Zhong & Xu, 2018), which formula is given in (1).

(1)

where

Pearson (xi, yi) = Pearson correlation coefficient of two variables x and y

xi = values of the Likert survey results from students as x-variable

xbar = mean of the Likert survey results from students as x-variable

yi = values of Likert survey results from students as y-variable

ybar = mean of the Likert survey results from students as y-variable

3. Result and Discussion

3.1. General Result Discussion

In this section, evaluation of the system as a whole is described. Table 3 gives the percentage of the quasi‑experimental responses during the short and odd semester. The percentage of students involved during the short semester is much higher than the regular odd semester. This result indicates that more students are highly involved during the short semester and they have more attention to the assignments and activities. During the short semester, the lecture sessions are also more intensive, twice a week, compared to the regular semester.

Semester	#Sent	#Received	Percentage
Short	101	89	88%
Odd	204	115	56%

Table 3. The percentage of questionnaires responses

In Table 4, the students’ attendance and pass rate percentage are given. Those percentages labeled by ‘2020’, are the impact observed during the implementation of the engagement sub-systems. Computational Thinking is a new subject starting from the 2020 academic year. The data shown have insignificant differences based on the results of the two-tailed test, it can be seen from the P-value for attendance and the pass rate which is more than 0.05 (common significance level). From Table 4 we also can infer how the behavioral engagement (attendance) in most of the subjects is increased slightly from an average of 92.57% to 94.5% and the pass rates involved in cognitive engagement also rose considerably from 77.86% to 91.25% on average.

An exceptional case occurs in ‘Advanced Web Programming’ subject. Research by (Wijanto, Karnalim, Ayub, Toba & Tan, 2021) showed that programming subject was the most difficult subject, especially in an online environment. By further investigation, the students are more interested in assignments (practical sessions) than attending the lecture sessions. This results in line with the cognitive engagement that the students have shown in the course rating sub-system. The students in this subject preferred to have discussions (by using online forums) than lectures. We also compare the pass rate percentage before and after the engagement sub-systems are implemented. The engagement sub-systems are highly appreciated by the students and facilitate interactions between lecturers and students. In all subjects that we observe, the sub-systems provide the facilities to increase the pass rate percentage.

Semester	Subject	Attendance (%)		Pass Rate (%)
		2019	2020	2019	2020
Short	Ethics	100	100	100	100
Short	Statistics	100	100	84	92
Short	Advanced Web programming	87	82	57	82
Odd	Software engineering	96	98	96	98
Odd	Database	90	96	86	96
Odd	Advanced Web programming	86	82	48	70
Odd	Strategic Algorithms	89	96	74	96
Odd	Computational Thinking	-	100	-	96
Average		92.57	94.25	77.86	91.25
P-Value for two-tailed test		0.65		0.11

Table 4. Comparison of attendance and pass rate percentage of the courses during the study

We are using the Pearson Correlation Coefficient in (1) to measure the students’ engagement and the perceived quality of the developed engagement sub-systems (Söllner et al., 2018). The top-10 selected correlation rank of those aspects can be seen in Table 5.

From the results in Table 5, we can infer that to ensure students’ engagement at least three aspects need to be maintained during an online session throughout a semester, namely: attendance list, course rating, and assignment. To support those aspects, some interactive facilities need to be provided, namely: notification and forum discussion. The overall correlation rank of the questionnaire can be seen in Figure 6.

Alongside the students’ engagement evaluation, we also evaluate the students’ satisfaction with the system. In Figure 7, survey results of the short semester are compared to survey results of the odd semester. Among the given statements, in the short semester, S4 and S8 have the highest value (4.10 of 5). The students thought that the menu choices and course rating features are helpful features. The lowest value (3.47 of 5) is resulted by S10, the vast majority of students encountered difficulty in assignment feature.

Correlation	Description	Coefficient	Engagement Aspects	Technology Driver
S6-S8	Attendance list-course rating	0.75	Behavioral-Cognitive	Interactivity
S10-S11	Assignment-discussion	0.70	Cognitive-Emotional	Quality of exercise
S3-S10	Notification-assignment	0.67	Behavioral-Cognitive	Interactivity
S3-S11	Notification-discussion	0.54	Behavioral-Emotional	Interactivity
S8-S10	Course rating-assignment	0.47	Cognitive-Cognitive	Learning materials
S6-S10	Attendance list-assignment	0.44	Behavioral-Cognitive	Interactivity
S3-S8	Notification-course rating	0.43	Behavioral-Cognitive	Interactivity
S6-S11	Attendance list-discussion	0.41	Behavioral-Emotional	Interactivity
S3-S6	Notification-attendance list	0.40	Behavioral-Behavioral	Interactivity
S8-S11	Course rating-discussion	0.35	Cognitive-Emotional	Learning materials

Table 5. Top-10 selected correlation rank

Figure 6. The complete correlation rank of S1-S11. Correlation coefficients are replaced
by symbols according to the value (darker dots show stronger correlations)

Figure 7. Mean scores of S1 to S11

In the odd semester, the highest value (4.37 of 5) is obtained by S8. Same as in the short semester, students argued that the course rating features are helpful. Most of the students gave the lowest value (3.80 of 5) for S3, students thought that the notification feature is not very helpful. When comparing all of the statements (S1 to S11), the survey results of the odd semester are higher than the survey results of the short semester. Feature improvements that were performed after the short semester have increased the performance and usefulness of the system.

Figure 8. Rank of helpful features

The result of S12 to S15 can be seen in Figure 8 and 9. Figure 8 shows the ranking of the best feature as survey results. The chosen best feature in the short semester is the attendance list input through the forum (51%), while in the odd semester, the best feature is the course rating (49%). Feature improvement in course rating has enhanced the ease and usability of the whole system as can be seen in Figure 8.

Figure 9 shows the ranking of the hard features survey results. The chosen hard feature in the short semester is the assignment (30%), while in the odd semester, the hard feature is the tournament for quiz (28%). Feature improvement in the assignment (i.e., students may add comments) increases the ease and usability of the whole system as can be seen in Figure 9.

The overall performance of the system has increased to a 0.30 satisfaction point on average. This result suggests that by interactively involving the students (as users) during an online learning management system development would improve the overall performance of the software itself and at the same time also improve the students’ engagement during online courses.

Figure 9. Rank of hard features

3.2. Findings and Implications

Our research has shown important aspects in students’ engagement and the related technological supporting tools. From our research, the three important aspects of engagement, i.e., behavioral, emotional and cognitive, are all included in the top survey results, seen from the students’ perspectives. In Table 5, we can see that those aspects are dominant for the students during their study. In our engagement sub-systems those features are implemented properly in a number of blended-learning features, such as: discussion forum, attendance list, course rating and class assignment. Those features are really appreciated by the students and support their positive achievements in the subjects (Table 4).

A big challenge is in the programming-based subjects, such as Advance Web Programming, Software Engineering and Database which include conceptual and practical competencies. During our research period, we are experimenting with an integrated collaborative-project based approach for Software Engineering and Database subjects. We have chosen those subjects since the majority of the class members are at the same level (third semester). We created special project assignments which collaborate the contents of the subjects in group discussions of maximum three students. This arrangement gives positive results. It increases the students’ engagement and their passing rate. Further analysis of the developed engagement sub-systems is apparently convinced of this fact. Increased number of students’ engagement in discussion forums and the presence of lecturer / assistant during the online discussion are supporting factors for the students’ success rate.

4. Conclusion and Future Work

In this research, we show the continuous improvement approach during the development of engagement sub-systems in a learning management system. The engagement sub-systems consist of discussion forum, course rating and class assignment. The impact of each of those sub-systems is evaluated in terms of behavioral, emotional and cognitive aspects. An important finding is that behavioral (discussion forum and attendance list sub-system) and cognitive (course rating sub-system) aspects have great influences for the students’ activities (class assignment sub-system) which finally has a great impact on their cognitive performances. This result suggests that in online learning activities, regular and intense interactions between students and lecturers are very important for students’ achievements. For future work we plan to automate the behavioral and cognitive engagement evaluation aspects by employing sentiment analysis and enhancing students’ class assignment in group discussions with automatic workgroup forming using social network approach.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

This research is supported by the Ministry of Research and Technology/National Research and Innovation Agency Republic of Indonesia, Grant Number 8/E1/KPT/2020, 24 January 2020.

References

Arachchi, S.A., & Perera, I. (2018). Continuous Integration and Continuous Delivery Pipeline Automation for Agile Software Project Management. 2018 Moratuwa Engineering Research Conference (MERCon) (156‑161). Moratuwa: IEEE. https://doi.org/10.1109/MERCon.2018.8421965

Ayub, M., Toba, H., Wijanto, M.C., & Yong, S. (2017). Modelling students’ activities in programming subjects through educational data mining. Global Journal of Engineering Education, 19(3), 249-255. https://doi.org/10.1109/ICODSE.2017.8285881

Ayub, M., Toba, H., Wijanto, M.C., Yong, S., & Wijaya, B. (2019). Gamification for blended learning in higher education. World Transactions on Engineering and Technology Education (76-81).

Barcellos, M.P. (2020). Towards a Framework for Continuous Software Engineering. 34th Brazilian Symposium on Software Engineering (626-631). Natal: Association for Computing Machinery.
https://doi.org/10.1145/3422392.3422469

Barricelli, B.R., Cassano, F., Fogli, D., & Piccinno, A. (2019). End-user development, end-user programming and end-user software engineering: A systematic mapping study. Journal of Systems and Software, 149, 101-137. https://doi.org/10.1016/j.jss.2018.11.041

Demchenko, Y.Z., Surbiryala, J., Koulouzis, S., Shi, Z., Liao, X., & Gordiyenko, J. (2019). Teaching DevOps and Cloud Based Software Engineering in University Curricula. 15th International Conference on eScience (eScience) (548-552). IEEE. https://doi.org/10.1109/eScience.2019.00075

Fredricks, J.A., Blumenfeld, P.C., & Paris, A.H. (2004). School engagement: potential of the concept, state of the evidence. Review of Educational Research, 74(1), 59-109. https://doi.org/10.3102/00346543074001059

Henrie, C.R., Halverson, L.R., & Graham, C.R. (2015). Measuring student engagement in technology‑mediated learning: A review. Computers & Education, 90, 36-53. https://doi.org/10.1016/j.compedu.2015.09.005

Monteiro, S.B., Lima, A.C., Venturini, F.C., & de Oliveira, W.S. (2018). Continuous improvement of systems in maintenance using a proactive quality management. 11th International Conference on the Quality of Information and Communications Technology (QUATIC) (47-55). IEEE. https://doi.org/10.1109/QUATIC.2018.00017

Purarjomandlangrudi, A., Chen, D., & Nguyen, A. (2016). Investigating the drivers of student interaction and engagement in online courses: A study of state-of-the-art. Informatics in Education, 15(2), 269-286. https://doi.org/10.15388/infedu.2016.14

Roll, I., Macfadyen, L., & Sandilands, D. (2015). Evaluation the Relationship Between Course Structure, Learner Activity, and Perceived Value of Online Courses. Proceedings of the Second ACM Conference on Learning @ Scale (385-388). Vancouver: Association for Computing Machinery. https://doi.org/10.1145/2724660.2728699

Söllner, M., Bitzer, P., Janson, A., & Leimeister, J.M. (2018). Process is king: Evaluating the performance of technology-mediated learning in vocational software training. Journal of Information Technology, 33(3), 233-253. https://doi.org/10.1057/s41265-017-0046-6

Taylor, L., & Parsons, J. (2011). Improving Student Engagement. Current Issues in Education, 14(1), 1-32.

Wijanto, M.C., Karnalim, O., Ayub, M., Toba, H., & Tan, R. (2021). Transitioning from Offline to Online Learning: Issues from Computing Student Perspective. 2021 IEEE Global Engineering Education Conference (EDUCON) (1150-1155). Vienna, Austria: IEEE. https://doi.org/10.1109/EDUCON46332.2021.9453996

Xie, Y., & Lin, S.Y. (2018). Using word clouds to support students’ knowledge integration from online inquiry: an investigation of the process and outcome. Interactive Learning Environments, 27(4), 478-496. https://doi.org/10.1080/10494820.2018.1484774

Yang, J., Zhong, D., & Xu, Z. (2018). Application of the Pearson’s Method in Model Software Quality Data Analysis. IOP Conference Series Materials Science and Engineering, 439(3), 1-9.
https://doi.org/10.1088/1757-899X/439/3/032076

---