Email this article 
Hide
Show all
QUALITY AND SECURITY AS KEY FACTORS IN THE DEVELOPMENT OF COMPUTER AUDITS IN HIGHER EDUCATION INSTITUTIONS
, Javier Díaz1* , José Jácome2 1Universidad Nacional de La Plata (Argentina)
2Universidad Técnica del Norte (Ecuador)
Received June 2023
Accepted March 2024
Abstract
Higher Education Institutions (HEIs) need a specialized computer audit method to minimize quality and security risks and facilitate institutional evaluation and accreditation. This study aimed to develop a Computer Audit Method for HEIs (MAIIES) providing methodological support for the computer audit process. The MAIIES method includes planning, execution, communication of results, validation, and follow-up of the audit exercise with 47 activities. The validation phase resulted in an evaluation instrument with 42 variables for quality and 18 for security, forming a multivariate model measuring quality and security dimensions. The model comprises factors such as human, technical, contextual, confidentiality, integrity, and availability. The MAIIES method provides a comprehensive audit framework, facilitating compliance with quality and security standards and identifying areas of improvement. It offers a strategic approach for minimizing quality and security risks in HEIs through a comprehensive computer audit process, enabling institutional evaluation and accreditation by ensuring compliance with quality and security standards and identifying areas of improvement.
Keywords – Computer audit, Institutions of higher education, Quality, Security, Factors, Metrics.
To cite this article:
Imbaquingo, D., Díaz, J., & Jácome, J. (2024). Quality and security as key factors in the development of computer audits in higher education institutions. Journal of Technology and Science Education, 14(4), 965‑989. https://doi.org/10.3926/jotse.2275 |
----------
1. Introduction
The great demand for the use of Information and Communication Technologies (ICT) in various institutions of any environment has involved irregular behaviors occurring daily, which require monitoring and control processes such as audits to minimize risks now of using equipment, facilities, hardware, and software, in addition to seeking the best alternatives in terms of investment and proper use of technology.
Starting from this, it can be understood that organizations are increasingly dependent on information; therefore, protecting sensitive and valuable information becomes a strategic capacity that guarantees business sustainability, profitability, and the global value of a company (Hohan, Olaru & Pirnea, 2015). In this context, guaranteeing the good use of each of these services allows the sustainable progress of the organization, maintaining a competitive advantage, protecting the reputation, ensuring compliance, and applying laws and regulations (O’Hanley & Tiller, 2013).
An audit can be defined as accumulating and evaluating evidence of specific, quantifiable information carried out by independent and competent persons to determine and report the degree of correspondence between quantifiable information and established standards (Campos-Pacurucu, Narváez‑Zurita, Eràzo-Álvarez & Ordoñez-Parra, 2019; Rodríguez-Labrada, Cano-Inclán & Cuesta‑Rodríguez, 2018). In turn, computer audits allow the diagnosis and evaluation of the computer environment (hardware, software, databases, networks, facilities, etc.), in which those responsible for the computer area, administrators, accountants, general auditors, and coordinators of processes are executed in the organization. Their participation occurs in different phases of the process: planning, execution (information gathering), analysis of results, and finding useful evidence in preparing the final report (Arcentales-Fernández & Caycedo-Casas, 2017).
Over the years, the Computer Audit has gone from being a support activity in the financial area to being the protagonist in Information Technology (IT) processes; it is a fundamental activity in the growth of any organization that handles critical information and implements technological infrastructure and information systems, thus guaranteeing security, internal operational control, efficiency, effectiveness, continuity of operations, and risk management, which support decision-making and continuous improvement (Imbaquingo, Pedro, Diaz, Saltos & Arciniega, 2021). However, there is no standard and proven methodology for the audit of Higher Education Institutions (HEIs), and no guidelines allow comparison of the results obtained (Soy-i-Aumatell, 2003). Concerning Ecuadorian (HEIs), the topic has not been of interest so far, so the computer audit procedures they use have not been standardized.
HEIs must integrate their processes to ensure their correct action and sustainability projections(García & González, 2020) to transform and improve the social environment. In this context, three substantive functions are specified that are executed by the action of knowledge: teaching, research, and linking or extension (Ley Orgánica de Educación Superior [LOES], 2018). Starting from the substantive functions and considering that information systems management is developed in Higher Education Institutions (HEIs) in both the strategic and operational parts, the modules or systems play a leading role as main axes of management, thus becoming an essential requirement (García & González, 2020; Rodríguez-Labrada et al., 2018). The problem is evident as far as information is concerned, for which reason audits are used, which include the process of collecting and evaluating evidence to establish criteria on whether the Information System (IS) may or may not protect existing assets and information technology to maintain data integrity (Rodríguez-Labrada et al., 2018). However, only 12% of HEIs in Ecuador carry out computer audits periodically because their importance is unknown or they lack specialized departments in the area (Cadena, Córdova, Enríquez & Padilla, 2019).
Without a modern and focused method for HEIs, each audit team imposes its procedures and personal criteria, generating quality and security problems in audit information. Furthermore, the quality of audits and information security has been the subject of interest in academic, professional, and legal debates because of a series of corporate collapses and the low levels of results obtained in the execution of previous audits (Sulaiman, Mat-Yasin & Muhamad, 2018), thus generating a lack of definition of the control environment, inadequate definition of technological risks, lack of information, and adequate supervision of internal control. Therefore, it is necessary to develop a new method that can influence the optimal administration of HEIs.
The quality of computerized audit results is difficult to define, and to date, no one has been universally recognized (Sulaiman et al., 2018). However, the most supported concept states that it is the measurement of the success of the performance of the audit exercise (Havelka & Merhout, 2013), focused on the review and validation of the results obtained in the control exercise, which is applied to analyze whether the audit products meet the criteria of relevance, opportunity, and sufficiency; add value to the business; or provide objective, verified, and independent information for decision-making in the areas, processes, and activities related to the audited object (Imbaquingo, San Pedro, Díaz, Arciniega, Saltos & Ortega, 2022).
When discussing information security, the objective is to protect data through human and technical measures and procedures to guarantee an institution’s business sustainability, profitability, and value (Hohan et al., 2015). Therefore, for this investigation, the quality and security of the information within the audit exercise are considered an essential part of the method, identifying audit quality metrics associated with the human, technical, and environmental factors, and security metrics focused on the pillars of integrity, confidentiality, and availability.
The main contribution of this investigative work is the design of a computer audit method for Higher Education Institutions (MAIIES), which ensures the quality and security of the results based on computer audit techniques, good practices, and international reference frameworks. To meet this objective, three research questions have been raised: ¿What are the factors that impact audit quality? ¿What are the metrics to evaluate quality and security in computer audits? What are the activities to develop a computer audit process?
The rest of the paper is organized as follows. Section 2 reviews related work, and Section 3 describes the research materials and methods. Section 4 describes the MAIIES method, highlighting the statistical analysis for the definition of the method validation metrics in terms of quality and safety. Sections 5 and 6 discuss and conclude the study, respectively.
1.1. Related Work
1.1.1. IT Audit
Audits are tools to support decision-making and continuous improvement because they are designed to help and should not create any problems (Cienfuegos, Gómez & Millas, 2021). Therefore, HEIs need a method that adapts to their needs and is easy to follow and understand. However, the existing computer audit methodologies or standards are oriented to the productive sector, so they do not fully adapt to the educational environment and reality of HEIs (Gkrimpizi, Peristeras & Magnisalis, 2023)
Aliyu, Maglaras, He, Yevseyeva, Boiten, Cook et al. (2020) highlight that (HEIs) possess vast amounts of sensitive information and knowledge, making them prime targets for cyber threats targeting their research data, financial records, and IT resources. This reality underscores the ongoing struggle to balance open access to this information with the need to secure it against such threats, especially given the vulnerabilities in HEI IT infrastructures. To address this, the authors suggest a structured evaluation framework aimed at assessing the cybersecurity maturity levels of HEIs.
Among the most used methodologies in IT audits in HEIs are ITAF, ISO, ISSAI, and ITIL (Otero, 2018) which have been applied within HEIs as a component analysis method to improve the quality and effectiveness of the audit (Siyaya, Epizitone, Jali & Olugbara, 2021), to control operations and verify that the inherent risks are managed correctly (Taşkın & Sandıkkaya, 2023), all of which satisfy the high demands and competition in the market produced by these institutions. Another aspect is the application of an audit to analyze the accessibility of its institutional websites (Kurt, 2017; Sanchez-Puchol, Pastor‑Collado & Borrell, 2017), as well as to review the security of information within the IT service department (Ghazvini, Shukur & Hood, 2018) and the assurance of the quality of computer services (Widjajanto, Agustini-Santoso & Riiati, 2018). Furthermore, techniques useful for evaluating higher education were also applied in this study (Bates, 2018). Finally, these methodologies (Carpenter & McGregor, 2020) can be applied to knowledge areas and professional education reforms by offering a constructivist explanation of risk audit technologies (Saputra & Ismandra, 2023).
1.1.2. Quality of Audits
A quality audit can be defined as a comprehensive assessment process that examines the competence and independence of auditors, the effectiveness of audit testing procedures, and the reliability and relevance of the evidence gathered (Francis, 2023). Audit quality is a multidimensional concept influenced by inputs, processes, and the regulatory ecosystem, highlighting the complexity and layered nature of ensuring high‑quality audits (Francis, 2011).
It is complex because, unlike any other field of study, it is difficult to define. To date, there is no universally recognized concept, but it is related to standards applicable to auditing. The closest definition measures the success of process completion (Havelka & Merhout, 2013; Holm & Zaman, 2012). In the guide proposed by (Contact Committee of the Heads of the SAIs of the European Union, 2004), it is established that the quality of the audit starts with the process of identifying and managing the activities that will comply with the objectives and quality indicators established by the regulation and control entities, who ensure that the problems in the quality of the audits are directly related to how the process was designed. For Francis (2004), the definition of quality is related to all audit failures: the higher the failure rate, the lower the audit quality. It is worth mentioning that the idea of quality differs among those involved in the audit and must accommodate the needs of each organization, person, area, or process (Detzen & Gold, 2021). The framework proposed by the International Audit and Assurance Standards Board states that quality is compliant with standards, controls, and the ethics used during the process (International Auditing and Assurance Standards Board, 2014).
Previous studies on audit quality have identified three factors that directly affect the quality of audit results: human, technical, and contextual or environmental. Each factor has a group of metrics that evaluates and measures the quality of an audit exercise (Imbaquingo et al., 2021, 2022)
1.1.3. Information Security
Information security, also known as cybersecurity or IT security, involves protecting electronic data from various risks, including unauthorized access, use, disclosure, interception, and data loss (Salazar & Silvestre, 2017). This encompasses the safeguarding of both business and individual users’ confidential information. The core objectives of IT and information security are defined by three critical aspects: confidentiality, ensuring that information is accessible only to those authorized to have access; integrity, guaranteeing the accuracy and completeness of data; and availability, ensuring that authorized users have access to the information and its associated assets when needed (Taherdoost, 2022).
The significance of information security and cybersecurity management is increasing given the necessity to safeguard data, while the incidence of cyberattacks has escalated, as reported by global cybersecurity entities. Furthermore, awareness regarding the implementation of defensive strategies has significantly expanded (Antunes, Maximiano, Gomes & Pinto, 2021). The COVID-19 pandemic has further exacerbated cybersecurity challenges globally, due to the shift towards remote work, prompting an expedited digital transformation (Ahmad, 2020).
Although at the HEIs level, there are studies to guide the audit process and audit proposals for the evaluation of information security by applying methodologies such as COBIT, ISO, ITIL, and others (Haufe, Colomo-Palacios, Dzombeta, Brandis & Stantchev, 2022) none of them contemplate the specific services and processes that are developed within HEIs, to control and guarantee the security of technological assets against different threats and incidents, or to determine opportunities for improvement.
However, several studies discuss information security focused on security pillars: availability (Ahmed & Pathan, 2020; Kure, Islam & Razzaque, 2018) integrity (Eom, Hong, An, Park & Kim, 2019; Gunes, Kayisoglu & Bolat, 2021), and confidentiality (McLeod & Dolezel, 2022; Wagner & Eckhoff, 2019), which allow obtaining various metrics for each pillar to assess information security, including security policies, asset control, encryption, staff training, access control, monitoring plans, incident management, and compliance audits.
2. Materials and Methods
The methodology for the development of the MAIIES is based on the Framework of Technology, Organization, and Environment (TOE), which proposes the adoption of new technologies in organizations considering three aspects: technological, organizational, and environmental or environment (Palos-Sanchez, Reyes-Menendez & Saura, 2019). This is consistent with the bibliographic review and with the aspects to be considered in an audit process. Within the technological context, all the technical and technological tools used in the audit phases (Contact Committee of the Heads of the SAIs of the European Union, 2004; Normas Internacionales de Ética para Contadores [IESBA], 2021) are considered; in the organizational context, those involved in the audit process and the structure of HEIs (Harris & Williams, 2020; Knechel, Krishnan, Pevzner, Shefchik & Velury, 2013); and in the context or environment, everything related to the regulatory environment, organizational structure, and current regulations to audit (Esparza, Diaz, Egas, Sinchiguano & Misacango, 2020; Havelka & Merhout, 2013).
The methodology begins with a literature review to obtain a deep understanding of IT audit methodologies, the reference frameworks used by IT auditors, and their phases and activities. Figure 1 shows the flow to literature review.
Figure 1. Flow to literature review
Next, metrics that allow the evaluation of the quality and security of the results in computer audit processes carried out in HEIs and end with a statistical analysis to identify and define quality and security evaluation instruments in computer audits.
The reference frameworks chosen for the study were ISO 19011:2018, ISSAI 5300, ITAF, and IIA s. They are based on compliance with certain parameters, such as validity and compliance with the general structure of a computer audit, frequent use, and implementation in auditing processes. Within each referential framework, there are unique procedures for developing an audit process. However, the union of two or more frameworks or methodologies is necessary for an audit to be considered complete and successful. Consequently, for the creation of MAIIES, the activities of each framework were identified as the basis for the proposed method.
Several authors agree that an audit is structured in three phases: audit planning, audit execution, and results communication (Harris & Williams, 2020). However, for the development of the MAIIES, validation and follow-up phases were added, thus ensuring a complete method with feedback that included an evaluation based on quality indicators, security, and post-audit compliance. In addition, follow-up is considered to encourage appropriate responses to the findings identified in the audit and lay the foundation for future audit work (Contact Committee of the Heads of the SAIs of the European Union, 2004).
In previous studies, the factors and metrics of quality and security of the results in computer audit processes were identified through a literature review, in which the human, technical, and contextual factors stand out, and 94 metrics were grouped into each factor (Imbaquingo et al., 2021), along with a statistical analysis in which it was determined if the metrics were grouped correctly in the identified factors, allowing a reduction of dimensions based on their results (Imbaquingo et al., 2022). However, with the 64-resulting metrics, the analysis focuses on computer audit processes implemented in Ecuadorian HEIs using data processing techniques such as Mahalanobis distances, Confirmatory Factorial Analysis, and the Kruskal-Wallis test.
Mahalanobis Distances. These distances allow measurement of the number of standard deviations where the observations are located. Geometrically, Euclidean distance is the shortest distance between two points; however, it does not consider the correlation between highly correlated variables. The Mahalanobis distance differs from the Euclidean distance in that it considers correlations between variables [61, 62]. Each Mahalanobis distance is a scale-invariant metric that obtains the distance between a point generated by an x ∈ ℝp, p-variant probability distribution fx(.), and the mean μ = E(X) of the distribution. We assume that distribution fx(.) has second-order finite moments and the covariance matrix can be defined as ∑ = E(X – μ). Equation 1 defines the Mahalanobis distances are defined as:
(1) |
Confirmatory Factorial Analysis. In addition, confirmatory factor analysis (CFA) was carried out to correctly explain the factors that compose the whole structure, confirming its validity and reliability. In this formulation, there is a vector of observed responses Yi which is predicted by the unobserved latent variables ξ, through the model (see Equation 2):
(2) |
Where Y is a vector of dimension p × 1 of observed random variables, ξ is the unobserved latent variables, and Λ is a dimension matrix p × k with k equal to the number of unobserved latent variables. Also, as Y is constituted by a set of variables ξ that imperfectly explain Y, the model considers the error ∈. The model is commonly solved by a maximum likelihood (ML) estimation formulation generated by iterative minimization of the fitting function (FML) of the Equation 3:
(3) |
Where ΛΩΛ' is the variance-covariance matrix involved in the proposed factor analysis model, and R is the observed variance-covariance matrix. In this way, the model parameters are estimated by minimizing the distance between the variance-covariance implied in the model and the observed one (Rosseel, 2012; Yang-Wallentin, Joreskog & Luo, 2010).
Kruskal-Wallis Test. With the results obtained through the data treatment and the validation of the construct by the CFA, it is guaranteed that a data sample is valid, does not present alterations due to the influence of outliers, and is made up only of a set of variables that correctly explain the factors that are of interest in the investigation. Since these data come from non-ordinal variables, a non-parametric technique must be used to compare the groups determined by categorical variables. The Kruskal-Wallis test is a non‑parametric alternative to one-way ANOVA. It is assumed that the observations in each sample group are from a sample with the same distribution. Therefore, for this test, the null hypothesis was established based on the Equation 4:
(4) |
Where ηi is the median of the ith group defined by the categorical variable in the sample. In this case, the null hypothesis is equivalent to: “H0 : the samples come from identical populations”. We define n that represents the total number of observations n = ∑ki=1 ni, where ni represents the sample size of each group i = 1,2, …, k and k represents the number of groups to be compared. Ranks were obtained for each observation in ascending or descending order of magnitude when ties existed. In this way, R(Xij) represents the rank assigned to the j-th observation of the i-th group, Xij and Ri represent the sum of ranks assigned to the i-th group, Ri ∑nii=1 R(Xij) for i = 1,2, …, k. In this way, the static test T is defined on the Equation 5:
(5) |
where:
(6) |
If there are no ties, S2 it is simplified to the expression n(n + 1)/12 and the statistical test is reduced to Equation 7:
(7) |
Under the null hypothesis, H0 and the previously defined assumption, T it is distributed asymptotically to the chi-square distribution with k – 1 degrees of freedom T~χ 2k–1 (Lehmann, 2006; Nwobi & Akanno, 2021).
Dunn-Šidák Test. Finally, as a post hoc test for Kruskal-Wallis, we applied the Dunn–Šidák test for the comparison between more than two samples in a paired way, constituting in this way an alternative, where in case of reaching the level of significance at a general level, Dunn’s test is capable of contrasting each possible pair and identifying which pairs of groups present significant differences (Dunn, 1958). Moreover, Dunn’s test can provide even smaller confidence intervals than Tukey’s test. For a given FWER (wise error rate (FWER) error metric α, the Dunn–Šidák test defined as μi – μj can be calculated using the Equation 8 and 9:
 | (8) |
where
 | (9) |
y̅i and y̅j are the means of the samples considered, c is is the number of possible comparisons in the family, and the quantile tα',v is obtained from Student’s probability distribution t for a given parameter of degrees of freedom v. Finally, the confidence intervals for each possible Dunn-Šidák test (see Equation 10) were obtained as follows:
 | (10) |
3. Results
The MAIIES proposal is structured in five phases: planning, execution, communication of results, validation, and follow-up. Each phase encompasses a set of activities, and for the complete method, 47 are accounted for. These activities were verified using the Delphi Method, a technique that allows gathering information based on the opinions of experts in a specific area to obtain a consolidation of a given topic (Reguant & Torrado, 2016). Figure 2 shows the general scheme of the proposed method.
For the validation phase, a statistical analysis of the metrics obtained for the quality and security of information in computer audit processes implemented in HEIs was conducted. The database consists of 54 computer audit observations performed in 54 HEIs in Ecuador. The variables used to construct the audit evaluation model are proposed in (Imbaquingo et al., 2021, 2022; Stoel, Havelka & Merhout, 2012). Thus, the evaluation instrument was made up of 81 variables, of which eight were categorical, including the name of the institution, the area where it is located, the level of studies it offers, compliance with the performance of audits, the perception of the importance of performing audits, whether previous audits have been performed, the type of audit previously performed and the type of audit. The 81 variables were evaluated on a ten-level ordinal scale, where each of the variables proposed in (Imbaquingo et al., 2022) was scored to measure the quality dimension composed of human, technical and contextual factors. The information security dimension is based on confidentiality, integrity and availability based on the ISO 27000 standard. The variables distribution for each factor is presented in Tables 1 and 2.
Figure 2. MAIIES scheme
Variable | Factor |
Human factor | |
p1 | The audit team sought to involve the client throughout the audit process |
p2 | The audit team obtained the client’s agreement about the activities carried out |
p4 | The staff performing the audit had the necessary competencies to perform their work |
p5 | The auditor had soft skills (characteristics and personal competencies that demonstrate how the auditor works with others) |
p6 | The staff who performed the audit provided effective suggestions to the Institution |
p7 | The auditor was open-minded when receiving new ideas |
p8 | The auditor was sure of himself and his work |
p9 | The audit team retained its independence in appearance and action |
p10 | The audit team focused on the facts |
p11 | The audit team received support to achieve the goals |
p12 | The audit team demonstrated effort in conducting the audit |
p13 | The auditor was concerned about their training and continuous updating |
p14 | The auditor had national and international certifications in auditing and computer auditing |
p15 | Audit team members demonstrated knowledge of information security and data processing |
p16 | Differences with the client were dealt with in a timely, professional, and objective manner |
p17 | The audit team was available to meet the client’s requests |
p18 | Those involved in the audit had frequent communication |
p19 | The auditor engaged experts to support the audit process to obtain results and recommendations for the client |
p20 | The auditor followed policies and procedures that regulate its ethical and professional compliance |
| Technical Factor |
p21 | The audit team used templates and forms to document |
p22 | The audit findings and conclusions were an accurate reflection of the actual facts of the audited process |
p23 | The audit results were supported and documented with the evidence collected during the audit. |
p24 | The members of the audit team and those responsible for the institution ensured at all times the information |
p25 | The client positively received the findings, conclusions, and recommendations |
p26 | Resources for the audit were allocated according to the importance and complexity of the audit |
p27 | The system, process, or object audited was significant to the organization |
p28 | In the scope, all the elements necessary to audit successfully were addressed. |
p29 | The execution of the audit complied with the elements agreed upon in the scope |
p30 | The results were delivered at the right and established time |
p31 | The risk assessment model was understandable |
p32 | The audit plan took into account the risks related to the client |
p33 | The audit process was carried out with accuracy and precision |
p34 | The audit report was clear and concise with its results |
p35 | The scope, findings, and recommendations have been understandable to anyone who used the audit report. |
p36 | The audit was executed under the policies, standards, manuals, guidelines, and practices of computer auditing |
p37 | Checklists were complete, approved, and documented |
p38 | An expert reviewed the fieldwork |
p39 | The client or managers of the audited organization provided support for the collection of information |
p40 | Information and results from previous audits were available for review |
p41 | The objectives and scope of the audit were adequately specified |
p42 | The activities and tools for the audit were clearly described |
p43 | Audit team members had a clear and consistent understanding of the audit plan |
p44 | The audit budget and schedule were properly established |
p45 | The requirements of personnel and equipment assigned for the audit were evaluated |
p46 | The audit plan was prepared, reviewed, and approved by the supervisors, managers of the organization, and members of the audit team |
p47 | The audit team used an IT audit methodology to plan, manage and perform the audit |
p48 | The audit team used technological tools and new methodologies to carry out their work |
| Context Factor |
p49 | Through his reports, the auditor promoted an organizational culture based on good computer security practices |
p50 | The audit team had strict quality control procedures |
p51 | The audit team leader was committed to the quality control system |
p52 | The rules and regulations issued by control bodies were reflected in the audit plan |
p53 | The audit team knew the relevant information of laws and regulations that can have a significant impact on the audit objectives |
p54 | Disciplinary measures were applied in case of non-compliance with the audit plan or current legal regulations |
p55 | The audit cost was established in accordance with the complexity and the activities carried out. |
Table 1. Variables and factors proposed for the evaluation of the Quality dimension.
Variable | Factor |
Confidentiality Factor | |
p56 | Information security policies are applied within the institution |
p57 | The information security policies and procedures within the institution are updated periodically |
p58 | Information security responsibilities are delegated, documented, and formally delivered to all institution staff, depending on their position. |
p59 | Security policies and actions are applied to sensitive information of the institution |
p60 | Information access policies are updated and applied based on existing user roles |
p61 | An information security accreditation is available for all its computer systems |
p62 | Documented procedures are in place to follow in case of security incidents |
p63 | Information security compliance audits are performed |
p64 | Password management policies apply to end users of the institution |
p65 | Users accessing the network and the actions they perform are identified |
| Integrity Factor |
p66 | Access control is applied to the institution’s IT infrastructure and services |
p67 | Users, collaborators, and staff are trained and involved in information security issues |
p68 | Vulnerability analysis of the institution’s web services is carried out |
p69 | Plans for monitoring and managing the impact of security incidents in the institution are applied |
p70 | Inventory of all IT assets is updated and documented |
| Availability Factor |
p71 | Applications are available to protect all your IT solutions from malware |
p72 | Data backups are made |
p73 | The activities developed by the users are monitored |
Table 2. Variables and factors proposed for the evaluation of the Information Security dimension
Statistical analysis began by processing the data that constituted the database. Each audit carried out in the HEIs constitutes a multivariate observation; therefore, Mahalanobis Distances were used to detect atypical observations. A cutoff score of 128.5648 was established based on the distribution χ 2 conserved 99.9% of the distribution, where 0.01% of the furthest distances were considered outliers. In this way, by computing the Mahalanobis distances for the entire database, none were detected as atypical, so the final database comprised 54 audit observations from higher education institutions.
The instrument proposed was validated for a group of internal auditors from Ecuador (Imbaquingo et al., 2022). However, the present study was developed in a specialized manner for higher education institutions, so in the first instance ten variables that do not apply to the context of higher education were eliminated. Therefore, a new process of verification of the validity and reliability of the modified instrument was carried out, for which the Confirmatory Factor Analysis (CFA) technique was selected. The analysis began by verifying the assumptions of additivity, normality, linearity, homogeneity, and homoscedasticity. Figure 4 shows the results of the multivariate additivity analysis of the sample using a correlation matrix, which is presented in Figure 3.
Figure 3 shows that none of the pairs of questions reached very high or perfect correlation values close to 1, so the additivity hypothesis was accepted. The correlation values were close to 1; therefore, the additivity hypothesis was accepted. To verify the multivariate assumptions of normality, linearity, homogeneity, and homoscedasticity, the sham regression analysis was used. The results were observed using the histogram, the QQ diagram, and the scatter plot presented in Figure 4.
As can be seen, Figure 4a shows the histogram of the adjusted values from a regression performed using the quantiles of the distribution χ 2 is the response variable, and the ordinal variables of the instrument as predictors. These adjusted values were standardized, and subsequently, a histogram was obtained, whose values described a distribution similar to the normal; therefore, the assumption of normality was accepted. To observe the assumption of linearity, the Q-Q plot was used, which is the diagram obtained by plotting the quantiles of the real sample concerning theoretical quantiles obtained from a random sample of the distribution χ 2 for the same number of degrees of freedom of the sample. As shown in Figure 4b, when plotting the Q-Q plot, the quantiles were distributed similarly to a straight line with a slope of 1, so the assumption of linearity was accepted. Finally, the assumptions of homogeneity and homoscedasticity were observed using the scatter plot shown in Figure 4c, where the standardized residuals were projected based on the residuals obtained in the fit of the regression model. As can be seen, the residuals were arranged similarly in the four quadrants, and there were no pre-established groups or patterns identified; therefore, the assumptions of homogeneity and homoscedasticity were accepted (Guevara, Herrera, García & Quiña, 2020; Jácome, Herrera, Herrera, Caraguay, Basantes & Ortega, 2019).
Figure 3. Multivariate correlation matrix for each possible pair of items that comprise the instrument
a)  | b)  |
c)  | |
Figure 4. Parametric assumptions: (a) histogram of standardized values;
(b) quantile diagram (QQ Plot); (c) scatter plot (Scatter-plot)
Once the assumptions were verified, it was concluded that the sample was parametric and met the requirements for applying CFA as a technique for verifying the validity and reliability of the instrument. The CFA results for each dimension are presented in Figures 5 to 6 and Table 3.