An Analytical Hierarchy Process Model for the Evaluation of College Experimental Teaching Quality
Qingli Yin
Department of Laboratory and
Facility Management, Shandong Jianzhu University
China

Received December 2012
Accepted July 2013
Abstract
Taking into account the characteristics of college experimental teaching, through investigation and analysis, evaluation indices and an Analytical Hierarchy Process (AHP) model of experimental teaching quality have been established following the analytical hierarchy process method, and the evaluation indices have been given reasonable weights. An example is given, and the evaluation results show that the evaluation indices proposed in this paper are capable of reflecting objectively, exactly and reasonably experimental teaching quality, and of effectively promoting the quality of experimental teaching.
Keywords – experimental teaching quality, evaluation, weight, AHP model, consistency

1 Introduction
College experimental teaching plays an irreplaceable role in the cultivation of innovative talents, and experimental teaching quality has a direct impact on teaching quality as a whole. It is therefore necessary to introduce evaluations of experimental teaching quality. The key point of evaluating experimental teaching quality concerns how to improve it. According to the characteristics of experimental teaching, we have identified the core elements of experimental teaching in this paper, and provide a reasonable evaluation index system based on Analytical Hierarchy Process (AHP). Indices at all levels have been given reasonable weights based on a mathematical model, and the degree of influence each evaluation index has on experimental teaching quality has also been determined. Finally, the evaluation results of selected teachers are examined, based on a sample of evaluation data obtained using a mathematical model, which demonstrates the rationality and credibility of the evaluation results.
2 An AHP model for evaluation of experimental teaching quality
Although the evaluation of college experimental teaching quality has been previously discussed (Chen, 2009; Feng, Shi & Du, 2010; Ma, Liu & Lv, 2009; Qin & Shi, 2010; Zhang, Zhou, Han & Huang, 2011), every university has its own features, and therefore, the evaluation index system or certain weights may not be same. This highlights the importance of creating an evaluation system that is suitable for each university. This paper takes into account the students’ points of view to establish an AHP model of evaluating experimental teaching quality, according to the actual situation at our university, focusing on the position of the students in the evaluations.
2.1 Construction of evaluation system assessing experimental teaching quality
Based on the characteristics of experimental teaching, taking into account the recommendations of experts, instructional supervisors, teachers and students, this evaluation system can be summarized as four aspects consisting of 13 factors, the hierarchical structure of which is shown in Table 1.
Goal A 
Criterion B 
Alternative C 
Evaluation of experimental teaching quality 
Teaching attitude: B_{1} 
C_{11}: Teaches and cultivates people，sets strict demands and is worthy of being called a teacher. C_{12}: Engages in experimental teaching, is wellprepared and lectures seriously and fluently. C_{13}: Corrects lab reports in timely, serious manner and patiently gives guidance. 
Teaching contents: B_{2}

C_{21}: Is familiar with the experimental contents and use of instruments; provides guidance materials C_{22}: Contents evidence a reasonable design and are explained clearly and accurately. The theoretical course and the experimental course can be organically linked. The most important topics related to the subject are given an appropriate description. C_{23}: The emphasis on and difficulty of the experimental teaching process is prominent, the amount of contents is suitable for students to master and the level of difficulty is appropriate for students to understand. C_{24}: Comprehensively designed experiment contents are incorporated in the course and scientific research is introduced through experimental teaching. 

Teaching methods: B_{3} 
C_{31}: Is good at inspiring students to think, stimulates the students' intellectual curiosity through timely guidance during experiments, encourages them to participate in the discussion of experiments and express different views. C_{32}: Is good at guiding students as they analyze experimental phenomena and results, incorporating learned knowledge. C_{33}: Teaching is organized in a flexible and effective manner, students are taught according to their aptitudes and instruction follows a logical order. 

Teaching results: B_{4} 
C_{41}: Contribute to the consolidation of related theoretical knowledge by the students. C_{42}: Increased knowledge, developed thinking and improvement in the students' practical skills through experimental activities. C_{43}: Promote innovation by students and their ability to develop and design comprehensive experiments. 
Table 1. Evaluation system assessing experimental teaching quality
2.2 Design of comparison matrices
Comparison matrices are the basis for weight sorting, and they have a decisive influence on the final overall sort. Therefore, the design of comparison matrices is a very important aspect of AHP. To accurately design comparison matrices at all levels, they must be carefully and objectively analyzed, researched and corrected until consistency verifications produce satisfactory results. To make comparisons, we need a scale of numbers that indicates how many times more important or dominant one element is over another with respect to the criterion. One common scale (Saaty, 2008) is shown in Table 2.
Intensity of importance 
Definition 
Explanation 
1 
Equal importance 
Two factors contribute equally to the objective. 
3 
Somewhat more important 
Experience and judgment slightly favor one over another. 
5 
Much more important 
Experience and judgment strongly favor one over another. 
7 
Very much more important 
Experience and judgment very strongly favor one over another. Its importance is demonstrated in practice. 
9 
Absolutely more important 
The evidence favoring one over the other is of the highest possible validity. 
2, 4, 6, 8 
Intermediate values 
Compromise is needed 
Reciprocals of the above 
If activity i has one of the above nonzero numbers assigned to it when compared to activity j, then j has the reciprocal value when compared to i. 

Table 2. The fundamental scale of absolute numbers
The matrix of pairwise comparisons A = (a_{ij}) represents the intensities of the expert’s preference between individual pairs of criteria (alternatives) (B_{i} versus B_{j}, for all i, j = 1, 2, …, n). They are usually chosen according to a given scale (1/9, 1/8, …, 8, 9). Given n criteria (alternatives) {B_{1},B_{2},…, B_{n}}, a decisionmaker compares pairs of criteria (alternatives) for all the possible pairs, and a comparison matrix A is thus obtained, where the element a_{ij} shows the preference weight of B_{i} obtained by comparison with B_{j}.
Using the scale of relative importance shown in Table 2, a set of pairwise comparison matrices is created by synthesizing the recommendations of experts, instructional supervisors, teachers and students, as shown in Tables 37, where the comparison matrix A is the criteria matrix for the criteria with respect to the goal, the comparison matrices B_{1}, B_{2}, B_{3}, B_{4} are alternative matrices for alternatives with respect to each criterion. For example, we see criteria matrix A, in which the principal diagonal contains entries of 1, as each factor is of equal importance. The experts decide that B_{4}, teaching results, is somewhat more important than teaching attitude, which is rated as 3 in the cell B_{4}, B_{1} and 1/3 in B_{1}, B_{4}. They also decide that teaching contents is slight more important than teaching methods, assigning a score of 2 in the cell B_{2}, B_{3} and 1/2 in B_{3}, B_{2}. The other elements of criteria matrix A are obtained in a similar manner, as shown below.
A 
B_{1} 
B_{2} 
B_{3} 
B_{4} 
Priority vector W_{A} 
Consistency check indicators 
B_{1} 
1 
1/4 
1/2 
1/3 
b_{1} = 0.0994 
l_{max} = 4.02062 
B_{2} 
4 
1 
2 
2 
b_{2} = 0.4379 
CI = 0.00687 
B_{3} 
2 
1/2 
1 
1 
b_{3} = 0.2190 
CRA = 0.00764 < 0.1 
B_{4} 
3 
1/2 
1 
1 
b_{4} = 0.2437 

Table 3. Criteria matrix A and its consistency check
B_{1} 
C_{11} 
C_{12} 
C_{13} 
Priority vector W_{B}_{1} 
Consistency check indicators 
C_{11} 
1 
1/2 
1/4 
0.1429 
l_{max} = 3 
C_{12} 
2 
1 
1/2 
0.2857 
CI_{1 }= 0 
C_{13} 
4 
2 
1 
0.5714 
CR_{1 }= 0 
Table 4. Alternative matrix B_{1} and its consistency check
B_{2} 
C_{21} 
C_{22} 
C_{23} 
C_{24} 
Priority vector W_{B}_{2} 
Consistency check indicators 
C_{21} 
1 
1/2 
1/4 
1/2 
0.1111 
L_{max} = 4 
C_{22} 
2 
1 
1/2 
1 
0.2222 
CI_{2} = 0 
C_{23} 
4 
2 
1 
2 
0.4445 
CR_{2} = 0 
C_{24} 
2 
1 
1/2 
1 
0.2222 

Table 5. Alternative matrix B_{2} and its consistency check
B_{3} 
C_{31} 
C_{32} 
C_{33} 
Priority vector W_{B}_{3} 
Consistency check indicators 
C_{31} 
1 
4 
3 
0.6337 
l_{max} = 3. 0092 
C_{32} 
1/4 
1 
1 
0.1744 
CI_{3} = 0.0046 
C_{33} 
1/3 
1 
1 
0.1919 
CR_{3} = 0.0079 
Table 6. Alternative matrix B_{3} and its consistency check
B_{4} 
C_{41} 
C_{42} 
C_{43} 
Priority vector W_{B}_{4} 
Consistency check indicators 
C_{41} 
1 
1/3 
1/5 
0.10945 
l_{max} = 3.0037 
C_{42} 
3 
1 
1/2 
0.3090 
CI_{4} = 0.0018 
C_{43} 
5 
2 
1 
0.58155 
CR_{4} = 0.0032 
Table 7. Alternative matrix B_{4} and its consistency check
2.3 Relative weights and consistency check
It is important to note that AHP does not demand perfect consistency. Some inconsistency is allowed in random judgments. An inconsistency ratio of about 10 percent or less is usually considered “acceptable”. The consistency index (CI) is calculated according to the following equation CI = (l_{max}n)/(n1), where l_{max} is the largest eigenvalue of the comparison matrix, and n is the number of criteria. The Consistency Ratio (CR) is calculated using the equation CR = CI/RI. The RI is the random index representing the consistency of a randomly generated pairwise comparison matrix. It was derived by Saaty (1980) as average random consistency index (Table 8) calculated from a sample of 500 randomly generated matrices based on the AHP scale (Table 2).
n 
1 
2 
3 
4 
5 
6 
7 
8 
9 
RI 
0 
0 
0.58 
0.90 
1.12 
1.24 
1.32 
1.41 
1.45 
Table 8. Random index (RI)
The CR tells the decisionmaker how consistent he has been when making the pairwise comparisons. If CR<0.10, the decisionmaker’s pairwise comparisons are relatively consistent and the criterion is considered to have acceptable consistency. If CR > 0.10, the decisionmaker should seriously consider reevaluating his pairwise comparisons – the sources of inconsistency must be identified and resolved and the matrix reanalyzed. The priority vector WA (weights of the criteria corresponding to the goal) is the normalizing eigenvector corresponding to l_{max} of matrix A, and the consistency ratio of matrix A is CR_{A} = 0.00764, as shown in last two columns of Table 3, respectively. Similarly, the priority vectors W_{B}_{1}, W_{B}_{2}, W_{B}_{3}, W_{B}_{4} (weights of alternatives for each criterion) and their consistency check indicators are listed in last two columns of Tables 47, respectively.
2.4 Combinatorial weights and combinatorial consistency check
The priorities of each alternative for the goal are referred to as combinatorial weights, and the components of the combinatorial weight vector are calculated as described by Han (2005):
, where each value of b_{j} and c_{ij} is listed in Table 9. 
The combinatorial consistency ratio is calculated according to the formula:

CR = 0.00764 + (0.0994 × 0 + 0.4379 × 0 + 0.2190 × 0.0046 + 0.2437 × 0.0018)/(0.0994 × 0.58 + 0.4379 × 0.9 + 0.219 × 0.58 + 0.2437 × 0.58) = 0.0096 < 0.1; therefore, the combinatorial consistency is acceptable, and the result of the global ranking has satisfactory consistency. If CR > 0.10, the decisionmaker should seriously reconsider the model or reconstruct the comparison matrices so that they have a higher consistency ratio.
Alternative C 
Criterion B 
Combinatorial weights w_{i} 

B_{1} 
B_{2} 
B_{3} 
B_{4} 

b_{1} = 0.0994 
b_{2} = 0.4379 
b_{3} = 0.2190 
b_{4} = 0.2437 

C_{11} 
c_{11} = 0.1429 
c_{12} = 0 
c_{13} = 0 
c_{14 }= 0 
w_{1}_{ }= 0.0142 
C_{12} 
0.2857 
0 
0 
0 
w_{2}_{ }= 0.0284 
C_{13} 
0.5714 
0 
0 
0 
w_{3 }= 0.0568 
C_{21} 
0 
0.1111 
0 
0 
w_{4 }= 0.0487 
C_{22} 
0 
0.2222 
0 
0 
w_{5 }= 0.0973 
C_{23} 
0 
0.4445 
0 
0 
w_{6}_{ }= 0.1946 
C_{24} 
0 
0.2222 
0 
0 
w_{7 }= 0.0973 
C_{31} 
0 
0 
0.6337 
0 
w_{8}_{ }= 0.1388 
C_{32} 
0 
0 
0.1744 
0 
w_{9} = 0.0382 
C_{33} 
0 
0 
0.1919 
0 
w_{10} = 0.0420 
C_{41} 
0 
0 
0 
0.10945 
w_{11} = 0.0267 
C_{42} 
0 
0 
0 
0.3090 
w_{12} = 0.0753 
C_{43} 
0 
0 
0 
0.58155 
w_{13} = 0.1417 
Table 9. Combinatorial weights
2.5 Model evaluation
As presented in Table 3, it was observed that for students, “teaching contents” represented the most important criterion, followed closely by “teaching results” and “teaching methods”. It can be seen that students rank teaching results higher than teaching methods. Lastly, “teaching attitude” does not seem to be particularly important to students. This model is in agreement with the current situation at engineering universities, and it comprehensively reflects student evaluations of experimental teaching quality. If a teacher does not have a certain amount of theoretical knowledge and practical experience, he can not reconcile theory with practice in the process of teaching, his instructional process is more boring, and is not very effective.
3 Application of the AHP model
We wrote a questionnaire that requires students to evaluate their teachers, assigning them one of five grades (A, B, C, D and E; equivalent to very satisfied, satisfied, generally, dissatisfied, very dissatisfied) for each of the 13 alternatives shown in Table 1. Table 10 shows the evaluation results of 105 students for three teachers.
First, the statistical data are quantified with the values 5, 4, 3, 2, 1, which correspond to the ranking grades A, B, C, D and E. Accordingly, taking into account the membership function of the Cauchy distribution:

(*) 
Where a, b, a, b are constants to be determined. Supposing that the membership degree is 1 when the evaluation is grade A, this means that f(5) = 1; the membership degree is 0.70 when the evaluation is grade C ( f(3) = 0.70); and the membership degree is 0.10 when the evaluation is grade E (f(1) = 0.1). The values of a, b, a, b are then determined to be a = 2.8049, b = 0.4417, a = .5873 and b = 0.0548, respectively. The membership function is obtained by substituting the values of a, b, a, b in the formula (*):

The values of the membership function at x = 2, 4 are calculated as f(2) = 0.4640, f(4) = 0.8690, thus grades A, B, C, D and E are quantified with values {1, 0.869, 0.70, 0.464, 0.1}. The scores for each alternative for each teacher are calculated using the data in Table 10, listed in columns 3, 5 and 7. The total scores for each teacher are obtained by adding the products of each alternative score and its combinatorial weight w_{i} obtained from Table 9, shown in the last row of Table 10 as the evaluation result tallies for the current situation.
Alternative 
Teacher 1 
Score 
Teacher 2 
Score 
Teacher 3 
Score 
C_{11} 
68A 30B 6C 1D 
0.9403 
58A 28B 15C 4D 
0.9027 
53A 38B 6C 7D 1E 
0.8911 
C_{12} 
67A 31B 2C 5D 
0.9301 
30A 45B 21C 5D 4E 
0.8240 
52A 40B 8C 5D 
0.9017 
C_{13} 
71A 17B 15C 2D 
0.9257 
43A 13B 25C 16D 8E 
0.7621 
52A 32B 16C 5D 
0.8888 
C_{21} 
60A 30B 12C 3D 
0.9130 
38A 27B 17C 14D 9E 
0.7691 
48A 43B 11C 3D 
0.8996 
C_{22} 
65A 36B 4C 
0.9437 
41A 36B 24C 4D 
0.8661 
40A 51B 9C 5D 
0.8851 
C_{23} 
69A 16B 9C 7D 4E 
0.8843 
33A 45B 22C 5D 
0.8555 
51A 33B 14C 7D 
0.8831 
C_{24} 
68A 33B 2C 2D 
0.9429 
36A 36B 25C 6D 2E 
0.8359 
35A 47B 17C 4D 2E 
0.8552 
C_{31} 
71A 23B 5C 6D 
0.9264 
43A 38B 17C 4D 3E 
0.8579 
65A 23B 13C 4D 
0.9137 
C_{32} 
63A 27B 8C 7D 
0.9077 
31A 47B 21C 6D 
0.8507 
40A 43B 10C 9D 3E 
0.8461 
C_{33} 
65A 21B 17C 2D 
0.9150 
34A 45B 17C 3D 6E 
0.8285 
45A 33B 18C 5D 4E 
0.8476 
C_{41} 
62A 26B 13C 4D 
0.9100 
40A 37B 21C 7D 
0.8581 
49A 28B 23C 5D 
0.8738 
C_{42} 
69A 23B 9C 4D 
0.9252 
46A 35B 18C 5D 1E 
0.8708 
43A 26B 28C 4D 4E 
0.8329 
C_{43} 
57A 28B 13C 5D 2E 
0.8853 
31A 33B 26C 10D 5E 
0.7906 
38A 30B 20C 12D 5E 
0.8013 
Total score 

0.9136 

0.8359 

0.8681 
Table 10. Statistical data on experimental teaching quality
4 Conclusion
The evaluation indicators and their weights of experimental teaching quality are developed according to the AHP method. As compared to other methods, the greatest advantage of the analytical hierarchy process (AHP) is that it is able to combine both qualitative and quantitative methods and to consider all influencing factors as fully as possible. The evaluation results of the AHP method are more objective, scientific and rational, and it has been proven that the AHP method is the most appropriate when surveys need to account for a high degree of intuition and subjectivity. The evaluation results for the 3 teachers in section 3 are only from a student perspective; if we administer the same questionnaire to experts, instructional supervisors and colleagues of the 3 teachers, requesting their evaluations of the 3 teachers according to the 13 alternatives in Table 1, it would be possible to obtain evaluation results for the 3 teachers from the experts’ points of view, using the same AHP model developed in this paper. This would provide more objective evaluations for the 3 teachers, synthesizing the students’ and the experts’ evaluation results. The model established in this paper can be extended to the comprehensive evaluation of textbooks, teaching management, quality courses, and other situations requiring comprehensive evaluation. The AHP method has already been used in many applications (Saaty, 2008; ISAHP, 2009).
References
Chen, H. (2009). Constructions and applications of the evaluation model of practical teaching quality in colleges [in Chinese]. Research and Exploration in Laboratory, 28, 159161.
Feng, L.X., Shi, S.T., & Du, W.M. (2010). The model of teaching evaluation based on analytic hierarchy process [in Chinese]. Northwest Normal University, Natural Science Edition, 46, 1922.
Han, Z.G. (2005). Mathematical modeling method and its application [in Chinese]. Beijing, China: Higher Education Press.
ISAHP (2009). Proceedings of the 10th International Symposium on the AHP. ISAHP Web Site. ISAHP. August, 2009. Retrieved 20110105.
Ma, Z.L., Liu, H.D., & Lv, S.P. (2009). Building experiment teaching quality evaluation system in science and engineering high school [in Chinese]. Experimental Science and Technology, 7, 7881.
Qin, C.M., & Shi, Z.G. (2010). Constructing and practicing of experiment teaching quality evaluation system in colleges [in Chinese]. Heilongjiang Education, 5, 2223.
Saaty, T.L. (1980). The Analytic Hierarchy Process. New York, NY: McGraw Hill International.
Saaty, T.L. (2008). Decision making with the analytic hierarchy process. Int. J. Services Sciences, 1, 8398. http://dx.doi.org/10.1504/IJSSCI.2008.017590
Zhang, Y.J., Zhou, Y., Han, Y., & Huang, D.X. (2011). Research and practice on quality evaluation of experimental teaching in common colleges and universities [in Chinese]. Experimental Technology and Management, 28, 270‑273.
Citation: Yin, Q. (2013). An Analytical Hierarchy Process Model for the Evaluation of College Experimental Teaching Quality. Journal of Technology and Science Education (JOTSE), 3(2), 5965. http://dx.doi.org/10.3926/jotse.66
Online ISSN: 20136374 – Print ISSN: 20145349 – DL: B20002012

Author biography
Qingli Yin
This work is licensed under a Creative Commons Attribution 4.0 International License
Journal of Technology and Science Education, 20112024
Online ISSN: 20136374; Print ISSN: 20145349; DL: B20002012
Publisher: OmniaScience