Journal of Technology and Science Education

• Home
• About
• Login
• Archives
• Submissions
• Publication Fee
• Indexing & Statistics

Home > Vol 9, No 1 (2019) > Basir

ASSESSMENT OF OUTCOME-BASED INTEGRATED DESIGN PROJECT

Faculty of Civil Engineering, Universiti Teknologi MARA (Malaysia)

Received September 2018

Accepted December 2018

Abstract

Outcome-Based Education (OBE) implementation is one of the qualifying requirements for engineering programme accreditation in Malaysia. Implementation of the OBE in Integrated Design Project (IDP) is essential in producing high quality engineering graduates that are able to meet the challenges especially in the era of Industry 4.0 in Malaysia. IDP is the course offered to the final year students of undergraduate programme in Faculty of Civil Engineering, University Teknologi MARA. This paper presents the mapping of the addressing Course Outcomes (CO) and Programme Outcomes (PO) to the scopes of assessment in a capstone project (CP) for IDP. The explicitness in the measurement of CO and PO in the course is demonstrated. The rubrics for the assessment of CP are also presented. Finally, evaluation of student performance in IDP particularly CP is discussed through a sample analysis of the CO and PO attainments.

 

Keywords – Outcome-based, Capstone project, Undergraduates, Programme outcomes, Assessment.

To cite this article:

Basir, N., Lian, O.C., Salmizi, J.M., & Shaharin, H. (2019). Assessment of outcome-based integrated design project. Journal of Technology and Science Education, 9(1), 77-84. https://doi.org/10.3926/jotse.541

 

----------

1. 1. 1. Introduction

The Board of Engineers Malaysia (BEM) registers graduate engineers that are qualified from a recognised engineering programme under the Registration of Engineers Act 1967 (Laws of Malaysia, 1967 (Revised 2015)). The Engineering Accreditation Council (EAC) is the only recognised body delegated by BEM for accreditation of engineering degree programmes offered in Malaysia. EAC comprises of five stakeholders namely BEM, The Institution of Engineers, Malaysia (IEM), Industry Employers, Malaysian Qualification Agency (MQA) and the Public Service Department (Jabatan Perkhidmatan Awam Malaysia (JPA)). BEM is one of the signatories of Washington Accord (WA) since 2009. The accredited engineering degree programmes by EAC are given full recognition as equivalent qualifications by other signatories of WA including United State, United Kingdom, Australia, Hong Kong, Japan and Singapore.

The implementation of Outcome-based Education (OBE) which is promoted by WA becomes one of the mandatory requirements in an accreditation of engineering degree programmes outlined by EAC. Specific outcomes taking inputs from the stakeholders must be designed in one curriculum to produce high quality graduates that meet the needs of the students and stakeholders. Specifically, the teaching, learning and assessment methods shall be conducted consistently accordance to the appropriate graduate attributes listed in the Programme Educational Objectives (PEO), Program Outcomes (PO), Course Outcomes (CO) of the education content (BEM, 2017).

International Engineering Alliance presented a total of twelve differentiating characteristics for WA graduates attributes for engineer track as shown in Table 1 (Accord, 2013). The characteristics described with respective knowledge profiles were adopted as the POs in an undergraduate programme EC220 in Faculty of Civil Engineering (FCE), Universiti Teknologi MARA (UiTM). Integrated Design Project (IDP) is one the courses offered to the final year students of undergraduate programme in FCE, UiTM. This paper presents the assessment of a capstone project (CP) for IDP which is implemented in outcome-based manner. The mapping of the addressing COs and POs to the scopes of assessment is also demonstrated.

PO	Attribute	Description
1	Engineering Knowledge	Apply knowledge of mathematics, natural science, engineering fundamentals and an engineering specialisation as specified in WK1 to WK4 respectively to the solution of complex engineering problems;
2	Problem Analysis	Identify, formulate, conduct research literature and analyse complex engineering problems reaching substantiated conclusions using first principles of mathematics, natural sciences and engineering sciences (WK1 to WK4);
3	Design/ Development of Solutions	Design solutions for complex engineering problems and design systems, components or processes that meet specified needs with appropriate consideration for public health and safety, cultural, societal, and environmental considerations (WK5);
4	Investigation	Conduct investigation of complex engineering problems using research-based knowledge (WK8) and research methods including design of experiments, analysis and interpretation of data, and synthesis of information to provide valid conclusions;
5	Modern Tool Usage	Create, select and apply appropriate techniques, resources, and modern engineering and IT tools, including prediction and modelling, to complex engineering problems, with an understanding of the limitations (WK6);
6	The Engineer and Society	Apply reasoning informed by contextual knowledge to assess societal, health, safety, legal and cultural issues and the consequent responsibilities relevant to professional engineering practice and solutions to complex engineering problems (WK7);
7	Environment and Sustainability	Understand and evaluate the sustainability and impact of professional engineering work in the solutions of complex engineering problems in societal and environmental contexts. (WK7);
8	Ethics	Apply ethical principles and commit to professional ethics and responsibilities and norms of engineering practice (WK7);
9	Individual and Team Work	Function effectively as an individual, and as a member or leader in diverse teams and in multi-disciplinary settings;
10	Communication	Communicate effectively on complex engineering activities with the engineering community and with society at large, such as being able to comprehend and write effective reports and design documentation, make effective presentations, and give and receive clear instructions;
11	Project Management and Finance	Demonstrate knowledge and understanding of engineering management principles and economic decision- making and apply these to one’s own work, as a member and leader in a team, to manage projects in multidisciplinary environments;
12	Life Long Learning	Recognise the need for, and have the preparation and ability to engage in independent and life-long learning in the broadest context of technological change.
Note: WK denotes the knowledge profile

Table 1. Graduate attribute profiles for WA. (Accord, 2013)

This paper is organized as follows. Section 2 introduces background of the implementation of OBE. Section 3 presents the implementation of OBE in the course Integrated Design Project (IDP). Section 4 discusses the measurement of the POs and COs through a sample analysis for a capstone project in IDP. Finally, our works of this paper are summarized in the last section.

2. Outcome-Based Education

The implementation of OBE focuses on the achievement of graduate attributes in the teaching and learning processes. OBE incorporates innovative teaching methods such as problem-based leaning, capstone projects, oral presentations, professional talks and site visits. OBE approach at an institutional level always deals with generic graduate attributes whereas at the program level focuses on the specific discipline-oriented goals of the program such as the POs and COs. Continuous Quality Improvement (CQI) of an education could be then judged by government and accreditation agencies based on the assessment of student attributes through the implementation of OBE (El-Maaddawy & Deneen, 2017). Effectiveness of OBE implementation in promoting teaching and learning process relies on strong interconnection between the proper teaching, learning and assessment activities and the constructive mapping of the outcomes in terms of graduate attributes (Taras, 2015). An example of implementation model of OBE practices for the civil engineering students was demonstrated by Alias and Bhkari (2017).

Assessment is an essential process in OBE implementation. With clear and measureable outcomes, the results from the assessment could be used for the CQI, specifically in the teaching and learning process. Assessment methods for student performance are generally classified into direct methods and indirect methods (Easa, 2013). A set of indicators are used in a direct assessment method to examine and observe student knowledge or skills in a direct manner. Popular direct methods are final examination, test, quiz, viva, rubrics, presentation, project, and behavioral observations. An example of rubrics to evaluate students attributes for design courses and the capstone courses was developed (Platanitis & Pop-Iliev, 2010). Alternatively, indirect methods indicate student performance through perceptions and self‑assessment. Common indirect methods are entrance-exit surveys, alumni surveys and feedback forms. For example, Henry Samueli School of Engineering and Applied Science at the University of California, Los Angeles applied a qualitative assessment of The High School Summer Research Program through exit and alumni survey to determine the growth of the program, benefits and long-term impact of the program to the students (Kittur, Shaw & Herrera, 2017). A model with the method of peer assessment has been developed in the assessment cycle. This approach discussed the activities in peer assessment and proposed the peer assessment for the learning opportunities rather than grade analysis (Reinholz, 2016).

3. Integrated Design Project

Integrated Design Project (IDP) is the course offered to the final year students of a 4-year undergraduate programme in Faculty of Civil Engineering, University Teknologi MARA. IDP simulates a real world integrated design environment through a capstone project (CP). CP generally involves a process of solving civil engineering related design problems with considerations on safety; accessibility, constructability and sustainability; impact on the aesthetics, societal and environmental. CP also involves the study of cost‑effectiveness of the proposed design solutions. The conduct of the CP promotes the students to enroll as role-playing planners, managers, engineers, operators and maintenance community, fostering coordination and cooperation among team members. This section presents the scopes of assessment in the CP and the mapping of the associating COs and POs to each of the scopes of assessment.

3.1. Scopes of Project

The overall scopes of CP are relying on two main parts: (1) a final report and (2) a presentation of the design solutions. Figure 1 summarizes the scopes under the main components of the CP.

The integrated design components in the final report of CP are categorized as follows:

1. (a)Superstructures and substructures design

Structural analysis and design of a superstructure (i.e. building) by using modern tools, verified by manual calculations with the relevant design code of practices are expected. Structural key plans and detailings for the superstructure shall be prepared. Additionally, a proposal of processes or systems or materials that are environmental responsible (green building related) should also be included.

A design proposal for an appropriate substructure (i.e. foundation system) for the superstructure is also expected. Justification of the proposal could be based on a good interpretation of soil investigation data and geotechnical analysis of the given site.

1. (b)Infrastructures design

CP also includes design of infrastructure works namely road and earthwork, sewerage, water supply and drainage systems. The design of the infrastructures is carried out with the considerations of the authority requirements, study of the site and neighborhood topography, impacts to the societal and environmental aspects, accessibility and availability of local suppliers.

1. (c)Construction management

Cost-effectiveness study of the design in (a) and (b) and the planning of the construction phases for the design are incorporated in CP. The students are required to prepare the taking off, estimation, bill of quantities and the construction schedule for the project.

At the end of the IDP course, students are required to present their design solutions to a numbers of evaluators specifically professional engineers from construction industry. The presentation of CP is evaluated based on the students’ communication skill, sequential and organization of the presentation, ability to discuss and teamwork during the presentation session especially in answering the questions.

Figure 1. Scopes of the capstone project in IDP

3.2. Mapping of CO and PO

A total of four COs and seven POs are mapped for CP in the course IDP. The COs and POs reflect the potential and ideal graduate attributes derived from general contents of the course. It is noted that these outcomes are to be achieved by the students who have completed the course. Table 2 shows the descriptions written in a form of objectives of the four COs in IDP course. The COs appropriately emphasize the aims of the course in producing students with the abilities to solve complex design problems under constraints, while demonstrating effective communication and leadership skills.

Table 3 shows the CO-PO matrix and the addressing domains in CP. Mapping of the COs and POs is developed in a parallel way. One or more POs are mapped to a CO according to their relevancy to the particular CO. POs that are mapped under COs should be able to clearly explain the specific outcomes after completing CP.

CO	Descriptions
CO1	Define and formulate solutions to complex design problem.
CO2	Use the design standards with consideration of real world constraints in particular the environmental responsibilities of professional engineer and matters related to economic, legislation, lifespan, ethical, social etc.
CO3	Demonstrate effective communication skill through presentation and defense of design project.
CO4	Demonstrate leadership skills, ability to work independently and in a team through project design, presentation and defense of project findings.

Table 2. COs in IDP course

CO-PO Matrix		Descriptions for POs	Domains
CO1	PO2	Problem Analysis	C
	PO11	Project Management and Finance	C
CO2	PO3	Design/ Development of Solutions	C
	PO5	Modern Tool Usage	C
	PO12	Life Long Learning	C
CO3	PO10	Communication	A
CO4	PO9	Individual and Team Work	A
Note: C denotes Cognitive, A denotes Affective

Table 3. CO-PO Matrix for CP

4. Evaluation of Capstone Project

This section presents evaluation method for report and presentation in CP. Analysis results of student attainment in COs and POs for two semesters are also presented.

4.1. Assessment Rubrics

Evaluation of the graduate attributes is made through a standard assessment form with descriptive rubrics. The assessment form with rubrics is given to the students during the assignment of CP. Awareness on the outcomes of CP specifically based on CO-PO by both lecturers and students is essentially needed.

Table 4 shows the assessment of CP, inclusive the scopes (see also Figure 1), the mapping of CO-PO to the scopes and the associated marks. The assessment form is divided into two parts: report and viva.

It is noted that CO1 and CO2, with the addressing PO2, PO3, PO5, PO11 and PO12 are the expected outcomes after completing the design report. The students attain the skills of analysis and design, application of modern tools, understanding of fundamental in management and appreciation of life‑long learning in proposing design solutions via report. Although PO3 and PO5 are associated with CO2, the evaluation for each of the scopes is carried out explicitly. Specifically, independent rubrics are available for the evaluation for CO2-PO3 and CO2-PO5 for the scopes 1(a-d) and 1(g-j). For instance, for scope 1(b), assessments on the student ability to design with constraints through CO2-PO3 whereas the ability to use the modern tool for the design through CO2-PO5 are shown. Meanwhile, for the viva session, the students are measured on their communication skills, CO3-PO10 and ability to work as a team via CO4-PO9 are expected. The simplified rubrics for the assessment of CP are presented in Table 5.

1.Report - Analysis & Design		CO	PO	Marks
a.	Structural Key Plan	CO2	PO3 PO5	5
b.	Design Using Tool – Super and Sub-structures			10
c.	Manual Calculation – Super and Sub-structures			10
d.	Detailing			5
e.	Interpretation of Results	CO1	PO2	5
f.	Green Building	CO2	PO12	5
g.	Road & Earthwork	CO2	PO3 PO5	5
h.	Sewerage System			5
i.	Drainage System			5
j.	Water Supply System			5
k.	Taking off Quantities	CO1	PO11	5
l.	Bill of Quantities			5
m.	Estimation			5
n.	Project Planning			5
Total Marks for (1)				80
2-5 Viva - Defend		CO	PO	Marks
Communication Skill		CO3	PO10	5
Presentation Sequential		CO3	PO10	5
Discussion (Q&A)		CO4	PO9	5
Teamwork		CO4	PO9	5
Total Marks for (2-5)				20

Table 4. Assessment form of CP

4.2. Sample Analysis

Both direct assessment (i.e. projects, test, and presentation) and indirect assessment (i.e. exit-entrance surveys) are implemented in IDP course. Only the evidence of student achievement of CO-PO particularly on CP taken from assessment of IDP students is presented. The percentage of students attainment in COs and POs particularly after completing the course IDP is analyzed by using excel template at course level. The results of COs and POs for every course in the undergraduate programme EC220 are analyzed through a system MyCOPO at programme level. MyCOPO is a system utilized by FCE University Teknologi MARA to analyze individual student outcomes. Actions for CQI could be then planned if any students failed to achieve any POs.

Figure 2 shows the percentage of attainment of COs and POs specially in completing CP for two semesters. There are 159 and 215 final year students who have attended the course for the semester March to September 2016 and semester September to January 2017, respectively. The assessment of the COs and POs for the course was conducted by the authors through the assessment form (Table 4) and constructive rubrics (Table 5).

The results in Figure 2 reveal that all students adequately achieved all COs and POs (i.e. greater than 50%). The attainment of all COs and POs, based on IDP student performance, was on average 82%, with a standard deviation of 4%.

Item	Criteria (CO-PO)
1(a-d), 2(g-j)	Design with constraints (CO2-PO3) Accuracy in the design of structural elements with appropriate consideration for public health, safety, cultural, societal and environmental.
	Design Using Tools (CO2-PO5) Application of appropriate techniques, resources & modern engineering tools in performing design processes, analysis and justification of cost effective findings.
1e	Interpretation of Results (CO1-PO2) Interpretation of design outputs from manual calculation and computer software.
1f	Green Building (CO2-PO12) Engagement in life-long learning on the efficiency of resources through proposal of materials or systems in green building.
1(k-n)	Project management (CO1-PO11) Application of fundamental knowledge on management in taking off, bill of quantities, estimation, planning and scheduling.
2-3	Communication Skill (CO3-PO10) Presentation with correct pronunciation and present content of design project in logical and interesting sequence and attractive slides within allocated time.
4-5	Discussion (Q&A) (CO4-PO9) Teamwork in answering questions correctly and clear roles, constant transition between team members, effective cross reference each other.

Table 5. Simplified rubrics for CP

Figure 2. CO-PO attainment for CP

The highest attainment of about 86% was recorded for CO2-PO5 (student ability in the application of modern tools) for both semesters. Attainment CO3-PO10, pertaining to student communication skills, exhibited low attainment of about 77% for both semesters. Attainment CO1-PO2 (student ability in interpretation and analysis of results) was recorded as the lowest of approximately 76% for semester March to September 2016. This suggests that the activities such as presentation for project could be conducted more frequent throughout the semester before the final viva session to the panels.

5. Conclusions

This paper shows an evidence of the implementation of OBE in IDP. This paper also presents the guidelines for assessing graduate attributes assessment in civil engineering education particularly through a capstone project. Rubrics for making the assessment more efficient and judicious are presented. The mapping of CO-PO, assessment method and constructed rubrics have successfully measured student performance in completing their project. Results of an effective assessment of student attainment in COs and POs are important in the CQI for future teaching and learning processes. It is hoped that this paper will provide useful guidelines on assessment of graduate attributes for engineering programs.

Acknowledgement

Authors would like to thank Faculty of Civil Engineering, Universiti Teknologi MARA, Shah Alam, Selangor, Malaysia for the supports.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

References

Accord, W. (2013). Graduate attributes and professional competencies.

Alias, A., & Bhkari, N.M. (2017). A Model of Outcome-Based Education (OBE) for Engineering Education. Gading Journal for the Social Sciences, 11(02), 71-87.

BEM. (2017). Engineering Programme Accreditation Manual: Engineering Accreditation Council.

Easa, S.M. (2013). Framework and guidelines for graduate attribute assessment in engineering education. Canadian journal of civil engineering, 40(6), 547-556. https://doi.org/10.1139/cjce-2012-0485

El-Maaddawy, T., & Deneen, C.C. (2017). Outcomes-based assessment and learning: Trialling change in a postgraduate civil engineering course.

Kittur, H., Shaw, L., & Herrera, W. (2017). A new model for a multi-disciplinary engineering summer research program for high school seniors: program overview, effectiveness, and outcomes. Journal of STEM Education, 18(4).

Laws of Malaysia (1967). Act 138: Registration of Engineers Act 1967 (Revised 2015).

Platanitis, G., & Pop-Iliev, R. (2010). Establishing fair objectives and grading criteria for undergraduate design engineering project work: An ongoing experiment. International Journal Research & Reviews in Applied Science, 5(3), 271-288.

Reinholz, D. (2016). The assessment cycle: a model for learning through peer assessment. Assessment & Evaluation in Higher Education, 41(2), 301-315. https://doi.org/10.1080/02602938.2015.1008982

Taras, M. (2015). Excellence in University Assessment: Learning from award-winning practice, by David Carless. London Review of Education, 13(3), 59-61. https://doi.org/10.18546/LRE.13.3.09

---