Journal of Technology and Science Education

• Home
• About
• Login
• Archives
• Submissions
• Publication Fee
• Indexing & Statistics

Home > Vol 11, No 2 (2021) > Garmendia

ENGINEERING INSTRUCTOR PERCEPTION OF PROBLEM-
AND PROJECT- BASED LEARNING: LEARNING,
SUCCESS FACTORS AND DIFFICULTIES

UPV/EHU (Spain)

Received June2020

Accepted February 2021

Abstract

This work considers three research objectives: to analyze the perception of instructors of the incidence of PBL/PjBL on content learning and skill development; to identify the success factors that they believe promote learning when using them, as well as their importance; and to identify the difficulties they face, and the frequency with which they occur. The responses to a questionnaire administered to 50 instructors who participated in a specific training program were analyzed. The results show that the instructors’ perception is that both models contribute to a better understanding of the contents with regard to their practical application, and to a high level of skill development in their students, with the most favored being group work, decision-making, autonomous learning and problem solving. The instructors consider important success factors to be student involvement in their own learning from the very beginning, feedback from the professor, the tasks having been well-designed and team work and cooperation among students. The most common difficulties identified in our study correspond to the excessive workload associated with monitoring the students, and managing and developing within the established time the planning of their tasks and activities, although there is a medium level of incidence in this regard, and it may be due to the characteristics of the training program received. Exploring these aspects in greater depth in future investigations could facilitate the development of more effective teaching practices.

 

Keywords – Problem-Based Learning, Project-Based Learning, Skills, Success factors, Difficulties.

To cite this article:

Garmendia, M., Aginako, Z., Garikano, X., & Solaberrieta, E. (2021). Engineering instructor perception of problem- and project- based learning: Learning, success factors and difficulties. Journal of Technology and Science Education, 11(2), 315-330. https://doi.org/10.3926/jotse.1044

 

----------

1. 1. 1. Introduction

For the first time in 2015, the European Accreditation of Engineering Programmes (ENAEE) network established the accreditation criteria for engineering studies within the European Higher Education Space for bachelor’s and master’s degree studies. Among these criteria, two refer to the knowledge, comprehension and continuous updating of contents, while six of them, the vast majority, have to do with the development of professional skills, such as analysis, design, problem solving, research, the development of judgment, practical application of techniques and methods and communication. This new focus on learning engineering has been an important factor that has fueled a process of evolution in the teaching of engineering from a scenario in which it was primarily focused on the acquisition of technical knowledge, to a mixed model that incorporates a more practical approach oriented toward the professional activity and focused on the acquisition of skills.

Even though there is a wide variety of methodologies to develop professional skills, in the field of engineering, those with an important presence around the world are Problem-Based Learning (PBL) and Project-Based Learning (PjBL), due to their relationship and appropriateness for the development of professional competences in future graduates (Mills & Treagust, 2003). In some models, such as that of the University of Aalborg (Denmark), both strategies are combined throughout the training program (Kolmos, 2004).

Various definitions of PBL can be found in the professional literature. Barrows (1994), who was involved in the early stages of the development of PBL at McMaster University, defines the concept as student‑based instruction, which takes place in small groups, with the instructor acting as a facilitator, and organized around problems. According to the author, the method is basically implemented in three phases: to begin with, the students are presented the problems, even before they have acquired the theoretical knowledge. Professional reasoning skills are developed and the learning needs are identified, all within a cooperative environment with a tutor. The next phase is one of individual study, motivated by the previous phase. It finishes with a cooperative learning phase in which the knowledge acquired is applied to the problem and the learning is summarised (Barrows, 1994).

The PBL model is not unique; based on the theory developed by Barrows, each institution has established a PBL model according to its own context and objectives. Savin-Baden (2014) classified the different PBL practices into nine different types or modes, according to the way in which they are implemented, including PjBL among them. Both models share the fact that the learning process begins with a real situation (be it a problem or a project request) that the students must resolve or develop, working as a team, identifying their learning needs and acquiring them in an autonomous manner, in order to apply them to the problem or project and develop a viable solution or a final product (De Graaff & Kolmos, 2003). Both models therefore share the same main basic aspects of learning, with the main difference between them being that PBL is oriented towards the process, while PjBL is more focused to the product (Savin-Baden, 2007).

In recent years, numerous research studies have been published on the integration of PBL and PjBL in instruction. However, according to the review carried out by Beddoes, Jesiek and Borrego (2010), the status and trends in research on PBL/PjBL revealed that most of the publications were limited to describing initiatives for classroom implementation in specific courses, as opposed to those publications, which constitute a minority, that are dedicated to analyzing the effectiveness of the method in terms of student learning, and with regard to acquiring technical knowledge and developing professional skills.

In the educational areas of technology and sciences, many authors recommend the use of PBL/PjBL (Kolmos, 2004; Mills & Treagust, 2003; Akınoğlu & Tandoğan, 2007). The general conclusion that appears repeatedly in most of these studies with regard to the effect of PBL on student learning is summarized by Felder and Brent (2016: page 238) in the following manner: “Relative to traditionally taught students, student who participated in project-based learning did as well or slightly better on test of content knowledge and significantly better on assessment of conceptual understanding, metacognitive skills”. An example of this is the study by Galand, Frenay and Raucent (2012), which comparatively analyzed the learning by four cohorts of students (two PBL and two traditional learning). In their analysis, they evaluated the learning on three different levels: comprehension of the concepts, comprehension of the principles and application of knowledge. The results showed that students who had followed a PBL model demonstrated a statistically significant positive effect as compared to those who studied under a traditional education model in terms of the application of knowledge. With regard to content learning, PBL students showed no negative effects.

Along these same lines, in an analysis of several different articles, Harmer (2014) concludes that the use of PBL improves the academic results and contributes to the development of generic competences. Felder and Brent (2016) group them into five main competences, confirming their positive effect:

• •Team work (De Camargo Ribeiro, 2008; Helmi, Mohd-Yusof & Phang, 2016; Jun, 2010) 

• •Creative thinking: problem solving and finding innovative solutions (Warnock & Mohammadi-Aragh, 2016; Yadav, Subedi, Lundeberg & Bunting, 2011) 

• •Critical thinking: making evaluations and decisions based on evidence (Santos & Silva, 2018; Jun, 2010) 

• •Self-directed learning: identifying the needs of the learning itself, and identifying the necessary resources to meet them and learn in an autonomous manner (Dochy, Sergers, Van den Bossche & Gijbels, 2003; Fang, 2012) 

• •Oral and written communication (Zeng & Xu, 2010; Fang, 2012) 

Furthermore, with regard to the possible factors that might promote the learning of students when using PBL/PjBL, thus contributing to increasing the efficacy of the method, this line of research is still in its incipient stages. Strobel and Van Barnefeld (2009) suggest that research should take a closer look at the specific practices that are effective in order to properly guide students and thus reinforce the effectiveness of the method. Ravitz (2009), in turn, believes that future research must specify how PBL is used in different disciplines and contexts. He understands that studies are needed that provide information about the practice of the method, and believes that the specific mechanisms must be identified that contribute to its efficacy. In line with the above, Domènech-Casal, Lope and Mora (2019) suggest that the transition towards PjBL methodologies requires guidance and support that make it possible to identify the instructional elements that must be articulated in the educational proposals.

Mohd, Darus, Saip, Baharom, Puteh, Husin et al. (2017), based on the characteristics, evaluation and practice of PBL, propose in a fairly generic manner a list of possible success factors: self-directed learning, self-reflection, teaching skills, the role of the instructor, student-focused learning, constructivism, group training, group activities, knowledge sharing, proposed activities, instructor evaluation, co-evaluation and self-evaluation. Of these, in the context of their study, they select the four highest rated factors by their students: self-evaluation, the role of the facilitator, constructivism and group activities and suggest that future research should refine the list of factors, which should be analyzed in additional fields of knowledge.

Furthermore, in relation to the difficulties and challenges that arise when using PBL/PjBL in the classroom, Chen, Kolmos and Du (2020), in a qualitative meta-analysis of 108 research articles on PBL/PjBL applied to instruction in engineering, conclude that, in spite of the great variety of practices and variants of PBL/PjBL analyzed in their study, the challenges identified were not related to any particular practice, rather they corresponded to general difficulties present in most of them. They grouped those that occurred most commonly on an individual and institutional level into several categories:

• •A lack of training allowing instructors to learn how to transition from their expository role into one of a facilitator, how to design the course activities, how to facilitate team work among their students and how to find a balance between helping and influencing the work of their students. 

• •Difficulties in selecting effective evaluation methods (self- and co-evaluation, presentations, observation, etc.), especially for the evaluation of skills such as communication, team work, problem solving and autonomous learning. 

• •The need for ongoing training of students in PBL skills, such as team work, conflict management, how to communicate effectively on a team, self-learning, time management and team management, etc. 

• •Greater dedication of time and effort in order to provide professional guidance and practical experience, as well as to facilitate team work during the problem and project development process. 

• •Lack of support from departments and institutions, responding to the needs for support materials, resources and strategies to improve the effectiveness of PBL/PjBL. 

• •Difficulty for institutions to design an effective PBL curriculum. 

• •External limitations, such as the high student-instructor ratio, the lack of infrastructures for team work or the lack of technical and economic support. 

In agreement with these authors, further research is needed in order to determine the success factors in the current practice of PBL/PjBL and the sharing of effective strategies and experiences that make it possible to face the difficulties and challenges in instructional practice.

According to the review that was carried out, the present work intends to delve deeper into the aforementioned aspects and show the results obtained in PBL/PjBL implementations in engineering degree courses at the University of the Basque Country. The following study objectives have been considered:

1. 1.To analyze the perception of instructors with regard to the incidence of PBL/PjBL on content learning and skill development. 

2. 2.To identify the success factors that instructors believe promote the learning when using PBL/PjBL and their importance. 

3. 3.To identify the difficulties that instructors find when using PBL/PjBL and the frequency with which they occur. 

4. 4.To analyze whether there are any differences in the perception of the instructors who have implemented PBL or PjBL in the aforementioned aspects in the previous objectives. 

2. Methodology

2.1. The Sample and Context of the Study

The integration of PBL/PjBL has been carried out in a specific context, since the instructors of the courses targeted by this research have voluntarily participated in a specific training program on these methodologies (Garmendia, Barragués, Zuza & Guisasola, 2014). The program has four phases:

• •Introductory workshop (20 hours). The professors participate in an initial training workshop with experts, in which the participants experience the methodology while taking on the role of students and then analyze and debate the strategies followed from the perspective of the professor, discussing different strategies, methodological options and possible difficulties in their implementation. In addition to training on PBL/PjBL, different strategies are analyzed to develop team work, cooperative learning and problem-solving skills. 

• •Design of a PBL/PjBL teaching proposal (4 months of development). Each participant designs and plans at least 35% of the credits for their course, in a guided manner through deliverables that are evaluated by a tutor assigned by the training program, introducing improvements in the design until the final version is obtained. 

• •Implementation of the completed design (one quarter). The faculty implements the completed design in class. During this period, communication is maintained with the tutor and the rest of the participants either on-site or through a virtual forum, in order to share experiences and, in particular, the difficulties encountered and different ways of dealing with them. The tutors make an on-site observation in the classroom to offer feedback, comments and suggestions for improvement. 

• •Final validation and publication of the PBL/PjBL proposal (one month). Once the practical implementation has been completed, the participant makes a final report, analyzing and assessing the results obtained, the difficulties encountered and the modifications made to the design. The tutors and managers of the program evaluate the final proposal and give their approval for possible publication in the Methodological Resource Center at the University of the Basque Country. 

A total of 77 instructors participated in the program, who taught 57 courses belonging to the engineering degrees offered at 5 UPV/EHU centers, including industrial, telecommunications, organization, civil engineering, environmental engineering, mining and other fields. The implementation of PBL/PjBL impacted an average of 46% of the ECTS of the courses. By way of illustration, below are some examples of PBL/PjBL proposals that have been implemented, with an initial driving question, followed by a problem or project scenario, the topic being studied and the course in which it was implemented:

Driving question	Topic	Course
How would you ensure that the water supply reaches my 12th floor flat?	Design and calculation of a pumping facility	Fluid Mechanics Engineering
Are combined-cycle thermal power plants the solution to the current energy and environmental crisis?	Thermal steam, gas and internal combustion engines	Thermotechnology
What is the environmental impact of waste water treatment on a town?	Study of the environmental impact of a civil works project	Environmental Engineering
How can you have a cold beer in the desert?	Energy balance	Chemical Engineering

Table 1. Examples of Problem- and Project-Based Learning proposals

2.2. Data Collection

An on-line questionnaire of our own creation has been used, which was sent to engineering instructors participating in the program. This questionnaire consists of three parts, with items about each of the aspects being studied:

• •Evaluation of the learning achieved, quantified on a scale of 1 to 4 (1: Very little, 2: Little, 3: Quite a bit, 4: A lot) according to the degree in which the methodology contributed to the learning of the contents or to developing skills in the students. 

• •Evaluation of the success factors in the learning when using PBL/PjBL, indicating on a scale of 1 to 4 (1: Very little, 2: Little, 3: Quite a bit, 4: A lot) the extent to which said factors promoted learning. 

• •Evaluation of the difficulties in implementing PBL/PjBL, indicating on a scale of 1 to 5 (1: Never, 2: Rarely, 3: Sometimes, 4: Often, 5: Continuously) the frequency with which these difficulties were encountered when implementing the PBL/PjBL methodology in the classroom. 

Questions on the questionnaire were developed following the qualitative analysis of the final reports that the instructors presented at the end of the training program, in which they described the progress of the experience, offered evidence of the results obtained in terms of student learning and evaluated the implementation.

In the analysis of these reports, two members of the research team identified and categorized the success factors and the difficulties mentioned by the instructors (Aginako, Garmendia & Bezanilla, 2017), and a preliminary version of the questionnaire was developed. Two methods were used to validate the questionnaire. On the one hand, the first version of the questionnaire was sent to two professors participating in the program for consultation. Their comments and contributions were used to modify the wording of some of the items that might have been ambiguous, resulting in the final version, which was sent to the instructors who had participated in the program. As a second form of validation, the internal consistency of the final questionnaire was evaluated according to Cronbach’s alpha, which resulted in a value of 0.83 for those factors that promote learning and 0.89 for those related to difficulties, both of which are considered to be good according to George and Mallery (2003).

Of a total of 77 instructors who formed the study group, 50 responded the questionnaire: 18 were professors who used a PBL approach in their lectures while 32 were PjBL professors. The response index of 65% is high for an online questionnaire, being the mean response index between 20% and 47% for an online survey, according to Nulty (2008).

To analyze whether the differences between PBL and PjBL are statistically significant, the Mann-Whitney U test was applied and the effect size was calculated using Cohen’s d.

3. Results and Discussion

3.1. Perception of Instructors with Regard to the Incidence of PBL/PjBL on Content Learning and Skill Development

Below (Table 2) the mean scores are shown, those that were given by the instructors for the items corresponding to contents and skills. In the first column is the mean of all the responses received (N=50), in order of maximum to minimum score, in an effort to reveal which aspects related to contents and which skills were developed to a greater extent thanks to the methodologies. In the next two columns, the scores for each methodology, PBL (N=18) and PjBL (N=32), the percentage difference between them, the level of significance p, applying the Mann-Whitney U test to analyze whether the difference between the two methodologies is significant, and the effect size by means of Cohen’s d.

With regard to the contents, the highest score (3.48, which falls between “Quite a bit” and “A lot”) corresponds to the item “Establish relations between theory and practice”, and the lowest score (3.06, “Quite a bit”) is for “Understand theoretical contents”. The instructors’ perception coincides with the results of previous studies (Felder & Brent, 2016; Galand et al., 2012), and confirms the suitability of both methodologies in order to contribute to better comprehension of the contents in view of their practical application.

Based on your own experience, the active methodology helped the students to: Very little (1), Little (2), Quite a bit (3), A lot (4)
Item		MEAN			DIF (%)	Mann-Whitney U	Effect size
		ALL	PBL	PjBL
						p	Cohen’s d
CONTENTS	Establish relations between theory and practice	3.48	3.44	3.5	1.74	0.647	0.092
	Relate course contents and obtain an integrated vision	3.28	3.22	3.31	2.80	0.566	0.127
	Understand theoretical contents	3.06	3.11	3.03	-2.57	0.647	0.135
SKILLS	Improve their group work skills	3.26	3.11	3.34	7.40	0.137	0.387
	Decision-making with regard to a real situation	3.24	3.06	3.34	9.15	0.238	0.307
	Develop their autonomy for learning	3.24	3.28	3.22	-1.83	0.830	0.054
	Solve problems or offer solutions for real situations	3.22	3.06	3.31	8.17	0.230	0.309
	Investigate on their own with regard to the proposed work	3.06	3.06	3.06	0.01	0.912	0.029
	Develop communication skills (oral or written)	2.96	2.94	2.97	1.02	0.764	0.069

Table 2. Evaluation of the effect of PBL/PjBL on content learning and skill development

The evaluations of the influence that the use of PBL/PjBL had on skills development falls between 3 (Quite a bit) and 4 (A lot), with the highest scoring being group work, decision-making, autonomous learning and problem solving (all of which scored above a mean of 3.2 on a scale of 1 to 4). It should be noted that these are four of the five competences to which PBL contributes, according to Felder and Brent (2016). The fifth competence is communication, which in our case is the one developed the least, from the instructors’ perspective (2.96), although it is also highly positive, practically at level 3 (Quite a bit).

Furthermore, it is observed that the scores by PjBL professors are higher than those by PBL instructors on all items except for two, but it is not possible to draw a conclusive result about the possible greater suitability of PjBL over PBL, since the results of the statistical tests indicate no significant difference between the two methodologies, and the effect sizes are small. In relation to the suitability of both methods for engineering studies, some authors, like Perrenet, Bouhuijs and Smits (2000) and Mills and Treagust (2003), believe that opting for the exclusive use of one of them, such as ABP, cannot by itself meet the needs of engineering studies. According to these authors, a combined proposal using PBL in the beginning courses to give problems a real context, followed by PjBL in the more advanced courses to tackle complex, interdisciplinary problems, could be a very beneficial solution in engineering studies. In line with this approach, in some models, such as that of the University of Aalborg (Denmark), both strategies are combined throughout the training program.

3.2. Instructor Perception of Success Factors that Promote Learning

Table 3 shows the results of the instructor scores for the factors that promote learning when using PBL/PjBL. The mean of the responses from all instructors has been organized from most to least, in order to show the relative importance they give to each of these factors. The next two columns show the results by methodology. Statistical tests have also been conducted to compare the means and determine whether the differences between the two PBL and PjBL groups are statistically significant.

Indicate from your point of view the extent to which the following factors promote learning when using PBL/PjBL: Very little (1), Little (2), Quite a bit (3), A lot (4)
Item	MEANS			DIF (%)	Mann-Whitney U	Effect Size
	ALL	PBL	PjBL		p	Cohen’s d
Student involvement in their learning from the very beginning	3.58	3.56	3.59	0.84	0.972	0.009
Professor feedback	3.56	3.61	3.53	-2.22	0.589	0.132
Having designed the tasks well	3.50	3.33	3.59	7.81	0.215	0.309
Team work and cooperation among students	3.50	3.56	3.47	-2.53	0.526	0.158
Greater reflection required by the tasks	3.46	3.67	3.34	-8.99	0.067	0.473
Daily active work	3.40	3.44	3.38	-1.74	0.820	0.057
Having given the contents a real or practical focus	3.36	3.22	3.44	6.83	0.332	0.248
Continuous evaluation	3.34	3.33	3.34	0.30	0.886	0.034
Student autonomy in learning	3.34	3.28	3.38	3.05	0.672	0.106
The positive attitude of students	3.32	3.33	3.31	-0.60	0.947	0.017
Having presented the tasks performed professionally in the field of engineering	3.28	3.00	3.44	14.67	0.027*	0.584*
More demanding tasks (greater complexity)	2.82	2.78	2.84	2.16	0.726	0.089

Table 3. Evaluation of the success factors that promote learning when using PBL/PjBL

The results show that the instructors believe that all the factors promote learning at a level between “Quite a bit” and “A lot”, except for the item “More demanding tasks” which is between “Little” and “Quite a lot”.

The difference is statistically significant for the item “Having presented the tasks performed professionally in the field of engineering”, with a higher score for PjBL (3.44). In other words, instructors who have implemented PjBL believe it to be more important to introduce real scenarios taken from professional practice than those who have implemented PBL. There are no statistically significant differences for the rest of the items, and therefore we can state that the success factors that promote learning coincide in both models. Furthermore, the evaluation of the importance that they are given is high (between “Quite a bit” and “A lot”), which suggests that considering them could contribute to developing more effective instructional proposals in the future.

The highest rated factor is “Student involvement in their learning from the very beginning”. In this sense, one of the recommendations from the training workshops the instructors attended was to implement PBL/PjBL in the course from the beginning (Guisasola & Garmendia, 2014), according to one of the principles of the PBL method, which is for the problem or project to be the origin and the common thread of the learning process, and not its final application, as commonly occurs in more traditional approaches. It seems that this characteristic of this inductive method, from the perspective of the instructors who have experienced it, is crucial in order to achieve greater student involvement.

The following two factors are related to instructors, with the highest rated being “Professor feedback”. This aspect is also mentioned as important in the study by Mohd et al. (2017), as part of the factor “Facilitator role”. In this methodology, feedback has to be given continuously and on several different occasions throughout the learning process, and should not be limited solely to grading tasks, but rather a continuous tracking of the evolution of the process must be carried out, which is centered on evaluating whether the learning is progressing, identifying the difficulties that the students encounter, answering any questions and correcting errors in time. In this sense, Glew (2003) and Wan Hamiza, Williams and Sher (2017) indicate that PBL can fail if there is not sufficient guidance from the instructor, an idea that is echoed by Henry, Tawfik, Jonassen, Winholtz and Khanna (2012), who conclude that prompt, appropriate feedback can improve the performance of students, correcting mistakes in their learning in time, so that they can apply this to subsequent problem modules.

The next factor, “Having designed the tasks well”, implies developing a well-thought out program of activities with a high level of detail, in which the most appropriate type of tasks must be determined in order to achieve the learning outcomes. Problematic scenarios and project proposals must be developed, determining the dynamics to follow in the classroom and on the student work teams, and carrying out detailed, careful time line planning (Bell, 2010). Both PBL and PjBL require a great deal of effort on behalf of instructors and students and in order to implement it, careful planning and a well-managed project are required (Alves, Sousa, Moreira, Carvalho, Cardoso, Pimenta et al., 2016).

Keeping in mind that PBL is carried out while working on small teams of students when tackling problems or implementing projects, it is no wonder that the next highest rated factor is “Team work and cooperation among students”. With regard to the group work guide, according to Wilkie (2004), instructors have identified the need to develop a different focus for group facilitation, and to shape the success of the student learning process during PBL, it is essential to establish efficient communication with them (Hung, Harpole Bailey & Jonassen, 2003). Furthermore, other tutor skills have to be developed for group work; according to Mayo, Donnelly, Nash and Schwartz (1993), they include the following: promoting group awareness in their own processes, promoting feedback within the groups, helping groups to define appropriate topics for study and guiding groups towards the integration of the learning in which they have been engaged (Hung et al., 2003). The tutor must be capable of facilitating productive cooperative relations (Wilkerson & Hundert, 1997), and an atmosphere of confident work (Schmidt & Moust, 1995).

All four factors mentioned are scored above 3.5, and would therefore be the most important aspects to bear in mind when implementing a PBL/PjBL proposal, according to the instructors who have experience with the methodology. The rest of the factors, except for the last one, are scored between “Quite a bit: 3” and “A lot: 4”, and therefore, even though they are not among the four most important, they would still be factors that, perhaps to a lesser extent, should also be taken into account in order to achieve more effective PBL in practice: proposing tasks that imply greater reflection on the part of students, developing a program of activities that involves them in the daily work, evaluating this work on a continuous basis, giving the contents a real or practical focus, incorporating professional work in the tasks and contributing to the development of autonomous learning in students.

3.3. Instructor Perception of the Difficulties and Challenges when Using PBL/PjBL

Table 4 presents the results of the instructor evaluations, in which they quantify the frequency with which they have encountered a series of difficulties when using PBL/PjBL (1: Never, 2: Rarely, 3: Sometimes, 4: Often, 5: Continuously). In order to see the relative importance of each difficulty, the first column shows the mean for the joint evaluation of the two methodologies, organized from highest to lowest. In the next two columns are the results according to methodology, and the result of the statistical tests comparing the means in order to determine whether the differences between the PBL and PjBL groups are statistically significant.

The mean of the responses is mostly between “Rarely: 2” and “Sometimes: 3”, on a Likert scale of 1 to 5, which means that the instructors consider the frequency with which they have encountered these difficulties to be low. These results differ in part from those found by Chen et al. (2020), since although practically all of them can be classified into one of the categories in which they grouped them, or appear as difficulties cited by authors in the sample of articles analyzed, in their meta-analysis, they were presented as “high frequency,” when in our case they are present in a couple of difficulties very close to the “Sometimes” range (the item with the highest frequency has a value of 3.1, with 3 being Sometimes), and in the rest with a lower frequency, such as “Rarely” or “Never”.

To check whether the perception of the difficulties was different between instructors who used PBL and PjBL, the Mann-Whitney U test for comparison of means was performed, observing that there are no statistically significant differences in any of the items between the two groups compared. This proves that instructors assess the difficulties experienced in a similar way, regardless of the methodology followed. This result coincides with that of Chen et al. (2020), when they conclude that, despite the wide variety of PBL/PjBL practices and variants analyzed in their study, the challenges identified were not related to a particular practice, but rather corresponded to general difficulties present in most of them.

The three most common difficulties correspond to the same category related to the work overload associated with monitoring students, and managing and developing the planning of their tasks and activities within the established time frame. These difficulties coincide with one of the categories mentioned by Chen et al. (2020): Increased dedication of time and effort during the problem and project development process. Mentzer, Czerniak and Brooks (2017) also mention that instructors frequently reference the lack of time as a major difficulty. According to Bell (2010), thorough and careful planning is essential for project development and student success, and this entails a dedication of time and effort that some instructors may not be able to take on.

The fourth most common difficulty, “Working with less autonomous students” is also mentioned by Chen et al. (2020) and other researchers, who noted that in the self-learning process, students have difficulties with identifying problems and transferring knowledge into practical solutions (Ahern, 2010; Bledsoe & Flick, 2012; Hu, Ortiz & Sriraman, 2014; Lutsenko, 2018). In this sense, Said, Adikan, Mekhilef and Abdrahim (2005: page 135) state: “One of the most pertinent questions to address is how to successfully instill a habit of critical thinking in students. This is easier said than done, due to the nature of the current education system, and it is difficult to undo old habits induced by a tradition of rote learning”.

Indicate the frequency with which you have encountered these difficulties when implementing PBL/PjBL methodology in the classroom: Never (1), Rarely (2), Sometimes (3), Often (4), Continuously (5)
Item	MEANS			DIF (%)	Mann-Whitney U	Effect Size
	ALL	PBL	PjBL		p	Cohen’s d
Undertaking the excess work load associated with the monitoring of continuous evaluation	3.10	2.72	3.31	21.68	0.093	0.564
Managing time, tasks and student activities	3.08	3.17	3.03	-4.28	0.342	0.179
Implementing the scheduled planning (doing all the activities within the established time)	2.98	3.06	2.94	-3.86	0.675	0.137
Working with students who are not very autonomous	2.84	2.72	2.91	6.76	0.595	0.164
Operating in an inappropriate classroom for group work	2.82	2.61	2.94	12.50	0.402	0.243
Working with a large number of students	2.78	2.50	2.94	17.50	0.326	0.301
Giving fast feedback to students on the activities performed	2.64	2.72	2.59	-4.72	0.593	0.14
Changing the way students work when they did not respond as expected	2.62	2.67	2.59	-2.73	0.705	0.074
Evaluating skills (team work, oral communication, autonomous learning, etc.)	2.54	2.33	2.66	13.84	0.259	0.385
Involving students	2.46	2.56	2.41	-5.84	0.597	0.15
Changing roles: from the focus of teaching to guided learning	2.46	2.61	2.38	-9.04	0.437	0.217
Detecting students who did not do the work	2.22	2.22	2.22	-0.16	0.883	0
Redirecting unfavorable or resistant attitudes by students	2.18	2.06	2.25	9.46	0.461	0.174
Redirecting errors that have been made in the design	2.16	2.17	2.16	-0.48	0.93	0.012
Using new assessment tools or techniques (rubrics, co-evaluation, etc.)	2.16	2.11	2.19	3.62	0.766	0.089
Evaluating content learning	2.14	2.06	2.19	6.42	0.601	0.146
Accessing computer resources or network connection	2.02	2.17	1.94	-10.58	0.67	0.216
Coordinating the evaluation with other course groups that did not use PBL/PjBL methodology	2.00	2	2	0.00	0.95	0
Forming and reconstituting work groups	1.98	2	1.97	-1.56	0.845	0.036
Managing conflicts among students	1.92	1.67	2.06	23.75	0.086	0.531
Accessing bibliographic resources	1.72	1.89	1.63	-13.97	0.312	0.322

Table 4. Frequency of difficulties encountered when using PBL/PjBL

The next two difficulties cited refer to inadequate classrooms for group work, and having to work with a large number of students, which also coincides with one of the categories of Chen et al. (2020): External limitations, such as the high student-instructor ratio, the lack of infrastructures for team work or the lack of technical and economic support. Institutions should keep in mind that implementing PBL/PjBL methodology requires physical spaces for students, both for teaching and for carrying out projects, and specific materials, access to laboratories, tools and equipment (Nunes de Oliveira, 2011; Spronken-Smith & Kingham, 2009; Graham, 2010).

Likewise, the difficulty for instructors to assess teamwork skills, oral communication, learning autonomy, etc. has been identified. This also coincides in this respect with the meta-analysis by Chen et al. (2020), who point out the difficulty of choosing effective assessment methods. Other authors also mention the difficulty in measuring the development of transferable skills by students and the real results of learning during a course (Chaparro-Peláez, Iglesias-Pradas, Pascual-Miguel & Hernández-García, 2013; Kunberger, 2013; Macias‑Guarasa, San-Segundo, Montero, Ferreiros & Cordoba, 2006; Marti, Gurguí, Gil, Hernández-Sabaté, Rocarias & Poveda, 2015).

Furthermore, it should be indicated that the instructors in the sample were trained in team work and cooperative learning strategies (Garmendia et al., 2014). Perhaps it is for this reason that the difficulties related to managing work on student teams are those least common on the questionnaire, and are cited in only two very specific aspects: “Forming and/or reconstituting work groups” and “Managing conflicts among students”, with a very low frequency (1.98 and 1.82, respectively, with 2 being “Rarely”).

One important difference related to the differences cited in the meta-analysis by Chen et al. (2020) is that, in our case, the instructors do not mention at any time the lack of training or the lack of institutional support as one of the difficulties they experienced. This is probably due to the fact that they participated in a specific innovation program that had institutional support and enough resources to meet their training needs: from an introductory workshop to support from tutors during the process of designing the PBL/PjBL proposal, and during its implementation.

In line with the above, it is likely that the frequency of the difficulties encountered in our study would vary in other contexts, especially if the instructors implement PBL/PjBL without any previous training, or had less intensive training or training with other characteristics. In any case, we suggest that instructor training in PBL/PjBL must include the analysis of the possible difficulties that might be encountered in its implementation, offering guidelines and strategies for overcoming them. Thus, we coincide with Mentzer et al. (2017) when they state that it takes time to learn how to use PBL in practice, even two or three years in order to change their comprehension and teaching practices, and there is a need to develop long-term and even ongoing and collaborative models for instructor training. Finally, support can also be of a political nature, with recognition of instructors for their efforts in implementing a new curricular focus. The need for sponsorship and executive support cannot be underestimated (Abrahamson, 1998).

4. Conclusions

The analysis of the results obtained according to the research objectives of this work has led us to make the following general conclusions:

• •With regard to content learning, the perception of instructors is that both methodologies, PBL and PjBL, contribute at a level of between “Quite a bit” and “A lot” to establishing relations between theory and practice, relating the course contents and obtaining an integrated vision, and to understanding theoretical contents, confirming the suitability of both methodologies in order to contribute to better comprehension of contents with a view to their practical application. Likewise, the instructors believe that they contribute to a high level of skill development in their students, with those promoted the most being group work, decision-making, autonomous learning and problem solving. 

• •Various success factors have been identified that can contribute to developing more effective teaching practices in the future. These factors are the same for both the PBL and PjBL models, and the most important are: student involvement in their own learning from the very beginning, feedback from the professor, the tasks having been well-designed and team work and cooperation among students. 

• •Difficulties have been identified in implementing PBL/PjBL, and the frequency with which they occur has been determined in order to quantify their relative importance. The three most common difficulties in our study correspond to the same category related to the work overload associated with monitoring students, and managing and developing the planning of their tasks and activities within the established time frame. These show a medium level of frequency between “Sometimes” and “Often”. The rest of the difficulties showed a lesser frequency, which contrasts with previous studies in which they are mentioned with a high level of frequency. This low incidence of difficulties seems to be due to the training program received by the instructors, which included tutor support during the entire design, development and implementation process. We coincide with other authors in that instructor training in PBL/PjBL should be developed in continuous medium to long-term models, and that they should have a cooperative focus to promote the exchange of experiences. 

• •The perception of instructors is similar when it comes to using PBL or PjBL, and no statistically significant differences are observed in the results obtained, or with regards to the content or skill development learning achieved, success factors or difficulties in use. 

Finally, we believe that it is necessary to continue to analyze and identify which success factors promote student learning when using PBL/PjBL and what difficulties instructors can encounter with their implementation and how to avoid them, so that this knowledge can give way to more effective teaching practices in the future.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

References

Abrahamson, S. (1998). Obstacles to Establishing Problem-based Learning. Journal of Dental Education. 62(9), 656-659. https://doi.org/10.1002/j.0022-0337.1998.62.9.tb03228.x

Aginako, Z., Garmendia, M., & Bezanilla, M.J. (2017). Difficulties teachers have when the implement a PBL or PjBL approach in the classroom at engineering schools. INTED2017 Proceedings, 6411-6421. https://doi.org/10.21125/inted.2017.1482

Ahern, A.A. (2010). A Case Study: Problem-Based Learning for Civil Engineering Students in Transportation Courses. European Journal of Engineering Education, 35(1), 109-116. https://doi.org/10.1080/03043790903497328

Alves, A., Sousa, R., Moreira, F., Carvalho, M.A., Cardoso, E., Pimenta et al. (2016). Managing PBL difficulties in an Industrial Engineering and Management program. Journal of Industrial Engineering and Management, 9(3), 586-611. https://doi.org/10.3926/jiem.1816

Akınoğlu, O., & Tandoğan, R.Ö. (2007). The Effects of Problem-Based Active Learning in Science Education on Students’ Academic Achievement, Attitude and Concept Learning. Eurasian Journal of Mathematics, Science and Technology Education, 3(1), 71-81. https://doi.org/10.12973/ejmste/75375

Barrows, H.S. (1994). Practice-based Learning: Problem-based Learning Applied to Medical Education. Springfield, IL: Southern Illinois University, School of Medicine.

Beddoes, K.D., Jesiek, B.K., & Borrego, M. (2010). Identifying opportunities for collaborations in international engineering educational research on problem- and project-based learning. Interdisciplinary journal of problem-based learning, 4(2). https://doi.org/10.7771/1541-5015.1142

Bell, S. (2010). Project-based learning for the 21st century: Skills for the future. The Clearing House, A Journal of Educational Strategies, Issues and Ideas, 83(2), 39-43. https://doi.org/10.1080/00098650903505415

Bledsoe, K.E., & Flick, L. (2012). Concept Development and Meaningful Learning Among Electrical Engineering Students Engaged in a Problem-Based Laboratory Experience. Journal of Science Education and Technology, 21(2), 226-245. https://doi.org/10.1007/s10956-011-9303-6

Chaparro-Peláez, J., Iglesias-Pradas, S., Pascual-Miguel, F.J., & Hernández-García, Á. (2013). Factors Affecting Perceived Learning of Engineering Students in Problem-Based Learning Supported by Business Simulation. Interactive Learning Environments, 21(3), 244-262. https://doi.org/10.1080/10494820.2011.554181

Chen, J., Kolmos, A., & Du, X. (2020). Forms of implementation and challenges of PBL in engineering education: a review of literature, European Journal of Engineering Education. https://doi.org/10.1080/03043797.2020.1718615

De Camargo Ribeiro, L.R. (2008). Electrical engineering students evaluate problem-based learning (PBL). International Journal of Electrical Engineering Education, 45(2), 152-161. https://doi.org/10.7227/IJEEE.45.2.7

De Graaf, E., & Kolmos, A. (2003). Characteristics of problem-based learning. International Journal of Engineering Education, 19(5), 657-662.

Dochy, F., Sergers, M., Van den Bossche, P., & Gijbels, D. (2003). Effects of problem-based learning: a meta-analysis.Learning and instruction,13(5), 533-588. https://doi.org/10.1016/S0959-4752(02)00025-7

Domènech-Casal, J., Lope, S., & Mora, L. (2019). Qué proyectos STEM diseña y qué dificultades expresa el profesorado de secundaria sobre Aprendizaje Basado en Proyectos. Revista Eureka sobre Enseñanza y Divulgación de las Ciencias, 16(2), 2203. https://doi.org/10.25267/Rev_Eureka_ensen_divulg_cienc.2019.v16.i2.2203

Fang, N. (2012). Improving engineering students’ technical and professional skills through project-based active and collaborative learning. International Journal of Engineering Education, 28(1), 26-36.

Felder, R.M., & Brent, R. (2016). Teaching and learning STEM. A practical guide. San Francisco, CA: Jonh Wiley & Sons.

Galand, B., Frenay, M., & Raucent, B. (2012). Effectiveness of problem-based learning in engineering education: A comparative study on three levels of knowledge structure. International Journal of Engineering Education, 28(4), 939-947.

Garmendia, M., Barragués, J.I., Zuza, K., & Guisasola, J. (2014). Proyecto de formación del profesorado universitario de Ciencias, Matemáticas y Tecnología, en las metodologías de Aprendizaje Basado en Problemas y Proyectos. Enseñanza de las ciencias: revista de investigación y experiencias didácticas, 113-129. https://doi.org/10.5565/rev/ensciencias.911

George, D., & Mallery, P. (2003). SPSS for Windows step by step: answers to selected exercises. A Simple Guide and Reference, 63, 1461-1470.

Glew, R.H. (2003). The problem with problem‐based medical education: Promises not kept. Biochemistry and Molecular Biology Education, 31(1), 52-56. https://doi.org/10.1002/bmb.2003.494031010158

Guisasola, G., & Garmendia, M. (2014). Aprendizaje basado en problemas, proyectos y casos: diseño e implementación de experiencias en la universidad. Servicio Editorial de la Universidad del País Vasco/Euskal Herriko Unibertsitateko Argitalpen Zerbitzua.

Graham, R. (2010). UK approaches to engineering project-based learning. White Paper sponsored by the Bernard M. Gordon/MIT Engineering Leadership Program. Available at: http://www.rhgraham.org/RHG/Recent_publications_files/MIT%20White%20Paper%20-%20UK%20PjBL%20April%202010_1.pdf (Accessed: February 2019)

Harmer, N. (2014). Project-based learning: Literature review. Plymouth University. Availabe at: https://www.plymouth.ac.uk/uploads/production/document/path/2/2733/Literature_review_Project-based_learning.pdf (Accessed: July 2017)

Helmi, S.A., Mohd-Yusof, K., & Phang F.A. (2016). Enhancement of team-based problem solving skills in engineering students through cooperative problem-based learning. International Journal of engineering education, 32(6), 2401-2414.

Henry, H.R., Tawfik, A.A., Jonassen, D.H., Winholtz, R.A., & Khanna, S. (2012). I know this is supposed to be more like the real world, but...: student perceptions of a PBL implementation in an undergraduate materials science course. Interdisciplinary Journal of Problem-based Learning, 6(1). https://doi.org/10.7771/1541-5015.1312

Hu, J., Ortiz, A.M., & Sriraman, V. (2014). Course Implementing PBL in a Concrete Construction Course. In Proceeding of 2014 ASEE Annual Conference & Exposition. https://peer.asee.org/implementing-pbl-in-a-concrete-construction-course

Hung, W., Harpole Bailey, J., & Jonassen, D.H. (2003). Exploring the tensions of problem‐based learning: insights from research. New Directions for Teaching and Learning, 2003(95), 13-23. https://doi.org/10.1002/tl.108

Jun, H. (2010). Improving undergraduates’ teamwork skills by adapting project-based learning methodology. In 2010 5th International Conference on Computer Science and Education (ICCSE), 652-655. IEEE. https://doi.org/10.1109/ICCSE.2010.5593527

Kolmos, A. (2004). Estrategias para desarrollar currículos basados en la formulación de problemas y organizados en base a proyectos. Educar, 33, 77-96. https://doi.org/10.5565/rev/educar.262

Kunberger, T. (2013). Revising a Design Course from a Lecture Approach to a Project-Based Learning Approach. European Journal of Engineering Education, 38(3), 254-267. https://doi.org/10.1080/03043797.2013.800020

Lutsenko, G. (2018). Case Study of a Problem-Based Learning Course of Project Management for Senior Engineering Students. European Journal of Engineering Education, 43(6), 895-910. https://doi.org/10.1080/03043797.2018.1454892

Macias-Guarasa, J., San-Segundo, R., Montero, J.M., Ferreiros, J., & Cordoba, R. (2006). Tools and Strategies for Improving PBL Laboratory Courses with a High Student-To-Faculty Ratio. In Proceedings Frontiers in Education 35th Annual Conference, F2C7-F2C12. Indiana: Wiley IEEE press.

Mayo, P., Donnelly, M.B., Nash, P.P., & Schwartz, R.W. (1993). Student perceptions of tutor effectiveness in a problem‐based surgery clerkship. Teaching and Learning in Medicine: An International Journal, 5(4), 227‑233. https://doi.org/10.1080/10401339309539628

Marti, E., Gurguí, A., Gil, D. Hernández-Sabaté, A., Rocarias, J., & Poveda, F. (2015). PBL On Line: A Proposal for the Organization, Part-Time Monitoring and Assessment of PBL Group Activities. Journal of Technology and Science Education, 5(2), 87-96. https://doi.org/10.3926/jotse.145

Mentzer, G.A., Czerniak, C.M., & Brooks, L. (2017). An Examination of Teacher Understanding of Project Based Science as a Result of Participating in an Extended Professional Development Program: Implications for Implementation. School Science and Mathematics, 117(1-2), 76-86. https://doi.org/10.1111/ssm.12208

Mills, J. E., & Treagust D. F. (2003). Engineering education: Is problem-based or project-based learning the answer? Australasian journal of engineering education, 3(2), 2-16.

Mohd, H., Darus, N.M., Saip, M.A., Baharom, F., Puteh, N., Husin, M.Z. et al. (2017). Success Factors of Problem based Learning for IT Courses: Measurements on PBL Characteristics, PBL Assessments and PBL Practices. Journal of Engineering and Applied Sciences, 12(21), 5514-5517.

Nulty, D.D. (2008). The adequacy of response rates to online and paper surveys: what can be done? Assessment and Evaluation in Higher Education, 33(3), 301-314. https://doi.org/10.1080/02602930701293231

Nunes de Oliveira, J.M. (2011). Nine years of Project-based Learning in engineering educaction. Revista de docencia Universitaria, 9(1), 45-55. https://doi.org/10.4995/redu.2011.6179

Ravitz. J. (2009). Introduction: Summarizing Findings and Looking Ahead to a New Generation of PBL Research. Interdisciplinary Journal of Problem-Based Learning, 3(1). https://doi.org/10.7771/1541-5015.1088

Perrenet, J.C., Bouhuijs, P.A.J., & Smits, J.G.M.M. (2000). The suitability of problem-based learning for engineering education: Theory and practice. Teaching in Higher Education, 5(3), 345-358. https://doi.org/10.1080/713699144

Said, S.M., Adikan, F.R.M., Mekhilef, S., & Abdrahim, N. (2005). Implementation of the Problembased Learning Approach in the Department of Electrical Engineering, University of Malaya. European Journal of Engineering Education. 30(1), 129-136. https://doi.org/10.1080/03043790512331313895

Santos, D.M.B., & Silva, C.A.S. (2018). Problem-Based Learning in a Computer Engineering Program: Quantitative Evaluation of the Students’ Perspective. IEEE Latin America Transactions, 16(7), 2061-2068. https://doi.org/10.1109/TLA.2018.8447377

Savin-Baden, M. (2007). Challenging models and perspectives of problem-based learning. In De Graaff, E., & Kolmos, A. (Eds.), Managements of change: Implementation of problem-based and project-based learning in engineering education (9-29). Rotterdam, Países Bajos: Sense Publishers. https://doi.org/10.1163/9789087900922_003

Savin-Baden, M. (2014). Using Problem-Based Learning: New Constellations for the 21st Century. The Journal on Excellence in College Teaching , 25(3-4), 197-219.

Schmidt, H.G., & Moust, J.H. (1995). What Makes a Tutor Effective? A Structural Equations Modelling Approach to Learning in Problem-Based Curricula. Annual Meeting of the American Educational Research Association, San Francisco, CA. https://files.eric.ed.gov/fulltext/ED385189.pdf https://doi.org/10.1097/00001888-199508000-00015

Spronken-Smith, R., & Kingham, S. (2009). Strengthening teaching and research links: The case of a pollution exposure inquiry Project. Journal of Geography in Higher Education, 33(2), 241-253. https://doi.org/10.1080/03098260802276813

Strobel, J., & Van Barneveld, A. (2009). When is PBL more effective? A meta-synthesis of meta-analyses comparing PBL to conventional classrooms. Interdisciplinary Journal of Problem-based Learning, 3(1). https://doi.org/10.7771/1541-5015.1046

Wan Hamiza, W.M.Z., Williams, A., & Sher, W. (2017). Introducing PBL in engineering education: Challenges Lecturers and students confront. International Journal of Engineering education, 33(3), 974-983.

Warnock, J.N., & Mohammadi-Aragh, M.J. (2016). Case study: Use of problem-based learning to develop students’ technical and professional skills. European Journal of Engineering Education, 41-2, 142-153. https://doi.org/10.1080/03043797.2015.1040739

Wilkerson, L., & Hundert, E.M. (1997). Becoming a problem-based tutor: Increasing self-awareness through faculty development. In Boud, D., & Feleti, G.I. (Eds.), The challenge of problem-based learning (2ª ed., 160-72). London: Kogan Page.

Wilkie, K. (2004). Becoming Facilitative: Shifts in lercturers’ approach to facilitating problem-based learning. In Savin-Baden, M., & Wilkie, K. (Eds.), Challenging Research in Problem-based Learning (81-92). Maidenhead, UK: Open University Press.

Yadav, A., Subedi, D., Lundeberg, M.A., & Bunting, C.F. (2011). Problem‐based learning: Influence on students’ learning in an electrical engineering course. Journal of Engineering Education, 100(2), 253-280. https://doi.org/10.1002/j.2168-9830.2011.tb00013.x

Zeng, L., & Xu, C. (2010). Problem-based learning in civil engineering education. Proceedings of 2010 2nd International Conference on Education Technology and Computer (ICETC) (3, V3-41-43). IEEE. https://doi.org/10.1109/ICETC.2010.5529601

---