Unveiling the impact of design methods on problem-solving performance in undergraduate STEM education

UNVEILING THE IMPACT OF DESIGN METHODS ON PROBLEM‑SOLVING PERFORMANCE IN UNDERGRADUATE STEM EDUCATION

Roberto Duran-Novoa1* , Felipe Torres2

1Universidad Técnica Federico Santa María (Chile)

2Universidad de Santiago de Chile (Chile)

Received October 2023

Accepted March 2024

Abstract

Problem-solving is at the core of engineering design, being fundamental for systematic innovation. During their education, students are taught numerous methods and tools, despite that literature shows debatable results regarding their real impact. Consequently, this study aims to quantify the relative impact of design methods on undergraduate students’ problem-solving performance and determine if this influence goes beyond their background and the problem’s complexity.

Utilising novelty, variety, and quality as criteria, the work done by 144 students was evaluated, solving two problems using three methods. The results show a performance improvement of up to 46% when working with methods that guide solution development through design principles. The context and the student method preference did not affect their performance, while the increment in the problem’s difficulty improved novelty and variety (15% and 11%) but reduced quality (34%). Surprisingly, the best‑performance method was the least preferred, indicating the need of exploring the relationship between performance and actual use.

The study results validate the work invested in teaching design methods, indicating the characteristics of the most efficient ones, beyond expert opinion. The structure of the study allows replication and could help future comparison of results.

 

Keywords – Design methods, Problem-solving, STEM, Undergraduate education.

To cite this article:

Duran-Novoa, R., & Torres, F. (2024). Unveiling the impact of design methods on problem-solving performance in undergraduate STEM education. Journal of Technology and Science Education, 14(3), 861-882. https://doi.org/10.3926/jotse.2473

 

----------

    1. 1. Introduction

Problem-solving is at the core of engineering design, being fundamental for systematic innovation. During their education, students are taught methods and tools to guide their problem-solving activities, which has shown positive effects on their results (Ananda, Rahmawati & Khairi, 2023; Bourgeois-Bougrine, Buisine, Vandendriessche, Glaveanu & Lubart, 2017; Scott, Leritz & Mumford, 2004). This tendency has also been supported by less traditional educational approaches, many of them based on active learning principles, which also have its own difficulties and barriers (Valero, 2022).

However, the relative efficiency of the methods taught and their relationship with the specific context or users who participate in each study is still unclear. For example, can it be determined if there is a better method to teach? which criteria could evaluate it? how relevant is the discipline studied? can the conclusions obtained be applied to the workplace?

Most of the studies that compare methods’ performance work with a small number of participants and do not follow similar procedures, impeding any statistical analysis, while reaching conflicting conclusions (Cano-Moreno, Arenas-Reina, Sánchez-Martínez & Cabanellas-Becerra, 2021; Chulvi, Royo, Agost, Felip & García-García, 2022; Dumas & Schmidt, 2015; Kannengiesser & Gero, 2015).

Consequently, this study aims to quantify the relative impact of design methods on students’ problem‑solving performance and establish if this influence goes beyond the student background and problem complexity. The previous could validate the work invested in teaching design methods and point towards which ones to teach.

1.1. Literature Review

Problem-solving is a skill increasingly demanded by society, which requires students to be adequately prepared to engage with today’s engineering challenges (Baer, 2015; Bozkurt-Altan & Tan, 2020; Cheng & Yang, 2019; Dumas, Schmidt & Alexander, 2016; Valero, 2022). This has been considered by educational institutions, especially since problem-based learning and its advantages have been studied and applied in multiple contexts (Ananda et al., 2023; Connors, Burns & Campitelli, 2011; Falk, Dym, Agogino, Frey, Leifer, Now et al., 2014; Fiore, Graesser & Greiff, 2018; Ince, 2018; Graesser, Fiore, Greiff & Andrews‑Todd, 2018; Priemer, Eilerts, Filler, Pinkwart, Rösken-Winter, Tiemann et al., 2019; Thorndahl & Stentoft, 2020).

Several studies have compared the results differences when solving problems using different methods and parameters, such as creativity effectiveness and idea generation (Genco, Hölttä-Otto & Seepersad, 2012; Linsey, Clauss, Kurtoglu, Murphy, Wood & Markman, 2011a; Scott et al., 2004; Toh & Miller, 2015). Frequently, relevant variables are ignored, like the effect of expert evaluation who assess results using their own tools and judgement (Bourgeois-Bougrine et al., 2017; Chulvi, González-Cruz, Mulet & Aguilar‑Zambrano, 2013; Gero, Jiang & Williams, 2013; Mulet, Chulvi, Royo & Galán, 2016). Also, most of the studies cannot be replicated, making it difficult to establish useful areas to explore further and to arrive at general design principles (Crilly, 2015; Daly, Adams & Bodner, 2012), and most of them are carried out with engineering students, despite the possible concentration of projects around similar topics, and the need for multidisciplinary teamwork (Santulli & Langella, 2011).

Another line of studies has sought to determine the impact of design methods more systematically, considering different variables to broaden their analysis (Barbara & Stefano, 2014; Benoni & Novoa, 2023; Chou, 2021). Linsey et al. (2011a,b) assessed the outcomes of brainstorming, brain-sketching, gallery, and C-Sketch / 6-3-5. The problem used for their experiment contained restrictions that increased the challenge (e.g. no electricity available). The output was evaluated considering the quality, quantity, variety, and novelty of ideas (Nelson, Wilson, Rosen & Yen, 2009). Their experiment consisted of 12 teams with six members each. They conclude that methods oriented to group work show better results than those oriented to individual work, pointing out that more experiments would be necessary to determine which methods might yield better performance results in terms of novelty, the variable with the lowest scores. Chulvi (2013) studied the work of designers and engineers attending a doctorate program in design and designers with professional experience (specific field is not indicated). The 48 participants were divided into 16 teams of three people each. Two teams acted as control groups, one using brainstorming, and the other working without method. Half of the 14 remaining teams were assigned SCAMPER, and the other half used the TRIZ Contradiction Matrix. The results were assessed by a team of experts, who measured novelty and utility as variables, employing the Analytical Hierarchy Process (AHP). Brainstorming, followed by SCAMPER obtained the best results; the no-method team obtained the lowest marks. Hence, employing a method might be better than using none, and among all of them, the most intuitive produces better outcomes. However, they indicate that more investigation is needed, mainly since Brainstorming was used by only one team. Duran-Novoa, Lozoya-Santos, Ramírez-Mendoza, Torres-Benoni and Vargas‑Martínez (2019) developed the same experiment in two contexts (countries), adding the possibility of choosing the design method as additional variable. In total, 108 participants were randomly divided into 36 teams of three members each, working with one of four methods: brainstorming, KJ‑technique, SCAMPER, and TRIZ. The solutions were measured in terms of novelty, variety, and quality; in both contexts, the most structured methods (SCAMPER and TRIZ) performed better, supporting the idea that methods’ impact goes beyond context. It is noteworthy that brainstorming and KJ-technique methods were preferred when participants were allowed to choose how to work, despite producing the most unsatisfactory results. This suggested that, for users, the ease of application and the understanding of the method fundamentals could be more important than generating a better outcome.

In consequence, several considerations could benefit the study. First, minimising the effect of evaluators’ bias in evaluation is necessary. Second, working in different contexts would lead to identifying the real impact of methods, especially if the results are not consistent. Third, exploring the effect of allowing the participants to select their work method could help academics to choose which design methods to teach, considering which ones are more likely to be applied in the workplace, despite how well they perform when the participants are required to use them under controlled circumstances. Finally, considering the effect of working with problems of different difficulty could help to estimate the extent of methods’ benefits in real-world situations.

As far as we know, no study has concurrently considered these variables. There is also a tendency to rely on intuitive approaches and simplify the analysis (e.g. expert opinions, participant interviews) while avoiding statistical analysis, replicability, and potential comparisons. Our intention is to overcome these limitations and deliver reliable results that can be valuable for future studies.

1.2. Research Question

As mentioned at the end of the introduction, it is expected to quantify the impact of design methods on students’ problem-solving performance and estimate the extent of this influence. The above can be posed as the following research question: How much impact do design methods have on students’ performance when solving problems of different difficulty?

To estimate the extent of this impact, the potential effects of the learning context on students will be considered (the method should surpass it), and the impact of allowing the students to select the method to work with (a preferred method might deliver relatively better results).

2. Methodology

The experiment compared the performance of students when using design methods to solve problems. The participants were divided into four groups, three worked using a design method, and one worked intuitively (see section 2.2). They had to find solutions in a short amount of time, aiming to favour the methods that deliver feasible solutions in a limited time frame (relatively more efficient) and to reduce the effect of non-controlled factors that occur in long experiments, such as influences among groups (Atilola, Tomko & Linsey, 2016), results of other sources of information applicable to the problem (Petre, 2004), and lack of repeatability (Linsey et al., 2011a). For each problem, the methods were assigned to some teams and chosen by others to identify possible relations between participants’ preferences and results. The control group did not change throughout the experiment.

2.1. Selected Design Methods

Design methods can be defined as “patterns of behaviour employed in inventing things of value that do not yet exist” (Gregory, 1966); applying them should produce “new design ideas based on the designer’s previous knowledge” (Sakae, Kato, Sato & Matsuoka, 2016).

The chosen methods have been used and studied in previous research, demonstrating a significant impact on user performance. They have been analysed and discussed in design-related conferences and journals and are frequently taught in STEM careers. As a result, although atomized, there is abundant information available to orient our analysis.

2.1.1. TRIZ – The Contradiction Matrix

The Theory of Inventive Problem Solving (TRIZ) proposes that there is a set of universal principles behind all inventions and that these principles can be identified and codified to make the inventive process more predictable (Altshuller, 1984; Belski, 2009; Cavallucci & Oget, 2013). It begins with the abstraction of a specific problem into a typical one (a contradiction), allowing to find a standard TRIZ solution, which should be developed into a specific solution for the particular problem, as presented in Figure 1.

TRIZ has been studied in Chulvi et al. (2013) and Duran-Novoa et al. (2019), working with its most popular tool, the Contradiction Matrix (MaTriz). This selection has the potential to allow comparing overall results and conclusions.

 

Figure 1. Basic representation of TRIZ’s strategy

2.1.2. SCAMPER

SCAMPER is a mnemonic that stands for Substitute, Combine, Adapt, Modify, But to other use, Eliminate, and Reverse. It was proposed by Alex Osborn as the Scamper method and later was denominated the SCAMPER Technique (Eberle, 1996). It establishes an algorithmic structure to solve problems; the user analyses the problem or its parts in simple terms, considering the seven SCAMPER principles to delimit the search areas to stimulate creative ideas for improvement or new developments (Table 1).

Letter

Meaning

General questions

S

Substitute

What can be substituted? Which elements could be added? Sounds? Lights?

C

Combine

What can be blended, mixed, or included? Which parts can be repeated, duplicated, triplicated, etc.?

A

Adapt

What is similar to the “objective?” Does its shape suggest something? A new colour, smell, texture?

M

Modify (also, Magnify or Minimize)

What can be made larger or smaller? Longer or thicker? More compact or shorter?

P

Put to other use

What are the other ways to use it? Does its shape suggest other uses?

E

Eliminate (also Elaborate)

Which parts can be eliminated? What if there is no product?

R

Reverse (also Rearrange)

What happens if the assembly is reversed? Can it be turned inside out? Upside down?

Table 1. SCAMPER questions, adapted from (Gero et al., 2013)

2.1.3. The Kj-Technique

The KJ-technique is a consensus-building method that helps to externalise and prioritise large quantities of ideas and information. It proposes solving problems first through a focus question, followed by creative work conducted through an individual idea-generation process (see Table 2); subsequently, the ideas are collectively analysed, deepening the most promising ones (Kawakita, 1991). Researchers have concluded that this method frequently achieves more and better-quality ideas because it prevents group work factors from reducing the idea-generation potential (Aslani, Naaranoja & Kekale, 2012; Diehl & Stroebe, 1987; Dunnette, Campbell & Jaastad, 1963; Sakae et al., 2016).

The KJ-technique was selected because it has a straightforward procedure that guides the designers’ actions. Still, it does not indicate search spaces or solution principles like SCAMPER or the MaTriz. Furthermore, it relates to the previously studied techniques (Duran-Novoa et al., 2019; Kunifuji, 2016; Linsey et al., 2011b; Sakae et al., 2016).

Step

Activity

Description

1

Question

On a board, the team defines the focus question (problem).

2

Alternatives

In silence, members propose alternatives using notes.

3

Grouping

In silence, the team groups similar or related alternatives.

4

Filtering

Duplicated ideas are deleted, while the related ones are linked (e.g., arrows).

5

Development

Each group of ideas is analysed as a potential solution, being expanded, restricted, or decomposed.

If necessary, the cycle repeats.

Table 2. The Five Steps of the KJ-technique, adapted from (Kawakita, 1991)

2.2. Participants

In total, 144 undergraduate students aged 19 to 23 participated in this study voluntarily, without compromising their academic work and having the possibility to withdraw at any moment. They were randomly paired into 72 teams of two persons, and later divided into four groups determined by their working method: the MaTriz, SCAMPER, KJ-technique, and the Control group which worked intuitively without method (see table 3). Participants were students from three institutions: Universidad del Desarrollo (UDD), Universidad de Santiago de Chile (USACH), and Universidad de Chile (UCH), Both UDD and UCH were 1st-year students, while USACH students were last year students (6-year program). None of them had experience working with the selected methods, which was validated through a brief survey and interview before the experiment.

After enrolment, each participant was assigned a code to determine the groups and teams, preventing any subsequent identification. The Control group was integrated by UDD students to allow comparing UDD‑method teams against UDD-control teams, maintaining unaltered the other variables. The number of participants impeded doing something similar with the other institutions.

Context

Program

Year

Problem-one

Problem-two

KJ Tech.

SCAMP.

MaTriz

Control group

KJ Tech.

SCAMP.

MaTriz

Control group

UCH

Architecture

1

6

6

6

0

8

8

2

0

UDD

Engineering, Architecture and Nutrition

1

4

11

9

18

6

12

6

18

USACH

Engineering

6

1

11

0

0

4

4

4

0

Teams by group (methods)

11

28

15

18

18

24

12

18

Teams total

72

72

Table 3. Distribution of methods and teams (both problems)

2.3. Problems

Both problems were adapted from literature, having enough differences in their proposed challenge to consider the second problem more difficult. Considering the STEM fields (science, technology, engineering, and mathematics), both problems avoid the need of specific technological or mathematical knowledge, as this can create barriers that hinder the understanding of the underlying methods and their impact.

2.3.1. Problem-One: Cutting Board.

Problem-one was “Design a cutting board that can be taken to the dining-room table”, previously employed in Duran-Novoa et al. (2019). The design should consider the kitchen and dining-room requirements (e.g., having mechanical resistance for kitchen use, allowing safe displacement, avoiding liquid spillage in the dining-room, etc.). Problem-one did not have additional restrictions other than serving its function, like cost, specific materials, weight, etc. This was done to create a favourable scenario for the free generation of ideas, reducing the probability of a team not proposing a solution. A scientific approach should yield benefits without being overtly influenced by technological or engineering knowledge.

2.3.2. Problem-Two: Peanut-Shelling Device

Problem-two was “Design a peanut-shelling device, targeting the throughput, separating the peanut shell from the nut, and using non-electrical energy sources”, previously employed in Linsey et al. (2011a,b). Problem-two was considered relatively more complex than Problem-one as it required finding a solution that met the objective within the constraints of energy sources, making it more applicable in a professional context. As a result, having an engineering background or “hands-on” experience was expected to have a positive impact.

2.4. Sample and Procedure

The experiment took place in the Institutions’ classrooms of each context. After receiving a brief introduction to the experiment, the control group was separated while the other participants were trained in the three methods (MaTriz, SCAMPER, and KJ-technique).

After the training, 30 teams were assigned a method to work with before explaining Problem-one, and 24 teams had the option to choose. The same procedure was developed before Problem-two, reversing the proportion (Table 4).

Method

Number of teams

(Problem-one)

Number of teams

(Problem-two)

Assigned

30

24

Selected

24

30

Control Group

18

18

Table 4. Team distribution by method assigned or selected (both problems)

 

Figure 2. Example of a proposal in the Idea sheet; it was obtained working with the MaTriz,
specifically the inventive principle “replacement of mechanical systems”

The teams working with methods could choose once for the first or second problem. When the groups had the choosing option, there were no restrictions on the choice; consequently, those teams that had the opportunity to choose in Problem-two could repeat the method used in Problem-one.

Figure 3 shows the general structure of the experiment. In more detail, it was divided into the following six stages:

Stage 1: A brief presentation of the experiment was given to the participants. The materials to be used were provided (e.g., drawing sheets, post-its, blackboards, etc.), and the participants were given general instructions. It was pointed out that we were evaluating the methods’ impact; consequently, rigour and honesty were expected. Groups (three methods, one control group) and participants teams were formed randomly.

Stage 2: Teams were sent to different classrooms with similar conditions. The classrooms were large enough to allow working without influencing another team. Within teams, they decided how to distribute their work.

Stage 3: The teams working with methods were trained in the MaTriz, SCAMPER, and the KJ-technique for 65 minutes. This training used a sample problem to show each method’s practical application, illustrating solution ideas with drawings or prototypes. The problem was “Design an office desk to alternate standing and sitting work”, already studied in Chulvi et al. (2013).

Stage 4: The Problem-one was introduced. The time allowed for participants was 45 minutes. The results were registered in idea sheets, where students drew solution proposals, including any explanation necessary (Figure 2). Each team could hand in up to two solutions for the problem, which should benefit the more productive teams since they had the possibility of selecting their proposals.

Stage 5: Students were given a 15-minute break. Afterwards, the same teams gathered to prepare to solve Problem-two. The teams with the option of deciding on a method had a maximum of five minutes to communicate their decision, while the teams who did not have the option were notified of their working method.

Stage 6: Problem-two was introduced, following the same rules described in Stage 4.

During the whole experiment, it was verified that the results were obtained using the assigned or selected methods, avoiding mixes or proposals obtained through inspiration that could be later justified by the method.

 

Figure 3. Structure of the experiment

2.5. Evaluation

Teams’ proposals were evaluated using a modified version of Nelson’s criteria (Nelson et al., 2009), which assigns a holistic score to each proposal based on partial evaluations of Novelty, Variety, Quality, and Quantity. The study evaluation omits Quantity due to the limited time frame, measuring it indirectly through restricting the number of proposed solutions: best methods should aid their users to be more productive and deliver more valid proposals to select from (Table 5). Each criterion was assessed on a 5-point scale, ranging from 0 to 4; 12 points was the maximum possible per proposal, 24 per team. Two independent evaluators conducted the evaluations, who had to agree on every score.

Although bias is always possible, it was minimised through the relative evaluation of the proposals. The evaluators scored each solution proposal relative to the 144 proposals received, not based on their prior knowledge. For example, a maximum score in novelty can be obtained only if no other team proposes a similar alternative; if the evaluator knows that a similar idea already exists in the market, it does not influence the proposal score. Similarly, if a team proposes an alternative based on electromagnetic properties and another based on fluids, its variety should be the highest. Regarding quality, all proposals could obtain a four if they are “feasible and easy to implement”, despite the possibility that one proposal could be more accessible to implement than another. Figure 2 presents a proposal that obtained a 4 in novelty (unique), a 4 in variety (different physical principles between proposals), and a 2 in quality (difficult to implement).

Aspect

General description

Measurement

Specific description

Score

Novelty

How unusual or unexpected an idea is

Similarity between the proposed ideas during the experiment

Very common (VC) more than 50%

0

Common (C) 26%-50%

1

Uncommon(U) 11%-25%

2

Very Unusual (VU) 2%-10%

3

Unique (N)

4

Variety

Explored solution space

during the idea generation

Physical principles, working principles (within physics), and proposed embodiments

Same working principles (SWP) and embodiment (E)

0

Same working principles and different embodiments (DE)

1

Same physical principles (SPP) and working principles

2

Same physical principles and different working principles (DWP)

3

Different physical principles (DPP) and different working principles

4

Quality

Feasibility of an idea and

expected performance

Feasibility and ease of Implementation considering cost and utility

Neither feasible (NF) nor easy to implement (NEI)

0

Hardly feasible (HF) and difficult to implement (HI)

1

Hardly feasible (HF) or difficult to implement (DI)

2

Feasible (F) and difficult to implement (DI)

3

Feasible (F) and easy to implement (EI)

4

Table 5. Evaluation criteria

Each proposal was evaluated following this specific order: first, measuring novelty, then variety, and last, quality, as shown in Figure 4 (the acronyms are described in Table 5). This sequence is more efficient for the evaluators in both scoring and registering. Novelty requires considering all proposals and classifying them to determine their similarities and assign a score; if not evaluated first, the evaluators would have to reclassify ideas after scoring the other criterion. Variety requires both teams’ ideas and grouping them facilitates their registration without interfering with the evaluation of quality, which is independent of each proposal.

 

Figure 4. Evaluation process (S means score). For example, the assigned score
was immediately zero if the problem function was not accomplished

3. Results

3.1. Problem-One: Cutting Board

There were significant differences in groups performance (Figure 5). The best results were obtained by the MaTriz and SCAMPER, both based on guiding the development of the solution through principles. Figure 5 shows the general score difference when principles are considered (MaTriz and SCAMPER) and when they are not (KJ-technique and Control group); having a similar dispersion, the principle-based methods mean was 54% higher. Table 8 shows examples of students’ proposals.

 

Figure 5. Tree diagram of Problem-one performance, grouping methods as principle-based
(MaTriz, SCAMPER) and not (KJ-technique, Control group)

3.1.1. Results by Criteria

Quality obtained the best results and novelty the lower. Using four as 100%, their difference is 32% (see Table 6). When methods were chosen, performance declined slightly but not significantly. Novelty was also the criterion with the highest dispersion, and where the method’s effect was more notorious (Table 7).

Descriptive statistics →

N

Minimum

Maximum

Mean

Std. Deviation

Variance

Novelty

144

0

4

1,85

1,140

1,300

Variety

144

0

4

2,51

0,982

0,965

Quality

144

0

4

3,13

1,026

1,052

Valid N (listwise)

144

 

 

 

 

 

Table 6. Descriptive Statistics (Novelty, Variety, and Quality), Problem-one

 

Sum of Squares

df

Mean Square

F

Novelty

Between Groups

61,356

3

20.452

22,983

Within Groups

124,582

140

0,890

 

Total

185,938

143

 

 

Variety

Between Groups

47,048

3

15,683

24,142

Within Groups

90,945

140

0,650

 

Total

137,993

143

 

 

Quality

Between Groups

28,855

3

9,618

11,070

Within Groups

121,638

140

0,869

 

Total

150,493

143

 

 

Table 7. ANOVA table for the criteria, Problem-one

Control Group

KJ-technique
 

Anti-spill tray with side container.
 

Side barriers with height.
 

Tongs with inclined gutters; it contains holder sleeves.
 

Cutting Guides, Non-slip Tray with Handles.

SCAMPER

MaTriz
 

Liquid retention rim; includes a strainer,
a bowl, and handles.
 

Inventive principles: Segmentation, taking out, mechanics substitution.