The influence of social style in evaluating academic presentations of engineering projects
|
Héctor Ortiz València, Águeda García Carrillo, Margarita González Benítez Dept of Engineering Projects of Universitat Politècnica de Catalunya Spain |
Received July 2012
Accepted September 2012
Abstract
An individual’s social style is determined by behavioral patterns in the interactions with their peers. Some studies suggest that social style may influence the way in which an individual’s performance is evaluated. We studied the effects that speakers’ and evaluators’ social styles have on the marks given for end-of-term presentations in a project engineering master’s course. The participants completed a self-evaluation exercise that classified their social styles into one of four categories: Driver, expressive, analytical, or amiable. Students individually rated the content and appearance of their classmates’ presentations. A statistical analysis of these scores revealed that the speaker’s social style had a significant effect on the marks received for content and appearance. The evaluator’s social style also demonstrated a statistically significant effect on the marks given for appearance, though not for content. Students with expressive social style received the highest scores, while the analytical style received the lowest scores. These results reiterate the necessity to train students as evaluators in order to reduce bias when evaluating their classmates and co-workers during their academic and professional careers.
Keywords – Social style, rating, evaluation, presentations.
----------
1 Introduction
An individual’s social style defines a set of behavioral patterns in their interactions with others. The social style model developed in the 1950s by Dr Merril (Tracom, 1991) uses simple questionnaires in order to evaluate two scales of social style. These scales were named assertiveness and sensitivity. Assertiveness is defined as the effort a person makes to influence the thoughts and actions of others. On the other hand, responsiveness describes the tendency to express one’s feelings.
The quadrants that result from the combination of the two scales -assertiveness and responsiveness- define four social styles: Analytical, driver, expressive, and amiable (see figure 1). People who are not very assertive and not very sensitive are considered to be analytical. These people tend to be prudent, reflexive, and objective, and may also be considered cold and indecisive. People with low responsiveness and high assertiveness are drivers. The members of this group are decisive, independent, sincere and efficient, but also are sometimes perceived as brusque in their interactions with others. People with high assertiveness and high emotional responsiveness make up the expressive group. These people are perceived as outgoing, enthusiastic, persuasive and spontaneous, while are sometimes seen as impertinent and distractive. Lastly, people with low assertiveness and high expressiveness compose the amiable group. These people are perceived as diplomatic, cooperative, and patient but also permissive and dependent.
The effect of social style on peer reviews has been investigated in previous studies, in industrial (Antonioni & Park, 2001) and academic environments (May & Gueldenzoph, 2006). However, to our knowledge, there are no studies in which this effect was analyzed on peer reviewers rating short in-class presentations. The reviewed literature allows us to foresee distortions in the marks given and received as a result of the combinations of the social styles of the speaker and evaluator. Specifically, the perceived similarity theory states that the similarities between individuals are an influential factor in the ratings given in similar pairs. These similarities can be identified either by behavioral or cognitive factors. In particular, the degree of perceived similarity has been described as a source of bias (Mumford, 1983) and imprecision (Orpen, 1994) in peer review.
The adaptation to the new European Higher Education Area (EHEA) requires the university professor to apply teaching methodologies in which the students are the protagonists of their own education. These methodologies introduce evaluation systems which allow for a more complete following of the individual’s learning process, facilitating their education and tutoring.
In addition to the main topics of the course, generic competencies of each degree program are also included. The level at which the skills have been learnt is also assessed (Scriven, 1967) (Hall & Burke, 2003) (Kaftan, Buck & Haack, 2006). All of the study plans at the School of Industrial Engineering of Barcelona (ETSEIB) include the seven generic competencies specified in the framework of the Universitat Politècnica de Catalunya, BarcelonaTech (UPC): Determined usage of information resources, sustainability and social commitment, efficient oral and written communication, self-learning, innovation and entrepreneurship, a third language (English), and teamwork.
The “project engineering” course in Industrial Organization masters’ degree (a second cycle engineering program planned to extinction with the incoming EHEA plan), has used active teaching methodologies since the year 2000: Problem and project-based education, cooperative learning, and group e-portfolios.
In this course, students are also trained on effective communications skill. The end-of-course evaluation specifically challenges on oral communication skills (Linguistic Services, 2012) with the aid of a poster (hereafter: Presentations).
Historically, in this course, the Presentations were not rated by peer reviews, although students were encouraged to orally critique the presentations at will. During the planning of the 2011-2012 academic year, the professors of the Projects course considered the possible benefits of structuring a peer review system for the Presentations; In particular, those for the competence of effective oral communication. In order to assess their peers, each student will have to use their knowledge and acquired abilities in a critical manner (Marín García, 2009).
A simple scoring rubric was elaborated to guide peer reviews of the Presentations (see Figure 2). Although a rubric is used to guide the evaluation, we must consider how the social style of each student could influence the evaluation of their peers. We decided to carry out a study to determine whether a student’s social style can bias the Presentation peer review. In the study’s initial considerations, the social styles of both the evaluator and the speaker being evaluated were identified as possible sources of bias.
In this article we present an exploratory study aimed to determine the influence of social style in the act of evaluating engineering project presentations in the context of higher education.
|
Group evaluated: |
|
|
|
|||
|
Evaluator: |
Evaluator group: |
|||||
|
Content points for the presentation |
0 - 10 |
|
Rate each speaker (0 to 10) |
|||
|
The current problem is clear and realistic |
|
Speaker |
Name |
Content |
Appearance |
|
|
Situation quality evaluation criteria |
|
1 |
|
|
|
|
|
Internal, external, and third party users are identified |
|
2 |
|
|
|
|
|
The desires of different users is presented |
|
3 |
|
|
|
|
|
Users affected by the project are identified |
|
4 |
|
|
|
|
|
Ergonomic design for the operators |
|
5 |
|
|
|
|
|
Interrelations with external systems developed |
|
6 |
|
|
|
|
|
Design for security |
|
7 |
|
|
|
|
|
Design for reliability |
|
8 |
|
|
|
|
|
Alternatives presented |
|
|||||
|
Description of the final solution |
|
|||||
|
Economic viability of the solution |
|
|||||
|
Technical viability of the solution |
|
|||||
|
Viability and environmental sustainability of the solution |
|
|||||
|
Appearance evaluation points for the presentation |
0 - 10 |
|||||
|
Adequate poster distribution |
|
|||||
|
Legibility of the poster |
|
|||||
|
Attractive composition |
|
|||||
|
Titles, author names, size according to norms |
|
|||||
|
Presentation under 20 minutes |
|
|||||
|
All members have relevant roles |
|
|||||
|
Cohesion between each member's intervention |
|
|||||
|
Group: usage of technical language |
|
|||||
|
Group: adequate pace |
|
|||||
2 methods
Six groups of six to eight students agreed to participate in the study during the end-of-course project presentations. During these presentations, all students presented parts of their projects in two to four minute speeches, followed by questions from a panel of reviewers. The rest of the class evaluated each student’s presentation, giving scores for content quality (content, hereafter) and performance quality during the presentation (appearance, hereafter). The marks were given on a 0 to 10 point scale. In order to encourage the evaluation process, the evaluators were instructed to assign integer values without score repetitions among the members of each group. Data were collected during the Presentation sessions.
A total of 32 students (26 male) participated in the study. The group carried out a total of 728 peer reviews. In the statistical analyses, the classification of the social style of each student was used as the independent variable, assessing its effect on the evaluations given and received for presentation content and appearance.
Before the presentations, the social style of each student was determined by means of a self-administered questionnaire on social style, distributed by Wilson Learning (online). This questionnaire consists of self-descriptive responses regarding 20 items. The questionnaire is completed and corrected in less than 20 minutes. The completion of the questionnaire supported a practical exercise on the social style concepts.
The same statistical analysis of the data was carried out for the ratings given for content and appearance in three phases: First, four factorial variance analyses (ANOVA) were used to separately describe the effect that the speakers’ and evaluators’ social styles have on evaluation. Second, a two-way ANOVA was used at four levels (social styles of the evaluator and the speaker) to break down the marginal effects of the social styles. Lastly, a post-hoc analysis was carried out on the average marks for each pair of social styles. The ANOVA analyses allow, by means of lineal regressions, a split of the variance of a dependent variable (in this case, the marks), explained by a number of independent factors (social styles of the evaluator and speaker, in the case of our study). The magnitude of the explained variance is compared with the residual error in order to determine its statistical significance (with reference to Fisher’s F distribution and expressed with p, the probability of a null hypothesis). When an analysis incorporates multiple comparisons, the significance threshold of p is adjusted by means of the Bonferroni technique to compensate the effect of repeated measurements.
3 results
The distribution of the social styles observed in the sample group was: 44% expressive (n=14, 11 males), 22 % driver (n=7, 4 males), 22 % amiable (n=7, 7 males) and 12% analytical (n=4, 4 males).
3.1 Single-factor ANOVA analysis
In the first analysis series (see Table 1), the comparison of content scores received in function of the speaker’s social style revealed a significant effect from the social style factor (F=5.879; p=0.001). The post-hoc analyses showed that subjects of the expressive social style received higher marks than the analytical and driver styles with a statistical significance which exceeds the threshold p<0.05, corrected by Bonferroni. There was also a tendency to receive higher marks than the amiable social style, though without reaching the threshold of significance (p=0.091).
Table 1. Results of the single-factor variance analysis. Descriptive statistics of the sample and results of the single-factor ANOVA tests. A total of 728 content and appearance evaluations were included in the mean score comparisons, split by the social style of the evaluator ans the speacker. Statistically significant differences in average marks were identitied between social styles. The * sign indicates a difference in averages with a significance of p<0.05 corrected according to Bonferroni (n=120). The ** sign indicates a significance of p<0.01; with the same Bonferroni correction
The analysis of appearance scores according to social style revealed a significant effect from the social style factor (F=4.601; p=0.003). Post-hoc analyses showed that the subjects of the expressive social style received higher marks than the analytical and driver styles with a statistical significance that exceeds the p<0.05 threshold (Bonferroni-corrected). Once again, there was a tendency of higher marks than those received by the amiable social style, though without reaching the threshold of significance (p=0.069).
The dependency analysis of content scores with respect to the evaluator’s social style showed a trend, though without achieving statistical significance (F=2.498, p=0.059). None of the post-hoc tests exceeded statistical significance.
The analysis of content scores according to the evaluator’s style showed no statistically significant difference.
The analysis of appearance scores given by evaluators according to social style revealed a significant effect from the social style factor (F=4.312; p=0.005). Post-hoc analyses showed that the subjects with expressive social style gave higher marks than the analytical and amiable social styles, with a statistical significance which exceeds the threshold p<0.05, corrected by Bonferroni.
3.2 Two-way ANOVA analysis
The 2x2 ANOVA variance analyses allow us to split the effects of social styles of evaluator and speaker, making the results independent from the distribution of social styles in the sample (see Figure 1). These analyses confirmed the significant effect that the social style of the evaluated subject has on the marks received for their presentation, both in content (F=3.35; p=0.019) and appearance (F=4.78; p=0.003). Analyzing the scores given according to the evaluator’s social style revealed a statistically significant effect on appearance scores (F=5.43; p=0.001) but not content scores (F=1.69; p=0.167) given for the presentations. The analysis of the gender factor did not reveal significant effects in content or appearance, whether including the gender of the speaker or the evaluator in the model. Further analyses of the gender factor were discarded due to a lack of statistical significance.
The post-hoc analysis of the factorial model including the social style of the speaker and the evaluator allowed us to analyze the social style effect, with independence of the evaluator-speaker style pairs. That is to say, the effect this factor has on each individual can be calculated, without being influenced by the distribution of other social styles in the sample.
In the case of the “appearance” evaluation, the analysis revealed that subjects with an expressive style gave and received higher marks than the other three social styles (see Figure 3). This difference is statistically significant for comparing marks given by amiable or analytical individuals, as well as marks received by analytical style students. The expressive social style assigns significantly higher “appearance” marks than their analytical and amiable classmates. The expressive social style also received significantly higher “appearance” marks than the amiable social style. At a group level, no statistically significant differences were observed in the marks given or received for content between social styles, although there was a tendency for the expressive social style to give and receive higher marks.
3.3 ANOVA analysis of style pairs
Table 2. Analysis by social style pairs. Characteristics of the average marks for each combination of social styles. There is a notable variability in marks depending on social style. The * sign indicates a difference in averages with a significance of p<0.05, corrected following Bonferroni (n=120)
The post-hoc analysis of style pair evaluations (that is, analyzing each possible combination of evaluator-speaker social styles separately) revealed a wider range of scores in each sub-group (see Table 2). In the assessment of presentation appearance, the lowest average marks were given in “amiable to analytical” evaluations (average ± standard dev. = 5.57 ± 2.38), while the highest marks were in “expressive to expressive” evaluations (7.40 ± 1.73). The comparison between these two is relevant (average increase of 1.83, SE=0.539), although its significance does not survive a strict Bonferroni correction (n=120; p=0.088). The rest of the comparisons between evaluator-speaker pair combinations neither do.
In content assessments, there is no difference that exceeds the strict threshold of significance, although there is a remarkable trend for the expressive social style to receive the highest content scores. The lowest average marks are observed in “analytical to driver” combinations (average ± standard deviation= 6.25 ± 1.96) and amiable to amiable (with the same descriptives), while the highest marks resulted from “expressive to expressive” evaluations (7.65 ± 1.67). The difference in marks between the “expressive to analytical” pair and the “expressive to expressive” pair is significant at the p <0.05 level corrected using the Bonferroni method (n=120), with an average difference of 1.17 points (SE=0.31, p=0.023).
4 discussion
In the assessments of academic project presentation, a large part of the variance is not explained by the social style of the evaluated subject or the evaluator. However, the social style of the individuals who participate in an act of evaluation of this nature does show a statistically significant effect at a general level and exceeds strict statistical thresholds in certain evaluator-speaker social style pairs.
The first set of analyses revealed that in our sample group there are significant differences in the marks given and received according to social style. The characteristics of our study group also revealed a non-homogeneous distribution of social styles. Almost half of the students were of the expressive type, while only 12% were of an analytical style. This disparity in group size is directly reflected in the types of evaluations given and received. With this distribution, any bias caused by the expressive style in the given or received evaluations will be magnified at the sample level.
The second analysis (two-way variance analysis) allowed for an isolation of the social style effect, independent of the distribution of these styles within the sample. This analysis showed significant differences in appearance scores given and received by expressive style individuals, who gave and received higher marks. In this analysis, there were no significant differences in content scores between social styles. On the other hand, in the style pair analysis there is a significant difference in content scores (despite the severity of the Bonferroni threshold at 120 comparisons). This difference indicates a very specific bias in the rating of analytical students by expressive students, in comparison with the marks they give to each other. This fact, combined with the high number of expressive students in our sample group, explains a large part of the differences observed in the initial analysis (see Table 1).
In the context of higher education, we perceive a double value in the application of the methods and results presented in this study. On the one hand, knowing the evaluation bias and its significance seems essential for designing fair strategies of peer-to-peer evaluation. With a simple analysis like the one carried out in this study, this bias shall be calculated and compensated wherever medium-size sample groups are available. On the other hand, there is remarkable educational value in the diffusion of results like these, in order to improve the students’ abilities to design and monitor interpersonal relationships between the members of their academic and professional teams.
This study presents relevant limitations. In the first place, the social style of each student was determined by means of a self-administered tool. By definition, the social style describes behavioral patterns as perceived by the individuals in their environment. Therefore, it is always preferable to obtain it through third party assessment tools. In this study we opted for a self-administered tool due to its simplicity and rapid administration. In second place, we have the size of the sample group. The exploratory nature of this study is limited by its small sample size. In various analyses, relevant tendencies to significant differences between subgroups were observed, though without reaching statistical significance. We hypothesize that a larger sample would increase the statistical power, reducing the standard error in the estimation of averages and, therefore, increasing the statistical significance of the results found in this study.
The high prevalence of expressive social style students (44%), along with the fact that the best marks are issued by the “expressive to expressive” social style pair, caused that the expressive group was, by long, the best rated. This phenomenon is in line with the theory of “perceived similarity”.
5 conclusions
In the assessment of short academic project presentations, the social style of the evaluator and the speaker play a significant role in the scores given for content and appearance. Raising awareness among the students on the sources of bias is recommendable for developing their evaluation skills. The evaluations based on presentation content suffered markedly lower influence from social styles than those based on appearance. Thus, content evaluation rubrics are once again shown to be key tools in guiding raters.
The expressive social style has proved predominant in this sample, though this result cannot be inferred to the entire student population. If the expressive social style was predominant, it could be confirmed that in this socio-demographic context this is a potential source of bias to keep in mind when considering using peer reviews in the context of education. The expressive subgroup would systematically receive higher marks than other subgroups. On the other hand, no significant differences were observed between genders.
References
Antonioni, D., & Park, J. (2001). The effects of personality similarity on peer ratings of contextual work behaviours. Personnel Psychology, 54, 331-360. http://dx.doi.org/10.1111/j.1744-6570.2001.tb00095.x
Hall, K., & Burke, W. (2003). Making formative assessment work - Effective practice in the primary classroom. Maidenhead, UK: Open University Press.
Kaftan, J., Buck, G., & Haack, A. (2006). Using formative assessments to individualize instruction and promote learning. Middle School Journal, 37(4), 44-49.
Linguistic Services. http://www.upc.edu/slt/comcomunicar Last consulted 10/08/2012.
Marín García, J.A. (2009). Students and professors as evaluators. Application in the evaluation of oral presentations. Spanish Journal of Pedagogy, 242, 79-98.
May, G.L., & Gueldenzoph, L.E. (2006). The effect of social style on peer evaluation ratings in project teams. Journal of Business Communication, 43(1), 4-20. http://dx.doi.org/10.1177/0021943605282368
Mumford, M.D. (1983). Social comparison theory and peer evaluation. Personnel Psychology, 36, 867-881. http://dx.doi.org/10.1111/j.1744-6570.1983.tb00516.x
Orpen, C. (1994). Perceived similarity: Its effect on the accuracy of peer evaluations among university students. International Journal of Educational Management, 8(3), 4-6. http://dx.doi.org/10.1108/09513549410062371
Scriven, M. (1967). The Methodology of Evaluation. Chicago: Rand McNally.
TRACOM (1991). The social style profile - technical report: Development, reliability, and validity. Denver, CO: The TRACOM Corporation.
Wilson Learning co.http://www.wilsonlearning.co.uk/docs/social_style_self_profile.pdf Last consulted 12/08/2012.
|
Citation: Ortiz , H., Garcia-Carrillo, A., & Gonzalez Benitez, M. (2012). The influence of social style in evaluating academic presentations of engineering projects. Journal Of Technology and Science Education (JOTSE), 2(2), 68-76. http://dx.doi.org/10.3926/jotse.50
On-line ISSN: 2013-6374 – Print ISSN: 2014-5349 – DL: B-2000-2012
|
Author biography
Héctor Ortiz València
Hector holds a Master in Telecommunications Engineering and is a PMI's accredited Project Management Professional. He works as a Program Manager in the United Technologies Corporation and as a part-time associate professor in the UPC (Universitat Politècnica de Catalunya, BarcelonaTech) teaching Project Engineering in the Industrial Engineering School of Barcelona.
Àgueda García Carrillo
Agueda García-Carrillo is Chemical Engineer and has received postgraduate university training in Technological Innovation. She holds a Master in Ergonomics and has received her PhD from Universitat Politècnica de Catalunya, BarcelonaTech (UPC). She is Researcher (UPC) in the field of Project Engineering Methodology and gives courses about this area at University.
Margarita González Benítez
Margarita Gonzalez has a degree in Chemical Science at the UCM and Science PhD at the UB. She is professor at the UPC in the Projects Engineering Department. She is the codirector of the Projects Engineering Research Group (GIIP) at the UPC and has been in charge of the Life Cycle Analysis area.
Journal of Technology and Science Education, 2012 (www.jotse.org)
Article's contents are provided on a Attribution-Non Commercial 3.0 Creative commons license. Readers are allowed to copy, distribute and communicate article's contents, provided the author's and Intangible Capital journal's names are included. It must not be used for commercial purposes. To see the complete licence contents, please visit http://creativecommons.org/licenses/by-nc/3.0/es/
Published by OmniaScience (www.omniascience.com)