Analysis of students’ generated questions in laboratory learning environments

Juan Antonio Llorens-Molina¹, Jesús María Llorens de Jaime², Isidora Sanz Berzosa¹

¹Escola Tècnica Superior d’Enginyeria Agronòmica i del Medi Natural, Universitat Politècnica de València

Valencia/Spain

²Facultad de Ciencias Políticas y Sociología, Universidad Complutense de Madrid

Madrid/Spain

Received February 2012
Accepted March 2012

Abstract

In order to attain a reliable laboratory work assessment, we argue taking the Learning Environment as a core concept and a research paradigm that considers the factors affecting the laboratory as a particularly complex educational context.

With regard to Laboratory Learning Environments (LLEs), a well known approach is the SLEI (Science Laboratory Environment Inventory). The aim of this research is to design and apply an alternative and qualitative assessment tool to characterize Laboratory Learning Environments in an introductory course of organic chemistry. An alternative and qualitative assessment tool would be useful for providing feed-back for experimental learning improvement; serving as a complementary triangulation tool in educational research on LLEs; and generating meaningful categories in order to design quantitative research instruments.

Toward this end,  spontaneous questions by students have been chosen as a reliable source of information. To process these questions, a methodology based on the Grounded Theory has been developed to provide a framework for characterizing LLEs.  This methodology has  been applied in two case studies. The conclusions lead us to argue for using more holistic assessment tools in both everyday practice and research. Likewise, a greater attention should be paid to metacognition to achieve suitable self-perception concerning students’ previous knowledge and manipulative skills.

Keywords -  Learning environments, metacognition, autonomy, science laboratory, student questions.  

----------

1           INTRODUCTION

1.1 Laboratory work from a comprehensive viewpoint of educational assessment

A comprehensive viewpoint of educational assessment emphasizes its role as a tool for learning improvement.  This is particularly relevant with regard to laboratory activities since these involve many different specific competences related to the following phases (Hofstein, 2004): (a) planning and design, (b) performance; (c) analysis and interpretation and (d) application. The execution of each of these activities is highly influenced by countless  contextual issues. Likewise, several generic competences have significant weight in laboratory activities, particularly those concerning communication skills, cooperative work, and decision making.  In addition, the influence of infrastructural issues concerning organization and management of educational resources are much greater than those involved in ordinary classroom activities. 

This complexity is a challenge that requires a new conceptual approach. Top priority should be placed on developing an accurate characterization of laboratory Learning Environment. In fact, we argue for taking this concept as a key issue and unit of analysis both in ordinary laboratory work assessment and educational research.

1.2 Laboratory Learning Environment as a key concept to analyze the experimental work

As discussed by Wilson (1995), the Learning Environment concept is a coherent, valid and reliable conceptual approach for: a) Carrying out an authentic and comprehensive assessment in ordinary learning (Bravo & Fernández del Valle, 2000), b) Designing research instruments to test innovative educational proposals.

Designing instruments to test educational models is particularly relevant in our educational context, as it is still strongly influenced by a conception of laboratory work taks as decontextualized hands-on activities, or simple testing of taught theoretical knowledge. Indeed, they are usually developed as a parallel learning process, more or less coordinated with the theoretical syllabus, and seldom explicitly integrated in a coherent learning sequence. This trend coexists with many innovative experiences whose goal is to correct these methodological deficiencies Thus, monitoring these changes in learning environments requires specific and suitable assessment tools. 

The characterization of laboratory learning environments (LLEs, from now on) cannot be restricted to basic issues concerning infrastructure or educational resources. The same is true with organizational models related to interactions between students and teachers, and among students themselves.  From a constructivist point of view, issues related to human and affective development, or to relationships in social and professional contexts should be taken into account. Powerful or rich learning environments have been described by several authors as Dochy, Segers, Van den Bossche and Struyven (2005) and Gijbels, Van den Watering, Dochy and Van den Bossche (2006). Moreover, attention paid to autonomy enhancement is especially noteworthy. In that respect, personal responsibility, initiative development, and decision making ability etc., are central issues in improving learning.

Many tools have been proposed to characterize the Learning Environments, such as CLES (Constructivist Learning Environment Survey) (Fraser, 2002). Some of the most representative tools-- LEI (Learning Environment Inventory) or CES (Classroom Environment Scale)-- are based on quantitative questionnaires. Nevertheless, other tools combine  qualitative and quantitative methods (Fraser, 2002). The nature and goals of LLEs lead us to consider the Mixed Methods Research paradigm, widely described by Onwuegbuzie and Leech (2005). The main feature of this emerging paradigm is a pragmatic approach based on the integration between qualitative and quantitative methods.

Regarding LLEs, the SLEI (Science Laboratory Environment Inventory) is an extensively used tool, which has been applied in many research studies using large samples of students Hofstein, Nahum and Shore (2001). This assessment tool consists of objective questionnaires whose items include the following scales: (a) Students’ cohesiveness; (b) Activity open-endedness; (c) Integration; (d) Clarity rules; (e) Infrastructure.

1.3 The characterization of the Laboratory Learning Environment from a Mixed Research Paradigm: Student generated questions as a useful data source

From the Mixed Research Paradigm, we argue for using triangulation as a strategy based on combining different methodologies (Fraser, Gidding & McRobbie, 1992). Indeed, there is  experimental evidences of the unreliability of LLEs assessment tools based only on students’ survey responses. According to Richardson (2004), questionnaire- based research should consider possible response bias in such instruments. Results may be influenced by affective issues as well as by students’ general understanding and vision of learning activities. Indeed, the influence of cultural issues and personal traits in surveys results has been studied extensively in order to  identify “response styles” (O’Neill, 1967).

A promising, alternative approach to achieving triangulation may be the analysis of spontaneous, students-generated questions (SGQs, from now on) during laboratory sessions (Harper, Etkina & Lin, 2003; Teixeira-Dias, Pedrosa de Jesús, Neri de Souza & Watts, 2005). According to Teixeira-Dias et al. (2005), SGQs can be considered an interactivity indicator, showing the degree in which the students organize their activities in a cooperative learning environment (Chin & Osborne, 2008). Furthermore, given the amount and quality of interactions are considered the most relevant features of an inquiry learning environment, SGQs analysis can be a useful way to characterize it.  As proposed by Pedrosa de Jesús, Neri de Souza, Teixeira-Dias and Watts (2004), two general types of questions can be distinguished:  questions that allow students to manage their cognitive process themselves--showing ability and autonomy development—and operative questions involving concrete tasks organization and manipulative procedures. From the standpoint of  laboratory learning, the following reflections justify the usefulness of the SGQs analysis:

·       SGQs are a reliable expression of greatest or lowest acquisition of previous necessary knowledge, abilities and skills. In addition, the degree to which information requested in SGQs pertains to laboratory work-- prelab content, laboratory manual, advice regarding equipment use, etc. (information which must be known by students without receiving any help)-- may be a good indicator of student engagement.  At the same time, SGQs analysis may be useful for evaluating educational resources (laboratory manuals, videotape recordings, etc.). Likewise, some aspects concerning to laboratory organization and management such as rule clarity, material and equipment availability, wastes storage, etc., can be analysed.

·       SGQs can help us analyse curricular integration of experimental activities, such as the suitability of prelabs.

·       SGQs can be useful indicators to identify the learning problems related to lab work methodology. SGQs  may reveal the degree to which experimental tasks are developed as an inquiry and reflective process,  or a “cookbook” approach. 

·       Lastly, SGQs reflect students’ ability to distinguish what is important from what is accessory. Thus, irrelevant aspects of concepts or experimental procedures are often overrated, hiding the most meaningful work issues.  This conceptual and operational overload can be related to what has been referred to in the literature as “noise” (Byrne, 1990).

1.4 Research Aims

This research aims to develop and apply a methodology based on SGQs analysis in order to make an overall assessment of laboratory learning environment as a tool for decision-making in the design and management of laboratory work activities. The final outcome of this analysis is to create an encoding framework to classify and interpret SGQs in order to be able to quantify in a simple descriptive way the most meaningful educational categories. This tool should be useful in the following ways: a) as a triangulation research strategy when used together with quantitative questionnaires;  b) for carrying out comparisons among different educational or alternative learning contexts; and c) for monitoring implementation of educational innovations activities, etc. from a holistic point of view.

2           Design and methodology

2.1 Context and design

This experiment was carried out in the first course at the Technical Superior School of Rural Environments and Enology, at the Polytechnic University of Valencia (Spain). The analysed laboratory sessions are part of a four- month course in introductory organic chemistry. The laboratory activities take place in 2-hour long sessions.  Attending student numbers have fluctuated between 8 and 26, though in most cases they have hovered around 15. Prelab activities consist of an audio visual tutorial together with an online test (Llorens-Molina, 2008).

A total amount of 377 SGQs from 25 laboratory sessions were collected over five consecutive years. The working method begins with writing down immediately, in situ, the SGQs addressed to teacher during laboratory sessions. Sometimes very short explanations and marginal notes are added to the SGQ text in order to facilitate the encoding process. 

2.2 Methodology

To make sense of SGQs, a qualitative research design based in the Grounded Theory (Glaser & Strauss, 2006) has been developed by means of an encoding process, developed by Charmaz (2006). From this theoretical point of view, the key process for turning data into theory is the stepwise refinement of the encoding framework. It concerns both code texts (they become progressively more inclusive and educational meaningful) and their relationships (progressively more diversified and complex). The encoding process starts with theoretical sampling and data collection, followed by initial encoding, a focused encoding, and ends with the making and hierarchical structuring of theoretical categories. The encoding process was carried out by means of software Atlas.ti (5.2) (Qualitative data analysis, 2002-2012). Below, this process is described in detail.

The starting material (primary documents) is the teacher report containing SGQs and their marginal notes. Each question (or a part to which can be attributed an independent meaning) is considered a quotation. From these quotations, an open encoding process leads to a set of initial codes. These may be keywords or language structures from quotations (“in vivo” encoding). For example:

·       “What do we must do now?”

·       “How do I switches on the heater?”

·       “Where is the paper filter?”

·       “The ice is added for it to precipitate, isn’t it?”

·       “Where are the oil wastes thrown out?”

Likewise, other initial codes may come from primary categories with educational meaning. For example:

·        The laboratory task goal is misunderstood: “Do we weigh the aqueous phase in biodiesel obtaining?” (Biodiesel is the non aqueous phase)

·        Hypothesis making: “Will the soap be obtained before if the temperature is higher?”

·        Requesting a word meaning clarification: “Is ethanol the same as 96º alcohol?”

From these initial codes, successive networks may be made. These networks allows us to establish different types of relationships among the codes: association, causality, contradiction. Likewise, the encoding process, itself, requires splitting and merging in order to create a code list.  The quotation analysis is then followed by an encoding by list. Code assignation is not a one-to-one relation, as several codes can be related to a single quotation. During the entire process a crucial role should be assigned to the memos as tools to guide the encoding process.

At the end, this process creates a network consisting of:

·       Codes, which are the most simple categories in SGQs classification

·       Code families, which bring together all codes showing significant educational relationships

·       Supercodes, which display higher levels of generality and abstraction,and more general issues in in LLEs characterization.

Document 1 (complementary materials) shows the encoding framework and explains the meaning of established categories. These codes, super codes and code families can be displayed by means of a network which shows the meaningful relationships among them (Document 2 in complementary materials).  In order to improve the display of this theoretical framework, the network from Atlas.ti is later transformed by means of CmapTools®. This software is normally used to make and share conceptual maps, and its graphic resources facilitate the expression of hierarchical and quantitative relationships among categories.

To identify the most interrelated codes, those that share a greater number of common quotations can be selected by means of the query tool in Atlas.ti, which allow to recover the co-occurrence among codes. This way, for each code, the ones which share at least a quotation are displayed (Document 3 in complementary materials).

By means of a simplified encoding framework, a quantitative approach was applied as an example to compare the LLEs of similar groups for two consecutive courses.

3           Results

3.1 Qualitative approach: making an encoding framework to describe LLEs

The encoding process of the SGQs from the 25 analyzed laboratory sessions leads to the proposed network shown in figure 1. (Codes, code families and supercodes are described in complementary materials, document 1; likewise, the network from Atlas.ti is shown in document 2).

Fig. 1.  Proposed encoding framework from SGQs by means of CmapsTools.

In this framework (Figure 1), three main realms (matching with supercodes) can be distinguished:

a) SGQs concerning conceptual and methodological issues. These are classified from three different viewpoints. According their purpose, several code families may be identified:

·       Questions intended to receive feed-back. They usually aim to confirm facts or manipulative processes before or after doing laboratory operations(code family 1.1.1.)

·       Questions seeking conceptual or methodological information or clarification. (code family 1.1.2.) Often, these questions consist of “What to do” or “How to do”.

·        Questions which are intended to ask for guidance about “how to do” and “what to do” referred to any experimental process (code family 1.1.3.)

b) Other minority but meaningful questions involve a participative and active attitude regarding laboratory organization and development, as well as an interest in delving further into conceptual and methodological issues (supercode 2 in figure 1).  They are, perhaps, the most representative of autonomy development.

c) Lastly, some relevant code families are related to infrastructural issues: safety and environmental care; organization and availability of material; and proper equipment use and functioning (supercode 3 in figure 1)

The meaning of the codes included in the first category (supercode 1) can be enriched taking into account the available information from the learning context. (code family 1.2. in figure 1). In that sense, these relationships may reveal learning deficiencies as the result of poor engagement in the subject-matter, inadequate coordination between theoretical and experimental contents, etc. Similarly, some questions reflect a lack of understanding of the purpose and goals of general and specific laboratory activities.  (code family 1.3. in figure 1).

3.2 Descriptive quantitative approach

To show a quantitative approach to SGQs analysis, several simplified categories combining the defined codes has been proposed in table 1.

Table 1. Simplified categories and their correspondence with the ones from encoding framework

These simplified categories have been applied to interpret year 2009 and 2010 SGQs analysis (Fig.2)

Fig. 2. SGQs occurrence (ratios respect the total amount of questions) in year 2009 and 2010 experiences for simplified categories

The quantitative significance of certain identified codes related to infrastructural issues or operational knowledge with which the students have no previous experience should be outlined. When the students’ attention is focused on these concerns, higher cognitive level thinking becomes difficult.  This problem is a well-known feature of LLEs. According to Byrne [16], during laboratory session, students may be subjected to a large amount of heterogeneous and poor, hierarchically- structured information inputs which are considered “noise”. These inputs are rarely important, and mask the essential meaning (“signal”) of the laboratory work.

In addition, the high incidence of the following types of questions should be highlighted:

·       Those intended to verify that real results match the expected results, or that the operation has been performed correctly.

These types of questions can be interpreted as an indication of low degree of autonomy. A significant number of students do not conceive the experimental observations as a fact to be interpreted, but rather as the consequence of something which may be well or badly done,  according to a  “cookbook” approach.

·       Methodological questions regarding the experimental process and questions concerning specific laboratory manipulations. These often involve operations from everyday life.

·       Questions requesting information available in the laboratory work context.

The item 1.2.4.: “Conceptual or methodological information where previous knowledge is unlikely” shows a very high score, but must be analyzed in light of the fact that the students were clear beginners with regard to laboratory work. In fact, the use of ordinary material and equipments were unknown to the majority of students.

A quantitatively significant and closely linked set of items (Fig.3) can be observed when the code’s cooccurrence is analyzed (document 3 in complementary materials).  Its educational meaning can be interpreted a signal of alarm regarding the students’ total understanding of laboratory task meaning and its function in the subject matter context.

 

Fig. 3. Set of codes with the stronger linkage and quantitative relevance in our educational context.

If the results from 2009 and 2010 are compared, important similarities in quantitative main codes and links can be appreciated. Nevertheless, as an example of the proposed tool usefulness, at least two relevant differences may be observed: The rate questions/student is much higher, and the triple linkage (1.1.1.1)-(1.2.1.)- (1.3.1) especially strong in academic year 2010. This fact reveals that little attention has been paid to prelab activities (Indeed, a significant decrease was observed from year 2009 to 2010 concerning the online tests marks associated to each one of prelab activities).

4           Conclusions

The theoretical framework established from the SGQ analysis provides noteworthy categories in line with categories indicated in the literature (Hofstein, Nahum & Shore, 2001).  Nevertheless, the proposed theoretical framework includes categories and relationships among them which can be a valuable contribution in order to a more detailed and deepen LLEs knowledge. In that respect, several critical issues can be distinguished:

·       Some code families are clear noise sources. More attention should be paid to their origin, and measured taken to guarantee minimum laboratory skills before taking a university course in general chemistry.  It is paradoxical that great effort has been devoted to research on misconceptions, while alternative frameworks, etc., and hands-on competences have been undervalued, even with regard to everyday and simple manipulative skills.

·       More effective display of information concerning waste disposal, material and reagents location etc. could help improve laboratory activities. Indeed, improvements in laboratory management must be linked to careful design of all relevant educational resources in order to achieve absolute consistency in information received by students regarding organizational and infrastructural aspects such as schedules, environmental protection, safety rules etc. ((e.g., many SGQs reflect inconsistency in labelling of reagents or glass materials and information provided in laboratory handbook, thus causing unnecessary confusion and loss of time.

·       The great frequency of SGQs aimed at receiving feed-back-- usually as “confirmatory” questions-- together with the poor cognitive level of students regarding conceptual or methodological inquiry is not surprising. This evidence should encourage us to promote the methodological and organizational changes that incorporation into the  European Space of Higher Education demands.

In summary, from the standpoint of educational practice, this methodology may be particularly useful for   implementing experimental syllabi  or methodological innovations in a learning context. These results lead us to argue for a radical transformation in the role of laboratory work in learning. Indeed, lab work must be integrated into a coherent process of constructing knowledge, beyond its traditional role of testing theory and hands-on skills development. Furthermore, a number of problems related to management laboratory issues can be detected by means of the impact of items in category 3 in encoding framework (Fig. 1). This information may be useful feed-back for laboratory instructors.   

In the research field, the proposed methodology may be useful as a qualitative way to be used together with quantitative tools in order to enhance the significance and internal validity of these tools. SGQ analysis can be an effective means of attaining a realistic and accurate characterization of LLEs in experimental control groups in educational research design, beyond their simple and aprioristic distinction based on the instructional proposals.

5           Complementary materials

The following complementary materials can be retrieved from: https://poliformat.upv.es/access/content/user/19877419/ELSIN 2010/

·       Document 1: Defined codes, supercodes and code families from five years long experience.

·       Document 2: Network from Atlas.ti

·       Document 3: Codes’ cooccurrence

REFERENCES

Bravo, A., & Fernández del Valle, J. (2000). La evaluación convencional frente a los nuevos modelos de evaluación auténtica. Psicothema, 12(2), 95-99.

Byrne, M. S. (1990). More effective practical work. Education in Chemistry, 27, 12-13.

Charmaz, K. (2006). Constructing grounded theory: A practical guide through qualitative analysis. London, UK: SAGE Publications Ltd.

Chin, C., & Osborne, J. (2008). Students’ questions: A potential resource for teaching and learning science. Studies in Science Education, 44(1), 1-39. http://dx.doi.org/10.1080/03057260701828101

Dochy, F., Segers, M., Van den Bossche, P., & Struyven, K. (2005). Students’ perceptions of a problem-based learning environments. Learning Environments Research, 8, 41-66. http://dx.doi.org/10.1007/s10984-005-7948-x

Fraser, B. J. (2002). Learning environments research: Yesterday, today and tomorrow. In Swee Chiew Goh & Myint Swe Khine, (Eds.), Studies in Educational Learning Environments. Perth, Australia: Murdoch University.

Fraser, B., Gidding, G. J., & McRobbie, C. J. (1992). The SLEI forms. In B. J. Fraser et al. (Eds.), The name assigned to the document by the author. This field may also contain sub-titles, series names, and report numbers.Assessing the climate of science laboratory classes. Perth, Australia: Key Centre for School Science and Mathematics, Curtin University of Technology. http://dx.doi.org/10.1142/9789812777133_0001

Gijbels, D., Van den Watering, G., Dochy, F., & Van den Bossche, F. (2006).  New learning environments and constructivism: The students’ perspective. Instructional Science, 34, 213-226. http://dx.doi.org/10.1007/s11251-005-3347-z

Glaser, B., & Strauss, A. (2006). Theoretical Sampling. In N. K. Denzin (Eds.), Sociological methods: A Sourcebook. New Brunswick, New Jersey: Transaction publishers.

Harper, K., Etkina, E., & Lin, Y. (2003). Encouraging and analyzing student questions in a large physics course. Meaningful Patterns for Instructors. Journal of Research in Science Teaching, 40, 776-791. http://dx.doi.org/10.1002/tea.10111

Hofstein, A. (2004). The laboratory in chemistry education: thirty years of experience with developments, implementation and research. Chemistry Education: Research and Practice, 5, 247-264.

Hofstein, A., Nahum, T. L., & Shore, R. (2001). Assessment of the learning environment of inquiry-type laboratories in high school chemistry. Learning Environments Research, 4, 193-207. http://dx.doi.org/10.1023/A:1012467417645

Llorens-Molina, J. A. (2008). Design and assessment of an online prelab model in general chemistry: A case study. Journal of the Research Center for Educational Technology, 4(2), 15-31.

O’Neill, H. W. (1967). Response style influence in public opinion surveys. Public Opinion Quartely, 31(1), 95-102. http://dx.doi.org/10.1086/267487

Onwuegbuzie, A., & Leech, N. (2005). On Becoming a Pragmatic Researcher: The importance of combining quantitative and qualitative research methodologies. International Journal of Social Research Methodology,  8(5), 375-387. http://dx.doi.org/10.1080/13645570500402447

Pedrosa de Jesús, M. H., Neri de Souza, F., Teixeira-Dias, J., & Watts, M. (2004). Students´ questions as organisers for small group learning in chemistry. Proceedings of the 7^th ECRICE, 3^rd  ECCE, Slovenia,  Ljubjljana.

Qualitative data analysis (2002-2012). Retrieved  January 30, 2012, from http://www.atlasti.com/

Richardson, J. T. E. (2004). Methodological issues in questionnaire-based research on student learning in higher education. Educational Psychology Review, 16(4), 347-358.

Teixeira-Dias, J., Pedrosa de Jesús, M. H., Neri de Souza, F., & Watts, M. (2005). Teaching for quality learning chemistry. International Journal of Science Education, 7, 1123-1137. http://dx.doi.org/10.1080/09500690500102813

Wilson, B. J. (1995). Why we talk about Learning environments? Educational Technology, 35(5), 25-30.

 

AUTHOR BIOGRAPHY

Juan Antonio Llorens-Molina

Doctor in Chemistry and professor at the ETSIAMN (Polytechnic University of Valencia, Spain). He has developed different tasks related to in-service training teachers for more than 20 years. Nowadays, he is researcher member of the Mediterranean Agroforestal Institute in the same university and belongs to IEMA (Group of Innovation in the Evaluation for the Improvement of Active Learning).

Jesús María Llorens de Jaime

Sociologist. Graduated from University of Valencia (Spain) and undergraduate in Social Anthropology at the Complutense University of Madrid (Spain).

Isidora Sanz Berzosa

Doctor in Biology and professor at the ETSIAMN (Polytechnic University of Valencia, Spain). She has developed most of her research activity in the Center of Chemical Ecology in the same university and nowadays is part of two research teams on essential oils. Also, she belongs to IEMA (Group of Innovation in the Evaluation for the Improvement of Active Learning).

Published by OmniaScience (www.omniascience.com)

Journal of Technology and Science Education, 2012 (www.jotse.org)

Article's contents are provided on a Attribution-Non Commercial 3.0 Creative commons license. Readers are allowed to copy, distribute and communicate article's contents, provided the author's and Intangible Capital journal's names are included. It must not be used for commercial purposes. To see the complete licence contents, please visit http://creativecommons.org/licenses/by-nc/3.0/es/