Journal of The Korean Association For Science Education (한국과학교육학회지)

The Korean Association for Science Education (한국과학교육학회)
권호 표지
DOI QR코드

DOI QR Code

Development of Framework and Rubric for Measuring Students' Level of Systems Thinking

학생들의 시스템 사고 수준 측정을 위한 Framework와 Rubric의 개발

  • Received : 2018.04.06
  • Accepted : 2018.05.16
  • Published : 2018.06.30
Download PDF
⟨ Previous Next ⟩

Abstract

The purposes of this study are 1) to identify systems thinking level and definition, 2) to develop a framework for the assessment of systems thinking level, and 3) to develop a rubric for scoring open-ended written responded test. In order to achieve these purposes, a total of 60 articles were analyzed by using the literature analysis framework. The systems thinking level and definition are identified through the results of systems thinking literature analysis. Based on the systems thinking level and definitions, the research derived a framework that includes the core ideas and evaluation content of each level. In addition, rubric for the scoring of open-ended response test items was revised and supplemented. It is concluded that a content validity test on the tools (systems thinking level and definition, framework for item development, rubric) has been developed in the study. The content validity was verified by 7-science education experts. According to the result of CVI, it was found to be more than .95 in all three tools. Based on the results of this study, the research will develop items that can measure students' level of systems thinking. The construct validity and criterion validity of the developed items should be verified systematically. The research could carry out a validation study for the systems thinking measurement related to the core competence emphasized in the 2015 revised curriculum.

이 연구의 목적은 학생들의 시스템 사고 수준을 측정하는 문항을 개발하기 위하여 첫째, 문헌 분석을 통해 시스템 사고 수준과 정의를 제시하고 둘째, 시스템 사고 정의와 수준에 근거하여 문항 개발을 위한 Framework을 개발하고 셋째, 시스템 사고 측정을 위한 열린 응답형 검사지를 채점할 수 있는 루브릭을 개발하는 것이다. 이를 위하여 문헌 분석 틀을 적용하여 국외 논문 60편에 대한 문헌 분석을 실시하였으며 분석된 내용을 근거로 시스템 사고 수준과 정의를 제시하였다. 그리고 시스템 사고 수준과 정의에 근거하여 각 수준별 핵심 아이디어와 평가 요소를 포함한 Framework을 개발하였다. 또한 순위선다형 문항 외에 열린 응답형 검사 문항의 채점을 위한 루브릭을 개발하였다. 루브릭은 Hung(2008)이 개발한 시스템 사고 측정을 위한 루브릭에 기반하여 연구팀에서 시스템 사고 수준과 정의에 적합하도록 문항과 위계를 수정하였다. 그리고 각 문항마다 1~5점 척도로 변환이 가능하도록 각 점수에 대한 정의를 제시하였다. 연구팀이 개발한 시스템 사고 수준과 정의, 문항 개발을 위한 Framework, 열린 응답형 검사지 채점을 위한 루브릭에 대하여 과학교육 전문가 7인에게 내용 타당도 검증을 의뢰하였으며 그 결과 CVI가 세 도구에서 모두 .95이상이 확인되었다. 이 연구 결과를 바탕으로 앞으로 학생들의 시스템 사고 수준을 측정할 수 있는 문항을 개발하고 개발된 문항에 대한 구인 타당도 및 준거 타당도 검증을 실시하고자 한다. 나아가 2015 개정 교육과정에서 강조하는 핵심 역량과 관련된 시스템 사고수준의 체계적인 측정을 위한 타당화 연구도 진행하고자 한다.

Keywords

References

  1. American Educational Research Association, & American Psychological Association. National Council for Measurement in Education (AERA, APA, & NCME).(1999). Standards for educational and psychological testing.
  2. Arter, J., McTighe, J.(2001). Scoring rubrics in the classroom: Using performance criteria for assessing and improving student performance. Thousand Oaks, CA: Corwin Press.
  3. Ben-Zvi Assaraf, O., & Orion, N. (2005a). Development of System Thinking Skills in the Context of Earth System Education. Journal of Research in Science Teaching, 42(5), 518-560. https://doi.org/10.1002/tea.20061
  4. Ben-Zvi Assaraf, O., & Orion, N. (2005b). A Study of Junior High Students’Perceptions of the water cycle. Journal of Geoscience Education, 53(4), 366-373. https://doi.org/10.5408/1089-9995-53.4.366
  5. Ben-Zvi Assaraf, O., & Orion, N. (2009). A Design Based Research of an Earth Systems Based Environmental Curriculum. Eurasia Journal of Mathematics, Science & Technology Education, 5(1), 47-62. https://doi.org/10.12973/ejmste/75256
  6. Ben-Zvi Assaraf, O., & Orion, N. (2010). Four Case Studies, Six Years Later: Developing System Thinking Skills in Junior High School and Sustaining Them over Time. Journal of Research in Science Teaching, 47(10), 1253-1280. https://doi.org/10.1002/tea.20383
  7. Bosch, O. J. H., King, C. A., Herbohn, J. L., Russell, I. W., & Smith, C. S. (2007). Getting the big picture in natural resource management-systems thinking as ‘method’for scientists, policy makers and other stakeholders. Systems Research and Behavioral Science, 24(2), 217-232. https://doi.org/10.1002/sres.818
  8. Bosch, O. J., Nguyen, N. C., Maeno, T., & Yasui, T. (2013). Managing complex issues through evolutionary learning laboratories. Systems Research and Behavioral Science, 30(2), 116-135. https://doi.org/10.1002/sres.2171
  9. Breslyn, W., McGinnis, R., McDonald, R., & Hestness, E. (2016). Developing a learning progression for sea level rise: A major impact of climate change. Journal of Research In Science Teaching, 53(10), 1471-1499. https://doi.org/10.1002/tea.21333
  10. Broks, A. (2016). Systems Theory Of Systems Thinking - General And Particular Within Modern Science And Technology Education (STE). Journal of Baltic Science Education, 15(4), 408-410.
  11. Cabrera, D., Colosi, L., & Lobdell, C. (2008). Systems thinking. Evaluation and program planning, 31(3), 299-310. https://doi.org/10.1016/j.evalprogplan.2007.12.001
  12. Checkland, P. (1981). Systems thinking, systems practice. Chichester: John Wiley & Son Ltd.
  13. Davis A., & Stroink, M. (2016). The Relationship between Systems Thinking and the New Ecological Paradigm. Systems Research and Behavioral Science, 33, 575-586. https://doi.org/10.1002/sres.2371
  14. Dori, Y. J., Tal, R. T., & Tsaushu, M. (2003). Teaching biotechnology through case studies - Can we improve higher order thinking skills of nonscience majors? Science Education, 87(6), 767-793. doi:10.1002/sce.10081.
  15. Fehring, R. J. (1987). Methods to validate nursing diagnoses. Heart and Lung, 16(6), 625-629.
  16. Gudovitch, Y. (1997). The global Carbon cycle as a model for teaching 'earth systems' in high school: Development, implementation, and evaluation. Unpublished master's thesis, the Weizmann Institute of Science, Rehovot, Israel [in Hebrew].
  17. Hung, W. (2008). Enhancing systems thinking skills with modelling. British Journal of Educational Technology, 39(6), 1099-1120. https://doi.org/10.1111/j.1467-8535.2007.00791.x
  18. Im, Y. & Lee, H. (2014). Development and analysis of effects of writing educational program for improving system thinking ability. Journal of Learner-Centered Curriculum and Instruction, 14, 407-427.
  19. Ison, R. (1999). Applying Systems Thinking to Higher Education. Systems Research and Behavioral Science, 16, 107-112. https://doi.org/10.1002/(SICI)1099-1743(199903/04)16:2<107::AID-SRES278>3.0.CO;2-E
  20. Jackson, M. (2003). Systems Thinking: Creative Holism for Manager. Chichester: John Wiley & Son Ltd.
  21. Jeon, J. & Lee, H. (2015). The development and application of STEAM education program based on systems thinking for high school students. Journal of the Korean Association for Science Education, 35(6), 1007-1018. https://doi.org/10.14697/jkase.2015.35.6.1007
  22. Kali, Y., Orion, N., & Eylon, B. (2003). Effect of Knowledge Integration Activities on Students’ Perception of the Earth’s Crust as a Cyclic System. Journal of Research in Science Teaching, 40(6), 545-565. https://doi.org/10.1002/tea.10096
  23. Kang, C., Lee, H., Yoon, I., & Kim, E. (2008). Analysis of conceptions related to Earth system and systems-thinking of high school student about water cycle. Journal of Science Education, 32(1), 61-72. https://doi.org/10.21796/jse.2008.32.1.61
  24. Kim, D. (1999). Introduction to Systems Thinking. Sydney: Pegasus Communications.
  25. Kim, D. (2005). Introspecsive reflection on applying systems thinking: Toward an incremental systems thinking. Journal of Institute of Governmental Studies, 11, 63-85.
  26. Kwon, Y., Kim, W., Lee, H., Byun, J., & Lee, I. (2011). Analysis of Biology Teachers’ Systems Thinking about Ecosystem. BIOLOGY EDUCATION, 39(4), 529-543. https://doi.org/10.15717/bioedu.2011.39.4.529
  27. Lee, H. & Lee, H. (2013). Revalidation of measuring instrument systems thinking and comparison of systems thinking between science and general high school students. Journal of the Korean Association for Science Education, 33, 1237-1247. https://doi.org/10.14697/jkase.2013.33.6.1237
  28. Lee, H., & Lee, H. (2016). Effects of Systems Thinking on High School Students’ Science Self-Efficacy. The Journal of The Korean Earth Science Society, 37(3), 133-145. https://doi.org/10.5467/JKESS.2016.37.3.133
  29. Lee, H., & Lee, H. (2017). Analysis and Effects of High School Students' Systems Thinking Using Iceberg(IB) Model. Journal of the Korean Association for Science Education, 37(4), 611-624. https://doi.org/10.14697/JKASE.2017.37.4.611
  30. Lee, H., Kwon, H., Park, K., & Lee, H. (2013). An instrument development and validation for measuring high school students’ systems thinking. Journal of the Korean Association for Science Education, 33, 995-1006. https://doi.org/10.14697/jkase.2013.33.5.995
  31. Lee, H., Kwon, Y., Oh, H., & Lee, H. (2011). Development and application of the educational program to increase high school students’ systems thinking skills: Focus on global warming. Journal of the Korean Earth Science Society, 32, 784-797. https://doi.org/10.5467/JKESS.2011.32.7.784
  32. Maani, K. E., & Maharaj, V. (2004). Links between systems thinking and complex decision making. System Dynamics Review, 20(1), 21-48. https://doi.org/10.1002/sdr.281
  33. Maani, K., & Cavana, R. Y. (2007). Systems thinking, system dynamics: Managing change and complexity. Prentice Hall.
  34. Mayer, V.J. (1995). Using the Earth system for integrating the science curriculum, Science Education, 79, 375-391. https://doi.org/10.1002/sce.3730790403
  35. Meadows, D. L., Booth Sweeney, L., & Martin Mehers, G. (2016). The climate change playbook. White River Junction, VT: Chelsea Green.
  36. Ministry of Education[MOE]. (2015). Science curriculum[no. 9]. Sejong: Author.
  37. Mitroff, I. I., & Linstone, H. A. (1993). The unbound mind. New York: Oxford University Press.
  38. Mohan, L., Chen, J., & Anderson, C. W. (2009). Developing a multi-year learning progression for carbon cycling in socio-ecological systems. Journal of Research in Science Teaching, 46(6), 675-698. https://doi.org/10.1002/tea.20314
  39. Mohan, L., Sharma, A., Jin, H., Cho, I., & Anderson, W. (2006). Developing a carbon cycling learning progression for K-12. Paper presented at the annual meeting of the National Association for Research on Science Teaching (San Francisco, CA).
  40. Moon, B., Jeong, J., Kyung, J., Koh, Y., Youn, S., Kim, H., & Oh, K. (2004). Related conception s to earth system and applying of systems thinking about carbon cycle of the preservice teachers. Journal of the Korean Earth Science Society, 25, 684-696.
  41. Nguyen, N., Bosch, O., & Maani, K. (2011). Creating ‘Learning Laboratories’ for Sustainable Development in Biosphere: A Systems Thinking Approach. Systems Research and Behavioral Science, 28, 51-62. https://doi.org/10.1002/sres.1044
  42. Nguyen, N., Graham, D., Ross, H., Maani, K., & Bosch, O. (2012). Educating Systems Thinking for Sustainability: Experience with a Developing Country. Systems Research and Behavioral Science, 29(1), 14-29. https://doi.org/10.1002/sres.1097
  43. O'Connor, J. & McDermmot, I. (1997). The art of systems thinking: Essential skills for creativity and problem solving. London, UK: Thorsons Publishers.
  44. Orion, N. (2002). An Earth Systems curriculum development model. In V.J. Mayer(Ed.), Global Science Literacy (pp.159-168). Kluwer Academic Publishers.
  45. Orion, N., & Basis, T. (2008). Characterization of High School Students' System Thinking Skills in the Context of Earth Systems. Presented in the 2008 NARST Annual Meeting. March, 2008. Baltimore, U.S.A.
  46. Ossimitz, G. (2000). Teaching system dynamics and systems thinking in Austria and Germany. In P. Davidsen, D. N. Ford & A. N. Mashayekhi (Eds.), Proceedings of the 18th International Conference System Dynamics Society_2000_4p.pdf.
  47. Park, B. & Lee, H. (2014). Development and application of systems thinking-based STEAM education program to improve secondary science gifted and talented students’ systems thinking skill. Journal of Gifted/Talented Education, 24, 421-444. https://doi.org/10.9722/JGTE.2014.24.3.421
  48. Park, K., Jeong, H., Jeon, J., & Lee, H. (2018). Development and application of systems thinking-based STEAM education program to improve Pre-service science teachers’ systems thinking skill. Teacher Education Research, 57(1), 108-128. https://doi.org/10.15812/ter.57.1.201803.108
  49. Plummer, J., & Maynard, L. (2014). Building a learning progression for celestial motion: An exploration of students’ reasoning about the Season. Journal of Research in Science Teaching, 51(7), 902-929. https://doi.org/10.1002/tea.21151
  50. Raudenbush, S. W., Rowan, B., & Cheong, Y. F. (1993). Higher order instructional goals in secondary schools: Class, teacher, and school influences. American Educational Research Journal, 30(3), 523-553. https://doi.org/10.3102/00028312030003523
  51. Resnick, L. B., & Resnick, D. P. (1992). Assessing the thinking curriculum: New tools for educational reform. In Changing assessments (pp. 37-75). Springer, Dordrecht.
  52. Richmond, B. (1993). Systems thinking: critical thinking skills for the 1990s and beyond. System Dynamics Review, 9, 113-133. https://doi.org/10.1002/sdr.4260090203
  53. Senge, P. M. (1996). The fifth discipline: Fieldbook. New York: Broadway Business.
  54. Senge, P. M. (2006). The fifth discipline : The art & practice of the learning organization. New York: Crown Business.
  55. Senge, P. M. (2012). Schools that learn(Updated and Revised): A fifth discipline fieldbook for educators, parents, and everyone who cares about education. New York: Doubleday.
  56. Shear, L. F. (1998). When Science Learners are Language Learners: Designing Linguistically Aware Instruction to Teach Science" the Knew Way". University of California, Berkeley May 1998.
  57. Sibley, D., Anderson, C., Heidemann, M., Merrill, J., Parker, J., & Szymanski, D. (2007). Box Diagrams to Assess Students’ Systems thinking about the Rock, Water and Carbon Cycle. Journal of Geoscience Education, 55(2), 138-146. https://doi.org/10.5408/1089-9995-55.2.138
  58. Stave, K., & Hopper, M. (2007). What Constitutes Systems Thinking? A Proposed Taxonomy. Proceedings of the 25th International Conference of the System Dynamics Society. Boston, MA, July 29-August 3, 2007.
  59. Sterman, J. D. (2000). Business dynamics: systems thinking and modeling for a complex world (No. HD30. 2 S7835 2000).
  60. Sweeney, L. & Sterman, D. (2000). Bathtub dynamics: initial results of a systems thinking inventory. System Dynamics Review, 16, 4, 249-286.
  61. Sweeney, L. B., & Meadows, D. (2010). The systems thinking playbook: Exercises to stretch and build learning and systems thinking capabilities. Chelsea Green Publishing.
  62. Vo, H. V., Chae, B., & Olson, D. L. (2006). Integrating systems thinking into IS education. Systems Research and Behavioral Science, 23(1), 107-121. https://doi.org/10.1002/sres.720
  63. Walker, G. H., Stanton, N. A., Jenkins, D. P., & Salmon, P. M. (2009). From telephones to iPhones: Applying systems thinking to networked, interoperable products. Applied Ergonomics, 40(2), 206-215. https://doi.org/10.1016/j.apergo.2008.04.003
  64. Zeidler, D. L., Lederman, N. G., & Taylor, S. C. (1992). Fallacies and student discourse: Conceptualizing the role of critical thinking in science education. Science Education, 76(4), 437-450. https://doi.org/10.1002/sce.3730760407
  65. Zohar, A., & Dori, Y. J. (2003). Higher order thinking skills and low-achieving students: Are they mutually exclusive?. The journal of the learning sciences, 12(2), 145-181. https://doi.org/10.1207/S15327809JLS1202_1

Cited by

  1. 문항 반응 분석을 활용한 초등학생과 중학생의 시스템 사고 검사 도구 타당도 검증 vol.67, pp.2, 2018, https://doi.org/10.25152/ser.2019.67.2.249
  2. 시스템 사고 평가 루브릭을 활용한 예비교사들의 지구 시스템 내 탄소 순환에 대한 시스템 사고 수준 분석 vol.39, pp.5, 2019, https://doi.org/10.14697/jkase.2019.39.5.599
  3. 물의 순환 학습 상황에서 초등학생의 시스템 사고의 특징 vol.39, pp.3, 2020, https://doi.org/10.15267/keses.2020.39.3.412