Journal of The Korean Association For Science Education (한국과학교육학회지)

The Korean Association for Science Education (한국과학교육학회)
권호 표지
DOI QR코드

DOI QR Code

Automated Scoring of Argumentation Levels and Analysis of Argumentation Patterns Using Machine Learning

기계 학습을 활용한 논증 수준 자동 채점 및 논증 패턴 분석

  • Received : 2021.03.02
  • Accepted : 2021.05.31
  • Published : 2021.06.30
Download PDF
⟨ Previous Next ⟩

Abstract

We explored the performance improvement method of automated scoring for scientific argumentation. We analyzed the pattern of argumentation using automated scoring models. For this purpose, we assessed the level of argumentation for student's scientific discourses in classrooms. The dataset consists of four units of argumentation features and argumentation levels for episodes. We utilized argumentation clusters and n-gram to enhance automated scoring accuracy. We used the three supervised learning algorithms resulting in 33 automatic scoring models. As a result of automated scoring, we got a good scoring accuracy of 77.59% on average and up to 85.37%. In this process, we found that argumentation cluster patterns could enhance automated scoring performance accuracy. Then, we analyzed argumentation patterns using the model of decision tree and random forest. Our results were consistent with the previous research in which justification in coordination with claim and evidence determines scientific argumentation quality. Our research method suggests a novel approach for analyzing the quality of scientific argumentation in classrooms.

이 연구는 과학적 논증 담화에 대한 자동 채점의 성능 개선 방향을 탐색하였으며, 자동 채점 모델을 활용하여 논증 담화의 양상과 패턴을 분석하였다. 이를 위해 과학적 논증 수업에서 발생한 학생 발화를 대상으로 논증 수준을 평가하는 자동 채점을 수행하였다. 이 자동 채점의 데이터셋은 4가지 단위의 논증 피처와 논증 수준 평가틀로 구성되었다. 특히, 자동 채점에 논증 패턴을 반영하기 위하여 논증 클러스터와 n-gram을 활용하였다. 자동 채점 모델은 3가지의 지도 학습 기법으로 구성되었으며, 그 결과 총 33개의 자동 채점 모델이 구성되었다. 자동 채점의 결과, 최대 85.37%, 평균 77.59%의 채점 정확도를 얻었다. 이 과정에서 논증 담화의 패턴이 자동 채점의 성능을 개선하는 주요한 피처임을 확인하였다. 또한, 의사결정 나무와 랜덤 포레스트의 모델을 통하여 과학적 논증 수준에 따른 논증의 양상과 패턴을 분석하였다. 이를 통하여 주장, 자료와 함께 정당화가 체계적으로 구성된 과학적 논증과 자료에 대한 활발한 상호작용이 이루어진 과학적 논증이 논증 수준의 발달을 이끈다는 점 등을 확인하였다. 이와 같은 자동 채점 모델의 해석은 논증 패턴을 분석하는 새로운 연구 방법을 제언하는 것이다.

Keywords

References

  1. Breiman, L. (2001). Random forests. Machine learning, 45(1), 5-32. https://doi.org/10.1023/A:1010933404324
  2. Choi, J., Song, H., & Nam, K. (2010). Formulaic Expressions in Korean. Discourse and Cognition, 17(2), 163-190.
  3. Clark, D. B., & Sampson, V. (2008). Assessing dialogic argumentation in online environments to relate structure, grounds, and conceptual quality. Journal of Research in Science Teaching, 45(3), 293-321. https://doi.org/10.1002/tea.20216
  4. Driver, R., Newton, P., & Osborne, J. (2000). Establishing the norms of scientific argumentation in classrooms. Science education, 84(3), 287-312. https://doi.org/10.1002/(SICI)1098-237X(200005)84:3<287::AID-SCE1>3.0.CO;2-A
  5. Duschl, R. (2008). Science education in three-part harmony: Balancing conceptual, epistemic, and social learning goals. Review of research in education, 32(1), 268-291. https://doi.org/10.3102/0091732X07309371
  6. Erduran, S., Simon, S., & Osborne, J. (2004). TAPping into argumentation: Developments in the application of Toulmin's argument pattern for studying science discourse. Science education, 88(6), 915-933. https://doi.org/10.1002/sce.20012
  7. Fauzi, M. A., Utomo, D. C., Setiawan, B. D., & Pramukantoro, E. S. (2017, August). Automatic essay scoring system using N-gram and cosine similarity for gamification based E-learning. In Proceedings of the International Conference on Advances in Image Processing (pp. 151-155), Bangkok, Thailand.
  8. Ford, M. (2008). Disciplinary authority and accountability in scientific practice and learning. Science Education, 92(3), 404-423. https://doi.org/10.1002/sce.20263
  9. Gonzalez-Howard, M. & McNeill, K. L. (2017, April). Variation in how teachers support critique in argumentation discussions. Paper presented at the annual meeting of the National Association for Research in Science Teaching(NARST), San Antonio, TX.
  10. Grooms, J., Sampson, V., & Enderle, P. (2018). How concept familiarity and experience with scientific argumentation are related to the way groups participate in an episode of argumentation. Journal of Research in Science Teaching, 55(9), 1264-1286. https://doi.org/10.1002/tea.21451
  11. Ha, M., Lee, G.-G., Shin, S., Lee, J.-K., Choi, S., Choo, J., Kim, N., Lee, H., Lee, J., Lee, J., Jo, Y., Kang, K., & Park, J. (2019). Assessment as a Learning-Support Tool and Utilization of Artificial Intelligence: WA3I Project Case. School Science Journal, 13(3), 271-282. https://doi.org/10.15737/SSJ.13.3.201908.271
  12. Henderson, J. B., & Osborne, J. F. (2019). Using computer technology to support the teaching and learning of argumentation in chemistry. In Erduran, S. (Eds), Argumentation in Chemistry Education (pp. 79-105). London: Royal Society of Chemistry.
  13. Jeong, A. C. (2003). The sequential analysis of group interaction and critical thinking in online. The American Journal of Distance Education, 17(1), 25-43. https://doi.org/10.1207/S15389286AJDE1701_3
  14. Kelly, G. J., Druker, S., & Chen, C. (1998). Students' reasoning about electricity: Combining performance assessments with argumentation analysis. International journal of science education, 20(7), 849-871. https://doi.org/10.1080/0950069980200707
  15. KICE(Korea Institute Of Curriculum & Evaluation). (2006). A Study on the Development and Introduction of an Automated Scoring Program(RRI 2006-6). Retrieved from http://kice.re.kr/resrchBoard/view.do?seq=19388&s=kice&m=030102
  16. Kil, H.-h. (2018). The Study of Korean Stopwords list for Text mining. The Korean Language and Literature, 78, 1-25.
  17. Kim, M., & Ryu, S. (2020). Teacher Feedback on Process-Centered Assessment for Scientific Argumentation. Journal of The Korean Association For Science Education, 40(3), 271-289. https://doi.org/10.14697/JKASE.2020.40.3.271
  18. Kim, S.-J. (2019). Exploring the Possibility of Using N-gram Feature for Automatic Scoring of Argumentative Writing Tasks. The research in writing, 41, 37-62.
  19. Lee, G.-G., Ha, H., Hong, H.-G., & Kim, H.-B. (2018). Exploratory Research on Automating the Analysis of Scientific Argumentation Using Machine Learning. Journal of The Korean Association For Science Education, 38(2), 219-234. https://doi.org/10.14697/JKASE.2018.38.2.219
  20. Lee, M., & Ryu. S. (2020). Automated Scoring of Scientific Argumentation Using Expert Morpheme Classification Approaches. Journal of The Korean Association For Science Education, 40(3), 321-336. https://doi.org/10.14697/JKASE.2020.40.3.321
  21. Liu, O. L., Rios, J. A., Heilman, M., Gerard, L., & Linn, M. C. (2016). Validation of automated scoring of science assessments. Journal of Research in Science Teaching, 53(2), 215-233. https://doi.org/10.1002/tea.21299
  22. Maeng, S., Park, Y.-S., & Kim, C.-J. (2013). Methodological Review of the Research on Argumentative Discourse Focused on Analyzing Collaborative Construction and Epistemic Enactments of Argumentation. Journal of The Korean Association For Science Education, 33(4), 840-862. https://doi.org/10.14697/jkase.2013.33.4.840
  23. McNeill, K. L., & Krajcik, J. (2007). Middle school students' use of appropriate and inappropriate evidence in writing scientific explanations. In Marsha C. Lovett & Priti Shah (Eds), Thinking with Data (pp. 233-265). NY: Taylor & Francis.
  24. Nam, J., Kwak, K.,, Jang, K., & Hand, B. (2008). The Implementation of Argumentation Using Science Writing Heuristic (SWH) in Middle School Science. Journal of The Korean Association For Science Education, 28(8), 922-936.
  25. Nielsen, J. A. (2013). Dialectical features of students' argumentation: A critical review of argumentation studies in science education. Research in Science Education, 43(1), 371-393. https://doi.org/10.1007/s11165-011-9266-x
  26. Osborne, J., Erduran, S., & Simon, S. (2004). Enhancing the quality of argumentation in school science. Journal of research in science teaching, 41(10), 994-1020. https://doi.org/10.1002/tea.20035
  27. Osborne, J. F., Henderson, J. B., MacPherson, A., Szu, E., Wild, A., & Yao, S. Y. (2016). The development and validation of a learning progression for argumentation in science. Journal of Research in Science Teaching, 53(6), 821-846. https://doi.org/10.1002/tea.21316
  28. Park, C., Kim, Y, Kim, J, Song, J., & Choi, H. (2015). R data mining. Seoul: Kyowoo.
  29. Park, E. L., & Cho, S. (2014). KoNLPy: Korean natural language processing in Python. In Proceedings of the 26th Annual Conference on Human & Cognitive Language Technology (pp. 133-136), Chuncheon, Korea.
  30. Ryu, S., & Sandoval, W. A. (2015). The influence of group dynamics on collaborative scientific argumentation. Eurasia Journal of Mathematics, Science and Technology Education, 11(3), 335-351.
  31. Sampson, V., & Clark, D. B. (2008). Assessment of the ways students generate arguments in science education: Current perspectives and recommendations for future directions. Science education, 92(3), 447-472. https://doi.org/10.1002/sce.20276
  32. Sampson, V., Enderle, P. J., & Walker, J. P. (2012). The development and validation of the assessment of scientific argumentation in the classroom (ASAC) observation protocol: A tool for evaluating how students participate in scientific argumentation. In M. S. Khine(2012).(Ed.), Perspectives on scientific argumentation (pp. 235-264). Dordrecht, Netherlands: Springer.
  33. Shin, H. S., & Kim, H.-J. (2012). Development of the Analytic Framework for Dialogic Argumentation Using the TAP and a Diagram in the Context of Learning the Circular Motion. Journal of the Korean Association for Science Education, 32(5), 1007-1026. https://doi.org/10.14697/jkase.2012.32.5.1007
  34. Toulmin, S. (1958). The uses of argument. Cambridge, UK: Cambridge University Press.
  35. Tsoumakas, G., & Vlahavas, I. (2007, September). Random k-labelsets: An ensemble method for multilabel classification. In Proceedings of the 18th European Conference on Machine Learning (pp. 406-417), Warsaw, Poland.
  36. Yang, I. H., Lee, H. J., Lee, H. N., & Cho, H. J. (2009). The development of rubrics to assess scientific argumentation. Journal of The Korean Association For Science Education, 29(2), 203-220.
  37. Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Cambridge, MA: Harvard University Press.
  38. Wachsmuth, H., Al Khatib, K., & Stein, B. (2016, December). Using argument mining to assess the argumentation quality of essays. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics (pp. 1680-1691), Osaka, Japan.
  39. Zohar, A., & Nemet, F. (2002). Fostering students' knowledge and argumentation skills through dilemmas in human genetics. Journal of Research in Science Teaching, 39(1), 35-62. https://doi.org/10.1002/tea.10008