한국과학교육학회지 (Journal of The Korean Association For Science Education)

  • 제43권1호
  • /
  • Pages.17-28
  • /
  • 2023
  • /
  • 1226-5187(pISSN)
  • /
  • 2288-8489(eISSN)
한국과학교육학회 (The Korean Association for Science Education)
권호 표지
DOI QR코드

DOI QR Code

컴퓨터 기반 평가와 지필평가 간 학생 응답 특성 탐색 -컴퓨터 기반 국가수준 학업성취도 평가 병행 시행 결과를 중심으로-

Exploring Differences of Student Response Characteristics between Computer-Based and Paper-Based Tests: Based on the Results of Computer-Based NAEA and Paper-Based NAEA

  • Jongho Baek (Korea Institute for Curriculum and Evaluation) ;
  • Jaebong Lee (Korea Institute for Curriculum and Evaluation) ;
  • Jaok Ku (Korea Institute for Curriculum and Evaluation)
  • 투고 : 2022.12.26
  • 심사 : 2023.02.16
  • 발행 : 2023.02.28
PDF 다운로드
⟨ 이전 논문 다음 논문 ⟩

초록

디지털 기반 지능 정보 사회로의 진입에 발맞추어 과학 교육과정에서는 과학과의 교과역량 함양을 강조하고 있으며, 역량 평가의 측면에서 컴퓨터 기반 평가가 관심을 받고 있다. 컴퓨터 기반 평가는 높은 실제성을 갖는 형태로 문항을 구현할 수 있고, 평가 결과를 데이터베이스로 축적하여 환류 체계를 구축함에 있어서도 이점이 있다. 다만, 평가 타당도 개선, 측정 효율성 저하, 관리 요소 증가 등의 문제를 해결할 필요가 있다. 본 연구에서는 학업성취도 평가가 지필평가에서 컴퓨터 기반 평가로 전환되는 과정에서 새로운 평가의 도입에 따른 학생들의 반응을 살펴보기 위해 2021년도에 시행된 학업성취도 평가의 지필평가와 컴퓨터 기반 평가의 병행시행 결과를 분석하였다. 특히, 동일한 문항을 평가 매체만을 변화시켰을 때 학생들의 성취에 미치는 영향, 컴퓨터 기반 평가의 장점을 살린 새로운 기능을 포함하여 문항을 구성했을 때의 변화가 학생들의 성취에 미치는 영향을 살펴보았다. 중학교 3학년 학생 7,137명이 지필평가와 2종의 컴퓨터 기반 평가 중 하나에 응시한 결과를 분석하였다. 평가 시행 후 집단별로 문항의 정답률과 변별도 평균을 산출하였으며, 학업성취도 출제 경험이 있는 과학교사 8명이 참여한 전문가 협의회를 통해 응답 특성에 대한 전문가 의견을 수렴하였다. 결과에 따르면 지필평가와 단순 모드 전환형 평가에서의 학생들의 성취 결과는 큰 차이는 없어 매체 효과가 거의 나타나지 않았다. 다만, 서답형 문항의 정답률이 컴퓨터 기반 평가에서 다소 높게 나타났음을 확인하였고, 이는 응답의 편이성과 관련되는 것으로 분석하였다. 한편, 컴퓨터 기반 평가 도입에 따라 새로운 기능들이 적용된 문항들에서 유사한 문항의 정답률과 차이가 10%p 이상인 문항들이 존재하였다. 학생들의 답지 반응을 분석한 바에 따르면, 이는 새로운 기능을 통해 마련한 혁신적인 문항들이 학생들의 이해 수준을 보다 세밀하게 파악한 결과로 볼 수 있었다. 결과를 토대로 컴퓨터 기반 평가를 도입하고 개발할 때 유의할 사항을 논의하고 시사점을 제시하였다.

In line with the entry into the digital-based intelligent information society, the science curriculum emphasizes the cultivation of scientific competencies, and computer-based test (CBT) is drawing attention for assessment of competencies. CBT has advantages to develop items that have high fidelity, and to establish a feedback system by accumulating results into the database. However, it is necessary to solve the problems of improving validity of assessment results, lowering measurement efficiency, and increasing management factors. To examine students' responses to the introduction of the new assessment tools in the process of transitioning from paper-based test (PBT) to CBT, in this study, we analyzed the results of the PBT and the CBT conducted in 2021 National Assessment of Educational Achievement (NAEA). In particular, we sought to find the effects on student achievement when only the mode of assessment was changed without change of items, and the effect on student achievement when the items were composed including technology enhanced features that take advantage of CBT. This study is derived through the analysis of the results of 7,137 third-grade middle school students taking one among the three kinds of assessments, which were the PBT or two kinds of CBT. After the assessment, the percentage of correct answers and the item discriminations were collected for each group, and expert opinions on characteristics of response were collected through the expert council involving 8 science teachers with experience in NAEA. According to the results, there was no significant difference between students' achievement results in the PBT and the CBT-M, which means simple mode conversion type of CBT, so it could be explained that the mode effect did not appear. However, it was confirmed that the percentage of correct answers for the construct response items was somewhat high in the CBT, and this result was analyzed to be related to the convenience of the response. On the other hand, there were the items with a difference of more than 10%p from the correct answer rate of similar items, among the items to which technology enhanced functions were applied following the introduction of CBT. According to the analysis of response rate of options, these results could be explained that the students' level of understanding could be more closely grasped through the innovative items developed through the technology enhanced function. Based on the results, we discussed some guidance to be considered when introducing CBT and developing items through CBT, and presented implications.

키워드

참고문헌

  1. ACARA(Australian Curriculum, Assessment and Reporting Authority) (2017). NAP sample assessment science literacy 2015: Technical report. Retrieved December 04, 2022, from https://www.nap.edu.au/docs/default-source/default-document-library/20170317-nap-sl-tech-report.pdf?sfvrsn=c0316d5e_4.
  2. ACARA (2018). National Assessment Program - Literacy and Numeracy (NAPLAN) 2018: Technical report. Australian Curriculum, Assessment and Reporting Authority. Retrieved May 29, 2020, from https://www.nap.edu.au/docs/default-source/default-document-library/2018_naplan_technical_report_full_v1.pdf?sfvrsn=0.
  3. ACER(Australian Council for Educational Research) (2016). NAPLAN online readability and layout study - literature review (p. 74).
  4. Ban, J., Kim, S., Kim, S. J., Lee, S., Kwak, C., Lee, J., Park, H., Jeon, M., Shin, D., Kim, M., Jeon, J., Yeom, J., & Kim, J.(2021). 2020 research service report for developing and using of contents for 2020 diagnosis-supplement system for basic academic skills. Chungnam National University Applied Measurement and Evaluation Center.
  5. Ban, J., Kim, S., Kim, M., Kim, S. J., Kim, J., Lee, S., Lee, Y., Lee, J., Yeom, J., Jeon, M., Lee, Y. J., & Jin, J.(2022). 2021 research service report for developing and using of contents for 2021 diagnosis-supplement system for basic academic skills. Chungnam National University Applied Measurement and Evaluation Center.
  6. Bennett, R. E., Ward, W. C., Rock, D. A., & LaHart, C. (1990). Toward a framework for constructed-response items. ETS Research Report Series(RR-90-7). Retrieved May, 27, 2020, from https://onlinelibrary.wiley.com/doi/epdf/10.1002/j.2333-8504.1990.tb01348.x.
  7. Bryant, W. (2017). Developing a strategy for using technology-enhanced items in large-scale standardized tests. Practical Assessment, Research, and Evaluation, 22(1), 1-10.
  8. Choi, H. (2013a). The development of science item on the computer-based performance assessment: A experiment on constructing circuits with an ammeter and a voltmeter. Journal of Science Education, 37(2), 348-358. https://doi.org/10.21796/JSE.2013.37.2.348
  9. Choi, H. (2013b). Developing science items of computer-based problem solving assessment. Korean Journal of Teacher Education, 29(4), 255-276. https://doi.org/10.14333/KJTE.2013.29.4.255
  10. Cotter, K. E., Centurino, V. A. S., & Mullis, I. V. S. (2020). Developing the TIMSS 2019 mathematics and science achievement instruments. In M. O. Martin, M. von Davier, & I. V. S. Mullis (Eds.), Methods and Procedures: TIMSS 2019 Technical Report (pp. 1.1-1.36). Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Lynch School of Education and Human Development, Boston College and International Association for the Evaluation of Educational Achievement (IEA).
  11. de Klerk, S. (2012). An overview of innovative computer-based testing. In T. J. H. M. Eggen, & B. P. Veldkamp (Eds.), Psychometrics in Practice at RCEC (pp. 137-150). Retrieved May 27, from https://ris.utwente.nl/ws/portalfiles/portal/5579570/Chapter_12.pdf.
  12. Fishbein, B., Martin, M. O., Mullis, I. V. S., & Foy, P. (2018). The TIMSS 2019 item equivalence study: Examining mode effects for computer-based assessment and implications for measuring trends. Large-scale Assessments in Education, 6(1), 1-23. https://doi.org/10.1186/s40536-018-0054-1
  13. Jewsbury, P., Finnegan, R., Xi, N., Jia, Y., Rust, K., Burg, S., Donahue, P., Mazzeo, J., Cramer, B., Lin, A., & Weil, N. (2020). 2017 NAEP transition to digitally based assessments in mathematics and reading at grades 4 and 8: Mode evaluation study. Retrieved January 11, 2021, from https://nces.ed.gov/nationsreportcard/subject/publications/main2020/pdf/transitional_whitepaper.pdf.
  14. Hardcastle, J., Herrmann-Abell, C. F., & DeBoer, G. E. (2017). Comparing student performance on paper-and-pencil and computer-based tests. Paper presented at the 2017 AERA Annual Meeting, San Antonio, TX. Retrieved January 14, 2021, from https://files.eric.ed.gov/fulltext/ED574099.pdf.
  15. Kim, H. K. (2021). The study of the mode effect between computer-based and paper-based science tests in TIMSS 2019. Journal of the Korean Chemical Society, 65(1), 48-57.
  16. Kim, K., Kim, D., Kim, W., Kim, M., Kim, J., Shin, D., Choi, I., Lee, I., Park, I., Choi, I., Shin, J., Han, J., Song, M., Han, K., Kim, H., & Park, G. (2013). Introduced plans for computer based national assessment of educational achievement (CRE 2013-5). Korea Institute for Curriculum and Evaluation.
  17. Kim, M. (2000). Strategies on screen design of learner-centered web-based instructional systems. Journal of Educational Technology, 16(4), 51-76. https://doi.org/10.17232/KSET.16.4.51
  18. Kozma, R. (2009). Transforming education: Assessing and teaching 21st century skills - Assessment call to action. In F. Scheuermann, & J. Bjornsson (Eds.), The Transition to Computer-Based Assessment (pp. 13-23). Luxembourg: Office for Official Publications of the European Communities.
  19. Lee, M., Dong, H., Park, I., Kim, W., Seo, M., Jung, H., Kim, K., Kang, M., Jang, E., Sung, K., Rim, H., Kim, S., Pae, J., Kim, S. Y., Lee, J., Park, J., Yang, K., Kang, T., Shin, Y, & Park, Y. (2017). A study on the enhancement of the NAEA system incorporating the aims of the revised 2015 national curriculum (CRE 2017-8). Korea Institute for Curriculum and Evaluation.
  20. Lee, C., Kim, H., Sang, K., & Choi, J. (2018). Comparison of item characteristics between computer-based assessment and paper-based assessment-focusing on mathematics and science subjects (ORM 2018-39-22). Korea Institute for Curriculum and Evaluation.
  21. Lee, J., Kim, J., Park, J., Sung, K., Lee, K., Lee, S., Jung, H., Choi, S., Kim, K., Ahn, Y., & Ha, M. (2020). Introducing the computer-based test for the national assessment of educational achievement: A developmental study for the eNAEA (RRE 2020-5). Korea Institute for Curriculum and Evaluation.
  22. Lee, J., Kim, Y., Kim, J., Nam, M., Park, J., Park, J. H., Sung, K., Lee, S., Jang, E., & Jung, H., (2021). Advancing the test tools for electronic national assessment of educational achievement (eNAEA) (RRE 2021-2). Korea Institute for Curriculum and Evaluation.
  23. McMillan, J. H. (2014). Classroom assessment: Principles and practice for effective standards-based instruction (6th Ed.). Boston, MA: Pearson.
  24. Ministry of Education(MOE) (2015). 2015 revised science curriculum. Ministry of Education 2015-74 [issue 9].
  25. Parshall, C. G., Davey, T., & Pashley, P. J. (2000). Innovative item types for computerized testing. In W. J. van der Linden & C. A. W. Glas (Eds.), Computerized Adaptive Testing: Theory and Practice (pp. 129-148). Netherlands: Kluwer Academic Publishers.
  26. Parshall, C. G., Harmes, J. C., Davey, T., & Pashley, p.J. (2010). Innovative items for computerized testing. In W. J. van der Linden & C. A. W. Glas(Eds.), Elements of Adaptive Testing(pp. 215-230). New York: Springer.
  27. Pommerich, M. (2004). Developing computerized versions of paper-and-pencil tests: Mode effects for passage-based Tests. The Journal of Technology, Learning, and Assessment, 2(6), 1-45.
  28. Ripley, M. (2009). Transformational computer-based testing. In F. Scheuermann, & J. Bjornsson (Eds.), The Transition to Computer-Based Assessment (pp. 85-91). Luxembourg: Office for Official Publications of the European Communities.
  29. Scalise, K., & Gifford, B. (2006). Computer-based assessment in E-Learning: A framework for constructing "intermediate constraint" questions and tasks for technology platforms. The Journal of Technology, Learning, and Assessment, 4(6). Retrieved March 23, 2020, from https://ejournals.bc.edu/index.php/jtla/article/view/1653/1495.
  30. Sireci, S. G., & Zenisky, A. L. (2006). Innovative item formats in computer-based testing: In pursuit of improved construct representation. In S. M. Downing & T. M. Haladyna (Eds.), Handbook of Test Development (pp. 329-347). Mahwah, NJ: Lawrence Erlbaum Associates Publishers.
  31. Song, M., Kim, S., Ku, J., Rim, H., Park, H., Han, J., Son, S., & Yang, S. (2014). OECD programme for international students assessment: PISA 2015 field trial report (RRE 2014-4-3). Korea Institute for Curriculum and Evaluation.
  32. Strain-Seymour, E., Way, W. D., & Dolan, R. P. (2009). Strategies and processes for developing innovative items in large-scale assessments. pearson education, Inc. Retrieved March 29, 2020, from https://images.pearsonassessments.com/images/tmrs/StrategiesandProcessesforDevelopingInnovativeItems.pdf.
  33. Zenisky, A. L., & Sireci, S. G. (2002). Technological innovations in large-scale assessment. Applied Measurement in Education, 15(4), 337-362. https://doi.org/10.1207/S15324818AME1504_02