Journal of The Korean Association For Science Education (한국과학교육학회지)

The Korean Association for Science Education (한국과학교육학회)
권호 표지
DOI QR코드

DOI QR Code

Exploring the Factors Influencing on the Accuracy of Self-Reported Responses in Affective Assessment of Science

과학과 자기보고식 정의적 영역 평가의 정확성에 영향을 주는 요소 탐색

  • Received : 2019.02.13
  • Accepted : 2019.05.05
  • Published : 2019.06.30
Download PDF
⟨ Previous Next ⟩

Abstract

This study reveals the aspects of subjectivity in the test results in a science-specific aspect when assessing science-related affective characteristic through self-report items. The science-specific response was defined as the response that appear due to student's recognition of nature or characteristics of science when his or her concepts or perceptions about science were attempted to measure. We have searched for cases where science-specific responses especially interfere with the measurement objective or accurate self-reports. The results of the error due to the science-specific factors were derived from the quantitative data of 649 students in the 1st and 2nd grade of high school and the qualitative data of 44 students interviewed. The perspective of science and the characteristics of science that students internalize from everyday life and science learning experiences interact with the items that form the test tool. As a result, it was found that there were obstacles to accurate self-report in three aspects: characteristics of science, personal science experience, and science in tool. In terms of the characteristic of science in relation to the essential aspect of science, students respond to items regardless of the measuring constructs, because of their views and perceived characteristics of science based on subjective recognition. The personal science experience factor representing the learner side consists of student's science motivation, interaction with science experience, and perception of science and life. Finally, from the instrumental point of view, science in tool leads to terminological confusion due to the uncertainty of science concepts and results in a distance from accurate self-report eventually. Implications from the results of the study are as follows: review of inclusion of science-specific factors, precaution to clarify the concept of measurement, check of science specificity factors at the development stage, and efforts to cross the boundaries between everyday science and school science.

이 연구는 자기보고식 검사를 통해 과학 관련 정의적 영역을 평가하려할 때 검사 결과에서 나타나는 주관성의 양상을 과학 특이적 측면에서 밝혔다. 과학 관련 개념이나 인식을 측정하려할 때 학생이 지닌 과학 특성, 본성에 대한 인식이 원인이 되어 나타나는 반응을 과학 특이적 반응으로 정의했다. 그 중에서 과학 특이적 반응이 특별히 측정 구인을 방해하거나 정확한 자기 보고를 벗어나게 하는 경우에 대해 탐색했다. 고등학교 1, 2학년 649명의 정의적 특성 및 심리적 특성을 검사한 양적 자료와 학생 44명을 면담한 질적 자료로부터 과학 특이적 요소로 인한 오차 결과를 도출했다. 학생이 일상과 과학 학습 경험으로부터 내면화한 과학에 대한 관점과 과학 특성은 검사 도구를 이루는 문항들과 상호작용한다. 그 결과 과학의 특성, 개인의 과학 경험, 검사 도구 속 과학이라는 세 측면에서 정확한 자기 보고를 방해하는 요소가 발견되었다. 과학 본질적 측면과 관련 있는 과학의 특성은 학생들이 과학을 보는 관점과 주관적으로 인식한 과학의 특성이 측정하려는 구인에 관계없이 문항에 반응하도록 한다. 학습자 측면에서 개인의 과학 경험은 학생이 지닌 과학 동기, 과학 경험과의 상호작용, 과학과 삶에 대한 인식으로 구성된다. 마지막으로 도구적 측면에서 검사 도구 속 과학은 과학 개념의 불명확성으로 인한 용어 혼동으로 연결되며 정확한 자기보고를 방해할 수 있다. 본 연구 결과에 의한 시사점으로 검사 문항에서 과학 특이적 요소의 포함 여부 검토, 측정 개념을 명확히 하기 위한 주의점, 개발 단계에서의 과학 특이성 요소 검토, 일상 과학과 학교 과학의 괴리를 줄이려는 노력 필요 등을 제안했다.

Keywords

GHGOBX_2019_v39n3_363_f0001.png 이미지

Figure 1. Science-specific factors that interfere with accurate self-report

Table 1. Research participants

GHGOBX_2019_v39n3_363_t0001.png 이미지

Table 2. Analysis materials

GHGOBX_2019_v39n3_363_t0002.png 이미지

Table 3. Contents of questionnaire

GHGOBX_2019_v39n3_363_t0003.png 이미지

Table 4. Contents of questions for semi-structured interview

GHGOBX_2019_v39n3_363_t0004.png 이미지

Table 5. Number of scenes, concepts, categories derived from students’ interview

GHGOBX_2019_v39n3_363_t0005.png 이미지

Table 6. Categories and concepts derived from student interviews

GHGOBX_2019_v39n3_363_t0006.png 이미지

Table 7. Students’ perception toward science

GHGOBX_2019_v39n3_363_t0007.png 이미지

Table 8. Students’ negative or intermediate response to the ‘general value of science’ items with high affective achievement

GHGOBX_2019_v39n3_363_t0008.png 이미지

Table 9. Item fit analysis on ‘general value of science’ by Rasch model

GHGOBX_2019_v39n3_363_t0009.png 이미지

Table 10. Some cases of students’ responses with ‘personal science experience’

GHGOBX_2019_v39n3_363_t0010.png 이미지

Table 11. Distribution of students in career course by level of instrumental motivation score in science

GHGOBX_2019_v39n3_363_t0011.png 이미지

Table 12. Group differences of affective⋅cognitive achievement and psychological scale according to level of instrumental motivation score in science

GHGOBX_2019_v39n3_363_t0012.png 이미지

Table 13. Cases of students’ responses with ‘science in test tool’

GHGOBX_2019_v39n3_363_t0013.png 이미지

Table 14. Distribution of students related to specific science area or subjects in science-related affective test

GHGOBX_2019_v39n3_363_t0014.png 이미지

Table 15. Group differences by specific science area related to groups in affective achievement

GHGOBX_2019_v39n3_363_t0015.png 이미지

Table 16. Inter-item consistency by subconstructs in specific science area related to groups

GHGOBX_2019_v39n3_363_t0016.png 이미지

References

  1. Abd-El-Khalick, F., Summers, R., Said, Z., Wang, S., & Culbertson, M. (2015). Development and large-scale validation of an instrument to assess Arabic-speaking students' attitudes toward science. International Journal of Science Education, 37(16), 2637-2663. https://doi.org/10.1080/09500693.2015.1098789
  2. Anastasi, A. (1988). Psychological testing(6th ed.). New York: Macmillan & Co.
  3. Au, Y. (2007). A search on social desirability according to administered mode and demonstrable condition of a psychology testing. Journal of Educational Evaluation, 20(4), 235-258.
  4. Bae, B., Lee, D., & Ham, K. (2015). Validation of the Korean short-version of social desirability scale(SDS-9) using the Rasch model. Korean Journal of Counseling, 16(6), 177-197.
  5. Brunetti, D., Schlottmann, R., Scott, A., & Hollrah, J. (1998). Instructed faking and MMPI-2 response latencies: The potential for assessing response validity. Journal of Clinical Psychology, 54, 143-153. https://doi.org/10.1002/(SICI)1097-4679(199802)54:2<143::AID-JCLP3>3.0.CO;2-T
  6. Choi, J., Hwang, S., Pai, D., Hwang, S. T., & Kim, Y. (2015). Diagnostic efficiency of personality disorder screening tool; The Korean version of self-report standardized assessment of personality-abbreviated scale: preliminary validation study. Journal of the Korean Neuropsychiatric Association, 54(4), 534-541. https://doi.org/10.4306/jknpa.2015.54.4.534
  7. Chun, E., Na, J., Joung, Y., & Song, J. (2015). Development and application of the measuring instrument for the analysis of science classroom culture from the perspective of ‘community of practice'. Journal of the Korean Association for Science Education, 35(1), 131-142. https://doi.org/10.14697/jkase.2015.35.1.0131
  8. Chung, S., & Shin, D. (2016). Trends of assessment research in science education, Journal of the Korean Association for Science Education, 36(4), 563-579. https://doi.org/10.14697/jkase.2016.36.4.0563
  9. Chung, S., & Shin, D. (2017). Cases of discrepancy in high school students' achievement in science education assessment: Focusing on testing tool in affective area. Journal of the Korean Association for Science Education, 37(5), 891-909. https://doi.org/10.14697/jkase.2017.37.5.891
  10. Cronbach, L. (1946). Response sets and test validity. Educational and Psychological Measurement, 6(3), 475-494. https://doi.org/10.1177/001316444600600405
  11. Crowne, D., & Marlowe, D. (1960). A new scale of social desirability independent of psychopathology. Journal of Consulting Psychology, 24, 349-354. https://doi.org/10.1037/h0047358
  12. Ferrando, P., & Chico, E. (2001). Detecting dissimulation in personality test scores: A comparison between person-fit indices and detection scales. Educational and Psychological Measurement, 61, 997-1012. https://doi.org/10.1177/00131640121971617
  13. Fives, H., Huebner, W., Birnbaum, A., & Nicolich, M. (2014). Developing a measure of scientific literacy for middle school students. Science Education, 98(4), 549-580. https://doi.org/10.1002/sce.21115
  14. George, M., & Skinner, H. (1990). Using response latency to detecting accurate response in a computerized lifestyle assessment. Computers in Human Behavior, 6, 167-175. https://doi.org/10.1016/0747-5632(90)90004-Z
  15. Holden, R., & Hibbs, N. (1995). International validity of response latencies for detecting fakers on a personality test. Journal of Research in Personality, 29, 362-372. https://doi.org/10.1006/jrpe.1995.1021
  16. Holden, R., & Kroner, D. (1992). Relative efficacy of differential response latencies for detecting faking on a self-report measure of psychopathology. Psychological Assessment, 4, 170-173. https://doi.org/10.1037/1040-3590.4.2.170
  17. Holden, R., Kroner, D., Fekken, G., & Popham, S. (1992). A model of personality test item response dissimulation. Journal of Personality and Social Psychology, 63, 272-279. https://doi.org/10.1037/0022-3514.63.2.272
  18. Hong, S. (1994). Interaction between science and technology: Technology as knowledge and science as practice. The Quarterly Changbi, 22(4), 329-350.
  19. Hough, L., Eaton, N., Dunnette, M., Kamp, J., & McCloy, R. (1990). Criterion-related validity of personality constructs and the effect of response distortion on those validities. Journal of Applied Psychology, 75, 581-595. https://doi.org/10.1037/0021-9010.75.5.581
  20. Hsu, L., Sanetelli, J., & Hsu, J. (1989). Faking detection validity and incremental validity of response latencies to MMPI subtle and obvious items. Journal of Personality Assessment, 53, 278-295. https://doi.org/10.1207/s15327752jpa5302_6
  21. Hui, C. H., & Triandis, H. C. (1985). The instability of response sets. Public Opinion Quarterly, 49, 253-260. https://doi.org/10.1086/268918
  22. Joo, Y., Kim, Y., Jeong, S., Shin, M., & Lee, C. (2001). Relationships between subjective symptoms and objective psychopathology in patients with schizophrenia. Journal of the Korean Neuropsychiatric Association, 40(4), 667-678.
  23. Kim, S., & Kim, H. (2016). Development of a science ethicality test for elementary school students. Journal of the Korean Association for Science Education, 36(1), 1-13. https://doi.org/10.14697/jkase.2016.36.1.0001
  24. Kim, H., & Lee, S. (1996). Secondary students' attitudes toward science-technology related issues in Korea. Journal of the Korean Association for Science Education, 16(4), 461-469.
  25. Kim, M., & Lee, H. (2006). A study of faking on normative and ipsative measures of personality for personnel selection. The Korean Journal of Industrial and Organizational Psychology, 19(3), 371-393.
  26. Kim, J., Jeong, H., Kim, Y., & Cho, Y. (2015). A study on application of NCS recruiting systems in public organization - based on NCS recruiting performance during the first half year in 2015. Journal of Skills and Qualifications, 4(1), 65-84.
  27. Kim, Y., Park, Y., Park, H., Shin, D., Jung, J., & Song, S. (2014). World of science education. Seoul: Book's hill.
  28. Kluger, A. Reilly, R., & Russell, C. (1991). Faking biodata tests: Are option-keyed instruments more resistant? Journal of Applied Psychology, 76, 889-896. https://doi.org/10.1037/0021-9010.76.6.889
  29. Lederman, N., Abd-El-Khalick, F., Bell, R., & Schwartz, R. (2002). Views of NOS questionnaire toward valid and meaningful assessment of learners' conceptions of NOS. Journal of Research in Science Teaching, 39, 497-521. https://doi.org/10.1002/tea.10034
  30. Lee, J., & Moon, S. (2015). An analysis on the university entrance examination system's change process and main contents of occasional-regular admissions. Comtemporary Educational Research, 27, 97-130.
  31. Lee, M., Sohn, W., & No, U. (2007). The Results from PISA 2006. Seoul: Korea Institute for Curriculum and Evaluation.
  32. Linacre, J. (2005). A user's guide to Winsteps Rasch-model computer programs. Retrieved from www.winsteps.com.
  33. London, M. (1997). London's career motivation theory: An update on measurement and research. Journal of Career Assessment, 5(1), 61-80. https://doi.org/10.1177/106907279700500105
  34. Markus, H. (1977). Self-schemata and processing information about the self. Journal of Personality and Social Psychology, 35, 63-78. https://doi.org/10.1037/0022-3514.35.2.63
  35. McDaniel, M., & Timm, H. (1990). Lying takes time: Predicting deception in biodata using response latency. Paper presented at the American Psychological Association Annual Convention, Boston.
  36. Messick, S. (1991). Psychology and methodology of response styles. In R. E. Snow, & D. E. Willey (Eds.), Improving inquires in social science(pp. 161-200). Hillsdale, NJ: Erlbaum.
  37. Messick, S. (1995). Standards of validity and the validity of standards in performance assessment. Educational Measurement: Issues and Practice, 14(4), 5-8. https://doi.org/10.1111/j.1745-3992.1995.tb00881.x
  38. Munsterberg, H. (1908). On the witness stand: Essays on psychology and crime. New York: Doubleday, Page & Co.
  39. National Science Teachers Association(NSTA) (2000). NSTA position statement of the nature of science. Retrieved July 12 2003, from http://www.nsta.org/159&psid=22
  40. Noe, R., & Ford, J. (1992). Emerging issues and new directions for training research. In G. Ferris, & K. Rowland (Eds.), Research in personnel and human resources management(pp. 345-384). Greenwich. CT: JAI Press.
  41. Osborne, J., Simon, S., & Tytler, R. (2009). Attitudes towards science: An update. Paper presented at the annual meeting of the American Educational Research Association, San Diego, California.
  42. Paulhus, D. (1984). Two component models of socially desirable responding. Journal of Personality and Social Psychology, 46, 589-609. https://doi.org/10.1037/0022-3514.46.3.598
  43. Paulhus, D. L. (2002). Socially desirable responding: The evolution of a construct. In H. I. Braum, D. N. Jackson, & D. E. Wiley (Eds.), The role of constructs in psychological and educational measurement(pp. 49-69). Mahwah, New Jersey: Lawrence Erbaum Associates.
  44. Pickering, A. (1992). Science as practice and culture. Chicago: The University of Chicago.
  45. Ratcliffe, M., & Grace, M. (2003). Science education for citizenship: teaching socio-scientific issues. Maidenhead, UK: McGraw-Hill International.
  46. Reise, S. P., & Flannery, W. P. (1996). Assessing person-fit on measures of typical performance. Applied Measurement in Education, 9(1), 9-26. https://doi.org/10.1207/s15324818ame0901_3
  47. Rust, J., & Golombok, S. (2014). Modern psychometrics: The science of psychological assessment(3rd). London: Routledge.
  48. Scheuneman, J. D. (1984). A theoretical framework for the exploration of causes and effects of bias in testing. Educational Psychologist, 19(4), 219-225. https://doi.org/10.1080/00461528409529298
  49. Schluf, Boaz, Hattie, J., & Dixon, R. (2008). Factors affecting responses to Likert type questionnaires: Introduction of the ImpExp, a new comprehensive model. Social Psychology of Education, 11(1), 59-78. https://doi.org/10.1007/s11218-007-9035-x
  50. Schwartz, R., Lederman, N., & Crawford, B. (2004). Developing views of nature of science in authentic context: An explicit approach of bridging the gap between nature of science and scientific inquiry. Science Education, 88, 610-645. https://doi.org/10.1002/sce.10128
  51. Seol, H., Kim, D., & Lee, S. (2006). Validation of the emotional empathy scale using Rasch rating scale model. Journal of Education Evaluation, 19(2), 179-201.
  52. Shin, S. Ha, M., & Lee, J. (2014). Difference analysis between groups and the generalizability of the instrument for measuring high school students attitude toward convergence. Journal of Learner-Centered Curriculum and Instruction, 14(5), 107-124.
  53. Shin, S., Ha, M., & Lee, J. (2016). The development and validation of instrument for measuring high school students' STEM career motivation. Journal of the Korean Association for Science Education, 36(1), 75-86. https://doi.org/10.14697/jkase.2016.36.1.0075
  54. Shin, Y., Kwak, Y., Kim, H., Lee, S., Lee, S. H., & Kang, H. (2017). Study on the development of test for indicators of positive experiences about science. Journal of the Korean Association for Science Education, 37(2), 335-346. https://doi.org/10.14697/jkase.2017.37.2.0335
  55. Son, E., Cha, J., & Kim, A. (2007). Test of construct equivalence of personality inventory in low and high socially desirable responding groups. The Korean Journal of Social and Personality Psychology, 21(2), 71-87.
  56. Stöber, J. (2001). The Social Desirability Scale-17 (SDS-17): Convergent validity, discriminant validity, and relationship with age. European Journal of Psychological Assessment, 17(3), 222-232. https://doi.org/10.1027//1015-5759.17.3.222
  57. Stricker, L. J. (1963). Acquiescence and social desirability response styles: Item characteristics, and conformity. Psychological Reports, 12, 319-341. https://doi.org/10.2466/pr0.1963.12.2.319
  58. Sweeney, P., & Moreland, R. (1980). Self-schemas and the perseverance of beliefs about the self. Paper presented at the meeting of American Psychological Association, Montreal.
  59. Vasilopoulos, N., Reilly, R., & Leaman, J. (2000). The influence of job familiarity and impression management on self-report measure scale scores and response latencies. Journal of Applied Psychology, 85, 50-64. https://doi.org/10.1037/0021-9010.85.1.50
  60. Yoo, D., Lee, J., & Kim, H. (2012). A study on the comparative analysis of the historical transformation process on employment pattern and characteristics by the period of Korea major enterprise. The Korean Academy of Business History, 27(4), 33-58.
  61. Zerbe, W. J., & Paulhus, D. L. (1987). Socially desirable responding in organizational behavior: A reconception. Academy of Management Review, 12, 250-264. https://doi.org/10.5465/amr.1987.4307820
  62. Zickar, M., & Drasgow, F. (1996). Detecting faking on a personality instrument using appropriateness measurement. Applied Psychological Measurement, 20, 71-87. https://doi.org/10.1177/014662169602000107