JOURNAL BROWSE
Search
Advanced SearchSearch Tips
A Study on the Utility of Relevance/Non-relevance Information in Homogeneous Documents
facebook(new window)  Pirnt(new window) E-mail(new window) Excel Download
 Title & Authors
A Study on the Utility of Relevance/Non-relevance Information in Homogeneous Documents
Moon, Sung-Been;
  PDF(new window)
 Abstract
This study examined the relative retrieval effectiveness after relevance feedback between two systems (Title/Abstract and Full-text) using four different sets of relevance judgment. Four relevance levels (not relevant, marginally relevant, relevant, highly relevant) are also used, each of which is determined by referees giving a relevance score to documents. This study also investigated how much the average precision was improved after relevance feedback when "marginally relevant" documents are included in the relevant class with the Title/Abstract system, and with the Full-text retrieval system as well. It is found that the Title/Abstract system benefited from relevance feedback with the marginally relevant documents. In case of the Title/Abstract system, the higher percentage of improvement was consistently obtained when including the marginally relevant documents in the relevance class, however the result was vice versa in case of the Full-text retrieval system. It implied that the marginally relevant documents in the relevant class had caused noises in the Full-text retrieval system.
 Keywords
relevance;relevance judgment;relevance feedback;full-text retrieval system;retrieval effectiveness;
 Language
Korean
 Cited by
 References
1.
문성빈 (1993). 적합성피드백을 이용한 전문검색시스템의 효율성 증진을 위한 연구. 정보관리학회지, 10(2), 43-67. (Moon, Sung-Been (1993). Enhancing performance of full-text retrieval systems using relevance feedback. Journal of the Korean Society for Information Management, 10(2), 43-67.)

2.
문성빈 (1997). 상이한 적합성 판정과 전문검색시스템의 평가에 관한 연구. 정보관리학회지, 14(2), 123-141. (Moon, Sung-Been (1997). Variations in relevance assessments and evaluation of the performance of full-text retrieval system. Journal of the Korean Society for Information Management, 14(2), 123-141.)

3.
Amati, G., & Crestani, F. (1999). Probabilistic learning for selective dissemination of information. Information Processing and Management, 35(5), 633-654. crossref(new window)

4.
Belkin, N.J. (1984). Cognitive models and information transfer. Social Science Information Studies, 4, 111-129. crossref(new window)

5.
Belkin, N.J., Cool, C., Kelly, D., Lin, S.J., Park, S.Y., Perez-Carballo, J., & Sikora, C. (2001). Iterative exploration, design and evaluation of support for query reformulation in interactive information retrieval. Information Processing and Management, 37(3), 403-434. crossref(new window)

6.
Blair, D.C., & Maron, M.E. (1990). Full-text information retrieval: further analysis and clarification. Information Processing and Management, 26(3), 437-447. crossref(new window)

7.
Borlund, P. (2003). The concept of relevance in IR. Journal of the American Society for Information Science and Technology, 54(10), 913-925. crossref(new window)

8.
Burgin, R. (1992). Variations in relevance judgements and evaluation of retrieval performance. Information Processing and Management, 28(5), 619-627. crossref(new window)

9.
Dang, E.K.F., Luk, R.W.P., Allan, J., Ho, K.S., Chan, S.C.F., Chung, K.F.L., & Lee, D.L. (2010). A new context-dependent term weight computed by boost and discount using relevance information. Journal of the American Society for Information Science and Technology, 61(12), 2514-2530. crossref(new window)

10.
Eisenberg, M.B. (1988). Measuring relevance judgments. Information Processing and Management, 24(4), 373-389. crossref(new window)

11.
Eisenberg, M.B., & Barry, C. (1988). Order effect: A study of the possible influence of presentation order on user judgments of document relevance. Journal of the American Society for Information Science, 39(1), 37-49.

12.
Greisdorf, H. (2003). Relevance threshold: A multi-stage predictive model of how users evaluate information. Information Processing and Management, 39(3), 403-423. crossref(new window)

13.
Harter, S.P. (1992). Psychological relevance and information science. Journal of the American Society for Information Science, 43(9), 602-615. crossref(new window)

14.
Harter, S.P. (1996). Variation in relevance assessments and the measurement of retrieval effectiveness. Journal of the American Society for Information Science, 47(1), 37-49. crossref(new window)

15.
Hjorland, B. (2010). The Foundation of the concept of relevance. Journal of the American Society for Information Science and Technology, 61(2), 217-237.

16.
Huang, X., & Soergel, D. (2013). Relevance: An improved framework for explicating the notion. Journal of the American Society for Information Science and Technology, 64(1), 18-35. crossref(new window)

17.
Kekalainen, J., & Jarvelin, K. (2002). Using graded relevance assessments in IR evaluation. Journal of the American Society for Information Science and Technology, 53(13), 1120-1129. crossref(new window)

18.
Lopez-Pujalte, C., Guerrero Bote, V. P., & Moya Anegon, F. (2002). A test of genetic algorithms in Relevance Feedback. Information Processing and Management, 38(6), 793-805. crossref(new window)

19.
Lopez-Pujalte, C., Guerrero Bote, V.P., & Moya Anegon, F. (2003). Order-based fitness functions for genetic algorithms applied to relevance feedback. Journal of the American Society for Information Science and Technology, 54(2), 152-160. crossref(new window)

20.
Maglaughlin, K.L., & Sonnenwald, D.H. (2002). User perspectives on relevance criteria: A comparison among relevant, partially relevant, and not relevant. Journal of the American Society for Information Science and Technology, 53(5), 327-342. crossref(new window)

21.
Maron, M.E. (1988). Probabilistic design principles for conventional and full-text retrieval systems. Information Processing and Management, 24(3), 249-255. crossref(new window)

22.
Mizzaro, S. (1997). Relevance: The whole history. Journal of the American Society for Information Science, 48(9), 810-832. crossref(new window)

23.
Quiroga, L.M., & Mostafa, J. (2002). An experiment in building profiles in information filtering: The role of context of user relevance feedback. Information Processing and Management, 38(5), 671-694. crossref(new window)

24.
Salton. G., & Buckley, C. (1990). Improving retrieval performance by relevance feedback. Journal of the American Society for Information Science, 41(4), 288-297. crossref(new window)

25.
Saracevic, T. (1975). Relevance: A review of and a framework for the thinking on the notion in information science. Journal of the American Society for Information Science, 26(6), 321-343. crossref(new window)

26.
Saracevic, T. (2007a). Relevance: A review of the literature and a framework for thinking on the notion in information science. Part II: Nature and manifestations of relevance. Journal of the American Society for Information Science and Technology, 58(13), 1915-1933. crossref(new window)

27.
Saracevic, T. (2007b). Relevance: A review of the literature and a framework for thinking on the notion in information science. Part III: Behavior and effects of relevance. Journal of the American Society for Information Science and Technology, 58(13), 2126-2144. crossref(new window)

28.
Schamber, L. (1994). Relevance and information behavior. Annual Review of Information Science and Technology, 29(1), 3-48.

29.
Schamber, L., Eisenberg, M.B., & Nilan, M.S. (1990). A re-examination of relevance: Toward a dynamic, situational, definition. Information Processing & Management, 26(6), 755-776. crossref(new window)

30.
Shaw, W.M. Jr., Wood, R.E., & Tibbo, H.R. (1991). The cystic fibrosis database: Content and research opportunities. LISR, 13, 347-366.

31.
Sormunen, E. (2002). Liberal relevance criteria of TREC-counting on negligible documents? In M. Beaulieu, R. Baeza-Yates, S. Myaeng, and K. Jarvelin (Eds.), Proceedings of the SIGIR 2002 (pp. 324-330). New York: ACM.

32.
Sormunen, E., Kekalainen, J., Koivisto, J., & Jarvelin, K. (2001). Document text characteristics affect the ranking of the most relevant documents by expanded structured queries. Journal of Documentation, 57(3), 358-376. crossref(new window)

33.
Spink, A., & Greisdorf, H. (2001). Regions and levels: Measuring and mapping users' relevance judgments. Journal of the American Society for Information Science and Technology, 52(2), 161-173. crossref(new window)

34.
Spink, A., & Losee, R.M. (1996). Feedback in information retrieval. Annual Review of Information and Science and Technology, 31, 33-78.

35.
Spink, A., Greisdorf, H., & Bateman, J. (1998). From highly relevant to nonrelevant: Examining different regions of relevance. Information Processing and Management, 34(5), 599-622. crossref(new window)

36.
Swanson, D.R. (1986). Subjective versus objective relevance in bibliographic retrieval system. The Library Quarterly, 56, 389-398. crossref(new window)

37.
Tang, R., Shaw, W.M., & Vevea, J.L. (1999). Towards the identification of the optimal number of relevance categories. Journal of the American Society for Information Science, 50(3), 254-264. crossref(new window)

38.
Vakkari, P., & Sormunen, E. (2004). The influence of relevance levels on the effectiveness of interactive information retrieval. Journal of the American Society for Information Science and Technology, 55(11), 963-969. crossref(new window)

39.
Voorhees, E. (2001). Evaluation by highly relevant documents. In W. Croft, D. Harper, D. Kraft, and J. Hobel (Eds.), Proceedings of the SIGIR 2001 (pp. 74-82). New York: ACM.

40.
Voorhees, E.M., & Harman, D.K. (Eds.) (2005). TREC: Experiment and evaluation in information retrieval. Cambridge: MIT Press.

41.
Xu, Y., & Chen, Z. (2006). Relevance judgment: What do information users consider beyond topicality. Journal of the American Society for Information Science and Technology, 57(7), 961-973. crossref(new window)

42.
Xu, Y., & Wang, D. (2008). Order effect in relevance judgment. Journal of the American Society for Information Science and Technology, 59(8), 1264-1275. crossref(new window)