Advanced SearchSearch Tips
Has Retrieval Technology in Vertical Site Search Systems Improved over the Years? A Holistic Evaluation for Real Web Systems
facebook(new window)  Pirnt(new window) E-mail(new window) Excel Download
 Title & Authors
Has Retrieval Technology in Vertical Site Search Systems Improved over the Years? A Holistic Evaluation for Real Web Systems
Mandl, Thomas; Womser-Hacker, Christa; Gatzke, Natalia;
  PDF(new window)
Evaluation of retrieval systems is mostly limited to laboratory settings and rarely considers changes of performance over time. This article presents an evaluation of retrieval systems for internal Web site search systems between the years 2006 and 2011. A holistic evaluation methodology for real Web sites was developed which includes tests for functionality, search quality, and user interaction. Among other sites, one set of 20 Web site search systems was evaluated three times in different years and no substantial improvement could be shown. It is surprising that the communication between site and user still leads to very poor results in many cases. Overall, the quality of these search systems could be improved, and several areas for improvement are apparent from our evaluation. For a comparison, Google’s site search function was also tested with the same tasks.
Site Search;Information Retrieval;Evaluation;
 Cited by
Adar, E., Teevan, J., & Dumais, S. T. (2008). Large scale analysis of web revisitation patterns. Proc. ACM Conf. on Human Factors in Computing Systems (CHI) (pp. 1197-1206). ACM Press: New York.

Armstrong, T. G., Alistair, M., Webber, W., & Zobel, J. (2009). Improvements that don't add up: Ad-hoc retrieval results since 1998. Conference on Information and Knowledge Management (CIKM) (pp. 601-610). ACM Press: New York.

Armstrong, T. G., Moffat, A., Webber, W., & Zobel, J. (2009). Has Adhoc retrieval improved since 1994? Proc. Annual Intl. SIGIR Conf., Jul 19-23. ACM Press: New York.

Borlund, P. (2013). Interactive information retrieval: An introduction. Journal of Information Science Theory and Practice (JISTAP), 1(3), 12-32.

Braschler, M., Herget, J., Pfister, J., Schäuble, P., Steinbach, M., & Stuker, J. (2006). Evaluation der Suchfunktion von Schweizer Unternehmens-Websites. Churer Schriften zur Informationswissenschaft. Switzerland: HTW Chur. Retrieved from

Braschler, M., Heuwing, B., Mandl, T., Womser-Hacker, C., Herget, J., Schäuble, P., & Stuker, J. (2007). Evaluation der Suchfunktion deutscher Unternehmens-Websites. Proceedings Wissensorganisation 09: "Wissen - Wissenschaft - Organisation" 12. Tagung der Deutschen ISKO (International Society for Knowledge Organization), Bonn, Oct. 19-21.

Cronbach, L. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297-334. crossref(new window)

Furtner, K., Mandl, T., & Womser-Hacker, C. (2015). Effects of auto-suggest on the usability of search in eCommerce. Proc. 14th International Symposium on Information Science (ISI 2015), Zadar, Croatia, May 2015 (pp. 178-190). Glückstadt.

Gätzke, N. (2011). Verbessert sich die Suchfunktion auf Internetseiten im Laufe der Zeit? Eine diachrone Analyse der Qualität von Sitesuche auf deutschen Unternehmens-Webseiten (Bachelors thesis). University of Hildesheim.

Jansen, B., Spink, A., & Saracevic, T. (2000). Real life, real users, and real needs: A study and analysis of user queries on the web. Information Processing and Management, 36, 207-227. crossref(new window)

Järvelin, K. (2009). Explaining user performance in information retrieval: Challenges to IR evaluation. Proceedings of the 2nd International Conference on the Theory of Information Retrieval, 2009 (pp. 289-296). Heidelberg: Springer, Lecture Notes in Computer Science vol. 5766.

Kelly, D. (2009). Methods for evaluating interactive information retrieval systems with users. Foundations and Trends in Information Retrieval, 3(1-2), 1-224.

Kemp, C., & Ramamohanarao, K. (2002). Long-term learning for Web search engines. In Principles and practice of knowledge discovery in databases (PKDD) [LNAI 2431] (pp. 263-274). Springer: Berlin Heidelberg. pp. 263-274.

Mandl, T., & Womser-Hacker, C. (2015). Information retrieval. In Encyclopedia of information science and technology (3rd ed.) (pp. 3923-3931). Hershey, PA: Idea Group Reference.

Robertson, S. (2008). On the history of evaluation in IR. Journal of Information Science, 34(4), 439-456. crossref(new window)

Shaikh, D. A., & Lenz, K. (2006). Where’s the search? Re-examining user expectations of Web objects. Usability News.

Uhl, J. (2010). Information Retrieval-Studie zur Evaluierung von Site-Search-Systemen (Masters thesis). University of Hildesheim.

Wittenberg, R. (1998). Grundlagen computerunterstützter Datenanalyse (2nd ed.). Stuttgart: Lucius und Lucius.

Zobel, J., Williams, H. E., & Kimberley, S. (2000). Trends in retrieval system performance. 23rd Australasian Computer Science Conference (ACSC), Jan 31 - Feb 3, Canberra, Australia (pp. 241-249).