Meta Analysis of Usability Experimental Research Using New Bi-Clustering Algorithm

Kim, Kyung-A;Hwang, Won-Il

  • Published : 2008.12.31


Usability evaluation(UE) experiments are conducted to provide UE practitioners with guidelines for better outcomes. In UE research, significant quantities of empirical results have been accumulated in the past decades. While those results have been anticipated to integrate for producing generalized guidelines, traditional meta-analysis has limitations to combine UE empirical results that often show considerable heterogeneity. In this study, a new data mining method called weighted bi-clustering(WBC) was proposed to partition heterogeneous studies into homogeneous subsets. We applied the WBC to UE empirical results and identified two homogeneous subsets, each of which can be meta-analyzed. In addition, interactions between experimental conditions and UE methods were hypothesized based on the resulting partition and some interactions were confirmed via statistical tests.


Data mining;meta-analysis;clustering;usability evaluation


  1. Madeira, S. C. and Oliveira, A. L. (2004). Biclustering algorithms for biological data analysis: A survey, IEEE Transactions on Computational Biology and Bioinformatics, 1, 24-45
  2. Nielsen, J. and Molich, R. (1990). Heuristic evaluation of user interface, In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Empowering People, 249-256
  3. Lazzeroni, L. and Owen, A. (2000). Plaid models for gene expression data, Technical Report, Stanford University
  4. Lewis, C. (1982). Using the `thinking-aloud' method in cognitive interface design, Research Report RC9265, IBM T. J. Watson Research Center, New York
  5. Lewis, J. R. (2001). Introduction: Current issues in usability evaluation, International Journal of Human-Computer Interaction, 13, 343-349
  6. Kluger, Y., Basri, R., Chang, J. T. and Gerstein, M. (2003). Spectral biclustering of microarray data: Co- clustering genes and conditions, Genome Research, 13, 703-716
  7. Tang, C., Zhang, L., Zhang, A. and Ramanathan, M. (2001). Interrelated two-way clustering: An unsupervised approach for gene expression data analysis, In Proceedings of Second IEEE International Symposium on Bioinformatics and Bioengineering, 41-48
  8. Yang, J., Wang, H., Wang, W. and Yu, P. (2003). Enhanced bi-clustering on expression data, In Proceedings of Third IEEE Conference on Bioinformatics and Bioengineering, 321-327
  9. Polson, P. G., Lewis, C., Rieman, J. and Wharton, C. (1992). Cognitive walkthroughs: A method for theory- based evaluation of user interfaces, International Journal of Man-Machine Studies, 36, 741-773
  10. Hertzum, M. and Jacobsen, N. E. (2001). The evaluator effect: A chilling fact about usability evaluation methods, International Journal of Human-Computer Interaction, 13, 421-443
  11. Johnson, R. A. and Wichern, D. W. (2002). Applied Multivariate Statistical Analysis, 5th ed., Prentice-Hall, New Jersey
  12. Hartigan, J. A. (1972). Direct clustering of a data matrix, Journal of the American Statistical Association, 67, 123-129
  13. Hand, D. J., Mannila, H. and Smyth, P. (2001). Principles of Data Mining, The MIT Press, Massachusetts
  14. Cheng, Y. and Church, G. M. (2000). Bi-clustering of expression data, In Proceedings of the 8th International Conference on Intelligent Systems for Molecular Biology, 93-103
  15. Andre, T. S., Hartson, H. R. and Williges, R. C. (2003). Determining the effectiveness of the usability problem inspector: A theory-based model and tool for ¯nding usability problems, Human Factors: The Journal of the Human Factors and Ergonomics Society, 45, 455-482
  16. Hartson, H. R., Andre, T. S. and Williges, R. C. (2003). Criteria for evaluating usability evaluation methods, International Journal of Human-Computer Interaction, 15, 145-181