A Comparison of the Search Based Testing Algorithm with Metrics

메트릭에 따른 탐색 기반 테스팅 알고리즘 비교

Choi, HyunJae;Chae, HeungSeok

  • Received : 2015.12.11
  • Accepted : 2016.02.10
  • Published : 2016.04.15


Search-Based Software Testing (SBST) is an effective technique for test data generation on large domain size. Although the performance of SBST seems to be affected by the structural characteristics of Software Under Test (SUT), studies for the comparison of SBST techniques considering structural characteristics are rare. In addition to the comparison study for SBST, we analyzed the best algorithm with different structural characteristics of SUT. For the generalization of experimental results, we automatically generated 19,800 SUTs by combining four metrics, which are expected to affect the performance of SBST. According to the experiment results, Genetic algorithm showed the best performance for SUTs with high complexity and test data evaluation with count ${\leq}20,000$. On the other hand, the genetic simulated annealing and the simulated annealing showed relatively better performance for SUTs with high complexity and test data evaluation with count ${\geq}50,000$. Genetic simulated annealing, simulated annealing and hill climbing showed better performance for SUTs with low complexity.


search-based software testing;test data generation;software metric;structural testing


  1. Ali, Shaukat, et aI., "A systematic review of the application and empirical investigation of search-487 based test case generation," Software Engineering, IEEE Transactions on, Vol. 36, No. 6, pp.742-762, 2010.
  2. Xiao, Man, et aI., "Empirical evaluation of optimization algorithms when used in goal-oriented automated test data generation techniques," Empirical Software Engineering, Vol. 12, No. 2. pp. 183-239, 2007.
  3. Harman, Mark, et aI., "Testability transformation ," Software Engineering, IEEE Transactions on, Vol. 30, No. 1, pp. 3-16, 2004.
  4. The Motor Industry Software Reliability Association, Report 5: Software Metrics, MIRA Ltd., Feb. 1995.
  5. Kernighan, Brian, "JPL Institutional Coding Standard for the C Programming Language," California Insititute of Technology, Mar. 2009.
  6. Martin, Lockheed, "Joint strike fighter air vehicle c++ coding standards for the system development and demonstration program," 2005.
  7. Michael, Christoph C., Gary McGraw, and Michael Schatz, "Generating software test data by evolution," Software Engineering, IEEE Transactions on, Vol. 27, No. 12, pp. 1085-1110, 2001.
  8. Tlili, Marouane, Stefan Wappler, and Harmen Sthamer, "Improving evolutionary real-time testing," Proc of the 8th annual conference on Genetic and evolutionary computation, ACM, pp. 1917-1924, 2006.
  9. Harman, Mark, et aI., "Optimizing for the number of tests generated in search based test data generation with an application to the oracle cost problem," Software Testing, Verification, and Validation Workshops (ICSTW), 2010 Third International Conference on IEEE, pp. 182-191, 2010.
  10. Wegener, Joachim, Andre Baresel, and Harmen Sthamer, "Evolutionary test environment for automatic structural testing," Information and Software Technology, Vol. 43, No. 14, pp. 841-854, 2001.
  11. McMinn, Phil, "Search-based software test data generation: a survey," Software testing, Verification and reliability, Vol. 14, No. 2, pp. 105-156, 2004.
  12. Harman, Mark, and Phil McMinn, "A theoretical and empirical study of search-based testing: Local, global, and hybrid search," Software Engineering, IEEE Transactions on, Vol. 36, No. 2, pp. 226-247, 2010.
  13. Affenzeller, Michael, et aI., Genetic algorithms and genetic programming: modern concepts and practical applications, Crc Press, 2009.
  14. McMinn, Phil, et aI., "Input domain reduction through irrelevant variable removal and its effect on local, global, and hybrid search-based structural test data generation," Software Engineering, IEEE Transactions on, Vol. 38, No. 2, pp. 453-477, 2012.
  15. Lakhotia, Kiran, Mark Harman, and Hamilton Gross, "AUSTIN: An open source tool for search based software testing of C programs," Information and Software Technology, Vol. 55, No. 1, pp. 112-125, 2013.
  16. Lundy, Miranda, and Alistair Mees, "Convergence of an annealing algorithm," Mathematical Programming, Vol. 34, No. 1, pp. 111-124, Jan. 1986.
  17. Arlt, Stephan, and Martin Schaf, "Joogie: Infeasible code detection for java," Computer Aided Verification, Springer Berlin Heidelberg, pp.767-773, 2012.


Supported by : 정보통신기술진흥센터