Smart Structures and Systems

Techno-Press (테크노프레스)
권호 표지
DOI QR코드

DOI QR Code

Autonomous vision-based damage chronology for spatiotemporal condition assessment of civil infrastructure using unmanned aerial vehicle

  • Mondal, Tarutal Ghosh (Lyles School of Civil Engineering, Purdue University) ;
  • Jahanshahi, Mohammad R. (Lyles School of Civil Engineering, Purdue University)
  • Received : 2019.10.07
  • Accepted : 2020.03.22
  • Published : 2020.06.25
⟨ Previous Next ⟩

Abstract

This study presents a computer vision-based approach for representing time evolution of structural damages leveraging a database of inspection images. Spatially incoherent but temporally sorted archival images captured by robotic cameras are exploited to represent the damage evolution over a long period of time. An access to a sequence of time-stamped inspection data recording the damage growth dynamics is premised to this end. Identification of a structural defect in the most recent inspection data set triggers an exhaustive search into the images collected during the previous inspections looking for correspondences based on spatial proximity. This is followed by a view synthesis from multiple candidate images resulting in a single reconstruction for each inspection round. Cracks on concrete surface are used as a case study to demonstrate the feasibility of this approach. Once the chronology is established, the damage severity is quantified at various levels of time scale documenting its progression through time. The proposed scheme enables the prediction of damage severity at a future point in time providing a scope for preemptive measures against imminent structural failure. On the whole, it is believed that the present study will immensely benefit the structural inspectors by introducing the time dimension into the autonomous condition assessment pipeline.

Keywords

Acknowledgement

This study was supported in part by a support from Bentley Systems, Inc. The authors would like to thank Dr. Santiago Pujol, Dr. Aishwarya Puranam and Mr. Rih-Teng Wu for their help to conduct the experiment and data collection.

References

  1. Adhikari, R.S., Bagchi, A. and Moselhi, O. (2014), "Automated condition assessment of concrete bridges with digital imaging", Smart Struct. Syst., Int. J., 13(6), 901-925. https://doi.org/10.12989/sss.2014.13.6.901
  2. Agrawal, A.K., Kawaguchi, A. and Chen, Z. (2010), "Deterioration rates of typical bridge elements in New York", J. Bridge Eng., 15(4), 419-429. https://doi.org/10.1061/(ASCE)BE.1943-5592.0000123
  3. Aliakbar, M., Qidwai, U., Jahanshahi, M.R., Masri, S. and Shen, W.M. (2016), "Progressive image stitching algorithm for vision based automated inspection", Proceedings of 2016 International Conference on Machine Learning and Cybernetics (ICMLC), 1, 337-343.
  4. Aliakbar, M., Qidwai, U. and Jahanshahi, M.R. (2019), "An evaluation of image-based structural health monitoring using integrated unmanned aerial vehicle platform", Struct. Control Health Monitor., 26(1), e2276. https://doi.org/10.1002/stc.2276
  5. Bay, H., Tuytelaars, T. and Van Gool, L. (2006), "Surf: Speeded Up Robust Features", Proceedings of the European Conference on Computer Vision, Springer, Berlin, Heidelberg.
  6. Besuner, P.M. (1976), "Residual life estimates for structures with partial thickness cracks", Proceedings of Mechanics of Crack Growth, ASTM International.
  7. Boller, C., Starke, P., Dobmann, G., Kuo, C.M. and Kuo, C.H. (2015), "Approaching the assessment of ageing bridge infrastructure", Smart Struct. Syst., Int. J., 15(3), 593-608. https://doi.org/10.12989/sss.2015.15.3.593
  8. Bottou, L. (2010), "Large-scale machine learning with stochastic gradient descent", Proceedings of COMPSTAT'2010, Physica-Verlag HD.
  9. Cha, Y.J., Choi, W. and Buyukozturk, O. (2017), "Deep learning-based crack damage detection using convolutional neural networks", Comput.-Aided Civil Infrastruct. Eng., 32(5), 361-378. https://doi.org/10.1111/mice.12263
  10. Cha, Y.J., Choi, W., Suh, G., Mahmoudkhani, S. and Buyukozturk, O. (2018), "Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types", Comput.-Aided Civil Infrastruct. Eng., 33(9), 731-747. https://doi.org/10.1111/mice.12334
  11. Chen, F.C. and Jahanshahi, M.R. (2018), "NB-CNN: deep learning-based crack detection using convolutional neural network and Naive Bayes data fusion", IEEE Transact. Indust. Electron., 65(5), 4392-4400. https://doi.org/10.1109/TIE.2017.2764844
  12. Choi, J., Yeum, C.M., Dyke, S.J. and Jahanshahi, M.R (2018), "Computer-aided approach for rapid post-event visual evaluation of a building facade", Multidisciplinary Digital Publishing Institute, 18(9), 3017. https://doi.org/10.3390/s18093017
  13. Cruse, T.A. and Besuner, P.M. (1975), "Residual life prediction for surface cracks in complex structural details", J. Aircraft, 12(4), 369-375. https://doi.org/10.2514/3.44458
  14. Dhawan, S.K., Bindal, A., Bhalla, S. and Bhattacharjee, B. (2019), "Expected residual service life of reinforced concrete structures from current strength considerations", Adv. Struct. Eng., 22(7), 1631-1643. https://doi.org/10.1177/1369433218818001
  15. Duan, Y., Chen, Q., Zhang, H., Yun, C.B., Wu, S. and Zhu, Q. (2019), "CNN-based damage identification method of tied-arch bridge using spatial-spectral information", Smart Struct. Syst., Int. J., 23(5), 507-520. https://doi.org/10.12989/sss.2019.23.5.507
  16. Esser, B. and Huston, D.R. (2005), "Versatile robotic platform for structural health monitoring and surveillance", Smart Struct. Syst., Int. J., 1(4), 325-338. https://doi.org/10.12989/sss.2005.1.4.325
  17. Fischler, M.A. and Bolles, R.C. (1981), "Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography", Communicat. ACM, 24(6), 381-395. https://doi.org/10.1145/358669.358692
  18. Ghorbani, R., Matta, F. and Sutton, M.A. (2015), "Full-field deformation measurement and crack mapping on confined masonry walls using digital image correlation", Experim. Mech., 55(1), 227-243. https://doi.org/10.1007/s11340-014-9906-y
  19. Girshick, R. (2015), "Fast R-CNN", Proceedings of the IEEE International Conference on Computer Vision.
  20. Girshick, R., Donahue, J., Darrell, T. and Malik, J. (2014), "Rich feature hierarchies for accurate object detection and semantic segmentation", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
  21. Glorot, X. and Bengio, Y. (2010), "Understanding the difficulty of training deep feedforward neural networks", Proceedings of the 13th International Conference on Artificial Intelligence and Statistics.
  22. He, K., Zhang, X., Ren, S. and Sun, J. (2001), "Deep residual learning for image recognition", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
  23. Hosang, J.H., Benenson, R. and Schiele, B. (2017), "Learning non-maximum suppression", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
  24. Jahanshahi, M.R. and Masri, S.F. (2013), "A new methodology for non-contact accurate crack width measurement through photogrammetry for automated structural safety evaluation", Smart Mater. Struct., 22(3), 035019. https://doi.org/10.1088/0964-1726/22/3/035019
  25. Jahanshahi, M.R. and Masri, S.F. (2014), "Image-based crack detection", Google Patents, 22(3), 8,873,837.
  26. Jahanshahi, M.R., Kelly, J.S., Masri, S.F. and Sukhatme, G.S. (2009), "A survey and evaluation of promising approaches for automatic image-based defect detection of bridge structures", Struct. Infrastruct. Eng., 5(6), 455-486. https://doi.org/10.1080/15732470801945930
  27. Jahanshahi, M.R., Masri, S.F. and Sukhatme, G.S. (2011), "Multiimage stitching and scene reconstruction for evaluating defect evolution in structures", Struct. Health Monitor., 10(6), 643-657. https://doi.org/10.1177/1475921710395809
  28. Jahanshahi, M.R., Masri, S.F., Padgett, C.W. and Sukhatme, G.S. (2013), "An innovative methodology for detection and quantification of cracks through incorporation of depth perception", Mach. Vision Applicat., 24(2), 227-241. https://doi.org/10.1007/s00138-011-0394-0
  29. Jahanshahi, M.R., Chen, F.C., Joffe, C. and Masri, S.F. (2017a), "Vision-based quantitative assessment of microcracks on reactor internal components of nuclear power plants", Struct. Infrastruct. Eng., 13(8), 1013-1026. https://doi.org/10.1080/15732479.2016.1231207
  30. Jahanshahi, M.R., Chen, F.C., Ansar, A., Padgett, C.W., Clouse, D. and Bayard, D.S. (2017b), "Accurate and robust scene reconstruction in the presence of misassociated features for aerial sensing", J. Comput. Civil Eng., 31(6), 04017056. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000702
  31. Jahanshahi, M.R., Shen, W.M., Mondal, T.G., Abdelbarr, M., Masri, S.F. and Qidwai, U.A. (2017c), "Reconfigurable swarm robots for structural health monitoring: a brief review", Int. J. Intel. Robot. Applicat., 1(3), 287-305. https://doi.org/10.1007/s41315-017-0024-8
  32. Kim, H., Ahn, E., Shin, M. and Sim, S.H. (2019), "Crack and noncrack classification from concrete surface images using machine learning", Struct. Health Monitor., 18(3), 725-738. https://doi.org/10.1177/1475921718768747
  33. Kong, X. and Li, J. (2018), "Vision-based fatigue crack detection of steel structures using video feature tracking", Comput.-Aided Civil Infrastruct. Eng., 33(9), 783-799. https://doi.org/10.1111/mice.12353
  34. Kong, X. and Li, J. (2019), "Non-contact fatigue crack detection in civil infrastructure through image overlapping and crack breathing sensing", Automat. Constr., 99, 125-139. https://doi.org/10.1016/j.autcon.2018.12.011
  35. Lee, B.Y., Kim, Y.Y., Yi, S.T. and Kim, J.K. (2013), "Automated image processing technique for detecting and analysing concrete surface cracks", Struct. Infrastruct. Eng., 9(6), 567-577. https://doi.org/10.1080/15732479.2011.593891
  36. Levenberg, K. (1944), "A method for the solution of certain nonlinear problems in least squares", Quarterly Appl. Mathe., 2(2), 164-168. https://doi.org/10.1090/qam/10666
  37. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollar, P. and Zitnick, C.L. (2001), "Microsoft COCO: Common Objects in Context", Proceedings of the European Conference on Computer Vision, Springer, Cham.
  38. Lowe, D.G. (2004), "Distinctive image features from scaleinvariant keypoints", Int. J. Comput. Vision, 60(2), 91-110. https://doi.org/10.1023/B:VISI.0000029664.99615.94
  39. Marquardt, D.W. (1963), "An algorithm for least-squares estimation of nonlinear parameters", J. Soc. Indust. Appl. Mathe., 11(2), 431-441. https://doi.org/10.1137/0111030
  40. Matheron, G. (1975), "Random Sets and Integral Geometry", 91(4), 1301-1328.
  41. Minkowski, H. (1989), "Volumen und oberflache", Proceedings of Ausgewahlte Arbeiten zur Zahlentheorie und zur Geometrie, Springer, Vienna, Austria.
  42. Mohanty, S., Chattopadhyay. A. and Peralta, P. (2010), "Adaptive residual useful life estimation of a structural hotspot", J. Intel. Mater. Syst. Struct., 21(3), 321-335. https://doi.org/10.1177/1045389X09357972
  43. Myung, H., Wang, Y., Kang, S.C. and Chen, X. (2014), "Survey on robotics and automation technologies for civil infrastructure", University of Canterbury, Mechanical Engineering.
  44. Otsu, N. (1979), "A threshold selection method from gray-level histograms", IEEE Transactions on Systems, Man, and Cybernetics, 9(1), 62-66. https://doi.org/10.1109/TSMC.1979.4310076
  45. Ren, S., He, K., Girshick, R. and Sun, J. (2015), "Faster R-CNN: Towards real-time object detection with region proposal networks", Proceedings of Advances in Neural Information Processing Systems.
  46. Salembier, P. (1990), "Comparison of some morphological segmentation algorithms based on contrast enhancement", Proceedings of 5. European Signal Processing Conference.
  47. Spencer Jr., B.F., Hoskere, V. and Narazaki, Y. (2019), "Advances in computer vision-based civil infrastructure inspection and monitoring", Eng., 5(2), 199-222. https://doi.org/10.1016/j.eng.2018.11.030
  48. Shan, B., Zheng, S. and Ou, J. (2016), "A stereovision-based crack width detection approach for concrete surface assessment", KSCE J. Civil Eng., 20(2), 803-812. https://doi.org/10.1007/s12205-015-0461-6
  49. Szegedy, C., Ioffe, S., Vanhoucke, V. and Alemi, A.A. (2001a), "Inception-v4, Inception-Resnet and the impact of residual connections on learning", Proceedings of the 31st AAAI Conference on Artificial Intelligence.
  50. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. and Wojna, Z. (2001b), "Rethinking the inception architecture for computer vision", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
  51. Tanathong, S. and Lee, I. (2013), "Translation-based KLT tracker under severe camera rotation using GPS/INS data", IEEE Geosci. Remote Sensing Lett., 11(1), 64-68. https://doi.org/10.1109/LGRS.2013.2245858
  52. Van Uitert, R. and Bitter, I. (2007), "Subvoxel precise skeletons of volumetric data based on fast marching methods", Medical Physics, 34(2), 627-638. https://doi.org/10.1118/1.2409238
  53. Wang, X., Balakrishnan, N. and Guo, B. (2014), "Residual life estimation based on a generalized Wiener degradation process", Reliabil. Eng. Syst. Safety, 124, 13-23. https://doi.org/10.1016/j.ress.2013.11.011
  54. Wu, R.T., Singla, A., Jahanshahi, M.R., Bertino, E., Ko, B.J., Verma, D. (2019), "Pruning deep convolutional neural networks for efficient edge computing in condition assessment of infrastructures", Comput.-Aided Civil Infrastruct. Eng., 34(9), 774-789. https://doi.org/10.1111/mice.12449
  55. Yamaguchi, T. and Hashimoto, S. (2010), "Fast crack detection method for large-size concrete surface images using percolation-based image processing", Mach. Vision Applicat., 21(5), 797-809. https://doi.org/10.1007/s00138-009-0189-8
  56. Yeum, C.M., Choi, J. and Dyke, S.J. (2017), "Autonomous image localization for visual inspection of civil infrastructure", Smart Mater. Struct., 26(3), 035051. https://doi.org/10.1088/1361-665X/aa510e
  57. Yeum, C.M., Choi, J. and Dyke, S.J. (2019), "Automated region-of-interest localization and classification for vision-based visual assessment of civil infrastructure", Struct. Health Monitor., 18(3), 675-689. https://doi.org/10.1177/1475921718765419
  58. Yu, S.N., Jang, J.H. and Han, C.S. (2007), "Auto inspection system using a mobile robot for detecting concrete cracks in a tunnel", Automat. Constr., 16(3), 255-261. https://doi.org/10.1016/j.autcon.2006.05.003
  59. Zhu, Z., German, S. and Brilakis, I. (2011), "Visual retrieval of concrete crack properties for automated post-earthquake structural safety evaluation", Automat. Constr., 20(7), 874-883. https://doi.org/10.1016/j.autcon.2011.03.004