ETRI Journal

  • 제37권6호
  • /
  • Pages.1143-1153
  • /
  • 2015
  • /
  • 1225-6463(pISSN)
  • /
  • 2233-7326(eISSN)
한국전자통신연구원 (Electronics and Telecommunications Research Institute)
권호 표지
DOI QR코드

DOI QR Code

Fixed Homography-Based Real-Time SW/HW Image Stitching Engine for Motor Vehicles

  • Suk, Jung-Hee (Information & Communications Core Technology Research Laboratory, ETRI) ;
  • Lyuh, Chun-Gi (Information & Communications Core Technology Research Laboratory, ETRI) ;
  • Yoon, Sanghoon (SoC Platform Research Center, KETI) ;
  • Roh, Tae Moon (Information & Communications Core Technology Research Laboratory, ETRI)
  • 투고 : 2014.01.29
  • 심사 : 2015.11.11
  • 발행 : 2015.12.01
PDF 다운로드
⟨ 이전 논문 다음 논문 ⟩

초록

In this paper, we propose an efficient architecture for a real-time image stitching engine for vision SoCs found in motor vehicles. To enlarge the obstacle-detection distance and area for safety, we adopt panoramic images from multiple telegraphic cameras. We propose a stitching method based on a fixed homography that is educed from the initial frame of a video sequence and is used to warp all input images without regeneration. Because the fixed homography is generated only once at the initial state, we can calculate it using SW to reduce HW costs. The proposed warping HW engine is based on a linear transform of the pixel positions of warped images and can reduce the computational complexity by 90% or more as compared to a conventional method. A dual-core SW/HW image stitching engine is applied to stitching input frames in parallel to improve the performance by 70% or more as compared to a single-core engine operation. In addition, a dual-core structure is used to detect a failure in state machines using rock-step logic to satisfy the ISO26262 standard. The dual-core SW/HW image stitching engine is fabricated in SoC with 254,968 gate counts using Global Foundry's 65 nm CMOS process. The single-core engine can make panoramic images from three YCbCr 4:2:0 formatted VGA images at 44 frames per second and frequency of 200 MHz without an LCD display.

키워드

참고문헌

  1. R. Szeliski, "Image Alignment and Stitching: A Tutorial," Microsoft Research, Tech. Rep., MSR-TR-2004-92, Oct. 2004.
  2. R. Szeliski, "Video Mosaics for Virtual Environments," IEEE Comput. Graph. Appl., vol. 16, no. 2, Mar. 1996, pp. 22-30. https://doi.org/10.1109/38.486677
  3. B.S. Kim, S.H. Lee, and N.I. Cho, "Real-Time Panorama Canvas of Natural Images," IEEE Trans. Consum. Electron., vol. 57, no. 4, Nov. 2011, pp. 1961-1968. https://doi.org/10.1109/TCE.2011.6131177
  4. S. Mann and R.W. Picard, "Virtual Bellows: Constructing High Quality Stills from Video," Proc. IEEE Int. Conf. Image Process., Austin, TX, USA, vol. 1, Nov. 13-16, 1994, pp. 363-367.
  5. S. Chen, "Quicktime VR: An Image-Based Approach to Virtual Environment Navigation," Proc. SIGGRAPH, New York, USA, Aug. 1995, pp. 29-38.
  6. Y. Xu, X. Li, and Y. Tian, "Automatic Panorama Mosaicing with High Distorted Fisheye Images," Proc. Int. Conf. Natural Comput., Yantai, China, Aug. 10-12, 2010, pp. 3286-3290.
  7. S.K. Nayar, "Catadioptric Omnidirectional Camera," Proc. IEEE Conf. Comput. Vis. Pattern Recogn., San Juan, Puerto Rico, June 17-19, 1997, pp. 482-488.
  8. T. Kawanishi et al., "Generation of High-Resolution Stereo Panoramic Images by Omnidirectional Sensor Using Hexagonal Pyramidal Mirrors," Proc. Int. Conf. Pattern Recogn., Brisbane, Australia, vol. 1, Aug. 16-20, 1998, pp. 485-489.
  9. S. Peleg and M. Ben-Ezra, "Stereo Panorama with a Single Camera," IEEE Conf. Comput. Vis. Pattern Recogn., Fort Collins, CO, USA, vol. 1, June 23-25, 1999, pp. 395-401.
  10. F. Huang et al., "Animated Panorama from a Panning Video Sequence," Int. Conf. Image Vis. Comput. New Zealand, Queenstown, New Zealand, Nov. 8-9, 2010, pp. 1-8.
  11. S. Peleg, M. Ben-Ezra, and Y. Pritch, "Omnistereo: Panoramic Stereo Imaging," IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 3, Mar. 2001, pp. 279-290. https://doi.org/10.1109/34.910880
  12. A. Ahmed et al., "Geometric Correction for Uneven Quadric Projection Surfaces Using Recursive Subdivision of Bezier Patches," ETRI J., vol. 35, no. 6, Dec. 2013, pp. 1115-1125. https://doi.org/10.4218/etrij.13.0112.0597
  13. S.J. Ha et al., "Panorama Mosaic Optimization for Mobile Camera Systems," IEEE Trans. Consum. Electron., vol. 53, no. 4, Nov. 2007, pp. 1217-1225. https://doi.org/10.1109/TCE.2007.4429204
  14. S.J. Ha et al., "Embedded Panoramic Mosaic System Using Auto-Shot Interface," IEEE Trans. Consum. Electron., vol. 54, no. 1, Feb. 2008, pp. 16-24. https://doi.org/10.1109/TCE.2008.4470018
  15. J.M. Seok and Y. Lee, "Visual-Attention-Aware Progressive RoI Trick Mode Streaming in Interactive Panoramic Video Service," ETRI J., vol. 36, no. 2, Apr. 2014, pp. 253-263. https://doi.org/10.4218/etrij.14.2113.0012
  16. D. Wagner et al., "Real-Time Panoramic Mapping and Tracking on Mobile Phones," IEEE Conf. Virtual Reality, Waltham, MA, USA, Mar. 20-24, 2010, pp. 211-218.
  17. M. Brown and D.G. Lowe, "Automatic Panoramic Image Stitching Using Invariant Features," Int. J. Comput. Vis., vol. 74, no. 1, Aug. 2007, pp. 59-73. https://doi.org/10.1007/s11263-006-0002-3
  18. D.G. Lowe, "Distinctive Image Features from Scale-Invariant Keypoints," Int. J. Comput. Vis., vol. 60, no. 2, 2004, pp. 91-110. https://doi.org/10.1023/B:VISI.0000029664.99615.94
  19. H. Bay et al., "SURF: Speeded-Up Robust Features," Comput. Vis. Image Understanding, vol. 110, no. 3, 2008, pp. 346-359. https://doi.org/10.1016/j.cviu.2007.09.014
  20. A. Shashua et al., EyeQ, Mobileye, 2010. Accessed Aug. 15, 2013. http://www.mobileye.com/technology/processing-platforms/eyeq/
  21. A. Shashua et al., EyeQ2, Mobileye, 2010. Accessed Aug. 15, 2013. http://www.mobileye.com/technology/processing-platforms/eyeq2/
  22. G.P. Stein, Y. Gdalyahu, and A. Shashua, "Stereo-Assist: Top-Down Stereo for Driver Assistance Systems," IEEE. Conf. Intell. Vehicles Symp., San Diego, CA, USA, 2010, pp. 723-730.
  23. J.-H. Suk et al., "An Efficient Architecture of Image Stitching Engine for Vis. SoC," Proc. Workshop Image Processing Image Understanding, Jeju, Rep. of Korea, Feb. 15-17, 2012, Index O-6.
  24. Homography, Wikipedia, 2013. Accessed May 30, 2014. https://en.wikipedia.org/wiki/Homography
  25. Image Stitching, Wikipedia, 2008. Accessed Aug. 30, 2013. https://en.wikipedia.org/wiki/Image_stitching
  26. International Organization for Standardization (ISO), ISO 26262 Road Vehicle-Functional Safety Geneva, Switzerland: ISO, 2011.
  27. K.H. Kwon et al., EISC, Advanced Digital Chips Inc., 2012. Accessed June 15. http://www.adc.co.kr/technology/eisc/eisc.php
  28. H.S. Park et al., "In-Vehicle AR-HUD System to Provide Driving-Safety Information," ETRI J., vol. 35, no. 6, Dec. 2013, pp. 1038-1047. https://doi.org/10.4218/etrij.13.2013.0041
  29. M.A. Fischler and R.C. Bolles, "Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography," Commun. ACM, vol. 24, no. 6, June 1981, pp. 381-395. https://doi.org/10.1145/358669.358692
  30. M. Zuliani, "RANSAC for Dummies," Matlab draft, Jan. 2012.
  31. K. Baker, "Singular Value Decomposition Tutorial," tutorial paper, Mar. 2005.
  32. Y.Y. Boykov and M. Jolly, "Interactive Graph Cuts for Optimal Boundary and Region Segmentation of Objects in N-D Images," Proc. IEEE Int. Conf. Comput. Vis., Vancouver, Canada, vol. 1, July 7-14, 2001, pp. 105-112.
  33. Alpha Compositing, Wikipedia, 2006. Accessed Aug. 30, 2013. https://en.wikipedia.org/wiki/Alpha_compositing#Alpha_blendin
  34. S. Ali and M. Hussain, "Panoramic Image Construction Using Feature Based Registration Methods," Proc. Int. Multitopic Conf., Islamabad, Pakistan, Dec. 13-15, 2012, pp. 209-214.
  35. Point Grey Research Inc. (2012, Dec. 7), Spherical Video System Ladybug2 and Ladybug3. Available: http://www.ptgrey.com
  36. O. Schreer et al., "Ultra-high-Resolution Panoramic Imaging for Format-Agnostic Video Production," Proc. IEEE, vol. 101, no. 1, Jan. 2013, pp. 99-114. https://doi.org/10.1109/JPROC.2012.2193850
  37. A. Akin et al., "Enhanced Omnidirectional Image Reconstruction Algorithm and its Real-Time Hardware," Euromicro Conf. Digital Syst. Des., Izmir, Turkey, Sept. 5-8, 2012, pp. 907-914.
  38. Y. Xu et al., "High-Speed Simultaneous Image Distortion Correction Transformations for a Multicamera Cylindrical Panorama Real-Time Video System Using FPGA," IEEE Trans. Circuits Syst. Video Technol., vol. 24, no. 6, June 2014, pp. 1061-1069. https://doi.org/10.1109/TCSVT.2013.2290576