전자공학회논문지 (Journal of the Institute of Electronics and Information Engineers)

  • 제54권8호
  • /
  • Pages.67-80
  • /
  • 2017
  • /
  • 2287-5026(pISSN)
  • /
  • 2288-159X(eISSN)
대한전자공학회 (The Institute of Electronics and Information Engineers)
권호 표지
DOI QR코드

DOI QR Code

추정된 시공간 배경 정보를 이용한 홀채움 방식

Hole-Filling Method Using Extrapolated Spatio-temporal Background Information

  • Kim, Beomsu (School of Electronic Engineering, Soongsil University) ;
  • Nguyen, Tien Dat (School of Electronic Engineering, Soongsil University) ;
  • Hong, Min-Cheol (School of Electronic Engineering, Soongsil University)
  • 투고 : 2017.04.25
  • 심사 : 2017.08.01
  • 발행 : 2017.08.25
PDF 다운로드
⟨ 이전 논문 다음 논문 ⟩

초록

본 논문에서는 추정된 시공간 배경 정보를 이용하여 자유 시점에서 합성영상을 생성하기 위한 홀채움 방식을 제안한다. 시간적 배경 정보를 추정하기 위해 비겹침 패치 기반의 배경 코드북을 이용한 새로운 시간적 배경 모델을 소개한다. 더불어, 공간적 배경 후보의 하한 및 상한 값의 제약 조건을 설정하는 깊이영상 기반 공간적 국부 배경 예측 방식에 대해 제안한다. 추정된 시간적 배경 정보와 공간적 배경 정보의 유사도를 비교하여 가려짐 배경 영역의 홀채움 과정을 수행한다. 또한 3-D 워핑 후 발생하는 컬러영상과 깊이영상간의 불일치 문제를 해결하기 위해 깊이영상 기반의 고스트 제거 필터를 기술한다. 마지막으로 잔여 홀을 채우기 위해 새로운 깊이 항을 포함한 우선순위 함수를 이용하여 인페인팅 방식이 적용된다. 실험 결과를 통해 기존의 홀채움 방식들과 비교하여 제안하는 방식의 객관적, 주관적 성능의 우수성을 확인할 수 있었다.

This paper presents a hole-filling method using extrapolated spatio-temporal background information to obtain a synthesized view. A new temporal background model using non-overlapped patch based background codebook is introduced to extrapolate temporal background information In addition, a depth-map driven spatial local background estimation is addressed to define spatial background constraints that represent the lower and upper bounds of a background candidate. Background holes are filled by comparing the similarities between the temporal background information and the spatial background constraints. Additionally, a depth map-based ghost removal filter is described to solve the problem of the non-fit between a color image and the corresponding depth map of a virtual view after 3-D warping. Finally, an inpainting is applied to fill in the remaining holes with the priority function that includes a new depth term. The experimental results demonstrated that the proposed method led to results that promised subjective and objective improvement over the state-of-the-art methods.

키워드

참고문헌

  1. C. Fehn, "Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D TV," in SPIE Int. Conf. Stereoscopic Display and Virtual Reality Syst. XI, pp. 93-104, Jan. 2004,
  2. P. Kauff, N. Atzpadin, C. Fehn, M. Muller, O. Schreer, A. Smolic, and R. Tanger, "Depth map creation and image-based rendering for advanced 3DTV services providing interoperability and scalability," Signal Process.: Image Commu., vol. 22, no. 2, pp. 217-234, Feb. 2007. https://doi.org/10.1016/j.image.2006.11.013
  3. C. Zhu and S. Li, "Depth image based view synthesis: new insights and perspectives on hole generation and filling," IEEE Trans. Broadcast., vol. 62, no. 1, pp. 82-93, Mar. 2016. https://doi.org/10.1109/TBC.2015.2475697
  4. L. Zhang and W. J. Tam, "Stereoscopic image generation based on depth images for 3D TV," IEEE Trans. Broadcast., vol. 51, no. 2, pp. 191-199, June 2005. https://doi.org/10.1109/TBC.2005.846190
  5. C. L. Zitnick, S. B. Kang, M. Uyttendaele, S. Winder, and R. Szeliski, "High-quality video view interpolation using a layered representation," ACM Trans. Graphics, vol. 23, no. 3, pp. 600-608, Aug. 2004. https://doi.org/10.1145/1015706.1015766
  6. M. Solh and G. AlRegib, "Hierarchical hole-filling for depth-based view synthesis in FTV and 3D video," IEEE. J. Selected Topic in Signal Process., vol.6, no.5, pp. 495-504, Sep. 2012. https://doi.org/10.1109/JSTSP.2012.2204723
  7. M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester, "Image inpainting," in Proc. ACM the 27th Int. Conf. Computer Graphics and Interactive Tech. (SIGGRAPH), pp. 417-424, July 2000.
  8. A. Criminisi, P. Perez, and K. Toyama, "Object removal by exemplarbased inpainting," in Proc. IEEE Int. Conf. Computer Vision and Pattern Recog. (CVPR), doi:10.1109/CVPR.2003.1211538, July 2003.
  9. C. Guillemot and O. Le Meur, "Image inpainting: Overview and recent advances," IEEE Signal Process. Mag., vol. 31, no. 1, pp. 127-144, Jan. 2014. https://doi.org/10.1109/MSP.2013.2273004
  10. Y. Mori, N. Fukushima, T. Fujii, and M. Tanimoto, "View generation with 3D warping using depth information for FTV," Signal Process.: Image Commu., vol. 24, no. 1-2, pp. 65-72, Jan. 2009. https://doi.org/10.1016/j.image.2008.10.013
  11. K. Oh, S. Yea, and Y.-S. Ho, "Hole filling method using depth based inpainting for view synthesis in free viewpoint television and 3-D video," in Proc. Picture Coding Symp., doi:10.1109/PCS.2009.5167450, July 2009.
  12. S. Zinger, L. Do, and P. H. N. de With, "Free-viewpoint depth image based rendering," J. Vis. Comm. Image Represent., vol. 21, no. 5-6, pp. 533-541, July 2010. https://doi.org/10.1016/j.jvcir.2010.01.004
  13. I. Daribo and H. Saito, "A novel inpainting-based layered depth video for 3DTV," IEEE Trans. Broadcast., vol. 57, no. 2, pp. 533-541, June 2011. https://doi.org/10.1109/TBC.2011.2125110
  14. I. Ahn and C. Kim, "A depth-based disocclusion filling method for virtual viewpoint image synthesis," Journal of the Institute of Electronics Engineers of Korea SP, vol. 48, no. 6, pp. 48-60, Nov. 2011.
  15. S. Choi, B. Ham, and K. Sohn, "Space-time hole filling with random walks in view extrapolation for 3D video," IEEE. Trans. Image Process., vol. 22, no. 6, pp. 2429-2441, June 2013. https://doi.org/10.1109/TIP.2013.2251646
  16. S. M. Muddala, M. Sjostrom, R. Olsson, "Virtual view synthesis using layered depth image generation and depth-based inpainting for filling disocclusions and translucent disocclusion," J. Vis. Commu. Image Represent., vol. 38, pp. 351-366, July 2016. https://doi.org/10.1016/j.jvcir.2016.02.017
  17. P. Mdjiki-Nya, M. Koppel, D. Doshkov, H. Lakshman, P. Merkle, K. Muller, and T. Wiegand, "Depth image-based rendering with advanced texture synthesis for 3-D Video," IEEE Trans. Multimedia, vol. 13, no. 3, pp. 453-465, June 2011. https://doi.org/10.1109/TMM.2011.2128862
  18. M. Koppel, X. Wang, D. Doshkov, T. Wiegand, and P. Ndjiki-Nya, "Consistent spatio-temporal filling of disocclusions in the multiview-video-plus-depth format," in Proc. IEEE Int. Workshop Multimedia Signal Process., pp. 25-30, Sep. 2012.
  19. M. Xi, L.-H. Wang, Q.-Q. Yang, D.-X. Li, and M. Zhang, "Depthimage-based rendering with spatial and temporal texture synthesis for 3DTV," EURASIP J. Image and Video Process., doi:10.1186/1687-5281-2013-9, Dec. 2013.
  20. C. Yao, T. Tillo, X. Jimin, "Depth map driven hole filling algorithm exploiting temporal correlation information," IEEE Trans. Broadcast., vol. 60, no. 2, pp. 394-404, June 2014. https://doi.org/10.1109/TBC.2014.2321671
  21. K. Kim, T. H. Chalidabhongse, D. Harwood, and L. Davis, "Background modeling and subtraction by codebook construction," in IEEE Int. Conf. Image Process., pp. 3061-3064, Oct. 2004.
  22. E. J. Fernandez-Sanchez, D. Javier, and R. Eduardo, "Background subtraction based on color and depth using active sensors," Sensors, vol. 13, no. 7, pp. 8895-8915, July 2013. https://doi.org/10.3390/s130708895
  23. A. S. Glassner, Ed., Graphics Gems I, Academic Press, Cambridge, MA, 1990.
  24. W. Zhou, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, "Image quality assessment: from error visibility to structural similarity," IEEE Trans. Image Process., vol. 13, no. 4, pp. 600-612, Apr. 2004. https://doi.org/10.1109/TIP.2003.819861