Journal of the Korea Academia-Industrial cooperation Society (한국산학기술학회논문지)

The Korea Academia-Industrial cooperation Society (한국산학기술학회)
권호 표지
DOI QR코드

DOI QR Code

Improvement of Frame Rate of Electro-Optical Sensor using Temporal Super Resolution based on Color Channel Extrapolation

채널별 색상정보 외삽법 기반 시간적 초해상도 기법을 활용한 전자광학 센서의 프레임률 향상 연구

  • Received : 2017.01.20
  • Accepted : 2017.05.12
  • Published : 2017.05.31
Download PDF
⟨ Previous Next ⟩

Abstract

The temporal super resolution is a method for increasing the frame rate. Electro-optical sensors are used in various surveillance and reconnaissance weapons systems, and the spatial resolution and temporal resolution of the required electro-optical sensors vary according to the performance requirement of each weapon system. Because most image sensors capture images at 30~60 frames/second, it is necessary to increase the frame rate when the target moves and changes rapidly. This paper proposes a method to increase the frame rate using color channel extrapolation. Using a DMD, one frame of a general camera was adjusted to have different consecutive exposure times for each channel, and the captured image was converted to a single channel image with an increased frame rate. Using the optical flow method, a virtual channel image was generated for each channel, and a single channel image with an increased frame rate was converted to a color channel image. The performance of the proposed temporal super resolution method was confirmed by the simulation.

시간적 초해상도 기법은 일반 카메라로 찍은 영상을 고속카메라로 찍은 영상과 같이 보일 수 있도록 프레임률을 증가시키는 방법이다. 전자광학 센서는 다양한 감시정찰 무기체계에 탑재되는데, 각 무기체계별 작전요구성능에 따라 필요로 하는 전자광학 센서의 공간적 해상도와 시간적 해상도가 달라진다. 대부분의 영상센서가 30~60 프레임/초로 영상을 촬영하기 때문에, 표적의 이동 및 변화가 이보다 더 빠른 경우 프레임률의 증가가 필요하다. 본 논문은 채널별 색상정보 외삽법을 활용하여 프레임률을 증가시키는 기법을 제안한다. DMD의 각 화소를 카메라 센서의 각 화소와 정합한 후, 카메라 센서의 베이어 패턴에 맞추어 각 채널별로 화소 그룹을 분리한다. DMD를 이용해 일반 카메라의 한 프레임이 채널별로 서로 다른 연속된 노출 시간을 가지도록 조절하여, 촬영한 영상을 프레임률이 증가한 단일채널 영상으로 변환한다. 옵티컬 플로우 기법을 활용하여 각 채널별로 시점에 맞는 가상의 영상을 생성하여, 프레임률이 증가한 단일채널 영상을 컬러채널 영상으로 만들었다. 시뮬레이션을 통해 제안된 시간적 초해상도 기법의 성능을 확인하였다.

Keywords

References

  1. D. Mahajan, F. C. Huang, W. Matusik, R. Ramamoorthi, and P. Belhumeur, "Moving gradients: a path-based method for plausible image interpolation." Proceedings of SIGGRAPH, 2009. DOI: https://doi.org/10.1145/1531326.1531348
  2. T. Brox, A. Bruhn, N. Papenberg, and J. Weickert, "High accuracy optical flow estimation based on a theory for warping." European Conference on Computer Vision (ECCV), pp. 25-36, 2004. DOI: https://doi.org/10.1007/978-3-540-24673-2_3
  3. C. Wang, L. Zhang, Y. He, and Y. P. Tan, "Frame rate up-conversion using trilateral filtering." IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), Vol. 20, Issue. 6, pp. 886-893, 2010. DOI: https://doi.org/10.1109/TCSVT.2010.2046057
  4. G. Bub, M. Tecza, M. Helmes, P. Lee, and P. Kohl, "Temporal pixel multiplexing for simultaneous high-speed, high-resolution imaging." Nature methods, Vol. 7, Issue. 3, pp. 209-211, 2010. DOI: https://doi.org/10.1038/nmeth.1429
  5. J. Gu, Y. Hitomi, T. Mitsunaga, and S. Nayar, "Coded rolling shutter photography: Flexible space-time sampling." IEEE International Conference on Computational Photography (ICCP), 2010. DOI: https://doi.org/10.1109/ICCPHOT.2010.5585094
  6. D. Reddy, A. Veeraraghavan, and R. Chellappa, "P2C2: Programmable pixel compressive camera for high speed imaging." Computer Vision and Pattern Recognition (CVPR), 2011. DOI: https://doi.org/10.1109/CVPR.2011.5995542
  7. D. Liu, J. Gu, Y. Hitomi, M. Gupta, T. Mitsunaga, and S. Nayar, "Efficient Space-Time Sampling with Pixel-wise Coded Exposure for High Speed Imaging." IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), Vol. 36, Issue. 2, pp. 248-260, 2013. DOI: https://doi.org/10.1109/TPAMI.2013.129
  8. T. Brox, A. Bruhn, N. Papenberg, and J. Weickert. High accuracy optical flow estimation based on a theory for warping. European Conference on Computer Vision (ECCV), pp. 25-36, 2004. DOI: https://doi.org/10.1007/978-3-540-24673-2_3
  9. A. Bruhn and J. Weickert. Towards ultimate motion estimation: Combining highest accuracy with real-time performance. International Conference on Computer Vision (ICCV), pp. 749-755, 2005. DOI: https://doi.org/10.1109/iccv.2005.240
  10. L. Xu, J. Jia, and Y. Matsushita, "Motion detail preserving optical flow estimation." IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), Vol. 34, Issue. 9, pp. 1744-1757, 2012. DOI: https://doi.org/10.1109/TPAMI.2011.236