ETRI Journal

Electronics and Telecommunications Research Institute (한국전자통신연구원)
권호 표지
DOI QR코드

DOI QR Code

Joint frame rate adaptation and object recognition model selection for stabilized unmanned aerial vehicle surveillance

  • Gyu Seon Kim (Department of Electrical and Computer Engineering, Korea University) ;
  • Haemin Lee (Department of Electrical and Computer Engineering, Korea University) ;
  • Soohyun Park (Department of Electrical and Computer Engineering, Korea University) ;
  • Joongheon Kim (Department of Electrical and Computer Engineering, Korea University)
  • Received : 2023.04.03
  • Accepted : 2023.08.08
  • Published : 2023.10.20
Download PDF
⟨ Previous Next ⟩

Abstract

We propose an adaptive unmanned aerial vehicle (UAV)-assisted object recognition algorithm for urban surveillance scenarios. For UAV-assisted surveillance, UAVs are equipped with learning-based object recognition models and can collect surveillance image data. However, owing to the limitations of UAVs regarding power and computational resources, adaptive control must be performed accordingly. Therefore, we introduce a self-adaptive control strategy to maximize the time-averaged recognition performance subject to stability through a formulation based on Lyapunov optimization. Results from performance evaluations on real-world data demonstrate that the proposed algorithm achieves the desired performance improvements.

Keywords

Acknowledgement

IITP funded by the Korea government (MSIT) (No.2022-0-00907) and by the National Research Foundation of Korea (2019M3E3A1084054).

References

  1. X. Wu, W. Li, D. Hong, R. Tao, and Q. Du, Deep learning for unmanned aerial vehicle-based object detection and tracking: a survey, IEEE Geosci. Remote Sens. Mag. 10 (2021), no. 1, 91-124.
  2. M. Choi, W. J. Yun, and J. Kim, Delay-sensitive and power-efficient quality control of dynamic video streaming using adaptive super-resolution, arXiv preprint 2021. arXiv:2110.05783.
  3. S. Park, C. Park, S. Jung, J. H. Kim, and J. Kim, Workloadaware scheduling using Markov decision process for infrastructure-assisted learning-based multi-UAV surveillance networks, IEEE Access 11 (2023), 16533-16548.
  4. Y. Ji, X. Zhong, Z. Kou, S. Zhang, H. Li, and Y. Yang, Efficiency-boosting federated learning in wireless networks: a longterm perspective, IEEE Trans. Veh. Technol. 2023 (2023), 1-14.
  5. Z. Fang, J. Wang, Y. Ren, Z. Han, H. V. Poor, and L. Hanzo, Age of information in energy harvesting aided massive multiple access networks, IEEE J. Sel. Areas Commun. 40 (2022), no. 5, 1441-1456. https://doi.org/10.1109/JSAC.2022.3143252
  6. Y. Zhang, J. Wang, L. Zhang, Y. Zhang, Q. Li, and K.-C. Chen, Reliable transmission for NOMA systems with randomly deployed receivers, IEEE Trans. Commun. 71 (2023), no. 2, 1179-1192. https://doi.org/10.1109/TCOMM.2022.3230847
  7. J. Wang, L. Bai, J. Chen, and J. Wang, Starling flocks-inspired resource allocation for ISAC-aided green ad hoc networks, IEEE Trans. Green Commun. Netw. 7 (2023), no. 1, 444-454.
  8. X. Wu, R. He, Z. Sun, and T. Tan, A light CNN for deep face representation with noisy labels, IEEE Trans. Inf. Forens. Sec. 13 (2018), no. 11, 2884-2896. https://doi.org/10.1109/TIFS.2018.2833032
  9. Y. Zhang, M. Wang, Y. Cao, X. Qiu, W. Zhou, and Z. Li, A light-weight deep CNN object detection framework based on dense connections, (Proceedings of IEEE International Conference on Signal, Information and Data Processing (ICSIDP), Chongqing, China), 2019, pp. 1-5.
  10. Y. He, P. Liu, Z. Wang, Z. Hu, and Y. Yang, Filter pruning via geometric median for deep convolutional neural networks acceleration, (Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA), 2019, pp. 4340-4349.
  11. D. Kim, J. Kim, J. Kwon, and T. H. Kim, Depth-controllable very deep super-resolution network, (Proceedings of International Joint Conference on Neural Networks (IJCNN), Budapest, Hungary), 2019, pp. 1-8.
  12. S. Park, Y. Kang, J. Park, and J. Kim, Self-controllable superresolution deep learning framework for surveillance drones in security applications, EAI Endorsed Trans. Sec. Safety 7 (2020), no. 23, e5.
  13. S. Jung and J. Kim, Adaptive and stabilized real-time superresolution control for UAV-assisted smart harbor surveillance platforms, J. Real-Time Image Process. 18 (2021), 1815-1825. https://doi.org/10.1007/s11554-021-01163-2
  14. I. V. Saetchnikov, E. A. Tcherniavskaia, and V. V. Skakun, Object detection for unmanned aerial vehicle camera via convolutional neural networks, IEEE J. Miniaturiz. Air Space Syst. 2 (2020), no. 2, 98-103.
  15. P. Zhu, L. Wen, D. Du, X. Bian, F. H, Q. Hu, and H. Ling, Detection and tracking meet drones challenge, IEEE Trans. Pattern Anal. Mach. Intell. 44 (2021), 7380-7399. https://doi.org/10.1109/TPAMI.2021.3119563
  16. S. E. Benton, E. Rogers, and D. H. Owens, Lyapunov stability theory for linear repetitive processes - the 2D equation approach, (Proceedings of IEEE European Control Conference (ECC), Karlsruhe, Germany), 1999, pp. 4768-4773.
  17. L. Jiao, F. Zhang, F. Liu, S. Yang, L. Li, Z. Feng, and R. Qu, A survey of deep learning-based object detection, IEEE Access 7 (2019), 128837-128868. https://doi.org/10.1109/ACCESS.2019.2939201
  18. J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, You only look once: unified, real-time object detection, (Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA), 2016, pp. 779-788.
  19. R. Girshick, J. Donahue, T. Darrell, and J. Malik, Rich feature hierarchies for accurate object detection and semantic segmentation, (Proceedings of IEEE Conference on Computer vision and Pattern Recognition (CVPR), Columbus, OH, USA), 2014, pp. 580-587.
  20. W. Budiharto, A. Gunawan, J. S. Suroso, A. Chowanda, A. Patrik, and G. Utama, Fast object detection for quadcopter drone using deep learning, (Proceedings of International Conference on Computer and Communication Systems (ICCCS), Nagoya, Japan), 2018, pp. 192-195.
  21. J. Shin, M. J. Piran, H.-K. Song, and H. Moon, UAV-assisted and deep learning-driven object detection and tracking for autonomous driving, (Proceedings of International ACM Mobicom Workshop on Drone Assisted Wireless Communications for 5G and Beyond, Sydney, Australia), 2022, pp. 7-12.
  22. D. Lao and G. Sundaramoorthi, Minimum delay object detection from video, (Proceedings of IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Rep. of Korea), 2019, pp. 5097-5106.
  23. J. Lee, B. Varghese, and H. Vandierendonck, ROMA: run-time object detection to maximize real-time accuracy, (Proceedings of IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA), 2023, pp. 6405-6414.
  24. K. S. Kim, D. Kwon, Y. Kim, J. Kim, and J. Kim, Self-adaptive machine learning operating systems for security applications, (Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, Canada), 2018, pp. 6946-6950.
  25. M. J. Neely, Stochastic network optimization with application to communication and queueing systems, Synth. Lect. Commun. Netw. 3 (2010), no. 1, 1-211. https://doi.org/10.1007/978-3-031-79995-2
  26. J. Kim, G. Caire, and A. F. Molisch, Quality-aware streaming and scheduling for device-to-device video delivery, IEEE/ACM Trans. Netw. 24 (2016), no. 4, 2319-2331.
  27. J. Kim, Y. J. Mo, W. Lee, and D. Nyang, Dynamic security-level maximization for stabilized parallel deep learning architectures in surveillance applications, (Proceedings of IEEE Symposium on Privacy-Aware Computing (PAC), Washington, DC, USA), 2017, pp. 192-193.
  28. M. J. Neely, Energy optimal control for time-varying wireless networks, IEEE Trans. Inf. Theory 52 (2006), no. 7, 2915-2934. https://doi.org/10.1109/TIT.2006.876219
  29. M. J. Neely, A. S. Tehrani, and A. G. Dimakis, Efficient algorithms for renewable energy allocation to delay tolerant consumers, (Proceedings of IEEE International Conference on Smart Grid Communications (SGC), Gaithersburg, MD, USA), 2010, pp. 549-554.
  30. J. Koo, J. Yi, J. Kim, M. A. Hoque, and S. Choi, REQUEST: seamless dynamic adaptive streaming over HTTP for multihomed smartphone under resource constraints, (Proceedings of ACM International Conference on Multimedia (MM), Mountain View, CA, USA), 2017, pp. 934-942.
  31. J. Koo, J. Yi, J. Kim, M. A. Hoque, and S. Choi, Seamless dynamic adaptive streaming in LTE/Wi-Fi integrated network under smartphone resource constraints, IEEE Trans. Mob. Comput. 18 (2019), no. 7, 1647-1660. https://doi.org/10.1109/TMC.2018.2863234
  32. L. Ma, D. Meng, S. Zhao, and B. An, Visual localization with a monocular camera for unmanned aerial vehicle based on landmark detection and tracking using YOLOv5 and DeepSORT, Int. J. Adv. Robot. Syst. 20 (2023), no. 3. DOI 10.1177/17298806231164831
  33. Y. Song and G. Wang, A note on passivity theory via Lyapunov like function, (Proceedings of Chinese Control Conference (CCC), Beijing, China), 2010, pp. 210-212.