DOI QR코드

DOI QR Code

VLM-based Object-centric 6DoF 3D Visual Grounding

VLM 기반 객체중심 6DoF 3D Visual Grounding

  • Seongyeop Lee (Industry-Academic Cooperation Foundation, Hanyang University ERICA) ;
  • Hyunsoo Shin (Research Institute of AI Convergence, Hanyang University) ;
  • Gyubeom Jo (Department of Robotics, Hanyang University ERICA) ;
  • Ji-Hun Bae (Department of Robotics, Hanyang University ERICA) ;
  • Sungon Lee (Department of Robotics, Hanyang University ERICA)
  • Received : 2025.09.05
  • Accepted : 2026.05.18
  • Published : 2026.08.31

Abstract

3D Visual Grounding aims to localize target objects in 3D scenes from natural language descriptions. Existing methods mainly provide object location information and do not explicitly capture object orientation, limiting their applicability to robotics. To address this limitation, we propose a VLM-based object-centric 6DoF 3D Visual Grounding framework that generates object-centric 6DoF poses from RGB multi-view images and text prompts. The framework reconstructs a 3D scene and camera viewpoints, extracts an object point cloud through mask-based multi-view projection with erosion and voting, and defines an object-centric coordinate frame by combining OBB-based geometric cues with semantic direction inference from a VLM enhanced via Egocentric Instruction Tuning. Experiments on seven objects show that the proposed method generates reliable object-centric 6DoF poses for objects with clear semantic directions.

Keywords

Acknowledgement

This research was financially supported by the Ministry of Trade, Industry & Energy (MOTIE, Korea) under the "Scale-up Technology Commercialization Program" (P0026386), supervised by the Korea Institute for Advancement of Technology (KIAT), and by the Technology Development Program (00507790) funded by the Ministry of Trade, Industry & Energy (MOTIE, Korea).

References

  1. T. Brown et al., "Language models are few-shot learners," Advances in neural information processing systems, vol. 33, pp. 1877-1901, Dec. 2020, DOI: 10.18653/v1/2021.mrl-1.1.
  2. Z. Yang, L. Li, K. Lin, J. Wang, C.-C. Lin, Z. Liu, and L. Wang, "The dawn of lmms: Preliminary explorations with gpt-4v(ision)," arXiv preprint arXiv:2309.17421, 2023, DOI: 10.48550/arXiv.2309.17421.
  3. Q. Feng, V. Ablavsky, and S. Sclaroff, "Cityflow- nl: Tracking and retrieval of vehicles at city scale by natural language descriptions," arXiv preprint arXiv:2101.04741, Jan., 2021, DOI: 10.48550/arXiv.2101.04741.
  4. L. Kong et al., "Robo3d: Towards robust and reliable 3d perception against corruptions," in Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France, pp. 19994-20006, 2023, DOI: 10.1109/ICCV51070.2023.01830.
  5. D. Z. Chen, A. X. Chang, and M. Nießner, "Scanrefer: 3d object localization in rgb-d scans using natural language," in European conference on computer vision, Glasgow, UK, pp. 202-221, 2020, DOI: 10.1007/978-3-030-58565-5_13.
  6. P. Achlioptas, A. Abdelreheem, F. Xia, M. Elhoseiny, and L. Guibas, "Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes," in European conference on computer vision, Glasgow, UK, pp. 422-440, 2020, DOI: 10.1007/978-3-030-58452-8_25.
  7. A. Dai et al., "Scannet: Richly-annotated 3d reconstructions of indoor scenes," in Proceedings of the IEEE conference on computer vision and pattern recognition, Honolulu, USA, pp. 2432-2443, 2017, DOI: 10.1109/CVPR.2017.261.
  8. Z. Zhu et al., "3d -Vista: Pre-trained transformer for 3d vision and text alignment," in Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France, pp. 2911-2921, 2023, DOI: 10.1109/ICCV51070.2023.00272.
  9. J. Yang et al., "Llm-grounder: Open-vocabulary 3d visual grounding with large language model as an agent," in 2024 IEEE International Conference on Robotics and Automation, Yokohama, Japan, pp. 7694-7701, 2024, DOI: 10.1109/ICRA57147.2024.10610443.
  10. Z. Yuan et al., "Visual programming for zero-shot open-vocabulary 3d visual grounding," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, pp. 20623-20633, 2024, DOI: 10.1109/CVPR52733.2024.01949.
  11. R. Xu et al., "Vlm-grounder: A vlm agent for zero-shot 3d visual grounding," in Conference on Robot Learning, vol. 270, pp. 3961-3985, 2025, DOI: 10.48550/arXiv.2410.13860.
  12. R. Li, S. Li, L. Kong, X. Yang, and J. Liang, "Zero-shot 3d visual grounding from vision-language models," arXiv preprint arXiv: 2505.22429, May 2025, DOI: 10.48550/arXiv.2505.22429.
  13. J. H. Jung et al., "Is 'right' right? enhancing object orientation understanding in multimodal large language models through egocentric instruction tuning," in 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, USA, pp. 14257-14267, 2025, DOI: 10.1109/CVPR52734.2025.01330.
  14. A. Jain, N. Gkanatsios, I. Mediratta, and K. Fragkiadaki, "Bottom up top down detection transformers for language grounding in images and point clouds," in European Conference on Computer Vision, Tel Aviv, Israel, pp. 417-433, 2022, DOI: 10.1007/978-3-031-20059-5_24.
  15. O. Unal, C. Sakaridis, S. Saha, and L. Van Gool, "Four ways to improve verbovisual fusion for dense 3d visual grounding," in European Conference on Computer Vision, Milan, Italy, pp. 196-213, 2024, DOI: 10.1007/978-3-031-73116-7_12.
  16. S. Peng et al., "Openscene: 3d scene understanding with open vocabularies," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Vancouver, Canada, pp. 815-824, 2023, DOI: 10.1109/CVPR52729.2023.00085.
  17. J. Kerr, C. M. Kim, K. Goldberg, A. Kanazawa, and M. Tancik, "Lerf: Language embedded radiance fields," in Proceedings of the IEEE/CVF international conference on computer vision, Paris, France, pp. 19729-19739, 2023, DOI: 10.1109/ICCV51070.2023.01807.
  18. S. Tong et al., "Eyes wide shut? exploring the visual shortcomings of multimodal llms," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Seattle, USA, pp. 9568-9578, 2024, DOI: 10.1109/CVPR52733.2024.00914.
  19. T.-Y. Lin et al., "Microsoft coco: Common objects in context," in European conference on computer vision, Zurich, Switzerland, pp. 740-755, 2014, DOI: 10.1007/978-3-319-10602-1_48.
  20. C. Schuhmann et al., "Laion-5b: An open large-scale dataset for training next generation image-text models," Advances in neural information processing systems, vol. 35, pp. 25278-25294, 2022, DOI: 10.5555/3600270.3602103.
  21. Y. Huang et al., "Egoexolearn: A dataset for bridging asynchronous ego-and exo-centric view of procedural activities in real world," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, pp. 22072-22086, 2024, DOI: 10.1109/CVPR52733.2024.02084.
  22. Y. Mu et al., "Embodiedgpt: Vision-language pre-training via embodied chain of thought," Advances in Neural Information Processing Systems, vol. 36, pp. 25081-25094, 2023, DOI: 10.5555/3666122.3667212.
  23. J. L. Schonberger and J.-M. Frahm, "Structure-from-motion revisited," in Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, USA, pp. 4104-4113, 2016, DOI: 10.1109/CVPR.2016.445.
  24. T. Ren et al., "Grounded sam: Assembling open-world models for diverse visual tasks," arXiv preprint arXiv:2401.14159, Jan. 2024, DOI: 10.48550/arXiv.2401.14159.
  25. S. Liu et al., "Grounding dino: Marrying dino with grounded pre-training for open-set object detection," in European conference on computer vision, Milan, Italy, pp. 38-55, 2024, DOI: 10.1007/978-3-031-72970-6_3.
  26. N. Ravi et al., "Sam 2: Segment anything in images and videos," in International Conference on Learning Representations, arXiv preprint arXiv:2408.00714, 2024, DOI: 10.48550/arXiv.2408.00714.
  27. A. Marafioti et al., "Smolvlm: Redefining small and efficient multimodal models," in Second Conference on Language Modeling, 2025, DOI: 10.48550/arXiv.2504.05299.