Journal of Korea Game Society (한국게임학회 논문지)

Korea Game Society (한국게임학회)
권호 표지
DOI QR코드

DOI QR Code

A Study on Load Distribution of Gaming Server Using Proximal Policy Optimization

Proximal Policy Optimization을 이용한 게임서버의 부하분산에 관한 연구

  • Received : 2019.03.13
  • Accepted : 2019.05.07
  • Published : 2019.06.19
Download PDF
⟨ Previous Next ⟩

Abstract

The gaming server is based on a distributed server. In order to distribute workloads of gaming servers, distributed gaming servers apply some algorithms which divide each of gaming server's workload into balanced workload among the gaming servers and as a result, efficiently manage response time and fusibility of server requested by the clients. In this paper, we propose a load balancing agent using PPO(Proximal Policy Optimization) which is one of the methods from a greedy algorithm and Policy Gradient which is from Reinforcement Learning. The proposed load balancing agent is compared with the previous researches based on the simulation.

게임 서버는 분산 서버를 기본으로 하고 있다. 분산 게임서버는 서버의 작업 부하를 분산하기 위한 일련의 알고리즘에 의해 각 게임 서버의 부하를 일정하게 나누어서 클라이언트들의 요청에 대한 서버의 응답시간 및 서버의 가용성을 효율적으로 관리한다. 본 논문에서는 시뮬레이션 환경에서 기존 연구 방식인 Greedy 알고리즘과, Reinforcement Learning의 한 줄기인 Policy Gradient 중 PPO(Proximal Policy Optimazation)을 이용한 부하 분산 Agent를 제안하고, 시뮬레이션 한 후 기존 연구들과의 비교 분석을 통해 성능을 평가하였다.

Keywords

KGOHCL_2019_v19n3_5_f0001.png 이미지

[Fig. 1] DNN(Deep Neural Network)

KGOHCL_2019_v19n3_5_f0002.png 이미지

[Fig. 2] Reinforcement Learning

KGOHCL_2019_v19n3_5_f0003.png 이미지

[Fig. 3] PPO algorithm

KGOHCL_2019_v19n3_5_f0004.png 이미지

[Fig. 4] ProGReGA Algorithm

KGOHCL_2019_v19n3_5_f0005.png 이미지

[Fig. 5] ml-agents

KGOHCL_2019_v19n3_5_f0006.png 이미지

[Fig. 6] Proposed Algorithm process

KGOHCL_2019_v19n3_5_f0007.png 이미지

[Fig. 7] Decision Making Algorithm

KGOHCL_2019_v19n3_5_f0008.png 이미지

[Fig. 8] Simulation environment

KGOHCL_2019_v19n3_5_f0009.png 이미지

[Fig. 9] ProGReGA Algorithm Distributed result

KGOHCL_2019_v19n3_5_f0010.png 이미지

[Fig. 10] Proposed Algorithm distributed result

KGOHCL_2019_v19n3_5_f0011.png 이미지

[Fig. 11] ProGReGA Algorithm and Proposed Algorithm Fragmented cells

KGOHCL_2019_v19n3_5_f0012.png 이미지

[Fig. 12] ProGReGA Algorithm and Proposed Algorithm Fragmented Groups

[Table 1] Fragmented cells and groups

KGOHCL_2019_v19n3_5_t0001.png 이미지

References

  1. Dongil Shin, Dongkyoo Shin, Minsoo Kim, Jaehong Jang, Hyunsook Yoon, Junghoon Lee,Changwan Han, "A Research on Implementation of Load Balancing Middleware for the Onlie Game Server", The Korean Institute of Information Scientists and Engineers, Vol. 27, No 2-1, p178-180, 2000.10
  2. K.H. Yang, K.H. Shim, D.O. Ko, I.K. Park, J.S. Kim, "Technical Trend of Online Game Server", Electronics and Telecommunications Trends, Volume 16, No 4, p14-22 2001.8
  3. Lim Soo Jung, Hong Dong Chul, Kim Soo Sung, Kim Sung Joo, Yu Seung Han, Joon Taek Han, Jang Tae Moo, Network analysis in 3D MMORPG online games", [KOCCA]Research Report, Korea Create Content Agency, 2010
  4. Kang Jung Joong, "Online Game Server", Game Academy at Korea Game Industry Development Institute, p8-16, 2005.6
  5. Jong-Gwan Choi, Hye-Young Kim, Won-Sik Woo, "A Study of a Game User Oriented Load Balancing Scheme on MMORPG", Journal of Korea Game Society, Volume 12, No 3, p69-76 2012.6 https://doi.org/10.7583/JKGS.2012.12.3.69
  6. Carlos Eduardo Benevides Bezerra, Claudio Fernando Resin Geyer, "A Load Balancing Scheme for massively multiplayer online games", Multimedia Tools and Applications, Volume 45, Issue 1-3, p 263-289 2009.10 https://doi.org/10.1007/s11042-009-0302-z
  7. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov, "Proximal Policy Optimization Algorithms", ArXiv 2017, arXiv:1707.06347v2, 2017.8
  8. "Introducing: Unity Machine Learning Agents Toolkit", Unity Blog, 2017.9.19., 2019.3.7, https://blogs.unity3d.com/2017/09/19/introducing-unity-machine-learning-agents/
  9. Beob Kyun Kim, Hang Jin Jang, Kang Soo You, "Load Balancing in Seamless Game with MigAgent", Journal of Internet Computing and Services, Volume 7, No 6, p51-62, 2006.12
  10. Jeongjin Lee, Gilsoo Doo, Dongun Ann, Seungjong Chung,"Design of Dynamic Map-Divide System for Load Distribution of MMORPG (Massively Multi-player Online Role Playing Game)", The Korean Institute of Information Scientists and Engineers, Vol 32, No. 1, p802-804, 2005.7
  11. J.Y. Lim, I.K. Park, J.Y. Chung, K.H. Shim, "Technical Trend of Distributed Game Server", Electronics and Telecommunications Trends, Volume 20, No 4, p93-102, 2005.8
  12. Ashish Revar, Malay Andhariya, Dharmendra Sutariya, "Load Balancing in Grid Environment using Machine Learning - Innovative Approach", International Journal of Computer Applications, Volume 8, No.10, p31-34, 2010.10 https://doi.org/10.5120/1241-1731
  13. Carlos Eduardo Benevides Bezerra, Joao Luiz Dihl Comba, Claudio Fernando Resin Geyer, "A Fine Granularity Load Balancing Technique for MMOG Servers Using a KD-Tree to Partition the Space",2009 VIII Brazilian Symposium on Games and Digital Entertainment, 2009.10