Coverage Path Planning for Maritime Search and Rescue Using Maskable Proximal Policy Optimization

Abstract

Maritime search and rescue (SAR) missions require rapid and resource-efficient exploration of wide search areas under severe uncertainty in target location, while actual operational time is highly constrained. To support effective decision-making in such conditions, search planning is commonly formulated as a grid-based coverage path planning problem. However, traditional heuristic-based search patterns used in practice, such as parallel track search and expanding square search, offer limited adaptability because they do not explicitly reflect probability distributions that evolve due to environmental factors such as wind and ocean currents. To address these limitations, this study proposes a maskable Proximal Policy Optimization (PPO) framework for efficient grid-based search, motivated by the operational characteristics of maritime SAR missions. The proposed approach integrates domain-specific action masking rules into the policy optimization process to restrict invalid or inefficient actions during learning and execution. By guiding the agent toward feasible movements and high-priority regions of the search space, the framework promotes structured exploration behavior and stable policy learning in large-scale grid environments with sparse and unevenly distributed targets. The learning environment is constructed to reflect realistic maritime search conditions by incorporating probability distributions derived from drift particle simulations, and the search problem is modeled as a Markov decision process. The proposed method is evaluated under multiple target distribution scenarios and compared with representative heuristic search strategies, including parallel track search, nearest neighbor, expanding square search, and 2-optimization. Search performance is primarily assessed in terms of total search cost, measured by the cumulative movement steps required to complete the search task. Experimental results demonstrate that the maskable PPO consistently achieves the lowest total search cost across all tested environments. In particular, it requires substantially fewer movement steps than parallel track search and nearest neighbor methods, while maintaining comparable or superior efficiency relative to expanding square search and 2-optimization, depending on the target distribution. These results indicate that the proposed approach effectively learns coverage paths that reflect the underlying probability distribution of the environment. Overall, this study demonstrates that action masking-based reinforcement learning provides a practical and scalable alternative to conventional heuristic search strategies for maritime SAR missions and establishes a foundation for adaptive search planning under realistic environmental uncertainty.

1. Introduction

In maritime search and rescue (SAR) situations, in most cases, the search target cannot continuously provide its current location, so the location of a reported or identified search target has high uncertainty. Considering that the survival time of the distressed person is very limited, the actual time available for search is extremely compressed. Therefore, path planning based on a rapid and efficient search strategy using search assets to effectively confirm the search target within the given time is essential. Among the search assets available for SAR missions, unmanned aerial vehicles have the advantages of immediate deployment, operation in hazardous areas without risk to human life, and detection capability based on high-resolution sensors, making them the most efficiently usable search assets in the field. Coverage path planning (CPP), which is the process of rationally organizing search paths using unmanned aerial vehicles, is considered a key technology that determines the efficiency and success rate of SAR missions.

In general, CPP is defined as the task of determining a path that visits all points within a region of interest while avoiding obstacles. This technique has been widely adopted in terrestrial environments where obstacles exist, such as robotic vacuum cleaning or precision agriculture. Unlike these settings, the maritime SAR environment is characterized as an open sea where obstacles are virtually non-existent. Consequently, the fundamental objective of CPP in maritime SAR is not complex obstacle avoidance, but rather to rapidly cover the entire allocated search area without omissions to eliminate any blind spots.

In the maritime SAR field, CPP traditionally started from heuristic search pattern methods. The International Aeronautical and Maritime Search and Rescue (IAMSAR) manual (International Maritime Organization, 2022) presents standard search patterns such as parallel track search and expanding square search. These methods have advantages in simplicity and real-time applicability, but they generate search paths based on the expected missing position of the search target. Since this does not generate search paths reflecting the probability distribution according to the uncertainty characteristics of the search target’s location, it is difficult to consider the generated paths as optimized. To complement this, recent studies have proposed various methods to improve efficiency and stability by enhancing existing heuristic patterns or combining them with optimization techniques.

For example, Kazemdehbashi (2024) proposed the adaptive grid-based decomposition algorithm, which reduces unnecessary cells generated by heuristic methods to shorten search time. Zhao et al. (2024) improved the smoothness and balance of paths generated through the Fuzzy random key evolutionary algorithm combined with fuzzy heuristic methods. However, these heuristic-based improvements only modify part of the existing forms and have the limitation of not directly improving the search path. In conclusion, heuristic methods generate paths based on the current location of the search target rather than its spatial distribution and do not reflect the probability of detection or probability of containment at each location. To maximize search success within the limited time of SAR missions, path planning that reflects the probability of detection at each location is required.

To generate optimal paths reflecting the probability distribution of the search space, recent research has actively applied reinforcement learning. Liu et al. (2024) used a probability of success deep Q-network (DQN) (Mnih et al., 2015) to allow the agent to repeatedly interact with the environment and learn reward feedback, gradually optimizing the search strategy. Ai et al. (2021) pointed out that existing rule-based search overly depends on predefined rules and proposed a deep reinforcement learning-based maritime SAR path planning method to overcome this. Through DQN, the global optimal path in the search area was learned, achieving higher search success rates and stable convergence performance compared to existing heuristic searches. Yang et al. (2025) applied Proximal Policy Optimization (PPO) (Schulman et al., 2017), a representative policy-gradient reinforcement learning algorithm, to unmanned aerial vehicle (UAV)-based search. PPO was evaluated as a suitable algorithm for path search problems due to its learning stability and sample efficiency, and the study reported improved search efficiency by 8.1% and 34.2% compared to DQN and Q-Learning, respectively. In this way, recent studies have been developing toward maximizing search efficiency by generating search paths that reflect environmental changes using reinforcement learning methods such as DQN and PPO.

However, reinforcement learning-based algorithms, such as DQN and PPO, face inherent limitations in efficiently exploring complex state spaces, often restricting their stable application to small-scale environments. For instance, DQN is prone to Q-value instability and overestimation in large environments, while PPO frequently exhibits inefficient exploration behaviors, such as repeatedly visiting already searched areas.

Notably, while prior studies (Yang et al., 2025; Ai et al., 2021) successfully applied reinforcement learning to path planning, these efforts were primarily conducted in relatively small grid environments, typically ranging from 10 × 10 to 20 × 20. However, simplifying the vast maritime SAR environment into such coarse grids makes it difficult to accurately reflect critical details, such as complex drift patterns and precise target distributions. Therefore, this study adopts a 50 × 50 (2,500 cells) environment to ensure high-fidelity spatial representation. In such a large-scale setting, the exploration space required for the agent to discover valid paths expands exponentially. Consequently, standard reinforcement learning agents consume an excessive number of episodes merely attempting to navigate the vast state space, often failing to converge to a meaningful policy within a feasible number of training steps due to the extreme sparsity of positive rewards.

To address these inefficiencies, maskable PPO (Huang and Ontañón, 2020) has been adopted as a theoretical foundation for restricting invalid actions. This method improves learning stability by utilizing a masking mechanism to preemptively block actions that are theoretically impossible. However, applying this general methodology to maritime SAR missions requires more than a direct application; it demands the design of specific action masking strategies tailored to the unique constraints of the maritime environment. While standard maskable PPO focuses on preventing invalid moves, the core objective of CPP in maritime SAR is to generate efficient paths that avoid redundant searches and rapidly visit high-priority areas. Therefore, this study proposes a novel CPP framework that tailors the maskable PPO structure to maritime SAR missions. Specifically, a reinforcement learning environment was constructed using a weighted probability map based on the Leeway model, and three strategic masking rules were designed to optimize search efficiency: (1) boundary restriction, (2) revisit prevention, and (3) global unvisited tracking. By structurally excluding unproductive behaviors through these rules, this study aims to demonstrate that the proposed framework achieves superior exploration efficiency compared to traditional heuristic methods.

2. Materials and Methods

The procedure of this study consists of three stages: (1) reinforcement learning environment construction, (2) path learning and generation, and (3) performance evaluation, as shown in Fig. 1. The first stage is Reinforcement Learning Environment Modeling. In this study, to address maritime SAR missions, a weighted probability map reflecting the actual maritime environment is generated, and based on this, a reinforcement learning-based path optimization algorithm is trained and evaluated. For this purpose, a weight map is generated as the environment by reflecting maritime environmental factors such as currents, wind direction, and wind speed based on the Leeway model, and a Markov decision process (MDP) (Garcia and Rachelson, 2013) is defined. In this stage, based on the MDP structure composed of state, action, transition probability, reward, and discount factor (γ), a model learning environment is created so that the agent can learn in a search environment similar to an actual SAR operation.

Fig. 1. Flowchart of reinforcement model evaluation.

The second stage is Path Learning and Generation. In this stage, learning is performed using an action-masking-based PPO algorithm that preemptively blocks impossible or inefficient actions (e.g., moving outside the grid, revisiting already searched cells). After training is completed, the agent generates an optimal coverage path that maximizes search efficiency.

The third stage is Performance Evaluation. The maskable PPO-based search paths learned in the same environment are compared with paths generated using heuristic search methods (Parallel track search, expanding square search, nearest neighbor, 2-optimization) through key performance metrics such as the average number of steps to reach the target from each cell and the total number of search steps. Through this, the study verifies the effectiveness of the reinforcement learning-based path generation methodology using probabilistic search maps that reflect actual maritime environmental conditions. The following section describes the detailed methods for each stage. Before detailing the specific implementation of these stages, the fundamental concepts of CPP are first introduced to establish the theoretical context for the proposed framework.

2.1. Coverage Path Planning

CPP is a fundamental problem in robotics, defined as the task of determining a trajectory that ensures an agent visits every point within a region of interest while avoiding obstacles. As illustrated in Fig. 2, unlike standard point-to-point path planning, which aims to find the shortest or most energy-efficient route between two specific locations, the primary objective of CPP is to maximize the area covered by the agent’s sensors or effectors.

Fig. 2. Comparison of path planning strategies in a grid environment: (a) standard point-to-point path planning and (b) grid-based coverage path planning.

CPP approaches are generally categorized into exact cellular decomposition and approximate grid-based methods. Cellular decomposition divides the free space into simple geometric regions (e.g., trapezoids) and covers each using systematic patterns such as back-and-forth motions. In contrast, grid-based methods decompose the environment into a collection of uniform cells, representing the area as a grid map. This approach allows for flexible path generation in complex environments and is particularly suitable for integrating with probabilistic maps where each cell holds a specific value or weight.

In the specific context of maritime SAR, CPP faces unique constraints compared to terrestrial applications like robotic cleaning or agricultural monitoring. The maritime environment is typically characterized as an obstacle-free open sea, but it is highly dynamic due to the continuous drift of targets caused by wind and currents. Consequently, the goal of maritime CPP shifts from “static full coverage with obstacle avoidance” to “maximizing the cumulative probability of detection within a limited operation time.” To address these challenges, this study adopts a grid-based approach modeled as an MDP, enabling the agent to prioritize high-probability areas based on environmental dynamics rather than simply following a geometric pattern.

2.2. Problem Formulation

In this study, the CPP problem was modeled as an MDP environment. An MDP consists of state (S), action (A), transition probability (P), reward (R), and discount factor (γ), and these elements are defined through the interaction between the agent and the environment. The agent selects an action to perform in a given state, and as a result, the environment transitions to a new state S′and returns a reward. By repeating this process, the agent learns a policy based on reward feedback and improves its decision-making to maximize long-term cumulative reward. The schematic diagram of this interaction loop is shown in Fig. 3. In this study, the agent’s cumulative reward is defined as efficiently exploring as many high-weight cells as possible within a limited search time. The agent learns the optimal search path within the given environment, assuming that the initial search position is at the center of the grid area and that the search is executed after the target reaches a specific location.

Fig. 3. Schematic diagram of the reinforcement learning interaction loop.

2.2.1. Weighted Probability Map Generation

To accurately reflect the uncertainty of real-world maritime search environments, this study constructed a weighted probability map utilizing the Leeway model based on OpenDrift (Dagestad et al., 2018). The Leeway model is a probabilistic method that predicts drift trajectories by stochastically calculating dynamic environmental variables such as currents, wind direction, and wind speed.

In the simulation, the search target is represented as a cluster of multiple particles moving through the interaction of wind and currents. To simulate the physical variability inherent in actual maritime accidents, specific ‘wind response coefficients’ are applied to each particle. These coefficients account for variations in the floating object’s shape, draft, and surface area. Consequently, even with the same initial position, individual particles follow distinct trajectories over time, generating a realistic probabilistic distribution that diffuses naturally.

This continuous particle distribution undergoes a process of spatial discretization to be converted into an environment for the reinforcement learning agent. The entire search area is divided into a 50 × 50 grid, and the particle density within each cell is calculated and normalized into integer weights ranging from 0 to 5. This process transforms irregularly scattered drift data into intuitive information that the agent can identify as search priorities. An example of the generated state space is illustrated in Fig. 4.

Fig. 4. Example of the weighted probability map.

The constructed weighted probability map (WPM) plays a pivotal role in the proposed framework. It functions not merely as positional data, but as a heuristic signal providing the probability density of search success. This probabilistic information prevents the agent from engaging in inefficient random searches across the vast open sea. Instead, it guides the agent to optimize its path toward high-value regions characterized by high particle density, thereby establishing a foundation for maximizing search efficiency within limited operational time.

2.2.2. State Space

The state space for the proposed reinforcement learning framework is designed to represent the dynamic search environment based on the constructed WPM. As illustrated in Fig. 5, the state at time step t, denoted as St, encapsulates the global probability distribution of the search area, including the agent’s current position.

Fig. 5. Example of the state space.

The agent observes this environment as a single matrix-form input, where the grid consists of cells with weight values w ∈ {0, …, 5}. These values serve as a direct mapping of the potential reward magnitude, distinguishing between low-priority areas (value 0, empty sea) and high-priority targets (values near 5, potential distress signals). This high-dimensional state space provides the agent with global situational awareness, enabling it to identify spatial patterns and plan an optimal path from its current location to essential search targets. Consequently, the state space acts as a critical interface that transforms physical environmental data into a decision-making policy for efficient coverage path planning.

2.2.3. Action Space

In this study, the action space is defined as the agent’s movement in four directions (up, down, left, right) within the state space to perform the search, as depicted in Fig. 6. The agent starts from the central cell of the grid and moves to an adjacent cell with each action, and the visited cell is recorded as having been searched.

Fig. 6. Example of the action space.

The mission termination condition is defined as visiting all weighted cells at least once. This reflects the objective of fully searching all areas of interest within the search region in actual maritime SAR missions. With this setup, the reinforcement learning agent is designed to learn paths that efficiently cover the entire search area while minimizing unnecessary repetitions within a limited number of actions, similar to an actual SAR mission.

2.2.4. Transition Probability

In this study, the transition probability (P) refers to the probability that the agent moves to the next state after selecting a specific action in the current state. Since the state space is composed of a cell-based grid, the agent’s position is updated to an adjacent cell according to the chosen action. However, in actual maritime environments, the movement path is not always maintained in the intended direction due to disturbance factors such as wind direction and speed, currents, and global positioning system (GPS) errors. To reflect this uncertainty, this study introduces the concept of probabilistic transition.

Specifically, the probability that the agent moves in the intended direction of the chosen action is set to 1–ε, and the probability of being diverted to other surrounding cells due to disturbances is set to ε/n, as expressed in Eq. (1).

\(\begin{align}\mathrm{P}\left(\mathrm{s}_{\mathrm{t}+1} \mid \mathrm{s}_{\mathrm{t}}, \mathrm{a}_{\mathrm{t}}\right)\left\{\begin{array}{l}1-\varepsilon, \text { if } \mathrm{s}_{\mathrm{t}+1} \text { is the intended next cell } \\ \varepsilon / \mathrm{n}, \text { if } \mathrm{s}_{\mathrm{t}+1} \text { is one of } \mathrm{n} \text { neighboring cells } \\ 0, \text { otherwise }\end{array}\right.\end{align}\)       (1)

ε : disturbance-induced path error probability

n : number of alternative neighboring cells

This probabilistic transition model reflects physical factors such as agent instability, changes in wind direction and speed, and GPS errors in actual maritime search environments, thereby creating a more realistic reinforcement learning training environment.

2.2.5. Reward Function

The reward function in this study is designed to increase coverage efficiency and minimize redundant searches in reinforcement learning-based exploration. Several variables representing the current search step are used in reward calculation: Step count indicates the number of search steps performed so far, combo count represents the number of consecutive new cells explored, Total_{repeat visits} is the accumulated number of repeated visits, and Max_{explore steps} denotes the maximum allowable search steps in a scenario.

First, when a cell with a weight greater than 0 that has not yet been visited is explored for the first time, a reward for visiting a new cell is given. This reward is proportional to the weight of the cell, which represents its importance, and is defined as in Eq. (2).

r^new_t = w_t · η_new · α^N_step       (2)

w_t: current weight

η_new = 0.001

α = 0.9993

N_step: cumulative number of movement steps

A higher weight gives a larger reward, and a decay factor, such as 0.9993, is applied so that the reward gradually decreases with the number of moves. This encourages the agent to prioritize visiting cells with high exploration value in the early stages of learning. This design reflects the operational requirement to secure the maximum possible search area immediately after deployment.

Additionally, when consecutive new non-zero cells are explored, a consecutive exploration bonus is added. This reward is intended to increase the continuity of exploration and reduce inefficient back-and-forth paths, and is expressed as in Eq. (3).

r^combo_t = η_combo · N_combo(t)       (3)

η_combo = 10^–5

N_combo(t): number of consecutive newly visited

non-zero cells at time t

On the other hand, when a previously visited cell is explored again, a revisit penalty is applied. The penalty gradually increases with the number of repeated visits to prevent the agent from redundantly searching the same area. The revisit penalty is defined as in Eq. (4).

r^rep_t = η_rep · (1 + λN_rep(t))       (4)

η_rep = 2 × 10^–4

λ = 0.1

N_rep(t): cumulative number of revisits

Whenever the exploration progresses to the point that the coverage of all non-zero cells increases by 1%, a coverage reward is given. This serves to encourage the gradual expansion of the total search area.

r^cov_t = max(η_cov – p_t, 0)       (5)

η_cov = 0.05

p_t: coverage penalty (based on unexplored ratio)

Additionally, when the exploration is 100% complete, a full-coverage bonus is awarded. This bonus provides a higher reward the faster full coverage is achieved within the limited number of moves. This serves as a sparse reward to sustain the agent’s motivation for long-term path planning required for complete mission success.

r_t^bonus = η_base + η_term(N_max – N_step)       (6)

η_base = 0.02

η_term = 0.015

N_max : the episode length limit, defined as 10 times the maximum number of exploration steps

Finally, the total reward is calculated by summing all of the above components for individual actions, with long-term rewards reflecting the overall efficiency of the search.

r_t = r_t^new + r_t^combo – r_t^rep + r_t^cov + r_t^bonus       (7)

This reward structure is designed to resolve the trade-off between rapid initial exploration and exhaustive global coverage. Specifically, the decay factor in the new cell reward (reward_{new cell}) is intentionally introduced to incentivize the agent to rapidly maximize coverage area in the early stages of the episode, reflecting the urgency of the ‘Golden Time’ in SAR missions. Conversely, the coverage reward (reward_coverage) and full-coverage bonus are designed to induce long-term behavior, ensuring the agent does not settle for local optima but persists until the entire search region is fully cleared.

2.2.6. Discount Factor

In this study, the discount factor is a key component of the MDP and serves to adjust the agent to learn an optimal policy by considering not only immediate short-term rewards but also future cumulative rewards. The discount factor acts as a decay coefficient with a value between 0 and 1. When the discount factor is close to 1, the agent prioritizes long-term rewards, whereas when it is close to 0, the agent prioritizes immediate rewards. Therefore, in this study, the γ value was appropriately set so that the agent considers not only short-term search efficiency but also the long-term goal of achieving full coverage of the entire search area. The cumulative expected reward Gt at time step t is defined as follows.

\(\begin{align}G_t=r_{t+1}+{\gamma}r_{t+2}+{\gamma}^2r_{t+3}+...=\sum_{k=0}^{\infty}{\gamma}^kr_{t+k+1}\end{align}\)       (8)

In the above equation, r_t+k+1 denotes the immediate reward defined in Eqs. (2–7) at time step t+k+1, and γ ∈ (0, 1) is the discount factor that determines the relative importance of future rewards.

This discount factor–based reward accumulation structure guides the agent not to focus solely on maximizing short-term rewards, but to maintain a balance between long-term search performance (coverage completeness) and the efficiency of the search path.

2.3. Proposed Algorithm Structure

In this study, the existing PPO algorithm operating in an MDP environment was combined with an action masking module to implement the maskable PPO model. PPO is a representative policy-based reinforcement learning method in which the agent directly optimizes the policy function using experiences (state, action, reward) obtained through interactions with the environment.

The experiences generated during PPO training consist of the set of states s_t observed by the agent from the environment at each time step t, the actions a_t chosen accordingly, and the rewards r_t returned by the environment after performing the actions. These interactions accumulate to form an episode, and the agent gradually maximizes the expected return in the given environment by repeatedly accumulating and learning from episodes.

The conventional policy gradient algorithm has a structure that directly maximizes the expected return E[G_t] with respect to the policy parameters θ, and its general form is expressed as follows.

▽_θJ(θ) = E_t[▽_θlogπ_θ(a_t|s_t)A_t]       (9)

Here, A_t is the advantage function at time step t, and π_θ denotes the policy function. Conventional policy gradient methods have the limitation that excessively large policy updates can cause learning instability. To address this, PPO introduces a proximal constraint that limits the magnitude of policy updates, enabling stable and reliable learning.

PPO defines the ratio of action probabilities between the newpolicy π_θ and the old policy π_θold as follows:

\(\begin{align}\rho_{t}(\theta)=\frac{\pi_{\theta}\left(a_{t} \mid s_{t}\right)}{\pi_{\theta_{\text {old }}}\left(a_{t} \mid s_{t}\right)}\end{align}\)       (10)

However, since policy updates are not excessively large, inefficient behaviors such as continuously repeating initial actions can occur. To minimize this, an action masking module was added. The proposed masking rules follow three principles to ensure the efficiency and consistency of exploration.

First, the agent cannot perform actions that move it outside the grid. The grid has a size of W × H, and the agent’s state s_t = x_t, y_t must always satisfy 0 ≤ x_t < W and 0 ≤ y_t < H. Therefore, actions that move outside the boundaries of a boundary state are considered invalid and cannot be selected, reflecting boundary conditions that prevent leaving the actual search area. This constraint prevents unnecessary transitions and ensures stability in policy learning.

Second, if there are unvisited cells adjacent to the agent’s current location, the agent must move only to those unvisitedcells. This prevents inefficient behaviors such as repeatedly visiting already explored cells. The agent prioritizes selecting unvisited neighbor cells among the four adjacent directions (↑, ↓, ←, →), and visitation status is managed using a binary mask. That is, if at least one unvisited cell exists nearby, the policy π_θ(a|s_t) restricts the action space to allow movement only to those cells. Among the multiple unvisited candidates, the specific action is selected based on the probability distribution output by the policy network π_θ, which learns to prioritize cells with higher expected rewards through training.

Third, if all adjacent cells have already been visited, the agent can move only in the direction of the closest unvisited cell within the entire grid. This rule addresses situations where adjacent unvisited cells disappear as exploration progresses and serves to teach the agent continuity in covering the search area. The agent calculates the relative direction to the nearest unvisited cell based on Euclidean distance and allows only the action that moves in that direction. This encourages the agent to explore the entire area without omission and minimizes unnecessary repetitive or circular movements.

In summary, these three constraints are designed to simultaneously achieve boundary stability, local exploration efficiency, and global coverage. They constitute key action-masking rules that enhance the exploration quality of reinforcement learning-based coverage path planning.

2.4. Performance Analysis

To quantitatively evaluate the exploration performance of the proposed maskable PPO model, this study utilized two main metrics.

First, the average number of steps to reach a target weighted cell refers to the average number of moves required for the agent to reach a probabilistically selected target cell among the weighted cells in the search area. This reflects the number of search steps needed to locate a target, such as a distressed person or drifting vessel, in an actual maritime SAR mission, and serves as a key metric for assessing the speed and efficiency of path exploration.

Second, the total number of steps performed during the entire search process represents the total number of moves made by the agent until the end of a single episode. This metric directly corresponds to the time and energy consumption required during an actual mission and is therefore used to measure the overall operational efficiency of the proposed algorithm.

By comprehensively utilizing these two performance metrics, the study quantitatively compared and analyzed the degree of improvement in search efficiency, coverage completeness, and path optimization achieved by the proposed maskable PPO model relative to conventional PPO and heuristic-based search methods.

2.5. Experimental Setup for Comparative Study

To rigorously validate the performance of the proposed maskable PPO, this study compares it against four representative heuristic algorithms commonly utilized in maritime search operations and combinatorial optimization problems: Parallel track search, expanding square search, nearest neighbor, and 2-optimization.

The first two algorithms, parallel track search and expanding square search, are standard search patterns defined in the IAMSAR manual. Parallel track search is designed to cover large rectangular areas uniformly by navigating in straight parallel lines with fixed spacing, making it suitable for general scanning. In contrast, expanding square search expands outward in concentric squares from a specific datum, a strategy typically employed when the target’s location is known with some initial uncertainty.

The latter two algorithms, nearest neighbor and 2-optimization, approach the coverage path planning problem as a variant of the Traveling Salesperson Problem. Nearest neighbor employs a greedy heuristic, prioritizing the closest unvisited weighted cell at each step to minimize immediate travel distance. The 2-optimization algorithm further refines the path generated by Nearest Neighbor by iteratively swapping edges to eliminate path crossings and reduce the total travel distance, representing a more optimized heuristic approach. Visual representations of the trajectories generated by these four baseline algorithms are illustrated in Fig. 7.

Fig. 7. Trajectory visualization of the four baseline algorithms: (a) parallel track search, (b) expanding square search, (c) nearest neighbor, and (d) 2-optimization.

To ensure a fair and quantitative comparison between the reinforcement learning agent and these baselines, strictly identical operational constraints were applied, with a necessary distinction in the starting position to accommodate the structural characteristics of the algorithms. While the proposed maskable PPO, expanding square search, Nearest neighbor, and 2-optimization initiated the search from the center of the grid (25, 25), parallel track search commenced from the coordinate (0, 0). This exception was applied because parallel track search is inherently designed to sweep across the area from a boundary, making a central start inefficient for its standard pattern. Aside from the starting point, a unified mission termination condition was enforced, where the search was considered complete only when all cells with non-zero weights (w > 0) within the weighted probability map were visited at least once. Regarding parameter settings, the track spacing for parallel track search and expanding square search was fixed at one cell—equivalent to the sensor range—without additional tuning, while Nearest neighbor and 2-optimization were configured to minimize the total travel distance required to cover all weighted cells. This experimental design ensures that the comparative results reflect the intrinsic path planning efficiency of each algorithm under optimized operational conditions.

3. Results

3.1. Simulation and Weighted Probability Map Generation

In this study, experiments were conducted by simulating the area centered on the Shetland Islands in northern Scotland (latitude 60.3°N, longitude 1.3°W). As illustrated in Fig. 8, this region was selected as the experimental target because its complex maritime terrain, characterized by numerous scattered islands, significantly influences the drift paths of objects. These geographical features allow for diverse distributions of search targets, making the area suitable for validating the path efficiency of the proposed exploration model. In particular, it provides an optimal environment for comparing and analyzing changes in search performance depending on the presence of islands.

Fig. 8. Study area centered on the Shetland Islands.

In the simulation, 10,000 drifting objects with lifeboat-like characteristics were released within a 3 km radius around the starting point (red dot in Fig. 8; latitude 60.0°N, longitude 1.9°W). Assuming that the targets move according to actual conditions, their positions were simulated over 48 hours at 1-hour intervals based on the weather conditions of November 16, 2015. To ensure the fidelity of the drift trajectories corresponding to these specific environmental conditions, high-resolution forcing data were utilized within the OpenDrift framework. Specifically, surface current data were obtained from the NorKyst-800 model (800 m resolution). The efficacy of this model in resolving complex topographically steered currents in the North Sea and Norwegian coastal waters has been extensively validated in prior drift experiments and operational forecast assessments (Jones et al., 2016; Brekke et al., 2021; de Aguiar et al., 2023).

Furthermore, wind fields were derived from the applications of research to operations at the mesoscale (AROME) model (2.5 km resolution), which is established in drift modeling studies for effectively capturing local wind variations (Jones et al., 2016; Brekke et al., 2021). Following the calibration methodology of Jones et al. (2016), a wind uncertainty factor was set to 2.0 m/s to account for environmental variability, while horizontal diffusivity was deactivated to prioritize deterministic transport driven by these high-resolution fields.

The simulation results, shown in Fig. 9, indicated that at the 6-hour mark, the drift paths were unaffected by islands (central dispersion), while at the 44-hour mark, the drift paths were influenced by the islands’ terrain (terrain-influenced dispersion). These two drift patterns were used as the experimental environment.

Fig. 9. Simulation results: (a) 6-hour distribution and (b) 44-hour distribution.

The particle distributions at the two time points were redefined as an environment composed of a 50 × 50 grid for evaluating the path generation algorithm and converted into weight maps. The results of this conversion are shown in Fig. 10. In this conversion process, to ensure a fair comparison of search efficiency based on distribution morphology, this study set the physical dimensions of the search area represented by the 50×50 grid to be identical for both the 6-hour and 44-hour scenarios.

Fig. 10. Weight maps generated based on Fig. 9: (a) 6-hour distribution and (b) 44-hour distribution.

3.2. Model Training

To create the maskable PPO algorithm to be tested on the generated weight maps, a total of eight 50 × 50 training central-dispersion weight maps were used to train the model. To increase independence of the training data, a 20-hour 40-minute drift prediction simulation was conducted at a different location (latitude 34°, longitude 127°) for lifeboats and vessels at 1-minute intervals, using weather data from May 12, 2025. From the simulation results, eight time points were selected, as shown in Fig. 11, to train the maskable PPO model.

Fig. 11. Weight maps used for training: (a) step 103, (b) step 416, (c) step 539, (d) step 654, (e) step 658, (f) step 796, (g) step 843, and (h) step 971.

The specific hyperparameters used for training the maskable PPO model are detailed in Table 1. The model was implemented using the Stable Baselines3 Contrib library (Khambhayata et al., 2024). To ensure stable convergence on the large-scale 50 × 50 grid environment, a large batch size of 2,048 and a rollout buffer of 40,960 steps were utilized. Additionally, the discount factor (γ) was set to 0.999 to prioritize long-term coverage rewards over immediate gains, encouraging the agent to achieve complete exploration.

Table 1. Hyperparameters for maskable PPO

3.3. Model Performance

As shown in Fig. 12, the performance of the trained model demonstrates a gradual improvement in exploration efficiency over the course of the episodes. To ensure clarity in the analysis of learning trends and to mitigate the high variance typical of raw reinforcement learning data, a moving average with a window size of 200 (n=200) was plotted for all performance metrics.

Fig. 12. Performance of the trained model: (a) new reward, (b) repeat reward, (c) combo reward, (d) coverage reward, and (e) total reward.

First, the New cell reward maintained a stable high value or showed a moderate increase from approximately 2.3 to 2.5, indicating that the agent consistently maintained the capability to discover new areas throughout the learning process. In contrast, the repeat reward (penalty) showed a significant improvement, increasing from approximately −1.5 in the early stages to −0.3 in the later stages. This indicates that the agent successfully learned to minimize the penalty by avoiding unnecessary redundant paths.

Coverage reward, a key indicator of exploration completeness, gradually increased from 5.0 to the 7.0–8.0 range, demonstrating that the policy was optimized to cover a wider area within the same number of steps. The Combo reward, given for consecutive successful exploration actions, started at around 0.2 but showed an upward trend to values above 0.8, suggesting that the agent’s movement paths became structurally organized and its exploration strategy became more consistent.

Finally, the Total reward, representing the aggregate of these components, rose from approximately 6.0 in the initial phase to 8.0 in the final stages. This continuous upward trend confirms that the overall exploration strategy was gradually optimized. These results demonstrate that the proposed maskable PPO model can generate paths that meet the efficiency and stability requirements of actual SAR missions, outperforming conventional heuristic methods.

In the search-and-rescue setting considered in this study, each experiment represents a single mission in which the target vessel is unknown a priori. Accordingly, for each run, a single target weighted cell is randomly sampled from the grid to represent the target location. To obtain representative performance results, the experiment is repeated ten times under identical environmental conditions, each time with a newly sampled target location. The specific target points selected for the two types of distributions—central dispersion and terrain-influenced—are detailed in Tables 2 and 4, respectively. For these selected target points, the trained maskable PPO model was compared in performance with representative heuristic search methods commonly used in maritime search, including parallel track search, expanding square search, nearest neighbor, and 2-optimization.

Table 2. Target locations and distance metrics for the central dispersion environment (6-hour case)

Table 3. Algorithm performance comparison (6-hour case)

Table 4. Target locations and distance metrics for a terrain-influenced environment (44-hour case)

In this comparative analysis, the average step is defined as the number of steps required for the agent to reach the target weighted cell for the first time in a single run. The reported average step is computed as the arithmetic mean of the step counts obtained from the ten independent runs described above. The quantitative results are presented in Tables 3 and 5, and the corresponding generated search paths are visualized in Figs. 13 and 14, respectively. The comparison results indicate that maskable PPO substantially reduces the required steps compared to parallel track search and nearest neighbor, while showing competitive performance against expanding square search and 2-optimization depending on the target distribution. Notably, maskable PPO consistently achieves the lowest total search cost across both environments, indicating superior end-to-end operational efficiency.

Table 5. Algorithm performance comparison (44-hour case)

Fig. 13. Comparison of coverage path trajectories in the central dispersion environment (6-hour case): (a) maskable PPO, (b) nearest neighbor, (c) 2-optimization, (d) parallel track search, and (e) expanding square search.

Fig. 14. Comparison of coverage path trajectories in terrain-influenced environment (44-hour case): (a) maskable PPO, (b) nearest neighbor, (c) 2-optimization, (d) parallel track search, and (e) expanding square search.

Furthermore, comparing the total accumulated movements over the entire search process clearly highlighted the efficiency of maskable PPO. The maskable PPO completed the search of all ten target points with 1,076 moves, recording the lowest total movement, and accomplished the mission with approximately 2.3 times less search cost compared to parallel track search (2,486 moves) and expanding square search (2,483 moves). Additionally, the total search steps were lower than those of nearest neighbor (1,159 moves) and 2-optimization (1,108 moves), confirming that maskable PPO effectively learned a policy that minimizes unnecessary path expansion and moves efficiently not only for individual targets but across the entire search process. These results suggest that the PPO-based exploration model outperforms conventional heuristic methods in terms of both search efficiency and total movement cost.

To interpret the discrepancy in step counts between the two scenarios, recall that the physical distance represented by each grid cell was set to be identical, as described in Section 3.1. Therefore, the variation in performance is solely determined by the spatial distribution of the target particles. In the 44-hour terrain-influenced case, the complex coastline of the Shetland Islands constrained the particles, causing them to cluster in specific areas. This resulted in a distinct difference in both the locations and the number of non-zero weighted cells (targets) compared to the widely spread 6-hour central dispersion case. Consequently, the agent required fewer total steps to complete the coverage, despite the increased drift duration.

4. Discussion and Conclusions

In this study, to maximize the efficiency of maritime SAR missions, a reinforcement learning–based CPP framework was established using a drift probability map based on the Leeway model as the environment. To improve the initial exploration inefficiency of conventional reinforcement learning, a maskable PPO incorporating action masking was proposed. The comparative analysis with various heuristic algorithms yields the following key implications.

First, the proposed Maskable PPO demonstrated clear superiority in global operational efficiency, specifically regarding the total number of steps. As shown in the experimental results, Maskable PPO recorded the lowest total movement steps in both the central-dispersion (6-hour) and terrain-influenced (44-hour) environments. Notably, in the terrain-influenced scenario, Maskable PPO completed the search with only 1,076 steps, which is approximately 2.3 times more efficient than Parallel Track Search (2,486 steps) and Expanding Square Search (2,483 steps). Even compared to optimization-based heuristics like 2-Optimization (1,108 steps), Maskable PPO achieved lower or comparable total costs. This confirms that the proposed method effectively minimizes redundant paths and maintains continuity in coverage, directly contributing to time and energy savings in actual operations.

Second, regarding the ‘Average Step to Target,’ Maskable PPO showed robust exploration performance, although heuristic methods may be faster in specific geometric conditions. While Maskable PPO generally reached targets significantly faster than Parallel Track Search and Nearest Neighbor, specific heuristics exhibited lower average steps in certain cases. For instance, in the 6-hour central-dispersion environment, Expanding Square Search recorded a slightly lower average step (258.8) compared to Maskable PPO (284.2). This is likely because the Expanding Square pattern is geometrically optimized for expanding outward from the center, which aligns well with the centrally concentrated target distribution. However, unlike heuristics that rely on fixed patterns, Maskable PPO optimizes its policy based on the probability map. Thus, while it may not always be the fastest in every single trial due to the stochastic nature of target sampling, it provides a stable and reliable strategy that consistently prioritizes high-probability areas across diverse environments.

Third, the action masking technique served as a key mechanism for dramatically improving both exploration stability and learning speed. Conventional algorithms such as PPO or DQN often exhibit inefficiencies in large grid environments, repeatedly visiting already explored areas or attempting moves outside the grid during the early learning stages. The action masking rules introduced in this study structurally block such unproductive behaviors, guiding the agent to update its policy only within valid action spaces. As a result, this not only accelerated the convergence speed of learning but also substantially mitigated instability issues typically experienced by reinforcement learning models in large search spaces.

Fourth, Maskable PPO showed superior environmental adaptability in highly uncertain maritime conditions compared to heuristic search methods. Traditional heuristic methods like Parallel Track Search rely on fixed rules and cannot reflect changes in the probability distribution of the search targets. In contrast, Maskable PPO efficiently navigated to target points without unnecessary expansion of the search area, even in scenarios requiring complex path generation, such as terrain-influenced dispersions. This demonstrates that the proposed algorithm can actively incorporate the probabilistic distribution of search targets.

Fifth, this study utilized a high-fidelity simulation environment to ensure statistical reliability and to actively generate diverse target distribution patterns. Unlike real-world trials, which are restricted to specific environmental conditions, our simulation using 10,000 drift particles allowed us to intentionally model various complex spatial distributions. This capability to evaluate the algorithm across diverse probability maps confirms that the proposed framework provides a generalized solution robust to the spatial uncertainty inherent in maritime SAR missions.

However, this study is subject to limitations that explicitly define our future research directions. First, physical constraints such as sensor noise and communication latency were not fully modeled in the simulation. Second, the current framework operates as a single-agent system, which restricts simultaneous coverage capabilities in extensive search areas. To address these limitations, future research will focus on bridging the ‘Sim-to-Real’ gap by implementing the model on actual UAV hardware. Furthermore, we plan to extend the framework to Multi-Agent PPO for cooperative Multi-UAV CPP, thereby developing a more rapid and scalable maritime SAR system.

Author Contributions

Conceptualization: Oh Y, Lee I; Data curation: Oh Y, Oh J; Formal analysis, Investigation, Methodology: Oh Y; Project administration: Kim S; Resources: Oh J; Software: Kim S; Writing–original draft, Writing–review & editing: Oh Y, Lee I.

Conflicts of Interest

No potential conflict of interest relevant to this article was reported.

Funding

This research was supported by the Korea Institute of Marine Science & Technology Promotion (KIMST), funded by the Ministry of Oceans and Fisheries, Korea (RS-2022-KS221629).

Data Availability Statement

The data that support the findings of this study are available from the corresponding author upon reasonable request.